* george white <solx86 at palisad.com> [2009-11-19 23:17]: > I have a couple of suggestions for making the IPS packaging system > much faster. First, introduce a type called dirfile, which is a cpio > archive of multiple files in a single directory. These would be files > that are always replaced as a group and never otherwise preserved > individually. This would also make downloads more efficient by > allowing compression to act across a collection of files that contain > common strings and to reduce the overhead from a large number of tiny > transactions.
We're still getting timing data from the new transport, so it's a bit soon to decide on whether a new request-multiple-file request is necessary. Precomputing these, over an arbitrary set of potential origin versions, is a possibility; there are also others. (We're trying to avoid the additional operation, as it will affect our ability to serve content with a stock HTTP server.) > The problem I see is that there are far too many tiny > files, where the overhead of testing each one is so high, it hardly > makes it worth finding it in the download cache. It would be helpful to see some timings that show a local file-existence test is slower that an HTTP request for that file from a depot server. Maybe I'm misunderstanding the idea here. > The other suggestion is to use only 2 digits in the directory name, > instead of 6, for the second level of the download cache. You could > get away with 2 if large numbers of small files were aggregated into > dirfile's, otherwise, you might need 3. Six is way too many as there > is hardly ever more than 1 file in each leaf dire ctory, when > optimally, there should be a 100 or so. In fact, we've recently revised the file layout based on similar analysis. I believe this change will come with the next dev/ build. If you're interested, 7960 client and depot need different organization of files was the Bugzilla bug ID for this issue. There is a comment in Brock's update regarding the layout selected and its behaviour. Thanks for your feedback. Please subscribe to pkg-discuss at opensolaris.org if you have additional comments or ideas for improvement. - Stephen -- sch at sun.com http://blogs.sun.com/sch/
