Re: [compress] Changes to allow multithreaded creation of zip files

2014-12-18 Thread Kristian Rosenvold
+1 for moving to git btw :) Kristian 2014-12-15 11:38 GMT+01:00 Emmanuel Bourg ebo...@apache.org: Le 15/12/2014 11:36, Stefan Bodewig a écrit : [as an aside, maybe we should think about moving Compress to git] +1 Emmanuel

Re: [compress] Changes to allow multithreaded creation of zip files

2014-12-18 Thread Kristian Rosenvold
I just landed two commits for COMMONS-295 and COMMONS-296, which is the groundwork for proper parallel support. The icing on the cake (the high-level api with nThreads configuration and single point of entry) is still work-in-progress and can be seen at

Re: [compress] Changes to allow multithreaded creation of zip files

2014-12-16 Thread Kristian Rosenvold
Amazing digging; thanks a lot. At least for maven's code, introducing the concept of a root stream that will be the start of the physical zip stream can simplify things quite a lot. Kristian 2014-12-16 6:48 GMT+01:00 Stefan Bodewig bode...@apache.org: On 2014-12-15, Kristian Rosenvold wrote:

Re: [compress] Changes to allow multithreaded creation of zip files

2014-12-16 Thread Stefan Bodewig
On 2014-12-16, Kristian Rosenvold wrote: At least for maven's code, introducing the concept of a root stream that will be the start of the physical zip stream can simplify things quite a lot. Yes, I can see the same for my compress antlib (Ant tasks based on Commons Compress) as well. It

Re: [compress] Changes to allow multithreaded creation of zip files

2014-12-16 Thread Emmanuel Bourg
Le 16/12/2014 10:35, Stefan Bodewig a écrit : Coming to think of it again, cpio and tar don't have any concept of a central directory so a scatter/gather approach may even work OOTB. ar has this long file name thing that would need to be taken care of and 7z is pretty similar to zip, just a

Re: [compress] Changes to allow multithreaded creation of zip files

2014-12-15 Thread Benedikt Ritter
Hello Kristian, 2014-12-15 8:49 GMT+01:00 Kristian Rosenvold krosenv...@apache.org: I just thought I'd let you know I'm working on changes to allow multiple threads to output to different ZipArchiveOutputStream instances (backed by commons-io DeferredFileOutputStream or similar) and then

Re: [compress] Changes to allow multithreaded creation of zip files

2014-12-15 Thread Stefan Bodewig
On 2014-12-15, Kristian Rosenvold wrote: I just thought I'd let you know I'm working on changes to allow multiple threads to output to different ZipArchiveOutputStream instances (backed by commons-io DeferredFileOutputStream or similar) and then stitch the results back together as a single

RE: [compress] Changes to allow multithreaded creation of zip files

2014-12-15 Thread Gary Gregory
/divdivCc: /divdivSubject: [compress] Changes to allow multithreaded creation of zip files /divdiv /divI just thought I'd let you know I'm working on changes to allow multiple threads to output to different ZipArchiveOutputStream instances (backed by commons-io DeferredFileOutputStream or similar

Re: [compress] Changes to allow multithreaded creation of zip files

2014-12-15 Thread Kristian Rosenvold
APIs? Gary div Original message /divdivFrom: Kristian Rosenvold krosenv...@apache.org /divdivDate:12/15/2014 02:49 (GMT-05:00) /divdivTo: Commons Developers List dev@commons.apache.org /divdivCc: /divdivSubject: [compress] Changes to allow multithreaded creation of zip

Re: [compress] Changes to allow multithreaded creation of zip files

2014-12-15 Thread Stefan Bodewig
On 2014-12-15, Kristian Rosenvold wrote: There is also the issue of determining if the MANIFEST.MF file really needs to be the first file in the jar, which puts an interesting constraint on the parallelism. I have been unable to understand if the jar spec actually requires this or if it's

Re: [compress] Changes to allow multithreaded creation of zip files

2014-12-15 Thread sebb
On 15 December 2014 at 17:47, Kristian Rosenvold krosenv...@apache.org wrote: 2014-12-15 17:39 GMT+01:00 Stefan Bodewig bode...@apache.org: On 2014-12-15, Kristian Rosenvold wrote: There is also the issue of determining if the MANIFEST.MF file really needs to be the first file in the jar,

Re: [compress] Changes to allow multithreaded creation of zip files

2014-12-15 Thread Stefan Bodewig
On 2014-12-15, Kristian Rosenvold wrote: There is also the issue of determining if the MANIFEST.MF file really needs to be the first file in the jar, which puts an interesting constraint on the parallelism. I've found this for Ant which caused us to implement that logic (in 2001):

[compress] Changes to allow multithreaded creation of zip files

2014-12-14 Thread Kristian Rosenvold
I just thought I'd let you know I'm working on changes to allow multiple threads to output to different ZipArchiveOutputStream instances (backed by commons-io DeferredFileOutputStream or similar) and then stitch the results back together as a single Zip file. I'm currently just researching the