Re: [COMPRESS] Pack200 support in pack200 branch
> So we could do 1.21 as is followed by 1.22 with pack200 support.
>
+1
Lee
On 11, 1, 2020, at 11:59, Gary Gregory wrote:
> It's been quite a while since this thread started so I will review my
> branch today but might wait for 1.22.
>
> While we indicated that we would address building and running pack200 in
> some way on Java 14+ for 1.21, an option is to release 1.21 as is since
> there are quite a few changes there already and we are updating to Java 8.
>
> So we could do 1.21 as is followed by 1.22 with pack200 support.
> One step at time.
> Gary
> On Sun, Aug 30, 2020, 18:58 Gary Gregory wrote:
> > Hi All,
> >
> > As the next step in the pack200 branch, I've renamed the package
> > org.apache.harmony to org.apache.commons.compress.harmony to more easily
> > track potential future changes. If this looks acceptable, it can be merged
> > and documented.
> >
> > The builds are green here
> > https://github.com/apache/commons-compress/actions/runs/231719106
> >
> > Gary
> >
> >
> > On Mon, Aug 24, 2020 at 11:14 AM Gary Gregory
> > wrote:
> >
> >> Thanks Peter, I updated the branch and the build passes for me locally,
> >> the CI is building now here
> >> https://github.com/apache/commons-compress/actions/runs/19115
> >>
> >> Gary
> >>
> >>
> >> On Sun, Aug 23, 2020 at 11:40 PM Peter Lee wrote:
> >>
> >>> After some debugging I found this at
> >>> org.apache.harmony.pack200.Archive#171 :
> >>>
> >>> if (classes.size() > 0 && files.size() > 0) {
> >>> segmentUnitList.add(new SegmentUnit(classes, files));
> >>> }
> >>> Seems the Pack200 implementation in harmony requires existing of both
> >>> classes AND files at the same time. The tests are passing if I modified a
> >>> little bit here :
> >>>
> >>> if (classes.size() > 0 || files.size() > 0)
> >>> Not sure if this is a problem, cause I'm not familiar with Pack200 and
> >>> Apache Harmony.
> >>>
> >>> Lee
> >>> On 8 23 2020, at 11:27 , Gary Gregory wrote:
> >>> > Hi All,
> >>> >
> >>> > I created a branch called *pack200* which contains the Apache Harmony
> >>> > Pack200 code.
> >>> >
> >>> > If there are 2 failing unit test ATM if anyone wants to help out.
> >>> > Gary
> >>
> >>
>
Re: [COMPRESS] Pack200 support in pack200 branch
It's been quite a while since this thread started so I will review my
branch today but might wait for 1.22.
While we indicated that we would address building and running pack200 in
some way on Java 14+ for 1.21, an option is to release 1.21 as is since
there are quite a few changes there already and we are updating to Java 8.
So we could do 1.21 as is followed by 1.22 with pack200 support.
One step at time.
Gary
On Sun, Aug 30, 2020, 18:58 Gary Gregory wrote:
> Hi All,
>
> As the next step in the pack200 branch, I've renamed the package
> org.apache.harmony to org.apache.commons.compress.harmony to more easily
> track potential future changes. If this looks acceptable, it can be merged
> and documented.
>
> The builds are green here
> https://github.com/apache/commons-compress/actions/runs/231719106
>
> Gary
>
>
> On Mon, Aug 24, 2020 at 11:14 AM Gary Gregory
> wrote:
>
>> Thanks Peter, I updated the branch and the build passes for me locally,
>> the CI is building now here
>> https://github.com/apache/commons-compress/actions/runs/19115
>>
>> Gary
>>
>>
>> On Sun, Aug 23, 2020 at 11:40 PM Peter Lee wrote:
>>
>>> After some debugging I found this at
>>> org.apache.harmony.pack200.Archive#171 :
>>>
>>> if (classes.size() > 0 && files.size() > 0) {
>>> segmentUnitList.add(new SegmentUnit(classes, files));
>>> }
>>> Seems the Pack200 implementation in harmony requires existing of both
>>> classes AND files at the same time. The tests are passing if I modified a
>>> little bit here :
>>>
>>> if (classes.size() > 0 || files.size() > 0)
>>> Not sure if this is a problem, cause I'm not familiar with Pack200 and
>>> Apache Harmony.
>>>
>>> Lee
>>> On 8 23 2020, at 11:27 , Gary Gregory wrote:
>>> > Hi All,
>>> >
>>> > I created a branch called *pack200* which contains the Apache Harmony
>>> > Pack200 code.
>>> >
>>> > If there are 2 failing unit test ATM if anyone wants to help out.
>>> > Gary
>>
>>
Re: [COMPRESS] Pack200 support in pack200 branch
Hi All,
As the next step in the pack200 branch, I've renamed the package
org.apache.harmony to org.apache.commons.compress.harmony to more easily
track potential future changes. If this looks acceptable, it can be merged
and documented.
The builds are green here
https://github.com/apache/commons-compress/actions/runs/231719106
Gary
On Mon, Aug 24, 2020 at 11:14 AM Gary Gregory
wrote:
> Thanks Peter, I updated the branch and the build passes for me locally,
> the CI is building now here
> https://github.com/apache/commons-compress/actions/runs/19115
>
> Gary
>
>
> On Sun, Aug 23, 2020 at 11:40 PM Peter Lee wrote:
>
>> After some debugging I found this at
>> org.apache.harmony.pack200.Archive#171 :
>>
>> if (classes.size() > 0 && files.size() > 0) {
>> segmentUnitList.add(new SegmentUnit(classes, files));
>> }
>> Seems the Pack200 implementation in harmony requires existing of both
>> classes AND files at the same time. The tests are passing if I modified a
>> little bit here :
>>
>> if (classes.size() > 0 || files.size() > 0)
>> Not sure if this is a problem, cause I'm not familiar with Pack200 and
>> Apache Harmony.
>>
>> Lee
>> On 8 23 2020, at 11:27 , Gary Gregory wrote:
>> > Hi All,
>> >
>> > I created a branch called *pack200* which contains the Apache Harmony
>> > Pack200 code.
>> >
>> > If there are 2 failing unit test ATM if anyone wants to help out.
>> > Gary
>
>
Re: [COMPRESS] Pack200 support in pack200 branch
Thanks Peter, I updated the branch and the build passes for me locally, the
CI is building now here
https://github.com/apache/commons-compress/actions/runs/19115
Gary
On Sun, Aug 23, 2020 at 11:40 PM Peter Lee wrote:
> After some debugging I found this at
> org.apache.harmony.pack200.Archive#171 :
>
> if (classes.size() > 0 && files.size() > 0) {
> segmentUnitList.add(new SegmentUnit(classes, files));
> }
> Seems the Pack200 implementation in harmony requires existing of both
> classes AND files at the same time. The tests are passing if I modified a
> little bit here :
>
> if (classes.size() > 0 || files.size() > 0)
> Not sure if this is a problem, cause I'm not familiar with Pack200 and
> Apache Harmony.
>
> Lee
> On 8 23 2020, at 11:27 , Gary Gregory wrote:
> > Hi All,
> >
> > I created a branch called *pack200* which contains the Apache Harmony
> > Pack200 code.
> >
> > If there are 2 failing unit test ATM if anyone wants to help out.
> > Gary
Re: [COMPRESS] Pack200 support in pack200 branch
After some debugging I found this at org.apache.harmony.pack200.Archive#171 :
if (classes.size() > 0 && files.size() > 0) {
segmentUnitList.add(new SegmentUnit(classes, files));
}
Seems the Pack200 implementation in harmony requires existing of both classes
AND files at the same time. The tests are passing if I modified a little bit
here :
if (classes.size() > 0 || files.size() > 0)
Not sure if this is a problem, cause I'm not familiar with Pack200 and Apache
Harmony.
Lee
On 8 23 2020, at 11:27 , Gary Gregory wrote:
> Hi All,
>
> I created a branch called *pack200* which contains the Apache Harmony
> Pack200 code.
>
> If there are 2 failing unit test ATM if anyone wants to help out.
> Gary
[COMPRESS] Pack200 support in pack200 branch
Hi All, I created a branch called *pack200* which contains the Apache Harmony Pack200 code. If there are 2 failing unit test ATM if anyone wants to help out. Gary
Re: [compress] Pack200
On 2011-09-14, Emmanuel Bourg wrote: Le 14/09/2011 17:16, Stefan Bodewig a écrit : Emmanuel, is the Pack200Utils.normalize method I've just committed what you've been looking for? Yes that's what I had in mind, thank you. You might also want to support repacking a file to itself, that's a common use case. This should be possible by using the same File instance for both arguments to normalize. Maybe I should add an alternative method signature with a sngle File argument (and likely methods without a Map argument as well). I implemented this in the pack200 Ant tasks: https://svn.java.net/svn/deployment~svn/trunk/ant-tasks/pack200/src/main/java/org/jdesktop/deployment/ant/pack200/PackUtils.java Yes, looks pretty similar to what I have, I may want to look at my error handling a little coser, though. 8-) BTW, I've added pack200 support to the Compress Antlib (tasks and a pack200resource) and will push for a release once CC 1.3 is out. Thanks Stefan - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [compress] Pack200
On 2011-09-15, Emmanuel Bourg wrote: Le 15/09/2011 11:46, Stefan Bodewig a écrit : BTW, I've added pack200 support to the Compress Antlib (tasks and a pack200resource) and will push for a release once CC 1.3 is out. Pack200 tasks available out of the box with Ant would be really nice, The Compress Antlib is separate from the Ant distribution. http://ant.apache.org/antlibs/compress/index.html which doesn't say much. Current trunk has been following Commons Compress closely and thus already support Unix dump (undump, actually) and Pack200. Stefan - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [compress] Pack200
On 2011-09-05, Stefan Bodewig wrote: On 2011-09-05, Emmanuel Bourg wrote: That looks interesting. Does it provide a repack mode suitable for signing compressed jars? I assume you mean , | Note that packing and unpacking a JAR will in general alter the bytewise | contents of classfiles in the JAR. This means that packing and unpacking | will in general invalidate any digital signatures which rely on bytewise | images of JAR elements. In order both to sign and to pack a JAR, you | must first pack and unpack the JAR to normalize it, then compute | signatures on the unpacked JAR elements, and finally repack the signed | JAR. Both packing steps should use precisely the same options, and the | segment limit may also need to be set to -1, to prevent accidental | variation of segment boundaries as class file sizes change slightly. ` as in http://download.oracle.com/javase/1.5.0/docs/api/java/util/jar/Pack200.Packer.html I think the normalization step could be part of a util class inside the package. Some utility method that takes a JAR, packs it and unpacks it again to a new jar that would then need to get signed. Signing itself seems outside of compress' scope to me. Emmanuel, is the Pack200Utils.normalize method I've just committed what you've been looking for? Stefan - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [compress] Pack200
On 2011-09-05, Stefan Bodewig wrote: On 2011-09-05, Emmanuel Bourg wrote: That looks interesting. Does it provide a repack mode suitable for signing compressed jars? I assume you mean , | Note that packing and unpacking a JAR will in general alter the bytewise | contents of classfiles in the JAR. This means that packing and unpacking | will in general invalidate any digital signatures which rely on bytewise | images of JAR elements. In order both to sign and to pack a JAR, you | must first pack and unpack the JAR to normalize it, then compute | signatures on the unpacked JAR elements, and finally repack the signed | JAR. Both packing steps should use precisely the same options, and the | segment limit may also need to be set to -1, to prevent accidental | variation of segment boundaries as class file sizes change slightly. ` as in http://download.oracle.com/javase/1.5.0/docs/api/java/util/jar/Pack200.Packer.html I think the normalization step could be part of a util class inside the package. Some utility method that takes a JAR, packs it and unpacks it again to a new jar that would then need to get signed. Signing itself seems outside of compress' scope to me. Emmanuel, is the Pack200Utils.normalize method I've just committed what you've been looking for? Stefan - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [compress] Pack200
That looks interesting. Does it provide a repack mode suitable for signing compressed jars? Emmanuel Bourg Le 04/09/2011 08:04, Stefan Bodewig a écrit : Hi, I've just committed Converter*Stream implementations for Pack200[1] which is a bit unusual in several ways. First of all it will (by design of the format) only work on compressing valid jar files. Actually the result isn't likely to be compressed (in the sense of smaller than the original) at all but expects another step of GZip compression in most cases. The second difference to the other compressors is that the API provided by the Java classlib doesn't lend itself to streaming at all. There is a Packer/Unpacker that expects an InputStream and an OutputStream and converts from one to the other in a single blocking operation (even closing the input side when done). I have experimented with Piped*Streams as well as Ant/commons-exec-like stream pumping in order to provide a streaming experience but always ran into some edge cases where things broke down. I'll give one example below. The current implementation of Pack200CompressorInputStream will pass the wrapped input and an OutputStream writing to a cache to the Unpacker synchronously inside the constructor, consuming the input completely. It will then defer all read-operations to the cache. Likewise the Pack200CompressorOutputStream will buffer up all write operations in a cache and once finish() or close() is called the cache is converted to an InputStream that is then passed together with the originally wrapped output to the Packer and written synchronously. Caches can be in-memory (using ByteArray*Stream) or temporary files controlled by a constructor option with in-memory as the default and temp-files for cases where the archives are expected to be big. Because of this design the byte-count methods don't make any sense (do we count when data is written-to/read-from the cache or while the (Un)Packer is doing its work?) and haven't been implemented at all. The class names StreamMode and StreamSwitcher result from my attempts of using real streams and should be changed unless anybody else comes up with a working streaming solution. The biggest hurdle for any streaming solution is that there is always going to be some sort of intermediate buffer. Something picks up data written to the output stream and makes it available to the input stream side. Once the buffer is full, nothing can be written unless anybody reads input in a timely manner. In the case of a Pack200CompressorInputStream you don't have any control over when the user code is going to read the data and whether it is going to consume all of it at all. For example if the stream is wrapped in a ZipArchiveInputStream (it represents a JAR, after all), it is never going to be consumed completely because the archive contains ZIP data at the end that is ignored by the input stream implementation. There are more cases where the Pack/Unpack operation would end up blocked so I decided to only code the more robust indirect solution for now. Stefan [1] http://download.oracle.com/javase/1.5.0/docs/api/java/util/jar/Pack200.html - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [compress] Pack200
On 2011-09-04, sebb wrote: You must have put a lot of thought into this Well, I was without network access for almost two weeks 8-) it would be useful to record these design decisions and investigations in the code somewhere. e.g. as package Javadoc. The current package.html already contains the how but not the why, I'll make sure the rest is persisted soemwhere as well. Stefan - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [compress] Pack200
On 2011-09-05, Emmanuel Bourg wrote: That looks interesting. Does it provide a repack mode suitable for signing compressed jars? I assume you mean , | Note that packing and unpacking a JAR will in general alter the bytewise | contents of classfiles in the JAR. This means that packing and unpacking | will in general invalidate any digital signatures which rely on bytewise | images of JAR elements. In order both to sign and to pack a JAR, you | must first pack and unpack the JAR to normalize it, then compute | signatures on the unpacked JAR elements, and finally repack the signed | JAR. Both packing steps should use precisely the same options, and the | segment limit may also need to be set to -1, to prevent accidental | variation of segment boundaries as class file sizes change slightly. ` as in http://download.oracle.com/javase/1.5.0/docs/api/java/util/jar/Pack200.Packer.html I think the normalization step could be part of a util class inside the package. Some utility method that takes a JAR, packs it and unpacks it again to a new jar that would then need to get signed. Signing itself seems outside of compress' scope to me. Stefan - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
[compress] Pack200
Hi, I've just committed Converter*Stream implementations for Pack200[1] which is a bit unusual in several ways. First of all it will (by design of the format) only work on compressing valid jar files. Actually the result isn't likely to be compressed (in the sense of smaller than the original) at all but expects another step of GZip compression in most cases. The second difference to the other compressors is that the API provided by the Java classlib doesn't lend itself to streaming at all. There is a Packer/Unpacker that expects an InputStream and an OutputStream and converts from one to the other in a single blocking operation (even closing the input side when done). I have experimented with Piped*Streams as well as Ant/commons-exec-like stream pumping in order to provide a streaming experience but always ran into some edge cases where things broke down. I'll give one example below. The current implementation of Pack200CompressorInputStream will pass the wrapped input and an OutputStream writing to a cache to the Unpacker synchronously inside the constructor, consuming the input completely. It will then defer all read-operations to the cache. Likewise the Pack200CompressorOutputStream will buffer up all write operations in a cache and once finish() or close() is called the cache is converted to an InputStream that is then passed together with the originally wrapped output to the Packer and written synchronously. Caches can be in-memory (using ByteArray*Stream) or temporary files controlled by a constructor option with in-memory as the default and temp-files for cases where the archives are expected to be big. Because of this design the byte-count methods don't make any sense (do we count when data is written-to/read-from the cache or while the (Un)Packer is doing its work?) and haven't been implemented at all. The class names StreamMode and StreamSwitcher result from my attempts of using real streams and should be changed unless anybody else comes up with a working streaming solution. The biggest hurdle for any streaming solution is that there is always going to be some sort of intermediate buffer. Something picks up data written to the output stream and makes it available to the input stream side. Once the buffer is full, nothing can be written unless anybody reads input in a timely manner. In the case of a Pack200CompressorInputStream you don't have any control over when the user code is going to read the data and whether it is going to consume all of it at all. For example if the stream is wrapped in a ZipArchiveInputStream (it represents a JAR, after all), it is never going to be consumed completely because the archive contains ZIP data at the end that is ignored by the input stream implementation. There are more cases where the Pack/Unpack operation would end up blocked so I decided to only code the more robust indirect solution for now. Stefan [1] http://download.oracle.com/javase/1.5.0/docs/api/java/util/jar/Pack200.html - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [compress] Pack200
On 4 September 2011 07:04, Stefan Bodewig bode...@apache.org wrote: Hi, I've just committed Converter*Stream implementations for Pack200[1] which is a bit unusual in several ways. First of all it will (by design of the format) only work on compressing valid jar files. Actually the result isn't likely to be compressed (in the sense of smaller than the original) at all but expects another step of GZip compression in most cases. The second difference to the other compressors is that the API provided by the Java classlib doesn't lend itself to streaming at all. There is a Packer/Unpacker that expects an InputStream and an OutputStream and converts from one to the other in a single blocking operation (even closing the input side when done). I have experimented with Piped*Streams as well as Ant/commons-exec-like stream pumping in order to provide a streaming experience but always ran into some edge cases where things broke down. I'll give one example below. The current implementation of Pack200CompressorInputStream will pass the wrapped input and an OutputStream writing to a cache to the Unpacker synchronously inside the constructor, consuming the input completely. It will then defer all read-operations to the cache. Likewise the Pack200CompressorOutputStream will buffer up all write operations in a cache and once finish() or close() is called the cache is converted to an InputStream that is then passed together with the originally wrapped output to the Packer and written synchronously. Caches can be in-memory (using ByteArray*Stream) or temporary files controlled by a constructor option with in-memory as the default and temp-files for cases where the archives are expected to be big. Because of this design the byte-count methods don't make any sense (do we count when data is written-to/read-from the cache or while the (Un)Packer is doing its work?) and haven't been implemented at all. The class names StreamMode and StreamSwitcher result from my attempts of using real streams and should be changed unless anybody else comes up with a working streaming solution. The biggest hurdle for any streaming solution is that there is always going to be some sort of intermediate buffer. Something picks up data written to the output stream and makes it available to the input stream side. Once the buffer is full, nothing can be written unless anybody reads input in a timely manner. In the case of a Pack200CompressorInputStream you don't have any control over when the user code is going to read the data and whether it is going to consume all of it at all. For example if the stream is wrapped in a ZipArchiveInputStream (it represents a JAR, after all), it is never going to be consumed completely because the archive contains ZIP data at the end that is ignored by the input stream implementation. There are more cases where the Pack/Unpack operation would end up blocked so I decided to only code the more robust indirect solution for now. Sounds good. You must have put a lot of thought into this; it would be useful to record these design decisions and investigations in the code somewhere. e.g. as package Javadoc. Stefan [1] http://download.oracle.com/javase/1.5.0/docs/api/java/util/jar/Pack200.html - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
