Bug#634089: jh_manifest causes CRC error
+++ tony mancill [2012-01-06 13:26 -0800]: Thank you for in-depth look into this bug. Niels or Matthew, any concerns with me preparing an upload? A fix for libarchive-zip-perl has now been filed, so I guess we should wait for an upload at which point this bug should evaporate and can just be closed. Debian: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=654899 Upstream: https://rt.cpan.org/Public/Bug/Display.html?id=73797 Wookey -- Principal hats: Linaro, Emdebian, Wookware, Balloonboard, ARM http://wookware.org/ __ This is the maintainer address of Debian's Java team http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-java-maintainers. Please use debian-j...@lists.debian.org for discussions and questions.
Bug#634089: jh_manifest causes CRC error
On Fri Jan 06 13:26, tony mancill wrote: Thank you for in-depth look into this bug. Niels or Matthew, any concerns with me preparing an upload? Go right ahead Thanks, Matt Thanks, tony On 01/05/2012 08:14 PM, Wookey wrote: This seems to me to be a serious problem blocking all java-helper-using packages So, I took a look at what's going on and found the following: --snip-- We can fix it by explicitly removing and recreating it: verbose_print(Updating manifest in $jar); $zip-removeMember( 'META-INF/MANIFEST.MF' ) unless($new_manifest); $zip-removeMember( 'META-INF/' ); $mem = $zip-addString($var, 'META-INF/MANIFEST.MF'); $mem-desiredCompressionMethod(COMPRESSION_DEFLATED); $zip-addDirectory( 'META-INF/' ); # This on the other hand may fail. $zip-overwrite() == AZ_OK or error(Writing modified jar ($jar) failed$ which I hope is an acceptable fix. Patch attached. signature.asc Description: Digital signature __ This is the maintainer address of Debian's Java team http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-java-maintainers. Please use debian-j...@lists.debian.org for discussions and questions.
Bug#634089: jh_manifest causes CRC error
+++ tony mancill [2012-01-06 13:26 -0800]: On 01/05/2012 08:14 PM, Wookey wrote: Thank you for in-depth look into this bug. Niels or Matthew, any concerns with me preparing an upload? This seems to me to be a serious problem blocking all java-helper-using packages So, I took a look at what's going on and found the following: --snip-- We can fix it by explicitly removing and recreating it: verbose_print(Updating manifest in $jar); $zip-removeMember( 'META-INF/MANIFEST.MF' ) unless($new_manifest); $zip-removeMember( 'META-INF/' ); $mem = $zip-addString($var, 'META-INF/MANIFEST.MF'); $mem-desiredCompressionMethod(COMPRESSION_DEFLATED); $zip-addDirectory( 'META-INF/' ); # This on the other hand may fail. $zip-overwrite() == AZ_OK or error(Writing modified jar ($jar) failed$ which I hope is an acceptable fix. Patch attached. Having thought about it a bit, I'm not sure what happens with the above workaround if there _isn't_ a META-INF/ dir already present (does that ever happen)? Does the dir member entry end up after the file member? Is that a problem in zipfiles? And what if ther META-INF dir has other files in it (does that happen with java-helper? - will doing this break that? - I'm not sure what filesystem semantics Archive::Zip imposes)? It might be a good idea to test this on some more packages than just terraintool. And in the meantime the perl people are taking some interest in the real bug, which I believe is in Archive::Zip, not jh_manifest, so as we've already waited several months another week or so to see if that gets fixed is probably in order before uploading this workaround in jh_manifest. There are 227 packages which build-depend on javahelper, and presumably all/mostly use jh_manifest so a fix of some sort is quite urgent. Wookey -- Principal hats: Linaro, Emdebian, Wookware, Balloonboard, ARM http://wookware.org/ __ This is the maintainer address of Debian's Java team http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-java-maintainers. Please use debian-j...@lists.debian.org for discussions and questions.
Bug#634089: jh_manifest causes CRC error
Thank you for in-depth look into this bug. Niels or Matthew, any concerns with me preparing an upload? Thanks, tony On 01/05/2012 08:14 PM, Wookey wrote: This seems to me to be a serious problem blocking all java-helper-using packages So, I took a look at what's going on and found the following: --snip-- We can fix it by explicitly removing and recreating it: verbose_print(Updating manifest in $jar); $zip-removeMember( 'META-INF/MANIFEST.MF' ) unless($new_manifest); $zip-removeMember( 'META-INF/' ); $mem = $zip-addString($var, 'META-INF/MANIFEST.MF'); $mem-desiredCompressionMethod(COMPRESSION_DEFLATED); $zip-addDirectory( 'META-INF/' ); # This on the other hand may fail. $zip-overwrite() == AZ_OK or error(Writing modified jar ($jar) failed$ which I hope is an acceptable fix. Patch attached. signature.asc Description: OpenPGP digital signature __ This is the maintainer address of Debian's Java team http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-java-maintainers. Please use debian-j...@lists.debian.org for discussions and questions.
Bug#634089: jh_manifest causes CRC error
This seems to me to be a serious problem blocking all
java-helper-using packages
So, I took a look at what's going on and found the following:
The short version is that Archive::Zip is misbehaving as jh_manifest
uses it and making corrupt jars. A patch to workaround that is
attached. It may be that a fix in Archive::Zip is a better solution.
As it took me a while to get there I'll leave my working in in case it
helps anyone else working on the issue:
If I build the package in squeeze everything is fine.
If I build it in wheezy then running it:
terraintool
gives CRC error.
Looking a little more closely. This is OK:
java -jar /usr/share/terraintool/terraintool.jar
It seems that what the binfmt support does is run this:
jarwrapper /usr/share/terraintool/terraintool.jar
That give CRC error.
jarwrapper is just a shell script. The bit that dies is this:
fastjar xf /usr/share/terraintool/terraintool.jar META-INF/MANIFEST.MF
Niels Thykier suggested seeing if LD_PREloading
it was sugested to see if replacing memcpy with memmove using
LD_PRELoAD fixed the situation. I found that you do that like this:
Create a file named memcpy_hider.c containing
#include string.h
void* memcpy(void *dst, const void *src, size_t size)
{
memmove(dst,src,size);
}
Then compile it with
gcc -fPIC -O2 -c memcpy_hider.c
ld -G memcpy_hider.o -o memcpy_hider.so
Finally, to test it out, run your browser like
LD_PRELOAD=/abolute/path/to/memcpy_hider.so program
So that built OK. I ran:
LD_PRELOAD=/home/wookey/debian/wheezy/memcpy_hider.so fastjar xf
/usr/share/terraintool/terraintool.jar META-INF/MANIFEST.MF
and it still gives a CRC error :-(
So either I've done something wrong or the memcpy thing is not the
problem here?
This suggests that the _hider lib is being used
LD_PRELOAD=/home/wookey/debian/wheezy/memcpy_hider.so ldd /usr/bin/fastjar
linux-vdso.so.1 = (0x7fffc55ff000)
/home/wookey/debian/wheezy/memcpy_hider.so (0x7f8819ebf000)
libz.so.1 = /lib/libz.so.1 (0x7f8819c81000)
libc.so.6 = /lib/x86_64-linux-gnu/libc.so.6 (0x7f88198fc000)
/lib64/ld-linux-x86-64.so.2 (0x7f881a0c2000)
stracing the fastjar command shows it falling over as soon as it tries
to write some output when unpacking the jar
open(/usr/share/terraintool/terraintool.jar, O_RDONLY) = 3
read(3, PK\3\4, 4)= 4
read(3, \24\0\0\0\0\0S\217%@\0\0\0\0\2\0\0\0\0\0\0\0\t\0\4\0, 26) = 26
read(3, META-INF/, 9) = 9
lseek(3, 4, SEEK_CUR) = 43
read(3, PK, 2)= 2
write(2, Error! CRCs do not match! Got 1a..., 51Error! CRCs do not match! Got
1a6cd7b3, expected 0 ) = 51
So it's opened the jar, found the PKZIp header, found the META-INF
directory entry, skipped 4 bytes, found PK again and died.
Some perusal of the PKZIP header spec:
http://www.pkware.com/documents/casestudies/APPNOTE.TXT
od -tx1 -c /usr/share/terraintool/terraintool.jar |less
show that that header contians:
local file header signature 4 bytes (0x04034b50)
version needed to extract 2 bytes
general purpose bit flag2 bytes
compression method 2 bytes
last mod file time 2 bytes
last mod file date 2 bytes
crc-32 4 bytes
compressed size 4 bytes
uncompressed size 4 bytes
file name length2 bytes
extra field length 2 bytes
file name (variable size)
extra field (variable size)
B. File data
Immediately following the local header for a file
is the compressed or stored data for the file.
000 50 4b 03 04 14 00 00 00 00 00 53 8f 25 40 00 00
P K 003 004 024 \0 \0 \0 \0 \0 S 217 % @ \0 \0
020 00 00 02 00 00 00 00 00 00 00 09 00 04 00 4d 45
\0 \0 002 \0 \0 \0 \0 \0 \0 \0 \t \0 004 \0 M E
040 54 41 2d 49 4e 46 2f fe ca 00 00 50 4b 03 04 14
T A - I N F / 376 312 \0 \0 P K 003 004 024
This seems to say that this entry has uncompressed size 0 and
compressed size 2 - is that right?
and this is a header for another directory:
50 4b 03 04 0a 00 00 00 00 00
P K 003 004 \n \0 \0 \0 \0 \0
0002120 4e 8f 25 40 00 00 00 00 00 00 00 00 00 00 00 00
N 217 % @ \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0002140 08 00 00 00 6d 63 63 6f 6d 62 65 2f 50 4b 03 04
\b \0 \0 \0 m c c o m b e /
this has size 0 for both compressed and uncompressed.
doing unzip -t /usr/share/terraintool/terraintool.jar
Archive: /usr/share/terraintool/terraintool.jar
META-INF/: ucsize 0 csize 2 for STORED entry
continuing with compressed size value
Bug#634089: jh_manifest causes CRC error
For the record here is the E-Mail traffic on Debian-Java list (it seems, that there is a problem with jh_manifest) Now, I can reproduce the error when doing the following: * using debian/javabuild with: javatut.jar src * using debian/manifest with: usr/share/javatut/javatut.jar: Main-Class: net.zabuchy.Main But as soon as I remove debian/javabuild and debian/manifest and build with debian/rules: %: dh $@ --with javahelper override_dh_auto_build: jh_build -mnet.zabuchy.Main javatut.jar src everything works just fine. Any thoughts? Tomek Does it come back if you use jh_manifest to set the main class? Yes, jh_manifest is to blame: % ./javatut.jar Hello world % jh_manifest -cnet.zabuchy.Main javatut.jar % ./javatut.jar Error! CRCs do not match! Got 1a6cd7b3, expected 0 Tomek --- __ This is the maintainer address of Debian's Java team http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-java-maintainers. Please use debian-j...@lists.debian.org for discussions and questions.
