Hello. I've run into a strange but easily reproduced problem with the jar files generated by Maven. Essentially, if I generate a jar file containing a large number of files (>= 65536, in practice), then javac becomes unable to resolve classes from that jar file. This only occurs with jars produced by the Maven jar plugin, and only when the number of files is large (as demonstrated below).
An example build, using the maven exec plugin to generate a large number (65525) of files: http://waste.io7m.com/2013/11/12/jarbug.zip $ mvn -C clean verify [INFO] Scanning for projects... [INFO] ------------------------------------------------------------------------ [INFO] Reactor Build Order: [INFO] [INFO] resources [INFO] code [INFO] jarbug [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building resources 0.1.0 [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ resources --- [INFO] Deleting /home/m0/doc/dev/2013/11/jarbug/resources/target [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ resources --- [WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent! [INFO] skip non existing resourceDirectory /home/m0/doc/dev/2013/11/jarbug/resources/src/main/resources [INFO] [INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ resources --- [WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent! [INFO] Compiling 1 source file to /home/m0/doc/dev/2013/11/jarbug/resources/target/classes [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ resources --- [WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent! [INFO] skip non existing resourceDirectory /home/m0/doc/dev/2013/11/jarbug/resources/src/test/resources [INFO] [INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ resources --- [INFO] No sources to compile [INFO] [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ resources --- [INFO] No tests to run. [INFO] [INFO] >>> exec-maven-plugin:1.2.1:java (make-sources) @ resources >>> [INFO] [INFO] <<< exec-maven-plugin:1.2.1:java (make-sources) @ resources <<< [INFO] [INFO] --- exec-maven-plugin:1.2.1:java (make-sources) @ resources --- Create /home/m0/doc/dev/2013/11/jarbug/resources/target/classes/com/io7m/jarbug/things/0.txt Create /home/m0/doc/dev/2013/11/jarbug/resources/target/classes/com/io7m/jarbug/things/1.txt Create /home/m0/doc/dev/2013/11/jarbug/resources/target/classes/com/io7m/jarbug/things/2.txt ... Create /home/m0/doc/dev/2013/11/jarbug/resources/target/classes/com/io7m/jarbug/things/65522.txt Create /home/m0/doc/dev/2013/11/jarbug/resources/target/classes/com/io7m/jarbug/things/65523.txt Create /home/m0/doc/dev/2013/11/jarbug/resources/target/classes/com/io7m/jarbug/things/65524.txt [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ resources --- [INFO] Building jar: /home/m0/doc/dev/2013/11/jarbug/resources/target/resources-0.1.0.jar [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building code 0.1.0 [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ code --- [INFO] Deleting /home/m0/doc/dev/2013/11/jarbug/code/target [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ code --- [WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent! [INFO] skip non existing resourceDirectory /home/m0/doc/dev/2013/11/jarbug/code/src/main/resources [INFO] [INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ code --- [WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent! [INFO] Compiling 1 source file to /home/m0/doc/dev/2013/11/jarbug/code/target/classes [INFO] ------------------------------------------------------------- [ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /home/m0/doc/dev/2013/11/jarbug/code/src/main/java/com/io7m/jarbug/Main.java:[8,23] error: cannot find symbol [INFO] 1 error [INFO] ------------------------------------------------------------- [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] resources ......................................... SUCCESS [30.178s] [INFO] code .............................................. FAILURE [0.062s] [INFO] jarbug ............................................ SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 30.356s [INFO] Finished at: Tue Nov 12 12:32:47 UTC 2013 [INFO] Final Memory: 14M/274M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile (default-compile) on project code: Compilation failure [ERROR] /home/m0/doc/dev/2013/11/jarbug/code/src/main/java/com/io7m/jarbug/Main.java:[8,23] error: cannot find symbol [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :code From the command line: $ cat code/src/main/java/com/io7m/jarbug/Main.java package com.io7m.jarbug; public final class Main { public static void main( final String args[]) { System.out.println(MakeSources.class); } } $ javac -classpath resources/target/resources-0.1.0.jar code/src/main/java/com/io7m/jarbug/Main.java code/src/main/java/com/io7m/jarbug/Main.java:8: error: cannot find symbol System.out.println(MakeSources.class); ^ symbol: class MakeSources location: class Main 1 error $ unzip -l resources/target/resources-0.1.0.jar | grep MakeSources 1489 2013-11-12 12:32 com/io7m/jarbug/MakeSources.class $ unzip -l resources/target/resources-0.1.0.jar | wc -l 65542 The numbers given above are actually five more than the number of files, because the unzip command line tool produces five extra lines in the output. So resources-0.1.0.jar really contains 65537 files. However... $ pushd resources/target/classes $ javac ../../../code/src/main/java/com/io7m/jarbug/Main.java So it works fine if the classes and files are not in a jar file. Also: $ pushd resources/target/classes $ zip -r ../../../new.zip . $ popd $ javac -classpath new.zip code/src/main/java/com/io7m/jarbug/Main.java Although the new.zip has fewer files (65530) than the resources-0.1.0.jar, because the various metadata files that Maven inserts upon jar creation is not present: $ unzip -l new.zip | wc -l 65535 Increasing the number of files by 10, and recreating new.zip: $ rm new.zip $ mvn -C clean verify ... Create /home/m0/doc/dev/2013/11/jarbug/resources/target/classes/com/io7m/jarbug/things/65532.txt Create /home/m0/doc/dev/2013/11/jarbug/resources/target/classes/com/io7m/jarbug/things/65533.txt Create /home/m0/doc/dev/2013/11/jarbug/resources/target/classes/com/io7m/jarbug/things/65534.txt ... $ pushd resources/target/classes $ zip -r ../../../new.zip . $ popd $ unzip -l new.zip | wc -l 65545 $ javac -classpath new.zip code/src/main/java/com/io7m/jarbug/Main.java No error is reported, the program compiles correctly. Let's see if the 'jar' program can produce a working jar file: $ jar cf new.jar -C resources/target/classes . $ unzip -l new.jar | wc -l 65547 $ javac -classpath new.jar code/src/main/java/com/io7m/jarbug/Main.java Again, no issues. So what is it that the Maven jar plugin is doing that's producing apparently broken jar files? $ unzip -l new.jar | awk '{print $NF}' > new.jar.list $ unzip -l resources/target/resources-0.1.0.jar | awk '{print $NF}' > resources.jar.list $ diff -u new.jar.list resources.jar.list --- new.jar.list 2013-11-12 12:58:16.491221126 +0000 +++ resources.jar.list 2013-11-12 12:58:39.159215571 +0000 @@ -1,4 +1,4 @@ -new.jar +resources/target/resources-0.1.0.jar Name ---- META-INF/ @@ -65543,5 +65543,10 @@ com/io7m/jarbug/things/8006.txt com/io7m/jarbug/things/4304.txt com/io7m/jarbug/MakeSources.class +META-INF/maven/ +META-INF/maven/com.io7m.jarbug/ +META-INF/maven/com.io7m.jarbug/resources/ +META-INF/maven/com.io7m.jarbug/resources/pom.xml +META-INF/maven/com.io7m.jarbug/resources/pom.properties ------- files So the only thing that really differs is that the Maven jar contains maven metadata in the META-INF directory. If anyone has any suggestions as to why this should cause the above insane behaviour, I'd be glad to hear them! M --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
