[ https://issues.apache.org/jira/browse/YARN-8430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Marcelo Vanzin moved SPARK-24559 to YARN-8430: ---------------------------------------------- Affects Version/s: (was: 2.2.0) Component/s: (was: Spark Submit) Workflow: no-reopen-closed, patch-avail (was: no-reopen-closed) Key: YARN-8430 (was: SPARK-24559) Project: Hadoop YARN (was: Spark) > Some zip files passed with spark-submit --archives causing "invalid CEN > header" error > ------------------------------------------------------------------------------------- > > Key: YARN-8430 > URL: https://issues.apache.org/jira/browse/YARN-8430 > Project: Hadoop YARN > Issue Type: Bug > Reporter: James Porritt > Priority: Major > > I'm encountering an error when submitting some zip files to spark-submit > using --archive that are over 2Gb and have the zip64 flag set. > {{PYSPARK_PYTHON=./ROOT/myspark/bin/python > /usr/hdp/current/spark2-client/bin/spark-submit \}} > {{ --conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=./ROOT/myspark/bin/python \}} > {{ --master=yarn \}} > {{ --deploy-mode=cluster \}} > {{ --driver-memory=4g \}} > {{ --archives=myspark.zip#ROOT \}} > {{ --num-executors=32 \}} > {{ --packages com.databricks:spark-avro_2.11:4.0.0 \}} > {{ foo.py}} > (As a bit of background, I'm trying to prepare files using the trick of > zipping a conda environment and passing the zip file via --archives, as per: > https://community.hortonworks.com/articles/58418/running-pyspark-with-conda-env.html) > myspark.zip is a zipped conda environment. It was created using python with > the zipfile pacakge. The files are stored without deflation and with the > zip64 flag set. foo.py is my application code. This normally works, but if > myspark.zip is greater than 2Gb and has the zip64 flag set I get: > java.util.zip.ZipException: invalid CEN header (bad signature) > There seems to be much written on the subject, and I was able to write Java > code that utilises the java.util.zip library that both does and doesn't > encounter this error for one of the problematic zip files. > Spark compile info: > {{Welcome to}} > {{ ____ __}} > {{ / __/__ ___ _____/ /__}} > {{ _\ \/ _ \/ _ `/ __/ '_/}} > {{ /___/ .__/\_,_/_/ /_/\_\ version 2.2.0.2.6.4.0-91}} > {{ /_/}} > {{Using Scala version 2.11.8, Java HotSpot(TM) 64-Bit Server VM, 1.8.0_112}} > {{Branch HEAD}} > {{Compiled by user jenkins on 2018-01-04T10:41:05Z}} > {{Revision a24017869f5450397136ee8b11be818e7cd3facb}} > {{Url g...@github.com:hortonworks/spark2.git}} > {{Type --help for more information.}} > YARN logs on console after above command. I've tried both > --deploy-mode=cluster and --deploy-mode=client. > {{18/06/13 16:00:22 WARN NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable}} > {{18/06/13 16:00:23 WARN DomainSocketFactory: The short-circuit local reads > feature cannot be used because libhadoop cannot be loaded.}} > {{18/06/13 16:00:23 INFO RMProxy: Connecting to ResourceManager at > myhost2.myfirm.com/10.87.11.17:8050}} > {{18/06/13 16:00:23 INFO Client: Requesting a new application from cluster > with 6 NodeManagers}} > {{18/06/13 16:00:23 INFO Client: Verifying our application has not requested > more than the maximum memory capability of the cluster (221184 MB per > container)}} > {{18/06/13 16:00:23 INFO Client: Will allocate AM container, with 18022 MB > memory including 1638 MB overhead}} > {{18/06/13 16:00:23 INFO Client: Setting up container launch context for our > AM}} > {{18/06/13 16:00:23 INFO Client: Setting up the launch environment for our AM > container}} > {{18/06/13 16:00:23 INFO Client: Preparing resources for our AM container}} > {{18/06/13 16:00:24 INFO Client: Use hdfs cache file as spark.yarn.archive > for HDP, > hdfsCacheFile:hdfs://myhost.myfirm.com:8020/hdp/apps/2.6.4.0-91/spark2/spark2-hdp-yarn-archive.tar.gz}} > {{18/06/13 16:00:24 INFO Client: Source and destination file systems are the > same. Not copying > hdfs://myhost.myfirm.com:8020/hdp/apps/2.6.4.0-91/spark2/spark2-hdp-yarn-archive.tar.gz}} > {{18/06/13 16:00:24 INFO Client: Uploading resource > file:/home/myuser/.ivy2/jars/com.databricks_spark-avro_2.11-4.0.0.jar -> > hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/com.databri}} > {{cks_spark-avro_2.11-4.0.0.jar}} > {{18/06/13 16:00:26 INFO Client: Uploading resource > file:/home/myuser/.ivy2/jars/org.slf4j_slf4j-api-1.7.5.jar -> > hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/org.slf4j_slf4j-api-1.}} > {{7.5.jar}} > {{18/06/13 16:00:26 INFO Client: Uploading resource > file:/home/myuser/.ivy2/jars/org.apache.avro_avro-1.7.6.jar -> > hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/org.apache.avro_avro-}} > {{1.7.6.jar}} > {{18/06/13 16:00:26 INFO Client: Uploading resource > file:/home/myuser/.ivy2/jars/org.codehaus.jackson_jackson-core-asl-1.9.13.jar > -> > hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/org}} > {{.codehaus.jackson_jackson-core-asl-1.9.13.jar}} > {{18/06/13 16:00:26 INFO Client: Uploading resource > file:/home/myuser/.ivy2/jars/org.codehaus.jackson_jackson-mapper-asl-1.9.13.jar > -> > hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/o}} > {{rg.codehaus.jackson_jackson-mapper-asl-1.9.13.jar}} > {{18/06/13 16:00:26 INFO Client: Uploading resource > file:/home/myuser/.ivy2/jars/com.thoughtworks.paranamer_paranamer-2.3.jar -> > hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/com.tho}} > {{ughtworks.paranamer_paranamer-2.3.jar}} > {{18/06/13 16:00:26 INFO Client: Uploading resource > file:/home/myuser/.ivy2/jars/org.xerial.snappy_snappy-java-1.0.5.jar -> > hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/org.xerial.s}} > {{nappy_snappy-java-1.0.5.jar}} > {{18/06/13 16:00:26 INFO Client: Uploading resource > file:/home/myuser/.ivy2/jars/org.apache.commons_commons-compress-1.4.1.jar -> > hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/org.ap}} > {{ache.commons_commons-compress-1.4.1.jar}} > {{18/06/13 16:00:26 INFO Client: Uploading resource > file:/home/myuser/.ivy2/jars/org.tukaani_xz-1.0.jar -> > hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/org.tukaani_xz-1.0.jar}} > {{18/06/13 16:00:26 INFO Client: Source and destination file systems are the > same. Not copying hdfs:/user/myuser/release/alphagenspark.zip#ROOT}} > {{18/06/13 16:00:26 INFO Client: Uploading resource > file:/my/script/dir/spark/alphagen/foo.py -> > hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/foo.py}} > {{18/06/13 16:00:26 INFO Client: Uploading resource > file:/usr/hdp/current/spark2-client/python/lib/pyspark.zip -> > hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/pyspark.zip}} > {{18/06/13 16:00:26 INFO Client: Uploading resource > file:/usr/hdp/current/spark2-client/python/lib/py4j-0.10.4-src.zip -> > hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/py4j-0.10.4-src}} > {{.zip}} > {{18/06/13 16:00:26 WARN Client: Same path resource > file:/home/myuser/.ivy2/jars/com.databricks_spark-avro_2.11-4.0.0.jar added > multiple times to distributed cache.}} > {{18/06/13 16:00:26 WARN Client: Same path resource > file:/home/myuser/.ivy2/jars/org.slf4j_slf4j-api-1.7.5.jar added multiple > times to distributed cache.}} > {{18/06/13 16:00:26 WARN Client: Same path resource > file:/home/myuser/.ivy2/jars/org.apache.avro_avro-1.7.6.jar added multiple > times to distributed cache.}} > {{18/06/13 16:00:26 WARN Client: Same path resource > file:/home/myuser/.ivy2/jars/org.codehaus.jackson_jackson-core-asl-1.9.13.jar > added multiple times to distributed cache.}} > {{18/06/13 16:00:26 WARN Client: Same path resource > file:/home/myuser/.ivy2/jars/org.codehaus.jackson_jackson-mapper-asl-1.9.13.jar > added multiple times to distributed cache.}} > {{18/06/13 16:00:26 WARN Client: Same path resource > file:/home/myuser/.ivy2/jars/com.thoughtworks.paranamer_paranamer-2.3.jar > added multiple times to distributed cache.}} > {{18/06/13 16:00:26 WARN Client: Same path resource > file:/home/myuser/.ivy2/jars/org.xerial.snappy_snappy-java-1.0.5.jar added > multiple times to distributed cache.}} > {{18/06/13 16:00:26 WARN Client: Same path resource > file:/home/myuser/.ivy2/jars/org.apache.commons_commons-compress-1.4.1.jar > added multiple times to distributed cache.}}{{18/06/13 16:00:26 WARN Client: > Same path resource file:/home/myuser/.ivy2/jars/org.tukaani_xz-1.0.jar added > multiple times to distributed cache.}} > {{18/06/13 16:00:27 INFO Client: Uploading resource > file:/tmp/spark-6c26ae3b-7248-488f-bc33-9766251474bb/__spark_conf__4405623606341803690.zip > -> > hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/__spark_conf__.zip}} > {{18/06/13 16:00:27 INFO SecurityManager: Changing view acls to: myuser}} > {{18/06/13 16:00:27 INFO SecurityManager: Changing modify acls to: myuser}} > {{18/06/13 16:00:27 INFO SecurityManager: Changing view acls groups to:}} > {{18/06/13 16:00:27 INFO SecurityManager: Changing modify acls groups to:}} > {{18/06/13 16:00:27 INFO SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view permissions: Set(myuser); groups > with view permissions: Set(); users with modify permissions: Set(myuser); > groups with modify permissions: Set()}} > {{18/06/13 16:00:27 INFO Client: Submitting application > application_1528901858967_0019 to ResourceManager}} > {{18/06/13 16:00:27 INFO YarnClientImpl: Submitted application > application_1528901858967_0019}} > {{18/06/13 16:00:28 INFO Client: Application report for > application_1528901858967_0019 (state: ACCEPTED)}} > {{18/06/13 16:00:28 INFO Client:}} > {{ client token: N/A}} > {{ diagnostics: AM container is launched, waiting for AM container to > Register with RM}} > {{ ApplicationMaster host: N/A}} > {{ ApplicationMaster RPC port: -1}} > {{ queue: default}} > {{ start time: 1528923627110}} > {{ final status: UNDEFINED}} > {{ tracking URL: > http://myhost2.myfirm.com:8088/proxy/application_1528901858967_0019/}} > {{ user: myuser}} > {{18/06/13 16:00:29 INFO Client: Application report for > application_1528901858967_0019 (state: ACCEPTED)}} > {{18/06/13 16:00:30 INFO Client: Application report for > application_1528901858967_0019 (state: ACCEPTED)}} > {{18/06/13 16:00:31 INFO Client: Application report for > application_1528901858967_0019 (state: ACCEPTED)}} > {{18/06/13 16:00:32 INFO Client: Application report for > application_1528901858967_0019 (state: ACCEPTED)}} > {{18/06/13 16:00:33 INFO Client: Application report for > application_1528901858967_0019 (state: ACCEPTED)}} > {{18/06/13 16:00:34 INFO Client: Application report for > application_1528901858967_0019 (state: ACCEPTED)}} > {{18/06/13 16:00:35 INFO Client: Application report for > application_1528901858967_0019 (state: ACCEPTED)}} > {{18/06/13 16:00:36 INFO Client: Application report for > application_1528901858967_0019 (state: ACCEPTED)}} > {{18/06/13 16:00:37 INFO Client: Application report for > application_1528901858967_0019 (state: ACCEPTED)}} > {{18/06/13 16:00:38 INFO Client: Application report for > application_1528901858967_0019 (state: ACCEPTED)}} > {{18/06/13 16:00:39 INFO Client: Application report for > application_1528901858967_0019 (state: FAILED)}} > {{18/06/13 16:00:39 INFO Client:}} > {{ client token: N/A}} > {{ diagnostics: Application application_1528901858967_0019 failed 2 times due > to AM Container for appattempt_1528901858967_0019_000002 exited with > exitCode: -1000}} > {{For more detailed output, check the application tracking page: > http://myhost2.myfirm.com:8088/cluster/app/application_1528901858967_0019 > Then click on links to logs of each attempt.}} > {{Diagnostics: java.util.zip.ZipException: invalid CEN header (bad > signature)}} > {{Failing this attempt. Failing the application.}} > {{ ApplicationMaster host: N/A}} > {{ ApplicationMaster RPC port: -1}} > {{ queue: default}} > {{ start time: 1528923627110}} > {{ final status: FAILED}} > {{ tracking URL: > http://myhost2.myfirm.com:8088/cluster/app/application_1528901858967_0019}} > {{ user: myuser}} > {{18/06/13 16:00:39 INFO Client: Deleted staging directory > hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019}} > {{Exception in thread "main" org.apache.spark.SparkException: Application > application_1528901858967_0019 finished with failed status}} > {{ at org.apache.spark.deploy.yarn.Client.run(Client.scala:1187)}} > {{ at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1233)}} > {{ at org.apache.spark.deploy.yarn.Client.main(Client.scala)}} > {{ at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}} > {{ at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}} > {{ at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}} > {{ at java.lang.reflect.Method.invoke(Method.java:498)}} > {{ at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:782)}} > {{ at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)}} > {{ at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)}} > {{ at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)}} > {{ at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)}} > {{18/06/13 16:00:39 INFO ShutdownHookManager: Shutdown hook called}} > {{18/06/13 16:00:39 INFO ShutdownHookManager: Deleting directory > /tmp/spark-6c26ae3b-7248-488f-bc33-9766251474bb}} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org