[
https://issues.apache.org/jira/browse/YARN-8430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcelo Vanzin moved SPARK-24559 to YARN-8430:
----------------------------------------------
Affects Version/s: (was: 2.2.0)
Component/s: (was: Spark Submit)
Workflow: no-reopen-closed, patch-avail (was: no-reopen-closed)
Key: YARN-8430 (was: SPARK-24559)
Project: Hadoop YARN (was: Spark)
> Some zip files passed with spark-submit --archives causing "invalid CEN
> header" error
> -------------------------------------------------------------------------------------
>
> Key: YARN-8430
> URL: https://issues.apache.org/jira/browse/YARN-8430
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: James Porritt
> Priority: Major
>
> I'm encountering an error when submitting some zip files to spark-submit
> using --archive that are over 2Gb and have the zip64 flag set.
> {{PYSPARK_PYTHON=./ROOT/myspark/bin/python
> /usr/hdp/current/spark2-client/bin/spark-submit \}}
> {{ --conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=./ROOT/myspark/bin/python \}}
> {{ --master=yarn \}}
> {{ --deploy-mode=cluster \}}
> {{ --driver-memory=4g \}}
> {{ --archives=myspark.zip#ROOT \}}
> {{ --num-executors=32 \}}
> {{ --packages com.databricks:spark-avro_2.11:4.0.0 \}}
> {{ foo.py}}
> (As a bit of background, I'm trying to prepare files using the trick of
> zipping a conda environment and passing the zip file via --archives, as per:
> https://community.hortonworks.com/articles/58418/running-pyspark-with-conda-env.html)
> myspark.zip is a zipped conda environment. It was created using python with
> the zipfile pacakge. The files are stored without deflation and with the
> zip64 flag set. foo.py is my application code. This normally works, but if
> myspark.zip is greater than 2Gb and has the zip64 flag set I get:
> java.util.zip.ZipException: invalid CEN header (bad signature)
> There seems to be much written on the subject, and I was able to write Java
> code that utilises the java.util.zip library that both does and doesn't
> encounter this error for one of the problematic zip files.
> Spark compile info:
> {{Welcome to}}
> {{ ____ __}}
> {{ / __/__ ___ _____/ /__}}
> {{ _\ \/ _ \/ _ `/ __/ '_/}}
> {{ /___/ .__/\_,_/_/ /_/\_\ version 2.2.0.2.6.4.0-91}}
> {{ /_/}}
> {{Using Scala version 2.11.8, Java HotSpot(TM) 64-Bit Server VM, 1.8.0_112}}
> {{Branch HEAD}}
> {{Compiled by user jenkins on 2018-01-04T10:41:05Z}}
> {{Revision a24017869f5450397136ee8b11be818e7cd3facb}}
> {{Url [email protected]:hortonworks/spark2.git}}
> {{Type --help for more information.}}
> YARN logs on console after above command. I've tried both
> --deploy-mode=cluster and --deploy-mode=client.
> {{18/06/13 16:00:22 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable}}
> {{18/06/13 16:00:23 WARN DomainSocketFactory: The short-circuit local reads
> feature cannot be used because libhadoop cannot be loaded.}}
> {{18/06/13 16:00:23 INFO RMProxy: Connecting to ResourceManager at
> myhost2.myfirm.com/10.87.11.17:8050}}
> {{18/06/13 16:00:23 INFO Client: Requesting a new application from cluster
> with 6 NodeManagers}}
> {{18/06/13 16:00:23 INFO Client: Verifying our application has not requested
> more than the maximum memory capability of the cluster (221184 MB per
> container)}}
> {{18/06/13 16:00:23 INFO Client: Will allocate AM container, with 18022 MB
> memory including 1638 MB overhead}}
> {{18/06/13 16:00:23 INFO Client: Setting up container launch context for our
> AM}}
> {{18/06/13 16:00:23 INFO Client: Setting up the launch environment for our AM
> container}}
> {{18/06/13 16:00:23 INFO Client: Preparing resources for our AM container}}
> {{18/06/13 16:00:24 INFO Client: Use hdfs cache file as spark.yarn.archive
> for HDP,
> hdfsCacheFile:hdfs://myhost.myfirm.com:8020/hdp/apps/2.6.4.0-91/spark2/spark2-hdp-yarn-archive.tar.gz}}
> {{18/06/13 16:00:24 INFO Client: Source and destination file systems are the
> same. Not copying
> hdfs://myhost.myfirm.com:8020/hdp/apps/2.6.4.0-91/spark2/spark2-hdp-yarn-archive.tar.gz}}
> {{18/06/13 16:00:24 INFO Client: Uploading resource
> file:/home/myuser/.ivy2/jars/com.databricks_spark-avro_2.11-4.0.0.jar ->
> hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/com.databri}}
> {{cks_spark-avro_2.11-4.0.0.jar}}
> {{18/06/13 16:00:26 INFO Client: Uploading resource
> file:/home/myuser/.ivy2/jars/org.slf4j_slf4j-api-1.7.5.jar ->
> hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/org.slf4j_slf4j-api-1.}}
> {{7.5.jar}}
> {{18/06/13 16:00:26 INFO Client: Uploading resource
> file:/home/myuser/.ivy2/jars/org.apache.avro_avro-1.7.6.jar ->
> hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/org.apache.avro_avro-}}
> {{1.7.6.jar}}
> {{18/06/13 16:00:26 INFO Client: Uploading resource
> file:/home/myuser/.ivy2/jars/org.codehaus.jackson_jackson-core-asl-1.9.13.jar
> ->
> hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/org}}
> {{.codehaus.jackson_jackson-core-asl-1.9.13.jar}}
> {{18/06/13 16:00:26 INFO Client: Uploading resource
> file:/home/myuser/.ivy2/jars/org.codehaus.jackson_jackson-mapper-asl-1.9.13.jar
> ->
> hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/o}}
> {{rg.codehaus.jackson_jackson-mapper-asl-1.9.13.jar}}
> {{18/06/13 16:00:26 INFO Client: Uploading resource
> file:/home/myuser/.ivy2/jars/com.thoughtworks.paranamer_paranamer-2.3.jar ->
> hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/com.tho}}
> {{ughtworks.paranamer_paranamer-2.3.jar}}
> {{18/06/13 16:00:26 INFO Client: Uploading resource
> file:/home/myuser/.ivy2/jars/org.xerial.snappy_snappy-java-1.0.5.jar ->
> hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/org.xerial.s}}
> {{nappy_snappy-java-1.0.5.jar}}
> {{18/06/13 16:00:26 INFO Client: Uploading resource
> file:/home/myuser/.ivy2/jars/org.apache.commons_commons-compress-1.4.1.jar ->
> hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/org.ap}}
> {{ache.commons_commons-compress-1.4.1.jar}}
> {{18/06/13 16:00:26 INFO Client: Uploading resource
> file:/home/myuser/.ivy2/jars/org.tukaani_xz-1.0.jar ->
> hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/org.tukaani_xz-1.0.jar}}
> {{18/06/13 16:00:26 INFO Client: Source and destination file systems are the
> same. Not copying hdfs:/user/myuser/release/alphagenspark.zip#ROOT}}
> {{18/06/13 16:00:26 INFO Client: Uploading resource
> file:/my/script/dir/spark/alphagen/foo.py ->
> hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/foo.py}}
> {{18/06/13 16:00:26 INFO Client: Uploading resource
> file:/usr/hdp/current/spark2-client/python/lib/pyspark.zip ->
> hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/pyspark.zip}}
> {{18/06/13 16:00:26 INFO Client: Uploading resource
> file:/usr/hdp/current/spark2-client/python/lib/py4j-0.10.4-src.zip ->
> hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/py4j-0.10.4-src}}
> {{.zip}}
> {{18/06/13 16:00:26 WARN Client: Same path resource
> file:/home/myuser/.ivy2/jars/com.databricks_spark-avro_2.11-4.0.0.jar added
> multiple times to distributed cache.}}
> {{18/06/13 16:00:26 WARN Client: Same path resource
> file:/home/myuser/.ivy2/jars/org.slf4j_slf4j-api-1.7.5.jar added multiple
> times to distributed cache.}}
> {{18/06/13 16:00:26 WARN Client: Same path resource
> file:/home/myuser/.ivy2/jars/org.apache.avro_avro-1.7.6.jar added multiple
> times to distributed cache.}}
> {{18/06/13 16:00:26 WARN Client: Same path resource
> file:/home/myuser/.ivy2/jars/org.codehaus.jackson_jackson-core-asl-1.9.13.jar
> added multiple times to distributed cache.}}
> {{18/06/13 16:00:26 WARN Client: Same path resource
> file:/home/myuser/.ivy2/jars/org.codehaus.jackson_jackson-mapper-asl-1.9.13.jar
> added multiple times to distributed cache.}}
> {{18/06/13 16:00:26 WARN Client: Same path resource
> file:/home/myuser/.ivy2/jars/com.thoughtworks.paranamer_paranamer-2.3.jar
> added multiple times to distributed cache.}}
> {{18/06/13 16:00:26 WARN Client: Same path resource
> file:/home/myuser/.ivy2/jars/org.xerial.snappy_snappy-java-1.0.5.jar added
> multiple times to distributed cache.}}
> {{18/06/13 16:00:26 WARN Client: Same path resource
> file:/home/myuser/.ivy2/jars/org.apache.commons_commons-compress-1.4.1.jar
> added multiple times to distributed cache.}}{{18/06/13 16:00:26 WARN Client:
> Same path resource file:/home/myuser/.ivy2/jars/org.tukaani_xz-1.0.jar added
> multiple times to distributed cache.}}
> {{18/06/13 16:00:27 INFO Client: Uploading resource
> file:/tmp/spark-6c26ae3b-7248-488f-bc33-9766251474bb/__spark_conf__4405623606341803690.zip
> ->
> hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019/__spark_conf__.zip}}
> {{18/06/13 16:00:27 INFO SecurityManager: Changing view acls to: myuser}}
> {{18/06/13 16:00:27 INFO SecurityManager: Changing modify acls to: myuser}}
> {{18/06/13 16:00:27 INFO SecurityManager: Changing view acls groups to:}}
> {{18/06/13 16:00:27 INFO SecurityManager: Changing modify acls groups to:}}
> {{18/06/13 16:00:27 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(myuser); groups
> with view permissions: Set(); users with modify permissions: Set(myuser);
> groups with modify permissions: Set()}}
> {{18/06/13 16:00:27 INFO Client: Submitting application
> application_1528901858967_0019 to ResourceManager}}
> {{18/06/13 16:00:27 INFO YarnClientImpl: Submitted application
> application_1528901858967_0019}}
> {{18/06/13 16:00:28 INFO Client: Application report for
> application_1528901858967_0019 (state: ACCEPTED)}}
> {{18/06/13 16:00:28 INFO Client:}}
> {{ client token: N/A}}
> {{ diagnostics: AM container is launched, waiting for AM container to
> Register with RM}}
> {{ ApplicationMaster host: N/A}}
> {{ ApplicationMaster RPC port: -1}}
> {{ queue: default}}
> {{ start time: 1528923627110}}
> {{ final status: UNDEFINED}}
> {{ tracking URL:
> http://myhost2.myfirm.com:8088/proxy/application_1528901858967_0019/}}
> {{ user: myuser}}
> {{18/06/13 16:00:29 INFO Client: Application report for
> application_1528901858967_0019 (state: ACCEPTED)}}
> {{18/06/13 16:00:30 INFO Client: Application report for
> application_1528901858967_0019 (state: ACCEPTED)}}
> {{18/06/13 16:00:31 INFO Client: Application report for
> application_1528901858967_0019 (state: ACCEPTED)}}
> {{18/06/13 16:00:32 INFO Client: Application report for
> application_1528901858967_0019 (state: ACCEPTED)}}
> {{18/06/13 16:00:33 INFO Client: Application report for
> application_1528901858967_0019 (state: ACCEPTED)}}
> {{18/06/13 16:00:34 INFO Client: Application report for
> application_1528901858967_0019 (state: ACCEPTED)}}
> {{18/06/13 16:00:35 INFO Client: Application report for
> application_1528901858967_0019 (state: ACCEPTED)}}
> {{18/06/13 16:00:36 INFO Client: Application report for
> application_1528901858967_0019 (state: ACCEPTED)}}
> {{18/06/13 16:00:37 INFO Client: Application report for
> application_1528901858967_0019 (state: ACCEPTED)}}
> {{18/06/13 16:00:38 INFO Client: Application report for
> application_1528901858967_0019 (state: ACCEPTED)}}
> {{18/06/13 16:00:39 INFO Client: Application report for
> application_1528901858967_0019 (state: FAILED)}}
> {{18/06/13 16:00:39 INFO Client:}}
> {{ client token: N/A}}
> {{ diagnostics: Application application_1528901858967_0019 failed 2 times due
> to AM Container for appattempt_1528901858967_0019_000002 exited with
> exitCode: -1000}}
> {{For more detailed output, check the application tracking page:
> http://myhost2.myfirm.com:8088/cluster/app/application_1528901858967_0019
> Then click on links to logs of each attempt.}}
> {{Diagnostics: java.util.zip.ZipException: invalid CEN header (bad
> signature)}}
> {{Failing this attempt. Failing the application.}}
> {{ ApplicationMaster host: N/A}}
> {{ ApplicationMaster RPC port: -1}}
> {{ queue: default}}
> {{ start time: 1528923627110}}
> {{ final status: FAILED}}
> {{ tracking URL:
> http://myhost2.myfirm.com:8088/cluster/app/application_1528901858967_0019}}
> {{ user: myuser}}
> {{18/06/13 16:00:39 INFO Client: Deleted staging directory
> hdfs://myhost.myfirm.com:8020/user/myuser/.sparkStaging/application_1528901858967_0019}}
> {{Exception in thread "main" org.apache.spark.SparkException: Application
> application_1528901858967_0019 finished with failed status}}
> {{ at org.apache.spark.deploy.yarn.Client.run(Client.scala:1187)}}
> {{ at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1233)}}
> {{ at org.apache.spark.deploy.yarn.Client.main(Client.scala)}}
> {{ at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}}
> {{ at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
> {{ at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
> {{ at java.lang.reflect.Method.invoke(Method.java:498)}}
> {{ at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:782)}}
> {{ at
> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)}}
> {{ at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)}}
> {{ at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)}}
> {{ at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)}}
> {{18/06/13 16:00:39 INFO ShutdownHookManager: Shutdown hook called}}
> {{18/06/13 16:00:39 INFO ShutdownHookManager: Deleting directory
> /tmp/spark-6c26ae3b-7248-488f-bc33-9766251474bb}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
