Sebastian Kochman created SPARK-14602:
-----------------------------------------

             Summary: [YARN+Windows] Setting SPARK_YARN_CACHE_FILES exceeds 
command line length limit on Windows
                 Key: SPARK-14602
                 URL: https://issues.apache.org/jira/browse/SPARK-14602
             Project: Spark
          Issue Type: Bug
          Components: Deploy, Spark Submit, Windows, YARN
    Affects Versions: 2.0.0
         Environment: YARN on Windows
            Reporter: Sebastian Kochman


After change https://issues.apache.org/jira/browse/SPARK-11157, which removed a 
single large Spark assembly in favor of multiple small jars, when you try to 
submit a Spark app to YARN on Windows (using spark-submit.cmd), the app fails 
with the following error:

Diagnostics: The command line has a length of 12046 exceeds maximum allowed 
length of 8191. Command starts with: @set 
SPARK_YARN_CACHE_FILES=[...]/.sparkStaging/application_[...] Failing this 
attempt. Failing the application.

So basically a large number of jars needed for staging in YARN causes exceeding 
Windows command line length limit.

Please see more details in the discussion here:
https://issues.apache.org/jira/browse/SPARK-11157?focusedCommentId=15238151&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15238151



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to