Hyukjin Kwon created SPARK-18842:
------------------------------------
Summary: De-duplicate paths in classpaths in processes for
local-cluster mode to work around the length limitation on Windows
Key: SPARK-18842
URL: https://issues.apache.org/jira/browse/SPARK-18842
Project: Spark
Issue Type: Sub-task
Components: Spark Core, Tests
Reporter: Hyukjin Kwon
Priority: Minor
Currently, some tests are being failed and hanging on Windows due to this
problem. For the reason in SPARK-18718, some tests using {{local-cluster}} mode
were disabled on Windows due to the length limitation by paths given to
classpaths.
The limitation seems roughly 32K (see
https://blogs.msdn.microsoft.com/oldnewthing/20031210-00/?p=41553/ and
https://support.thoughtworks.com/hc/en-us/articles/213248526-Getting-around-maximum-command-line-length-is-32767-characters-on-Windows)
but executors were being launched with the command such as
https://gist.github.com/HyukjinKwon/5bc81061c250d4af5a180869b59d42ea in (only)
tests.
This length is roughly 40K due to the class paths. However, it seems there are
duplicates more than half. So, if we de-duplicate this paths, it is reduced to
roughly 20K.
Maybe, we should consider as some more paths are added in the future but it
seems better than disabling all the tests for now with minimised changes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
