[
https://issues.apache.org/jira/browse/HADOOP-8899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed El Baz updated HADOOP-8899:
---------------------------------
Attachment: HADOOP-8899.branch-1-win.classpath.patch
Please note the following:
- The generated Jar does not encapsulate any other jars, it simply references
all entries in the classpath in its Manifest, so this is just another way to
propagate the long list of Jars to the tasks to launch.
- Classpath elements are referenced in the generated Jar's Manifest in the same
order.
- This change is only gated on Windows where the Java classpath argument now
has the generated Jar (which references the other ones), instead of the long
list which is giving the problem. For non-Windows OS, the behavior is just the
same as before.
- We do not explicitly unjar the file, we feed it to Jave to lookup the
classpath elements from the manifest.
- The patch also includes testing performance changes (in TestJvmManager): in
Windows, launchTask is only affected by ~3%, in Linux no impact since the logic
is the same.
I have attached a patch with the changes (including test changes)
> Classpath exceeds maximum OS limit
> ----------------------------------
>
> Key: HADOOP-8899
> URL: https://issues.apache.org/jira/browse/HADOOP-8899
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 1-win
> Reporter: Ahmed El Baz
> Attachments: HADOOP-8899.branch-1-win.classpath.patch
>
>
> We hit errors in TT due to long classpath value. One example is Oozie trying
> to start a hive job, and it fails with the following error:
> java.io.IOException: Command exceeds the OS command length limit: 8192,
> command: "set CLASSPATH="….
> The classpath includes Hadoop Core + Hive Jars which are in the distributed
> cache. This is causing the classpath to be too long giving the error above. A
> viable long term fix is to generate a temporary JAR file in the task cache
> directory which includes all elements in the long classpath, and then provide
> this as the "-classpath" argument for the JVM to be spawned. This is gated
> for Windows only.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
