[
https://issues.apache.org/jira/browse/HADOOP-13070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15304825#comment-15304825
]
Ravi Prakash commented on HADOOP-13070:
---------------------------------------
Hi Sangjin! Thanks for taking this up. I look forward to all your improvements.
{{ApplicationClassLoader}} seems like its only being used by MR. I grepped in
Tez and Spark source, and didn't find any instances. Even if we were to do this
only for MR, it would be incredibly valuable. I feel it would also set a
precedent / pattern that other frameworks can then leverage.
If we were to focus on MR, do you know what are the common problematic
conflicting dependencies? One alternative approach would be to start 2 JVMs for
each MR-task: an MR-framework JVM and an MR-task JVM. We would do all
MR-framework specific work in the MR-framework JVM and send raw Map-Reduce
input key-value pairs over a socket and read output key value pairs over a
socket from the MR-task JVM. The MR specific code running in the MR-task JVM
would then be minimal and only needs to read over the socket and call the user
code. I know protobuf (required for serialization / deserialization) is often
the conflicting library, so it would be no help in that case. (We could still
shade this minimal set of libraries...... although I personally dislike shading
a lot).
I sure do wish Java 9 had something that would make it easier but I didn't see
anything.
> classloading isolation improvements for cleaner and stricter dependencies
> -------------------------------------------------------------------------
>
> Key: HADOOP-13070
> URL: https://issues.apache.org/jira/browse/HADOOP-13070
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
> Reporter: Sangjin Lee
> Assignee: Sangjin Lee
> Priority: Critical
> Attachments: classloading-improvements-ideas-v.3.pdf,
> classloading-improvements-ideas.pdf, classloading-improvements-ideas.v.2.pdf
>
>
> Related to HADOOP-11656, we would like to make a number of improvements in
> terms of classloading isolation so that user-code can run safely without
> worrying about dependency collisions with the Hadoop dependencies.
> By the same token, it should raised the quality of the user code and its
> specified classpath so that users get clear signals if they specify incorrect
> classpaths.
> This will contain a proposal that will include several improvements some of
> which may not be backward compatible. As such, it should be targeted to the
> next major revision of Hadoop.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
