[
https://issues.apache.org/jira/browse/HADOOP-13070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15305075#comment-15305075
]
Sangjin Lee commented on HADOOP-13070:
--------------------------------------
Thanks for the comments [~raviprak]! To answer your questions...
{quote}
ApplicationClassLoader seems like its only being used by MR. I grepped in Tez
and Spark source, and didn't find any instances. Even if we were to do this
only for MR, it would be incredibly valuable. I feel it would also set a
precedent / pattern that other frameworks can then leverage.
{quote}
If you meant that the only usage is hadoop itself, I believe that's correct.
Within hadoop, there are 3 usages today: MR task class isolation, hadoop run
jar class isolation, and more recently the NM aux service class isolation.
Since {{ApplicationClassLoader}} is part of the public API, other frameworks
can use it.
bq. If we were to focus on MR, do you know what are the common problematic
conflicting dependencies?
Unfortunately there are many to choose from, and quite a few of the well-known
ones fall into the problem category. Some of the more famous ones include guava
and jackson to name a couple.
But isolating class spaces has more benefits than simply preventing collisions.
Since we're afraid of breaking users, hadoop has been very slow/conservative in
upgrading any libraries it uses. As a result, we're stuck in the stone age for
many of the libraries we use. Isolation would give hadoop more freedom to
upgrade its dependencies without worrying about impacting users. That is of
course provided that the isolation mode becomes the default, which may still be
some time away.
{quote}
One alternative approach would be to start 2 JVMs for each MR-task: an
MR-framework JVM and an MR-task JVM. We would do all MR-framework specific work
in the MR-framework JVM and send raw Map-Reduce input key-value pairs over a
socket and read output key value pairs over a socket from the MR-task JVM. The
MR specific code running in the MR-task JVM would then be minimal and only
needs to read over the socket and call the user code.
{quote}
That is an interesting idea to solve this problem. I still worry about the
performance implication it has. Also, it still would not eliminate the problem
entirely. As you pointed out, even in that separate process you still need a
minimal amount of hadoop code which then pulls in the needed dependencies.
> classloading isolation improvements for cleaner and stricter dependencies
> -------------------------------------------------------------------------
>
> Key: HADOOP-13070
> URL: https://issues.apache.org/jira/browse/HADOOP-13070
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
> Reporter: Sangjin Lee
> Assignee: Sangjin Lee
> Priority: Critical
> Attachments: classloading-improvements-ideas-v.3.pdf,
> classloading-improvements-ideas.pdf, classloading-improvements-ideas.v.2.pdf
>
>
> Related to HADOOP-11656, we would like to make a number of improvements in
> terms of classloading isolation so that user-code can run safely without
> worrying about dependency collisions with the Hadoop dependencies.
> By the same token, it should raised the quality of the user code and its
> specified classpath so that users get clear signals if they specify incorrect
> classpaths.
> This will contain a proposal that will include several improvements some of
> which may not be backward compatible. As such, it should be targeted to the
> next major revision of Hadoop.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
