[
https://issues.apache.org/jira/browse/HADOOP-12406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14986295#comment-14986295
]
Nadeem Douba commented on HADOOP-12406:
---------------------------------------
Hi Vinod,
I was running this on top of YARN with the default configuration enabled on top
of Java 1.8 from Oracle. I think the new isolation strategy used in the newer
versions of the JVM for class loaders prevents the system class loader from
loading a job jar class. I didn't do anything fancy with configuration options
or anything else. The only thing I added were the remote debugging options to
allow me to attach to the map and reduce threads remotely and debug them step
by step. I can tell you that the job jar definitely isn't in the system's class
loader but in the current thread's context class loader. I verified this
observation by manually inspecting the class loader object from the system and
the one that comes with the thread context.
Cheers,
Nadeem
> AbstractMapWritable.readFields throws ClassNotFoundException with custom
> writables
> ----------------------------------------------------------------------------------
>
> Key: HADOOP-12406
> URL: https://issues.apache.org/jira/browse/HADOOP-12406
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Affects Versions: 2.7.1
> Environment: Ubuntu Linux 14.04 LTS amd64
> Reporter: Nadeem Douba
> Priority: Blocker
> Labels: bug, hadoop, io, newbie, patch-available
> Attachments: HADOOP-12406.patch
>
>
> Note: I am not an expert at JAVA, Class loaders, or Hadoop. I am just a
> hacker. My solution might be entirely wrong.
> AbstractMapWritable.readFields throws a ClassNotFoundException when reading
> custom writables. Debugging the job using remote debugging in IntelliJ
> revealed that the class loader being used in Class.forName() is different
> than that used by the Thread's current context
> (Thread.currentThread().getContextClassLoader()). The class path for the
> system class loader does not include the libraries of the job jar. However,
> the class path for the context class loader does. The proposed patch changes
> the class loading mechanism in readFields to use the Thread's context class
> loader instead of the system's default class loader.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
