[
https://issues.apache.org/jira/browse/TEZ-4376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated TEZ-4376:
------------------------------
Description:
When I try to start yarn nodemanager with tez shufflehandler configured, I get
this:
{code:java}
java.lang.NoClassDefFoundError:
org/apache/tez/shaded/javax/security/cert/X509Certificate
at
org.apache.tez.shaded.io.netty.util.internal.EmptyArrays.<clinit>(EmptyArrays.java:38)
at
org.apache.tez.auxservices.ShuffleHandler.defaultProcessId(ShuffleHandler.java:601)
at
org.apache.tez.auxservices.ShuffleHandler.serviceStart(ShuffleHandler.java:567)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxiliaryServiceWithCustomClassLoader.serviceStart(AuxiliaryServiceWithCustomClassLoader.java:76)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceStart(AuxServices.java:313)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceStart(ContainerManagerImpl.java:654)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:975)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:1054)
Caused by: java.lang.ClassNotFoundException:
org.apache.tez.shaded.javax.security.cert.X509Certificate
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at
org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:189)
at
org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:157)
... 15 more
{code}
this is because in netty 4, EmptyArrays class - which is originally referred to
by DefaultChannelId, here the stacktrace shows that I simply copied it to
ShuffleHandler code to insert some logging - refers to X509Certificate
I tried to add javax.security exclude as:
{code:java}
<relocation>
<pattern>javax</pattern>
<shadedPattern>org.apache.tez.shaded.$0</shadedPattern>
<excludes>
<exclude>javax.crypto.*</exclude>
<exclude>javax.security.*</exclude> <!-- HERE -->
</excludes>
</relocation>
{code}
but it didn't work, not sure why
*when I completely removed javax relocation, it started to work*
I'm tempted to make javax.security exclusion work, but first I need to
understand why javax relocation is needed at all
is there a chance that we want to package a custom javax to be used instead of
what is provided on nodemanager's classpath?
was:
When I try to start yarn nodemanager with tez shufflehandler configured, I get
this:
{code:java}
java.lang.NoClassDefFoundError:
org/apache/tez/shaded/javax/security/cert/X509Certificate
at
org.apache.tez.shaded.io.netty.util.internal.EmptyArrays.<clinit>(EmptyArrays.java:38)
at
org.apache.tez.auxservices.ShuffleHandler.defaultProcessId(ShuffleHandler.java:601)
at
org.apache.tez.auxservices.ShuffleHandler.serviceStart(ShuffleHandler.java:567)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxiliaryServiceWithCustomClassLoader.serviceStart(AuxiliaryServiceWithCustomClassLoader.java:76)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceStart(AuxServices.java:313)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceStart(ContainerManagerImpl.java:654)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:975)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:1054)
Caused by: java.lang.ClassNotFoundException:
org.apache.tez.shaded.javax.security.cert.X509Certificate
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at
org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:189)
at
org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:157)
... 15 more
{code}
this is because in netty 4, EmptyArrays class - which is referred to by
DefaultChannelId - refers to X509Certificate
I tried to add javax.security exclude as:
{code:java}
<relocation>
<pattern>javax</pattern>
<shadedPattern>org.apache.tez.shaded.$0</shadedPattern>
<excludes>
<exclude>javax.crypto.*</exclude>
<exclude>javax.security.*</exclude> <!-- HERE -->
</excludes>
</relocation>
{code}
but it didn't work, not sure why
*when I completely removed javax relocation, it started to work*
I'm tempted to make javax.security exclusion work, but first I need to
understand why javax relocation is needed at all
is there a chance that we want to package a custom javax to be used instead of
what is provided on nodemanager's classpath?
> Remove javax.security from the tez-auxservices shaded jar
> ---------------------------------------------------------
>
> Key: TEZ-4376
> URL: https://issues.apache.org/jira/browse/TEZ-4376
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
>
> When I try to start yarn nodemanager with tez shufflehandler configured, I
> get this:
> {code:java}
> java.lang.NoClassDefFoundError:
> org/apache/tez/shaded/javax/security/cert/X509Certificate
> at
> org.apache.tez.shaded.io.netty.util.internal.EmptyArrays.<clinit>(EmptyArrays.java:38)
> at
> org.apache.tez.auxservices.ShuffleHandler.defaultProcessId(ShuffleHandler.java:601)
> at
> org.apache.tez.auxservices.ShuffleHandler.serviceStart(ShuffleHandler.java:567)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxiliaryServiceWithCustomClassLoader.serviceStart(AuxiliaryServiceWithCustomClassLoader.java:76)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceStart(AuxServices.java:313)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceStart(ContainerManagerImpl.java:654)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:975)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:1054)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.tez.shaded.javax.security.cert.X509Certificate
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> at
> org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:189)
> at
> org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:157)
> ... 15 more
> {code}
> this is because in netty 4, EmptyArrays class - which is originally referred
> to by DefaultChannelId, here the stacktrace shows that I simply copied it to
> ShuffleHandler code to insert some logging - refers to X509Certificate
> I tried to add javax.security exclude as:
> {code:java}
> <relocation>
> <pattern>javax</pattern>
> <shadedPattern>org.apache.tez.shaded.$0</shadedPattern>
> <excludes>
> <exclude>javax.crypto.*</exclude>
> <exclude>javax.security.*</exclude> <!-- HERE -->
> </excludes>
> </relocation>
> {code}
> but it didn't work, not sure why
> *when I completely removed javax relocation, it started to work*
> I'm tempted to make javax.security exclusion work, but first I need to
> understand why javax relocation is needed at all
> is there a chance that we want to package a custom javax to be used instead
> of what is provided on nodemanager's classpath?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
