Hi,
When I run hibench join, I get an error that the classloader is already loaded.
HiBench’s Input/ output is on crail datanode, runs on another node, and has
RDMA communication.
So, disni is installed on crail node and Hibench node too.
However, When I ran hibench terasort, I didn't have this problem.
Could you tell me what the problem is?
<Log>
20/04/18 14:42:27 ERROR datasources.FileFormatWriter: Aborting job
18e95766-3878-4fc4-98d4-7b335c342a7d.
java.lang.UnsatisfiedLinkError: Native Library
/home/hibench/work/incubator-crail/apache-crail-1.3-incubating-SNAPSHOT/lib/libdisni.so
already loaded in another classloader
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1911)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1861)
at java.lang.Runtime.loadLibrary0(Runtime.java:871)
at java.lang.System.loadLibrary(System.java:1124)
at
com.ibm.disni.verbs.impl.NativeDispatcher.<clinit>(NativeDispatcher.java:36)
at
com.ibm.disni.verbs.impl.RdmaProviderNat.<init>(RdmaProviderNat.java:43)
at com.ibm.disni.verbs.RdmaProvider.provider(RdmaProvider.java:58)
at com.ibm.disni.verbs.RdmaCm.open(RdmaCm.java:49)
at
com.ibm.disni.verbs.RdmaEventChannel.createEventChannel(RdmaEventChannel.java:66)
at com.ibm.disni.RdmaCmProcessor.<init>(RdmaCmProcessor.java:48)
at com.ibm.disni.RdmaEndpointGroup.<init>(RdmaEndpointGroup.java:61)
at com.ibm.darpc.DaRPCEndpointGroup.<init>(DaRPCEndpointGroup.java:47)
at com.ibm.darpc.DaRPCClientGroup.<init>(DaRPCClientGroup.java:41)
at
com.ibm.darpc.DaRPCClientGroup.createClientGroup(DaRPCClientGroup.java:34)
at
org.apache.crail.namenode.rpc.darpc.DaRPCNameNodeClient.init(DaRPCNameNodeClient.java:46)
at org.apache.crail.core.CoreDataStore.<init>(CoreDataStore.java:126)
at org.apache.crail.CrailStore.newInstance(CrailStore.java:77)
at
org.apache.spark.storage.CrailDispatcher.org$apache$spark$storage$CrailDispatcher$$init(CrailDispatcher.scala:120)
at
org.apache.spark.storage.CrailDispatcher$.get(CrailDispatcher.scala:662)
at
org.apache.spark.shuffle.crail.CrailShuffleManager.registerShuffle(CrailShuffleManager.scala:52)
at org.apache.spark.ShuffleDependency.<init>(Dependency.scala:93)
Thanks,
Sohyun.
