Hi,
I'm having an LinkageError in spark trying to read a hive table that has
the external location in IGFS:
java.lang.LinkageError: loader constraint violation: when resolving field
"LOG" the class loader (instance of
org/apache/spark/sql/hive/client/IsolatedClientLoader$$anon$1) of the
referring class, org/apache/hadoop/fs/FileSystem, and the class loader
(instance of sun/misc/Launcher$AppClassLoader) for the field's resolved
type, org/apache/commons/logging/Log, have different Class objects for that
type
at
org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem.initialize(IgniteHadoopFileSystem.java:255)
>From what I can see the exception comes when spark tries to read a table
from Hive and then through IGFS and passing the "LOG" variable of the
FileSystem around to the HadoopIgfsWrapper (and beyond...).
The steps I followed to reach this error were:
- Create a file /tmp/test.parquet in HDFS
- Create an external table test.test in hive with location = igfs://igfs@
<host>/tmp/test.parquet
- Start spark-shell with the command:
- ./bin/spark-shell --jars
$IGNITE_HOME/ignite-core-2.6.0.jar,$IGNITE_HOME/ignite-hadoop/ignite-hadoop-2.6.0.jar,$IGNITE_HOME/ignite-shmem-1.0.0.jar,$IGNITE_HOME/ignite-spark-2.6.0.jar
- Read the table through spark.sql
- spark.sql("SELECT * FROM test.test")
Is there maybe a way to avoid having this issue? Has anyone used ignite
through hive as HDFS cache in a similar way?
