JackyYangPassion opened a new issue, #404: URL: https://github.com/apache/incubator-hugegraph-toolchain/issues/404
### Bug Type (问题类型) others (please comment below) ### Before submit - [X] I had searched in the [issues](https://github.com/apache/hugegraph-toolchain/issues) and found no similar issues. ### Environment (环境信息) - Server Version: v1.0.0 - Toolchain Version: v1.0.0 - Spark Version: spark-3.3.1-hadoop3 ### Expected & Actual behavior (期望与实际表现) **CMD** : sh bin/hugegraph-spark-loader.sh --master local --name spark-hugegraph-loader --file example/spark/struct.json --username admin --token admin --host 127.0.0.1 --port 8080 --graph graph-test ``` java.lang.IllegalArgumentException: Class is not registered: org.apache.spark.sql.types.StructType Note: To register this class use: kryo.register(org.apache.spark.sql.types.StructType.class); at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:503) at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:97) at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:540) at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:645) at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:387) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:593) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 23/01/01 14:17:42 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0) (192.168.1.5 executor driver): java.lang.IllegalArgumentException: Class is not registered: org.apache.spark.sql.types.StructType Note: To register this class use: kryo.register(org.apache.spark.sql.types.StructType.class); at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:503) at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:97) at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:540) at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:645) at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:387) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:593) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 23/01/01 14:17:42 ERROR TaskSetManager: Task 0 in stage 0.0 failed 1 times; aborting job 23/01/01 14:17:42 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 23/01/01 14:17:42 INFO TaskSchedulerImpl: Cancelling stage 0 23/01/01 14:17:42 INFO TaskSchedulerImpl: Killing all running tasks in stage 0: Stage cancelled 23/01/01 14:17:42 INFO DAGScheduler: ResultStage 0 (json at HugeGraphSparkLoader.java:234) failed in 1.370 s due to Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0) (192.168.1.5 executor driver): java.lang.IllegalArgumentException: Class is not registered: org.apache.spark.sql.types.StructType Note: To register this class use: kryo.register(org.apache.spark.sql.types.StructType.class); at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:503) at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:97) at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:540) at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:645) at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:387) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:593) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Driver stacktrace: 23/01/01 14:17:42 INFO DAGScheduler: Job 0 failed: json at HugeGraphSparkLoader.java:234, took 1.420781 s Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0) (192.168.1.5 executor driver): java.lang.IllegalArgumentException: Class is not registered: org.apache.spark.sql.types.StructType Note: To register this class use: kryo.register(org.apache.spark.sql.types.StructType.class); at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:503) at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:97) at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:540) at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:645) at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:387) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:593) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2672) at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2608) at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2607) at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2607) at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1182) at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1182) at scala.Option.foreach(Option.scala:407) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1182) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2860) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2802) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2791) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49) at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:952) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2228) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2323) at org.apache.spark.sql.catalyst.json.JsonInferSchema.infer(JsonInferSchema.scala:116) at org.apache.spark.sql.execution.datasources.json.TextInputJsonDataSource$.$anonfun$inferFromDataset$5(JsonDataSource.scala:110) at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169) at org.apache.spark.sql.execution.datasources.json.TextInputJsonDataSource$.inferFromDataset(JsonDataSource.scala:110) at org.apache.spark.sql.execution.datasources.json.TextInputJsonDataSource$.infer(JsonDataSource.scala:99) at org.apache.spark.sql.execution.datasources.json.JsonDataSource.inferSchema(JsonDataSource.scala:65) at org.apache.spark.sql.execution.datasources.json.JsonFileFormat.inferSchema(JsonFileFormat.scala:59) at org.apache.spark.sql.execution.datasources.DataSource.$anonfun$getOrInferFileFormatSchema$11(DataSource.scala:210) at scala.Option.orElse(Option.scala:447) at org.apache.spark.sql.execution.datasources.DataSource.getOrInferFileFormatSchema(DataSource.scala:207) at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:411) at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:228) at org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:210) at scala.Option.getOrElse(Option.scala:189) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:210) at org.apache.spark.sql.DataFrameReader.json(DataFrameReader.scala:361) at org.apache.spark.sql.DataFrameReader.json(DataFrameReader.scala:340) at org.apache.hugegraph.loader.spark.HugeGraphSparkLoader.read(HugeGraphSparkLoader.java:234) at org.apache.hugegraph.loader.spark.HugeGraphSparkLoader.load(HugeGraphSparkLoader.java:135) at org.apache.hugegraph.loader.spark.HugeGraphSparkLoader.main(HugeGraphSparkLoader.java:86) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.IllegalArgumentException: Class is not registered: org.apache.spark.sql.types.StructType Note: To register this class use: kryo.register(org.apache.spark.sql.types.StructType.class); at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:503) at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:97) at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:540) at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:645) at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:387) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:593) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) ``` but i find org.apache.spark.sql.types.StructType has registered in the source code : ``` conf.registerKryoClasses(new Class[]{ org.apache.hadoop.hbase.io.ImmutableBytesWritable.class, org.apache.hadoop.hbase.KeyValue.class, org.apache.spark.sql.types.StructType.class, StructField[].class, StructField.class, org.apache.spark.sql.types.LongType$.class, org.apache.spark.sql.types.Metadata.class, org.apache.spark.sql.types.StringType$.class, Class.forName( "org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage"), Class.forName("scala.reflect.ClassTag$$anon$1"), Class.forName("scala.collection.immutable.Set$EmptySet$"), Class.forName("org.apache.spark.sql.types.DoubleType$") ``` ### Vertex/Edge example (问题点 / 边数据举例) _No response_ ### Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构) _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@hugegraph.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
