successok opened a new issue, #5174: URL: https://github.com/apache/seatunnel/issues/5174
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. ### What happened 23/07/28 10:45:06 INFO discovery.AbstractPluginDiscovery: Discovery plugin jar: Hive at: file:/bigdata/local/seatunnel-2.3.1/connectors/seatunnel/connector-hive-2.3.1.jar 23/07/28 10:45:06 INFO discovery.AbstractPluginDiscovery: Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='Hive'} from classpath 23/07/28 10:45:07 WARN conf.HiveConf: HiveConf of name hive.masking.algo does not exist 23/07/28 10:45:07 INFO util.FileSystemUtils: Start Kerberos authentication using principal xxxx. and keytab xxxx.keytab 23/07/28 10:45:07 INFO security.UserGroupInformation: Login successful for user xxxx.COM using keytab file xxxxx..keytab. Keytab auto renewal enabled : false 23/07/28 10:45:07 INFO util.FileSystemUtils: Kerberos authentication successful Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(Lorg/apache/hadoop/hive/conf/HiveConf;)V at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.<init>(HiveMetaStoreProxy.java:58) at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getInstance(HiveMetaStoreProxy.java:74) at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:59) at org.apache.seatunnel.connectors.seatunnel.hive.source.HiveSource.prepare(HiveSource.java:123) at org.apache.seatunnel.core.starter.spark.execution.SourceExecuteProcessor.initializePlugins(SourceExecuteProcessor.java:104) at org.apache.seatunnel.core.starter.spark.execution.SparkAbstractPluginExecuteProcessor.<init>(SparkAbstractPluginExecuteProcessor.java:49) at org.apache.seatunnel.core.starter.spark.execution.SourceExecuteProcessor.<init>(SourceExecuteProcessor.java:51) at org.apache.seatunnel.core.starter.spark.execution.SparkExecution.<init>(SparkExecution.java:57) at org.apache.seatunnel.core.starter.spark.command.SparkTaskExecuteCommand.execute(SparkTaskExecuteCommand.java:59) at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) at org.apache.seatunnel.core.starter.spark.SeaTunnelSpark.main(SeaTunnelSpark.java:35) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:847) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:922) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:931) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 23/07/28 10:45:07 INFO spark.SparkContext: Invoking stop() from shutdown hook ### SeaTunnel Version seatunnel2.3.1 ### SeaTunnel Config ```conf env { spark.app.name = "apache-seatunnel-2.3.1_hive_to_dr" spark.yarn.queue = "root.hive" spark.executor.instances = 2 spark.executor.cores = 4 spark.driver.memory = "3g" spark.executor.memory = "4g" spark.ui.port = 1300 spark.sql.catalogImplementation = "hive" spark.hadoop.hive.exec.dynamic.partition = "true" spark.hadoop.hive.exec.dynamic.partition.mode = "nonstrict" spark.network.timeout = "1200s" spark.sql.sources.partitionOverwriteMode = "dynamic" spark.yarn.executor.memoryOverhead = 800m spark.kryoserializer.buffer.max = 512m spark.task.maxFailures=0 spark.executor.extraJavaOptions = "-Dfile.encoding=UTF-8" spark.driver.extraJavaOptions = "-Dfile.encoding=UTF-8" job.name = "apache-seatunnel-2.3.1_hive_to_dr" } source { Hive { table_name = "test.xxxx" metastore_uri = "thrift://xxxx:9083" result_table_name = "dwd_xxxx_t1" kerberos_principal = "xxxx" kerberos_keytab_path = "xxxx" hdfs_site_path = "/etc/hadoop/conf/hdfs-site.xml" } } transform { sql { query = "xxxxx" source_table_name = "xxx" result_table_name = "xxxx" } } sink { console { source_table_name = "xxxx" } } ``` ### Running Command ```shell bin/start-seatunnel-spark-2-connector-v2.sh -m local[*] -e client -c ./config/example02.conf ``` ### Error Exception ```log Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(Lorg/apache/hadoop/hive/conf/HiveConf;)V at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.<init>(HiveMetaStoreProxy.java:58) I have checked the directory of Spark and confirmed that there is a `hive-exe.jar` in it. Additionally, the local spark-shell is able to read Hive files correctly.I also copied `hive-exe.jar` to `seatunnel-2.3.1/lib` and `seatunnel-2.3.1/connectors/seatunnel/` ``` ### Zeta or Flink or Spark Version spark 2.4.7 by cloudera CDP 7.1.6 ### Java or Scala Version java 8 ### Screenshots  ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
