[
https://issues.apache.org/jira/browse/FLINK-25253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17457845#comment-17457845
]
ghost synth commented on FLINK-25253:
-------------------------------------
Thanks Chesnay Schepler,I added hadoop-mapreduce-client-core-3.1.1.jar to /lib
and it worked.
Maybe we should add some hints to the Hive Connector document. In some cases,
in addition to setting export HADOOP_CLASSPATH=`hadoop classpath`, we also need
to manually add some hadoop dependencies
Otherwise, there will be some confusing situations. Before the solution, I
always thought it might be a dependency conflict problem.
> An ClassNotFoundException of missing Hadoop class occurred when submitting to
> yarn
> ----------------------------------------------------------------------------------
>
> Key: FLINK-25253
> URL: https://issues.apache.org/jira/browse/FLINK-25253
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive, Deployment / YARN
> Affects Versions: 1.14.0
> Environment: Environment version:
> Hadoop 3.1.1
> Hive 3.1.1
> Flink 1.4.0
> kafka 2.6.1
>
> Reporter: ghost synth
> Priority: Minor
> Attachments: FlinkPlaySubmit.scala, flink lib.png,
> original-TropicalaLink-1.0-SNAPSHOT.jar, pom.xml, submit_log.log
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> I use the Hive Table connector to write hive from Kafka and submit to yarn
> successfully, but it will always report during execution
> *Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.mapred.JobConf*
> Before submitting, I executed "export HADOOP_CLASSPATH=`hadoop classpath`" to
> import hadoop dependencies.
> I found in the JM log that the classpath already contains Hadoop
> dependencies, but an exception still occurs
> The original jar that I submitted only contains code and does not contain
> dependencies,The program loads dependencies from the hadoop path and lib
> under the flink directory
> The attachment contains flink lib, code, jar and JM log
> Thanks
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
