[
https://issues.apache.org/jira/browse/FLINK-23449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17391381#comment-17391381
]
Kai Chen edited comment on FLINK-23449 at 8/2/21, 6:47 AM:
-----------------------------------------------------------
[~arvid] I checked https://issues.apache.org/jira/browse/FLINK-21319 and I
found these solutions which is also mentioned in
http://apache-flink.147419.n8.nabble.com/Flink-td7866.html:
1. Manually add
MRJobConfig.DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH("$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/\*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/\*")
to Yarn configuration file under yarn.application.classpath
2.Manually add hadoop-mapreduce-client-core.jar to user code jar.
Flink Hive connector can work with above solutions but it is not very
convenient to use (as spark hive). I think we should at least add these 2
solutions to the document.
was (Author: yuchuanchen):
[~arvid] I checked https://issues.apache.org/jira/browse/FLINK-21319 and I
found these solutions which is also mentioned in
http://apache-flink.147419.n8.nabble.com/Flink-td7866.html:
1. Manually add
MRJobConfig.DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH("$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*")
to Yarn configuration file under yarn.application.classpath
2.Manually add hadoop-mapreduce-client-core.jar to user code jar.
Flink Hive connector can work with above solutions but it is not very
convenient to use (as spark hive). I think we should at least add these 2
solutions to the document.
> YarnTaskExecutorRunner does not contains MapReduce classes
> -----------------------------------------------------------
>
> Key: FLINK-23449
> URL: https://issues.apache.org/jira/browse/FLINK-23449
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive, Deployment / YARN
> Affects Versions: 1.11.3
> Environment: flink-1.11
> flink on yarn cluster
> jdk1.8
> hive1.2.1
> hadoop2.7
> hadoop classes is provided with {{export HADOOP_CLASSPATH=`hadoop classpath`
> when submitting test APP. （described in
> [https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/hadoop.html]
> ）}}
> {{}}
> Reporter: Kai Chen
> Priority: Major
> Labels: pull-request-available
>
> I followed instructions described in
> [https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/hive]
> and tested hive streaming sink, met this exception
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.mapred.JobConf
> [http://apache-flink.147419.n8.nabble.com/Flink-td7866.html] met the same
> problem.
>
> I checked TM jvm envs and the code and found that flink only set up
> YARN_APPLICATION_CLASSPATH, but without MAPREDUCE_APPLICATION_CLASSPATH.
> See:
> [https://github.com/apache/flink/blob/ed39fb2efc790af038c1babd4a48847b7b39f91e/flink-yarn/src/main/java/org/apache/flink/yarn/Utils.java#L119]
>
> I think we should add MAPREDUCE_APPLICATION_CLASSPATH as well, as the same as
> spark does.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
