[
https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824923#comment-16824923
]
XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 9:00 AM:
--------------------------------------------------------------
h2. *Try2*
{code:java}
## remove old version kafka lib
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
## remove mapdreduce lib cache
rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache
## remove kafka-clients-0.8.2.1.jar in mapreduce.tar.gz(HDFS)
hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz .
hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz
/hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori
tar zxf mapreduce.tar.gz
rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar
tar zcf mapreduce.tar.gz hadoop/
hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce
## Restart kylin and try again
{code}
Though it looks so tricky, but this can work without repackage.
was (Author: hit_lacus):
h2. *Try2*
{code:java}
## remove old version kafka lib
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
## remove mapdreduce lib cache
rm -rf /data/hadoop/yarn/local/usercache/kylin-hdp3/appcache
## remove kafka-clients-0.8.2.1.jar in mapreduce.tar.gz(HDFS)
hadoop fs -copyToLocal /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz .
hadoop fs -mv /hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz
/hdp/apps/3.0.1.0-187/mapreduce/mapreduce.tar.gz.ori
tar zxf mapreduce.tar.gz
rm hadoop/share/hadoop/tools/lib/kafka-clients-0.8.2.1.jar
tar zcf mapreduce.tar.gz hadoop/
hadoop fs -copyFromLocal mapreduce.tar.gz /hdp/apps/3.0.1.0-187/mapreduce
{code}
In my env, this can work.
> NRT Streaming cannot work at HDP3.0
> -----------------------------------
>
> Key: KYLIN-3970
> URL: https://issues.apache.org/jira/browse/KYLIN-3970
> Project: Kylin
> Issue Type: Bug
> Components: NRT Streaming
> Affects Versions: v2.6.1
> Reporter: XiaoXiang Yu
> Assignee: XiaoXiang Yu
> Priority: Major
> Labels: HDP, Hadoop3
> Attachments: image-2019-04-24-15-38-10-098.png
>
>
> We found that when using
> [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz],
> NRT streaming cannot work in serval step for jar conflict problem.
> h2. *Env*
> |HDFS|3.1.1|
> |YARN|3.1.1|
> |MapReduce2|3.1.1|
> |Hive|3.1.0|
> |HBase|2.0.0|
> |ZooKeeper|3.4.6|
> |HDP|3.0.1.0|
>
>
> !image-2019-04-24-15-38-10-098.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
