[
https://issues.apache.org/jira/browse/KYLIN-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824897#comment-16824897
]
XiaoXiang Yu edited comment on KYLIN-3970 at 4/24/19 8:43 AM:
--------------------------------------------------------------
h2. *Try-1*
remove old version kafka-client.jar
{code:java}
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
{code}
We want to let mapper use kylin's classpath first, so we add
*mapreduce.job.user.classpath.first=true* in _kylin_job_conf.xml_
But this cause more conflict, we will got a NoSuchMethodError caused by Guava.
{code:java}
2019-04-24 08:06:04,378 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.COUNTRY 'US' => reducer 0 2019-04-24 08:06:04,378 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.CATEGORY 'BOOK' => reducer 1 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.DEVICE 'Andriod' => reducer 2 2019-04-24 08:06:04,379
INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.USER_GENDER 'Male' => reducer 3 2019-04-24
08:06:04,379 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.COUNTRY 'KOREA' => reducer 0 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.CATEGORY 'CLOTH' => reducer 1 2019-04-24 08:06:04,379
INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.DEVICE 'iOS' => reducer 2 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.USER_GENDER 'Female' => reducer 3 2019-04-24
08:06:04,381 FATAL [Thread-13]
org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread
Thread[Thread-13,5,main] threw an Error. Shutting down now...
java.lang.NoSuchMethodError:
com.google.common.hash.Hasher.putString(Ljava/lang/CharSequence;)Lcom/google/common/hash/Hasher;
at
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper$CuboidStatCalculator.putRowKeyToHLLNew(FactDistinctColumnsMapper.java:377)
at
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper$CuboidStatCalculator.run(FactDistinctColumnsMapper.java:411)
at java.lang.Thread.run(Thread.java:748) 2019-04-24 08:06:04,385 INFO
[Thread-13] org.apache.hadoop.util.ExitUtil: Exiting with status -1:
ExitException
{code}
was (Author: hit_lacus):
h2. *Try*
remove old version kafka-client.jar
{code}
mv /usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar
/usr/hdp/3.0.1.0-187/hadoop-mapreduce/.//kafka-clients-0.8.2.1.jar.old
{code}
add *mapreduce.job.user.classpath.first=true* in _kylin_job_conf.xml_
We will got a NoSuchMethodError caused by Guava
{code:java}
2019-04-24 08:06:04,378 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.COUNTRY 'US' => reducer 0 2019-04-24 08:06:04,378 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.CATEGORY 'BOOK' => reducer 1 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.DEVICE 'Andriod' => reducer 2 2019-04-24 08:06:04,379
INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.USER_GENDER 'Male' => reducer 3 2019-04-24
08:06:04,379 INFO [main]
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample output:
DEFAULT.SAMPLE_1.COUNTRY 'KOREA' => reducer 0 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.CATEGORY 'CLOTH' => reducer 1 2019-04-24 08:06:04,379
INFO [main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.DEVICE 'iOS' => reducer 2 2019-04-24 08:06:04,379 INFO
[main] org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper: Sample
output: DEFAULT.SAMPLE_1.USER_GENDER 'Female' => reducer 3 2019-04-24
08:06:04,381 FATAL [Thread-13]
org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread
Thread[Thread-13,5,main] threw an Error. Shutting down now...
java.lang.NoSuchMethodError:
com.google.common.hash.Hasher.putString(Ljava/lang/CharSequence;)Lcom/google/common/hash/Hasher;
at
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper$CuboidStatCalculator.putRowKeyToHLLNew(FactDistinctColumnsMapper.java:377)
at
org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper$CuboidStatCalculator.run(FactDistinctColumnsMapper.java:411)
at java.lang.Thread.run(Thread.java:748) 2019-04-24 08:06:04,385 INFO
[Thread-13] org.apache.hadoop.util.ExitUtil: Exiting with status -1:
ExitException
{code}
> NRT Streaming cannot work at HDP3.0
> -----------------------------------
>
> Key: KYLIN-3970
> URL: https://issues.apache.org/jira/browse/KYLIN-3970
> Project: Kylin
> Issue Type: Bug
> Components: NRT Streaming
> Affects Versions: v2.6.1
> Reporter: XiaoXiang Yu
> Assignee: XiaoXiang Yu
> Priority: Major
> Labels: HDP, Hadoop3
> Attachments: image-2019-04-24-15-38-10-098.png
>
>
> We found that when using
> [apache-kylin-2.6.1-bin-hadoop3.tar.gz|https://www.apache.org/dyn/closer.cgi/kylin/apache-kylin-2.6.1/apache-kylin-2.6.1-bin-hadoop3.tar.gz],
> NRT streaming cannot work in serval step for jar conflict problem.
> h2. *Env*
> |HDFS|3.1.1|
> |YARN|3.1.1|
> |MapReduce2|3.1.1|
> |Hive|3.1.0|
> |HBase|2.0.0|
> |ZooKeeper|3.4.6|
> |HDP|3.0.1.0|
>
>
> !image-2019-04-24-15-38-10-098.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
