[
https://issues.apache.org/jira/browse/KYLIN-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16791409#comment-16791409
]
Guangxu Cheng commented on KYLIN-3866:
--------------------------------------
{quote}While, I still not understand the requirment: in which case the
"mapreduce.application.classpath" is absent, and still not allow to use local
cmd to get the classpath? Usually Kylin is installed in a Hadoop client node,
which need the MR/Hive/HBase be well configured, the classpath configuration
should be correct for all Hadoop node. Could you please elaborate the problem
you encountered, and how common it is? Thank you!
{quote}
[~Shaofengshi] Thanks for your review.
1. "mapreduce.application.classpath" is not a required parameter. If the
parameter is not configured, the default value will be used. see:
[https://github.com/apache/hadoop/blob/5d8c8cd568ae54c1be366e786c738b63bd53e271/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java#L219]
In our cluster, we use the default value and do not need the user to configure
the parameter on the client side.
2. We are not deploying kylin on the nodes of the MR cluster because the MR
cluster is maintained by other departments within Tencent and we do not have
permission to deploy other applications on the nodes of the MR cluster. We also
don't know the installation path of the MR cluster.
In Tencent, Kylin is deployed separately, Hadoop/HBase/Hive client is deployed
on the kylin node, but the path of deploying Hadoop client may not be the same
as the MR cluster. So, the path obtained by the command "mapred classpath" is
only the path of Hadoop client not the path of the MR cluster deployment.
Now, kylin forces the user to configure this parameter. If it is not
configured, the path obtained by using the command "mapred classpath" (which
may be wrong value) will be assigned to the parameter, which I think it is
unreasonable.
So, it may be more reasonable for the user to decide whether to use the
parameter.
> Whether to set mapreduce.application.classpath is determined by the user
> ------------------------------------------------------------------------
>
> Key: KYLIN-3866
> URL: https://issues.apache.org/jira/browse/KYLIN-3866
> Project: Kylin
> Issue Type: Bug
> Reporter: Guangxu Cheng
> Priority: Major
> Attachments: KYLIN-3866.master.001.patch
>
>
> In the kylin environment, if there is no configuration parameter
> "mapreduce.application.classpath", then kylin will use the command "mapred
> classpath" to get the local classpath and set it to
> "mapreduce.application.classpath", However, it will cause the following two
> problems:
> 1. The local classpath may not be the same as the MapReduce cluster.
> 2. There is no need to configure the parameter
> "mapreduce.application.classpath" on the client side in some cluster.
> So, it may be a better choice for users to decide whether they need to
> configure this parameter or not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
