[jira] [Commented] (KYLIN-3393) Building streaming cube do not need to set “KAFKA_HOME” environment variable
[ https://issues.apache.org/jira/browse/KYLIN-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521816#comment-16521816 ] Shaofeng SHI commented on KYLIN-3393: - You can add KAFKA_HOME into $KYLIN_HOME/conf/setenv.sh, that won't pollute your environment. > Building streaming cube do not need to set “KAFKA_HOME” environment variable > - > > Key: KYLIN-3393 > URL: https://issues.apache.org/jira/browse/KYLIN-3393 > Project: Kylin > Issue Type: Improvement >Reporter: wuyingjun >Assignee: wuyingjun >Priority: Major > Attachments: 1.png, 2.png, KYLIN-3393.patch > > > You need hava KAFKA environment and set “KAFKA_HOME” environment variable > when building streaming cube, or else you can not submit a streaming cube > building job, like pic 1； > After we change the kafka dependency scope mode described in the > KYLIN-3393.patch , we can build streaming cube without a kafka environment in > the kylin service node successfully， like pic 2； -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3393) Building streaming cube do not need to set “KAFKA_HOME” environment variable
[ https://issues.apache.org/jira/browse/KYLIN-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509326#comment-16509326 ] Shen Yinjie commented on KYLIN-3393: In fact ,in our enviroment,we set HADOOP_HOME,HIVE_HOME,and related $PATH etc all in kylin.sh. When start Kylin, these enviroment variables could be loaded successfully.:) > Building streaming cube do not need to set “KAFKA_HOME” environment variable > - > > Key: KYLIN-3393 > URL: https://issues.apache.org/jira/browse/KYLIN-3393 > Project: Kylin > Issue Type: Improvement >Reporter: wuyingjun >Assignee: wuyingjun >Priority: Major > Attachments: 1.png, 2.png, KYLIN-3393.patch > > > You need hava KAFKA environment and set “KAFKA_HOME” environment variable > when building streaming cube, or else you can not submit a streaming cube > building job, like pic 1； > After we change the kafka dependency scope mode described in the > KYLIN-3393.patch , we can build streaming cube without a kafka environment in > the kylin service node successfully， like pic 2； -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3393) Building streaming cube do not need to set “KAFKA_HOME” environment variable
[ https://issues.apache.org/jira/browse/KYLIN-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501886#comment-16501886 ] Shaofeng SHI commented on KYLIN-3393: - Hi Yingjun, usually API will be stable, but implementation can change from version to version. With only compiling with public API, Kylin can work with different implementations. This is just HBase: Kylin compiles with HBase 1.1 API, but still can work on HBase 1.3's features. If you do need deploy Kylin with a Kafka client, that is easy, you can re-build a new package with Kafka client, and maybe modify the logic in "find-kafka-dependency.sh". > Building streaming cube do not need to set “KAFKA_HOME” environment variable > - > > Key: KYLIN-3393 > URL: https://issues.apache.org/jira/browse/KYLIN-3393 > Project: Kylin > Issue Type: Improvement >Reporter: wuyingjun >Assignee: wuyingjun >Priority: Major > Attachments: 1.png, 2.png, KYLIN-3393.patch > > > You need hava KAFKA environment and set “KAFKA_HOME” environment variable > when building streaming cube, or else you can not submit a streaming cube > building job, like pic 1； > After we change the kafka dependency scope mode described in the > KYLIN-3393.patch , we can build streaming cube without a kafka environment in > the kylin service node successfully， like pic 2； -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3393) Building streaming cube do not need to set “KAFKA_HOME” environment variable
[ https://issues.apache.org/jira/browse/KYLIN-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499041#comment-16499041 ] wuyingjun commented on KYLIN-3393: -- Hi，[~yimingliu] [~Shaofengshi] ，I know what you mean. But as Shaofeng said ， each node in our cluster have hive and hdfs client， we use slider to start hbase and kylin， the node contains kylin server is random. However, we do not hava kafka client in each node. so we can not run streaming cube in kylin server. I think if the client library has conficts with the server side， you probably can not running the kylin without keeping the dependency libraries. > Building streaming cube do not need to set “KAFKA_HOME” environment variable > - > > Key: KYLIN-3393 > URL: https://issues.apache.org/jira/browse/KYLIN-3393 > Project: Kylin > Issue Type: Improvement >Reporter: wuyingjun >Assignee: wuyingjun >Priority: Major > Attachments: 1.png, 2.png, KYLIN-3393.patch > > > You need hava KAFKA environment and set “KAFKA_HOME” environment variable > when building streaming cube, or else you can not submit a streaming cube > building job, like pic 1； > After we change the kafka dependency scope mode described in the > KYLIN-3393.patch , we can build streaming cube without a kafka environment in > the kylin service node successfully， like pic 2； -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3393) Building streaming cube do not need to set “KAFKA_HOME” environment variable
[ https://issues.apache.org/jira/browse/KYLIN-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16498889#comment-16498889 ] Billy Liu commented on KYLIN-3393: -- [~wuyingjun], Kylin does not depend on the Kafka Server, but the Kafka Client. So you do not need to install Kafka into kylin node, but copy the necessary client library. That's the same thing we are doing with Hadoop/Hive/HBase as Shaofeng said. Kylin keeps the dependency libraries out of the package to make it more compatible with the environment. If Kylin ships the fixed-version client libraries in binary, you have to recompile everything if the client library has conflicts with the server side. > Building streaming cube do not need to set “KAFKA_HOME” environment variable > - > > Key: KYLIN-3393 > URL: https://issues.apache.org/jira/browse/KYLIN-3393 > Project: Kylin > Issue Type: Improvement >Reporter: wuyingjun >Assignee: wuyingjun >Priority: Major > Attachments: 1.png, 2.png, KYLIN-3393.patch > > > You need hava KAFKA environment and set “KAFKA_HOME” environment variable > when building streaming cube, or else you can not submit a streaming cube > building job, like pic 1； > After we change the kafka dependency scope mode described in the > KYLIN-3393.patch , we can build streaming cube without a kafka environment in > the kylin service node successfully， like pic 2； -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3393) Building streaming cube do not need to set “KAFKA_HOME” environment variable
[ https://issues.apache.org/jira/browse/KYLIN-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496316#comment-16496316 ] Shaofeng SHI commented on KYLIN-3393: - Hi yingjun, I undrestand your point. But, to connect with Kafka server, you need install the client in the Kylin node before run it, this is the same as Hive, HBase and other services. The problem you are facing might be, your cluster default installes Hadoop/Hive/HBase clients on each node, but no Kafka clients. > Building streaming cube do not need to set “KAFKA_HOME” environment variable > - > > Key: KYLIN-3393 > URL: https://issues.apache.org/jira/browse/KYLIN-3393 > Project: Kylin > Issue Type: Improvement >Reporter: wuyingjun >Assignee: wuyingjun >Priority: Major > Attachments: 1.png, 2.png, KYLIN-3393.patch > > > You need hava KAFKA environment and set “KAFKA_HOME” environment variable > when building streaming cube, or else you can not submit a streaming cube > building job, like pic 1； > After we change the kafka dependency scope mode described in the > KYLIN-3393.patch , we can build streaming cube without a kafka environment in > the kylin service node successfully， like pic 2； -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3393) Building streaming cube do not need to set “KAFKA_HOME” environment variable
[ https://issues.apache.org/jira/browse/KYLIN-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495244#comment-16495244 ] wuyingjun commented on KYLIN-3393: -- Yes, I know . But hadoop and hbase are kylin storage components while I think kafka as A datasource should not need to install in the kylin server node. Now kylin can not directly use the kafka data source from another node unless we install kafka in the kylin server node. It is not comfortable. > Building streaming cube do not need to set “KAFKA_HOME” environment variable > - > > Key: KYLIN-3393 > URL: https://issues.apache.org/jira/browse/KYLIN-3393 > Project: Kylin > Issue Type: Improvement >Reporter: wuyingjun >Assignee: wuyingjun >Priority: Major > Attachments: 1.png, 2.png, KYLIN-3393.patch > > > You need hava KAFKA environment and set “KAFKA_HOME” environment variable > when building streaming cube, or else you can not submit a streaming cube > building job, like pic 1； > After we change the kafka dependency scope mode described in the > KYLIN-3393.patch , we can build streaming cube without a kafka environment in > the kylin service node successfully， like pic 2； -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3393) Building streaming cube do not need to set “KAFKA_HOME” environment variable
[ https://issues.apache.org/jira/browse/KYLIN-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495142#comment-16495142 ] Shaofeng SHI commented on KYLIN-3393: - hi yingjun, this is by design: Kylin doesn't embed these jars: it expects to use the jar from user's environment. This can avoid compitable issues in various hadoop releases. > Building streaming cube do not need to set “KAFKA_HOME” environment variable > - > > Key: KYLIN-3393 > URL: https://issues.apache.org/jira/browse/KYLIN-3393 > Project: Kylin > Issue Type: Improvement >Reporter: wuyingjun >Assignee: wuyingjun >Priority: Major > Attachments: 1.png, 2.png, KYLIN-3393.patch > > > You need hava KAFKA environment and set “KAFKA_HOME” environment variable > when building streaming cube, or else you can not submit a streaming cube > building job, like pic 1； > After we change the kafka dependency scope mode described in the > KYLIN-3393.patch , we can build streaming cube without a kafka environment in > the kylin service node successfully， like pic 2； -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3393) Building streaming cube do not need to set “KAFKA_HOME” environment variable
[ https://issues.apache.org/jira/browse/KYLIN-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495078#comment-16495078 ] wuyingjun commented on KYLIN-3393: -- [~yimingliu] , Please help me validating this bug. If it exsits , I want to submit a patch to kylin. > Building streaming cube do not need to set “KAFKA_HOME” environment variable > - > > Key: KYLIN-3393 > URL: https://issues.apache.org/jira/browse/KYLIN-3393 > Project: Kylin > Issue Type: Improvement >Reporter: wuyingjun >Assignee: wuyingjun >Priority: Major > Attachments: 1.png, 2.png, KYLIN-3393.patch > > > You need hava KAFKA environment and set “KAFKA_HOME” environment variable > when building streaming cube, or you can not submit a streaming cube building > job, like pic 1； > After we change the kafka dependency scope mode described in the > KYLIN-3393.patch , we can build streaming cube without a kafka environment > successfully， like pic 2； -- This message was sent by Atlassian JIRA (v7.6.3#76005)
