[jira] [Commented] (KYLIN-3137) Update hive-site.xml locating logic for CDH distro
[ https://issues.apache.org/jira/browse/KYLIN-3137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497653#comment-16497653 ] ASF subversion and git services commented on KYLIN-3137: Commit e0f29e29164d44581c133314ac01c419e4277c35 in kylin's branch refs/heads/master from shaofengshi [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=e0f29e2 ] KYLIN-3137 Spark cubing refine > Update hive-site.xml locating logic for CDH distro > -- > > Key: KYLIN-3137 > URL: https://issues.apache.org/jira/browse/KYLIN-3137 > Project: Kylin > Issue Type: Improvement > Components: Job Engine, Others, Spark Engine >Affects Versions: v2.2.0 >Reporter: Ruslan Dautkhanov >Assignee: Shaofeng SHI >Priority: Major > Labels: cdh, cloudera, configuration, hive > > Getting following exception while trying to build a cube > {noformat} > java.lang.RuntimeException: Cannot find hive-site.xml in > kylin_hadoop_conf_dir: /etc/hadoop/conf. In order to enable spark cubing, you > must set kylin.env.hadoop-conf-dir to a dir which contains at least > core-site.xml, hdfs-site.xml, hive-site.xml, mapred-site.xml, yarn-site.xml > at > org.apache.kylin.engine.spark.SparkExecutable.doWork(SparkExecutable.java:117) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125) > at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125) > at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:144) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} > I am using Kylin binaries for CDH downloaded from kylin.apache.org. > Yes, indeed hive-site.xml is not in /etc/hadoop/conf in Cloudera's > distribution for Hadoop. > hive-site.xml is in /etc/hive/conf, not in /etc/hadoop/conf > The other four files: > core-site.xml, hdfs-site.xml, mapred-site.xml, yarn-site.xml > can be found in /etc/hadoop/conf but, again, not hive-site.xml which is in > /etc/hive/conf . > Would be great to have this adjusted for CDH. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3137) Update hive-site.xml locating logic for CDH distro
[ https://issues.apache.org/jira/browse/KYLIN-3137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497441#comment-16497441 ] ASF subversion and git services commented on KYLIN-3137: Commit ad8d25f7c9646ad85395f5535fb28a97b8eb25bf in kylin's branch refs/heads/KYLIN-3137 from shaofengshi [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=ad8d25f ] KYLIN-3137 update kylin.properties for spark 2.1 > Update hive-site.xml locating logic for CDH distro > -- > > Key: KYLIN-3137 > URL: https://issues.apache.org/jira/browse/KYLIN-3137 > Project: Kylin > Issue Type: Improvement > Components: Job Engine, Others, Spark Engine >Affects Versions: v2.2.0 >Reporter: Ruslan Dautkhanov >Assignee: Shaofeng SHI >Priority: Major > Labels: cdh, cloudera, configuration, hive > > Getting following exception while trying to build a cube > {noformat} > java.lang.RuntimeException: Cannot find hive-site.xml in > kylin_hadoop_conf_dir: /etc/hadoop/conf. In order to enable spark cubing, you > must set kylin.env.hadoop-conf-dir to a dir which contains at least > core-site.xml, hdfs-site.xml, hive-site.xml, mapred-site.xml, yarn-site.xml > at > org.apache.kylin.engine.spark.SparkExecutable.doWork(SparkExecutable.java:117) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125) > at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125) > at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:144) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} > I am using Kylin binaries for CDH downloaded from kylin.apache.org. > Yes, indeed hive-site.xml is not in /etc/hadoop/conf in Cloudera's > distribution for Hadoop. > hive-site.xml is in /etc/hive/conf, not in /etc/hadoop/conf > The other four files: > core-site.xml, hdfs-site.xml, mapred-site.xml, yarn-site.xml > can be found in /etc/hadoop/conf but, again, not hive-site.xml which is in > /etc/hive/conf . > Would be great to have this adjusted for CDH. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3137) Update hive-site.xml locating logic for CDH distro
[ https://issues.apache.org/jira/browse/KYLIN-3137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496541#comment-16496541 ] ASF subversion and git services commented on KYLIN-3137: Commit f11aec308119426222a4bc9d9d6a28f551a16111 in kylin's branch refs/heads/KYLIN-3137 from shaofengshi [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=f11aec3 ] KYLIN-3137 Spark cubing refine > Update hive-site.xml locating logic for CDH distro > -- > > Key: KYLIN-3137 > URL: https://issues.apache.org/jira/browse/KYLIN-3137 > Project: Kylin > Issue Type: Improvement > Components: Job Engine, Others, Spark Engine >Affects Versions: v2.2.0 >Reporter: Ruslan Dautkhanov >Assignee: Shaofeng SHI >Priority: Major > Labels: cdh, cloudera, configuration, hive > > Getting following exception while trying to build a cube > {noformat} > java.lang.RuntimeException: Cannot find hive-site.xml in > kylin_hadoop_conf_dir: /etc/hadoop/conf. In order to enable spark cubing, you > must set kylin.env.hadoop-conf-dir to a dir which contains at least > core-site.xml, hdfs-site.xml, hive-site.xml, mapred-site.xml, yarn-site.xml > at > org.apache.kylin.engine.spark.SparkExecutable.doWork(SparkExecutable.java:117) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125) > at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125) > at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:144) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} > I am using Kylin binaries for CDH downloaded from kylin.apache.org. > Yes, indeed hive-site.xml is not in /etc/hadoop/conf in Cloudera's > distribution for Hadoop. > hive-site.xml is in /etc/hive/conf, not in /etc/hadoop/conf > The other four files: > core-site.xml, hdfs-site.xml, mapred-site.xml, yarn-site.xml > can be found in /etc/hadoop/conf but, again, not hive-site.xml which is in > /etc/hive/conf . > Would be great to have this adjusted for CDH. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3137) Update hive-site.xml locating logic for CDH distro
[ https://issues.apache.org/jira/browse/KYLIN-3137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496282#comment-16496282 ] Shaofeng SHI commented on KYLIN-3137: - I tested several ways to specify the hive-site.xml, but none of them work: * export HIVE_CONF_DIR=/etc/hive/conf && spark-submit ... * --conf spark.yarn.dist.files=path/to/your/hive-site.xml * --files path/to/your/hive-site.xml Only two way can work: * Put hive-site.xml to HADOOP_CONF_DIR (as-is way) * Put hive-site.xml to spark/conf > Update hive-site.xml locating logic for CDH distro > -- > > Key: KYLIN-3137 > URL: https://issues.apache.org/jira/browse/KYLIN-3137 > Project: Kylin > Issue Type: Improvement > Components: Job Engine, Others, Spark Engine >Affects Versions: v2.2.0 >Reporter: Ruslan Dautkhanov >Assignee: Shaofeng SHI >Priority: Major > Labels: cdh, cloudera, configuration, hive > > Getting following exception while trying to build a cube > {noformat} > java.lang.RuntimeException: Cannot find hive-site.xml in > kylin_hadoop_conf_dir: /etc/hadoop/conf. In order to enable spark cubing, you > must set kylin.env.hadoop-conf-dir to a dir which contains at least > core-site.xml, hdfs-site.xml, hive-site.xml, mapred-site.xml, yarn-site.xml > at > org.apache.kylin.engine.spark.SparkExecutable.doWork(SparkExecutable.java:117) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125) > at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125) > at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:144) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} > I am using Kylin binaries for CDH downloaded from kylin.apache.org. > Yes, indeed hive-site.xml is not in /etc/hadoop/conf in Cloudera's > distribution for Hadoop. > hive-site.xml is in /etc/hive/conf, not in /etc/hadoop/conf > The other four files: > core-site.xml, hdfs-site.xml, mapred-site.xml, yarn-site.xml > can be found in /etc/hadoop/conf but, again, not hive-site.xml which is in > /etc/hive/conf . > Would be great to have this adjusted for CDH. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3137) Update hive-site.xml locating logic for CDH distro
[ https://issues.apache.org/jira/browse/KYLIN-3137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16306400#comment-16306400 ] Ruslan Dautkhanov commented on KYLIN-3137: -- Yep that's how I worked around this - created a symlink. Although CM may recreate /etc/hadoop/conf as this directory is managed by Cloudera Manager and my symlink will be gone. I was thinking that Kylin should also autodetect and set HIVE_CONF_DIR (defaults to /etc/hive/conf for CDH), not just HADOOP_CONF_DIR.. > Update hive-site.xml locating logic for CDH distro > -- > > Key: KYLIN-3137 > URL: https://issues.apache.org/jira/browse/KYLIN-3137 > Project: Kylin > Issue Type: Bug > Components: General >Affects Versions: v2.2.0 >Reporter: Ruslan Dautkhanov > Labels: cdh, cloudera, configuration, hive > > Getting following exception while trying to build a cube > {noformat} > java.lang.RuntimeException: Cannot find hive-site.xml in > kylin_hadoop_conf_dir: /etc/hadoop/conf. In order to enable spark cubing, you > must set kylin.env.hadoop-conf-dir to a dir which contains at least > core-site.xml, hdfs-site.xml, hive-site.xml, mapred-site.xml, yarn-site.xml > at > org.apache.kylin.engine.spark.SparkExecutable.doWork(SparkExecutable.java:117) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125) > at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125) > at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:144) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} > I am using Kylin binaries for CDH downloaded from kylin.apache.org. > Yes, indeed hive-site.xml is not in /etc/hadoop/conf in Cloudera's > distribution for Hadoop. > hive-site.xml is in /etc/hive/conf, not in /etc/hadoop/conf > The other four files: > core-site.xml, hdfs-site.xml, mapred-site.xml, yarn-site.xml > can be found in /etc/hadoop/conf but, again, not hive-site.xml which is in > /etc/hive/conf . > Would be great to have this adjusted for CDH. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-3137) Update hive-site.xml locating logic for CDH distro
[ https://issues.apache.org/jira/browse/KYLIN-3137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16305928#comment-16305928 ] Shaofeng SHI commented on KYLIN-3137: - This is normal (not a problem of CDH). When submit Spark, only one HADOOP_CONF_DIR can be set. To be simple, just copy hive-site.xml to or create a simbolic link for it in that folder. > Update hive-site.xml locating logic for CDH distro > -- > > Key: KYLIN-3137 > URL: https://issues.apache.org/jira/browse/KYLIN-3137 > Project: Kylin > Issue Type: Bug > Components: General >Affects Versions: v2.2.0 >Reporter: Ruslan Dautkhanov > Labels: cdh, cloudera, configuration, hive > > Getting following exception while trying to build a cube > {noformat} > java.lang.RuntimeException: Cannot find hive-site.xml in > kylin_hadoop_conf_dir: /etc/hadoop/conf. In order to enable spark cubing, you > must set kylin.env.hadoop-conf-dir to a dir which contains at least > core-site.xml, hdfs-site.xml, hive-site.xml, mapred-site.xml, yarn-site.xml > at > org.apache.kylin.engine.spark.SparkExecutable.doWork(SparkExecutable.java:117) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125) > at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125) > at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:144) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} > I am using Kylin binaries for CDH downloaded from kylin.apache.org. > Yes, indeed hive-site.xml is not in /etc/hadoop/conf in Cloudera's > distribution for Hadoop. > hive-site.xml is in /etc/hive/conf, not in /etc/hadoop/conf > The other four files: > core-site.xml, hdfs-site.xml, mapred-site.xml, yarn-site.xml > can be found in /etc/hadoop/conf but, again, not hive-site.xml which is in > /etc/hive/conf . > Would be great to have this adjusted for CDH. -- This message was sent by Atlassian JIRA (v6.4.14#64029)