Hello Jiang, What's your HDInsight version? We verified HDI 3.5 and 3.6 are compatible, not tested on other versions.
HDI already has "SPARK_HOME" system environment variable set, and pointed to "/usr/hdp/current/spark-client". In this case, Kylin would use the "SPARK_HOME" set in the system, not the embedded one. If the system Spark is not a compatible version (v2.1), it may reports error. So suggest you to explicitly set SPARK_HOME to KYLIN_HOME/spark before start Kylin. Besides, when using community Spark on HDI, if it reports ClassNotFound, you need copy some vendor specific jars from system spark. For your second question, you can check the core-site.xml on HDI node, then use the fully qualified name. 2018-08-03 10:10 GMT+08:00 Jiang Xu <[email protected]>: > Hi guys, > > > > I am trying to set up the apache Kylin at Azure Hdinsight cluster, and I > followed the procedure on below link from Apache Kylin official web, and I > have some questions, and need your help to give some suggestions and > comments. > > Link I checked: http://kylin.apache.org/docs/tutorial/cube_spark.html, > the example is for Hortonworks HDP 2.4 Sandbox VM, and what I am using is > Azure Hdinsight cluster with “spark” type. > > Questions: > > 1, I set up one Azure Hdinsight cluster with “spark” type, is it ok to > install apache Kylin? > > 2, for the configuration, I have one question for your guy, I copied the > below screen for your reference, for the first line in the below rectangle, > for “Kylin,engine.spark-conf.spark.yarn.archive=hdfs:// > sandox.hortworks.com:8020/kylin/spark/spark-libs.jar”, how to configure > the namenode for azure hdinsight cluster case? > > > > BTW: > > I attached the error log when start the Kylin: > > > #### > root@hn0-kylina:/usr/local/apache-kylin-2.4.0-bin-hbase1x# > $KYLIN_HOME/bin/kylin.sh start > Retrieving hadoop conf dir... > KYLIN_HOME is set to /usr/local/apache-kylin-2.4.0-bin-hbase1x > Retrieving hive dependency... > Retrieving hbase dependency... > Retrieving hadoop conf dir... > Retrieving kafka dependency... > Retrieving Spark dependency... > find: тАШ/usr/hdp/current/spark-client/jarsтАЩ: No such file or directory > spark jars not found > #### > > Regards, > Jiangxu > -- Best regards, Shaofeng Shi 史少锋
