Hi Arnault, Thanks for pointing it out. You're right, the file structure of Spark is changed. Kylin 2.1 uses Spark 2.1, so we couldn't take the spark-assembly-1.6.3-hadoop2.6.0.jar .
In Spark 2.1, there is no such an assembly jar; instead, there are many small jars. If you want to cache and reuse them, need create an archive, then upload to HDFS and set the archive path. I followed an answer in this post: https://stackoverflow.com/questions/41112801/property-spark-yarn-jars-how-to-deal-with-it : 1. Create the archive: jar cv0f spark-libs.jar -C $SPARK_HOME/jars/ . 2. Upload to HDFS: hdfs dfs -put spark-libs.jar /some/path/ 3. Set spark.yarn.archive to hdfs:///some/path/spark-libs.jar In kylin.properties, set (replace "mycluster" with your HDFS): kylin.engine. spark-conf.spark.yarn.archive=hdfs://mycluster:8020/kylin/spark/spark-libs.jar 2017-10-04 16:18 GMT+08:00 Arnault <[email protected]>: > Hello, > > I want to try the Spark engine feature. Unfortunately the documentation is > not quiet updated if you use Kylin 2.1.0: > - Kylin file structure is not the same. There is no lib folder in the 2.1.0 > but a jars one. > - spark-assembly-1.6.3-hadoop2.6.0.jar is not available in the jars > folders. > > Do we have to take the spark-assembly-1.6.3-hadoop2.6.0.jar from 2.0.0 and > follow the same procedure or it's different with the 2.1.0 ? > > Thank you in advance. > > Arnault > > > > -- > Sent from: http://apache-kylin.74782.x6.nabble.com/ > -- Best regards, Shaofeng Shi 史少锋
