Kylin 2.1/2.2/2.3 compiles (and ships) with Spark 2.1; If you want to run with Spark 1.6, please use Kylin 2.0;
2018-03-01 6:45 GMT+08:00 Ted Yu <[email protected]>: > Please use vendor's forum. > > Thanks > > -------- Original message -------- > From: Sonny Heer <[email protected]> > Date: 2/28/18 2:35 PM (GMT-08:00) > To: [email protected] > Subject: Re: running spark on kylin 2.2 > > So when I run it with just spark-submit it gets further. but now there is > a API difference. Does Kylin 2.2 work with Spark 1.6.1 ? This is the > version that comes with HDP 2.4.2.0-258 > > ERROR: > Exception in thread "main" java.lang.NoSuchMethodError: > org.apache.spark.sql.hive.HiveContext.table(Ljava/lang/ > String;)Lorg/apache/spark/sql/Dataset; > at org.apache.kylin.engine.spark.SparkCubingByLayer.execute( > SparkCubingByLayer.java:167 > > > It appears it only supports spark 2.x? Please advise what we can do to > make this work on HDP 2.4... > > Thanks > > On Wed, Feb 28, 2018 at 2:07 PM, Sonny Heer <[email protected]> wrote: > >> I don't see spark-libs.jar under $KYLIN_HOME/spark/jars >> >> per this doc: http://kylin.apache.org/docs21/tutorial/cube_spark.html >> >> On Wed, Feb 28, 2018 at 10:30 AM, Sonny Heer <[email protected]> wrote: >> >>> Hi Billy >>> Looks like the current error is this: >>> >>> Error: Could not find or load main class org.apache.spark.deploy.yarn.A >>> pplicationMaster >>> >>> End of LogType:stderr >>> >>> Thanks >>> >>> On Wed, Feb 28, 2018 at 8:04 AM, Billy Liu <[email protected]> wrote: >>> >>>> Any exception in logs? >>>> >>>> With Warm regards >>>> >>>> Billy Liu >>>> >>>> >>>> 2018-02-28 22:53 GMT+08:00 Sonny Heer <[email protected]>: >>>> > Anyone know what I need to set in order for spark-submit to use the >>>> HDP >>>> > version of spark and not the internal one? >>>> > >>>> > currently i see: >>>> > >>>> > export HADOOP_CONF_DIR=/ebs/kylin/hadoop-conf && >>>> > /ebs/kylin/apache-kylin-2.2.0-bin/spark/bin/spark-submit >>>> > >>>> > >>>> > I see in the kylin.properties files: >>>> > ## Spark conf (default is in spark/conf/spark-defaults.conf) >>>> > >>>> > Although it doesn't how how I can change this to use the HDP >>>> spark-submit. >>>> > >>>> > Also HDP is on 1.6.1 version of spark and kylin internally uses 2.x. >>>> Not >>>> > sure if that matters during submit. I can't seem to get more than 2 >>>> > executors to run without it failing with other errors. We have about >>>> 44 >>>> > slots on our cluster. >>>> > >>>> > Also uncommented: >>>> > ## uncomment for HDP >>>> > >>>> > kylin.engine.spark-conf.spark.driver.extraJavaOptions=-Dhdp. >>>> version=current >>>> > >>>> > kylin.engine.spark-conf.spark.yarn.am.extraJavaOptions=-Dhdp >>>> .version=current >>>> > >>>> > kylin.engine.spark-conf.spark.executor.extraJavaOptions=-Dhd >>>> p.version=current >>>> > >>>> > see attached for other properties set. >>>> >>> >>> >> > -- Best regards, Shaofeng Shi 史少锋
