Re: Caused by: java.lang.NoClassDefFoundError: org/apache/spark/api/java/function/Function
这个参数配置了么，*kylin.engine.spark-conf.spark.yarn.archive，* 是否配置为自己集群所在的hdfs目录，而不是文档上的hdfs://sandbox.hortonworks.com:8020 On Wed, Aug 14, 2019 at 2:37 PM 邵志鹏 wrote: > 2019-08-14T14:31:09,188 INFO [http-bio-7070-exec-7] > org.apache.kylin.cube.CubeManager - Updating cube instance > 'kylin_sales_cube_clone_spark' > 2019-08-14T14:31:09,188 INFO [http-bio-7070-exec-7] > org.apache.kylin.cube.CubeManager - Remove segment > kylin_sales_cube_clone_spark[2012010100_20140201235500] > > Action - Delete Segement can Remove segment , and then rebuild it. > > Main problem is > Caused by: java.lang.ClassNotFoundException: > org.apache.spark.api.java.function.Function。 > > 不确定是否是jar报冲突，spark下面所有的jar都一起打包放到hdfs了，里面有spark-core。 > spark版本：2.3.2 > spark-core_2.11-2.3.2 > > > 在 2019-08-14 14:08:03，"邵志鹏" 写道： > > Hi kylin team: > > kylin-2.6.3 > 使用MapReduce已经成功构建smaple cube。 > 然后clone了一份kylin_sales_cube修改为Spark，build 报错。 > > Spark是使用脚本下载的，配置参考了 > http://kylin.apache.org/cn/docs/tutorial/cube_spark.html。 > > > $KYLIN_HOME/bin/download-spark.sh > jar cv0f spark-libs.jar -C $KYLIN_HOME/spark/jars/ . > hadoop fs -mkdir -p /kylin/spark/ > hadoop fs -put spark-libs.jar /kylin/spark/ > ... > > 报错日志： > Caused by: java.lang.NoClassDefFoundError: > org/apache/spark/api/java/function/Function > at > org.apache.kylin.engine.spark.SparkBatchCubingJobBuilder2.(SparkBatchCubingJobBuilder2.java:53) > ~[kylin-engine-spark-2.6.3.jar:2.6.3] > at > org.apache.kylin.engine.spark.SparkBatchCubingEngine2.createBatchCubingJob(SparkBatchCubingEngine2.java:44) > ~[kylin-engine-spark-2.6.3.jar:2.6.3] > at > org.apache.kylin.engine.EngineFactory.createBatchCubingJob(EngineFactory.java:60) > ~[kylin-core-job-2.6.3.jar:2.6.3] > at > org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:234) > ~[kylin-server-base-2.6.3.jar:2.6.3] > at > org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202) > ~[kylin-server-base-2.6.3.jar:2.6.3] > at > org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395) > ~[kylin-server-base-2.6.3.jar:2.6.3] > ... 77 more > Caused by: java.lang.ClassNotFoundException: > org.apache.spark.api.java.function.Function > at > org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1928) > ~[catalina.jar:7.0.91] > at > org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1771) > ~[catalina.jar:7.0.91] > at > org.apache.kylin.engine.spark.SparkBatchCubingJobBuilder2.(SparkBatchCubingJobBuilder2.java:53) > ~[kylin-engine-spark-2.6.3.jar:2.6.3] > at > org.apache.kylin.engine.spark.SparkBatchCubingEngine2.createBatchCubingJob(SparkBatchCubingEngine2.java:44) > ~[kylin-engine-spark-2.6.3.jar:2.6.3] > at > org.apache.kylin.engine.EngineFactory.createBatchCubingJob(EngineFactory.java:60) > ~[kylin-core-job-2.6.3.jar:2.6.3] > at > org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:234) > ~[kylin-server-base-2.6.3.jar:2.6.3] > at > org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202) > ~[kylin-server-base-2.6.3.jar:2.6.3] > at > org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395) > ~[kylin-server-base-2.6.3.jar:2.6.3] > > > 报错之后，再去提交build，又报新的错： > Caused by: org.apache.kylin.rest.exception.BadRequestException: The cube > kylin_sales_cube_clone_spark has segments > [kylin_sales_cube_clone_spark[2012010100_20140201235500]], but none of > them is READY. It's not allowed for parallel building > at > org.apache.kylin.rest.service.JobService.checkAllowParallelBuilding(JobService.java:422) > ~[kylin-server-base-2.6.3.jar:2.6.3] > at > org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:221) > ~[kylin-server-base-2.6.3.jar:2.6.3] > at > org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202) > ~[kylin-server-base-2.6.3.jar:2.6.3] > at > org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395) > ~[kylin-server-base-2.6.3.jar:2.6.3] > ... 77 more > > > Monitor里面是没有新的构建任务的。 > > Thanks. > > By 邵志鹏 > > > > > > > > >
Re：Kylin2.4版本建立model的时候，会生成一个如图以下的yarn任务，它的作用是？
Hi, You may click the“Reload Table”in Data Source Page, and make sure you have checked the “Calculate column cardinality”, it will recompute column cardinality again. Best wishes, Xiaoxiang Yu 发件人: 向伟靖 <865357...@qq.com> 答复: "user@kylin.apache.org" 日期: 2019年8月14日 星期三 15:26 收件人: user 主题: 回复：Kylin2.4版本建立model的时候，会生成一个如图以下的yarn任务，它的作用是？ 噢噢，如果数据量特别大，构建这个产生的任务特别大特别多，是不是可以省略这个流程步骤，因为我是直接把这个任务kill了，没有让它继续执行。如果后期需要重新这个任务，应该怎么启动呢？ -- 原始邮件 -- 发件人: "Johnson"; 发送时间: 2019年8月14日(星期三) 下午3:23 收件人: "user"; 主题: Re:Kylin2.4版本建立model的时候，会生成一个如图以下的yarn任务，它的作用是？ 据我所知，应该是在计算表中各列的基数，对后续build segment有帮主。 [cid:09D57C02@F9389966.ABB7535D] 在 2019-08-14 14:15:07，"向伟靖" <865357...@qq.com> 写道： 你好，我想问一下2.4版本建立model的时候，会生成如图以下的yarn任务，这个任务的作用是？ [cid:6FD2BA07@36CB1219.ABB7535D]
??????Kylin2.4????????model????????????????????????????yarn??????????????????
??kill -- -- ??: "Johnson"; : 2019??8??14??(??) 3:23 ??: "user"; : Re:Kylin2.4modelyarn?? build segment ?? 2019-08-14 14:15:07??"??" <865357...@qq.com> ?? 2.4modelyarn
Caused by: java.lang.NoClassDefFoundError: org/apache/spark/api/java/function/Function
Hi kylin team: kylin-2.6.3 使用MapReduce已经成功构建smaple cube。 然后clone了一份kylin_sales_cube修改为Spark，build 报错。 Spark是使用脚本下载的，配置参考了http://kylin.apache.org/cn/docs/tutorial/cube_spark.html。 $KYLIN_HOME/bin/download-spark.sh jar cv0f spark-libs.jar -C $KYLIN_HOME/spark/jars/ . hadoop fs -mkdir -p /kylin/spark/ hadoop fs -put spark-libs.jar /kylin/spark/ ... 报错日志： Caused by: java.lang.NoClassDefFoundError: org/apache/spark/api/java/function/Function at org.apache.kylin.engine.spark.SparkBatchCubingJobBuilder2.(SparkBatchCubingJobBuilder2.java:53) ~[kylin-engine-spark-2.6.3.jar:2.6.3] at org.apache.kylin.engine.spark.SparkBatchCubingEngine2.createBatchCubingJob(SparkBatchCubingEngine2.java:44) ~[kylin-engine-spark-2.6.3.jar:2.6.3] at org.apache.kylin.engine.EngineFactory.createBatchCubingJob(EngineFactory.java:60) ~[kylin-core-job-2.6.3.jar:2.6.3] at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:234) ~[kylin-server-base-2.6.3.jar:2.6.3] at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202) ~[kylin-server-base-2.6.3.jar:2.6.3] at org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395) ~[kylin-server-base-2.6.3.jar:2.6.3] ... 77 more Caused by: java.lang.ClassNotFoundException: org.apache.spark.api.java.function.Function at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1928) ~[catalina.jar:7.0.91] at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1771) ~[catalina.jar:7.0.91] at org.apache.kylin.engine.spark.SparkBatchCubingJobBuilder2.(SparkBatchCubingJobBuilder2.java:53) ~[kylin-engine-spark-2.6.3.jar:2.6.3] at org.apache.kylin.engine.spark.SparkBatchCubingEngine2.createBatchCubingJob(SparkBatchCubingEngine2.java:44) ~[kylin-engine-spark-2.6.3.jar:2.6.3] at org.apache.kylin.engine.EngineFactory.createBatchCubingJob(EngineFactory.java:60) ~[kylin-core-job-2.6.3.jar:2.6.3] at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:234) ~[kylin-server-base-2.6.3.jar:2.6.3] at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202) ~[kylin-server-base-2.6.3.jar:2.6.3] at org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395) ~[kylin-server-base-2.6.3.jar:2.6.3] 报错之后，再去提交build，又报新的错： Caused by: org.apache.kylin.rest.exception.BadRequestException: The cube kylin_sales_cube_clone_spark has segments [kylin_sales_cube_clone_spark[2012010100_20140201235500]], but none of them is READY. It's not allowed for parallel building at org.apache.kylin.rest.service.JobService.checkAllowParallelBuilding(JobService.java:422) ~[kylin-server-base-2.6.3.jar:2.6.3] at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:221) ~[kylin-server-base-2.6.3.jar:2.6.3] at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202) ~[kylin-server-base-2.6.3.jar:2.6.3] at org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395) ~[kylin-server-base-2.6.3.jar:2.6.3] ... 77 more Monitor里面是没有新的构建任务的。 Thanks. By 邵志鹏
Kylin2.4????????model????????????????????????????yarn??????????????????
2.4modelyarn
??????Re??Kylin2.4????????model????????????????????????????yarn??????????????????
??wiki?? -- -- ??: "Xiaoxiang Yu"; : 2019??8??14??(??)3:41 ??: "user@kylin.apache.org"; : Re??Kylin2.4modelyarn?? Hi, You may click the??Reload Table??in Data Source Page, and make sure you have checked the ??Calculate column cardinality??, it will recompute column cardinality again. Best wishes, Xiaoxiang Yu ??: ?? <865357...@qq.com> : "user@kylin.apache.org" : 2019??8??14?? ?? 15:26 ??: user : ??Kylin2.4modelyarn?? ??kill -- -- ??: "Johnson"; : 2019??8??14??(??) 3:23 ??: "user"; : Re:Kylin2.4modelyarn?? build segment ?? 2019-08-14 14:15:07??"??" <865357...@qq.com> ?? 2.4 modelyarn
Re:Caused by: java.lang.NoClassDefFoundError: org/apache/spark/api/java/function/Function
2019-08-14T14:31:09,188 INFO [http-bio-7070-exec-7] org.apache.kylin.cube.CubeManager - Updating cube instance 'kylin_sales_cube_clone_spark' 2019-08-14T14:31:09,188 INFO [http-bio-7070-exec-7] org.apache.kylin.cube.CubeManager - Remove segment kylin_sales_cube_clone_spark[2012010100_20140201235500] Action - Delete Segement can Remove segment , and then rebuild it. Main problem is Caused by: java.lang.ClassNotFoundException: org.apache.spark.api.java.function.Function。 不确定是否是jar报冲突，spark下面所有的jar都一起打包放到hdfs了，里面有spark-core。 spark版本：2.3.2 spark-core_2.11-2.3.2 在 2019-08-14 14:08:03，"邵志鹏" 写道： Hi kylin team: kylin-2.6.3 使用MapReduce已经成功构建smaple cube。 然后clone了一份kylin_sales_cube修改为Spark，build 报错。 Spark是使用脚本下载的，配置参考了http://kylin.apache.org/cn/docs/tutorial/cube_spark.html。 $KYLIN_HOME/bin/download-spark.sh jar cv0f spark-libs.jar -C $KYLIN_HOME/spark/jars/ . hadoop fs -mkdir -p /kylin/spark/ hadoop fs -put spark-libs.jar /kylin/spark/ ... 报错日志： Caused by: java.lang.NoClassDefFoundError: org/apache/spark/api/java/function/Function at org.apache.kylin.engine.spark.SparkBatchCubingJobBuilder2.(SparkBatchCubingJobBuilder2.java:53) ~[kylin-engine-spark-2.6.3.jar:2.6.3] at org.apache.kylin.engine.spark.SparkBatchCubingEngine2.createBatchCubingJob(SparkBatchCubingEngine2.java:44) ~[kylin-engine-spark-2.6.3.jar:2.6.3] at org.apache.kylin.engine.EngineFactory.createBatchCubingJob(EngineFactory.java:60) ~[kylin-core-job-2.6.3.jar:2.6.3] at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:234) ~[kylin-server-base-2.6.3.jar:2.6.3] at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202) ~[kylin-server-base-2.6.3.jar:2.6.3] at org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395) ~[kylin-server-base-2.6.3.jar:2.6.3] ... 77 more Caused by: java.lang.ClassNotFoundException: org.apache.spark.api.java.function.Function at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1928) ~[catalina.jar:7.0.91] at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1771) ~[catalina.jar:7.0.91] at org.apache.kylin.engine.spark.SparkBatchCubingJobBuilder2.(SparkBatchCubingJobBuilder2.java:53) ~[kylin-engine-spark-2.6.3.jar:2.6.3] at org.apache.kylin.engine.spark.SparkBatchCubingEngine2.createBatchCubingJob(SparkBatchCubingEngine2.java:44) ~[kylin-engine-spark-2.6.3.jar:2.6.3] at org.apache.kylin.engine.EngineFactory.createBatchCubingJob(EngineFactory.java:60) ~[kylin-core-job-2.6.3.jar:2.6.3] at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:234) ~[kylin-server-base-2.6.3.jar:2.6.3] at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202) ~[kylin-server-base-2.6.3.jar:2.6.3] at org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395) ~[kylin-server-base-2.6.3.jar:2.6.3] 报错之后，再去提交build，又报新的错： Caused by: org.apache.kylin.rest.exception.BadRequestException: The cube kylin_sales_cube_clone_spark has segments [kylin_sales_cube_clone_spark[2012010100_20140201235500]], but none of them is READY. It's not allowed for parallel building at org.apache.kylin.rest.service.JobService.checkAllowParallelBuilding(JobService.java:422) ~[kylin-server-base-2.6.3.jar:2.6.3] at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:221) ~[kylin-server-base-2.6.3.jar:2.6.3] at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202) ~[kylin-server-base-2.6.3.jar:2.6.3] at org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395) ~[kylin-server-base-2.6.3.jar:2.6.3] ... 77 more Monitor里面是没有新的构建任务的。 Thanks. By 邵志鹏
Re:Kylin2.4版本建立model的时候，会生成一个如图以下的yarn任务，它的作用是？
据我所知，应该是在计算表中各列的基数，对后续build segment有帮主。 在 2019-08-14 14:15:07，"向伟靖" <865357...@qq.com> 写道： | 你好，我想问一下2.4版本建立model的时候，会生成如图以下的yarn任务，这个任务的作用是？ |
????????????????????????????????
??Kylin
??????????????????????????????????????
-- -- ??: "??"<865357...@qq.com>; : 2019??8??15??(??) 12:03 ??: "user"; : ??Kylin
Re:Re: Caused by: java.lang.NoClassDefFoundError: org/apache/spark/api/java/function/Function
设置了kylin.engine.spark-conf.spark.yarn.archive，我也很奇怪，看上去就是没有加载到spark-libs.jar。 ## manually upload spark-assembly jar to HDFS and then set this property will avoid repeatedly uploading jar at runtime kylin.engine.spark-conf.spark.yarn.archive=hdfs://master:9000/kylin/spark/spark-libs.jar $ hadoop fs -ls hdfs://master:9000/kylin/spark/ Found 1 items -rw-r--r-- 3 admin supergroup 200878688 2019-08-14 17:16 hdfs://master:9000/kylin/spark/spark-libs.jar 在 2019-08-14 14:46:37，"Chao Long" 写道： 这个参数配置了么，kylin.engine.spark-conf.spark.yarn.archive，是否配置为自己集群所在的hdfs目录，而不是文档上的hdfs://sandbox.hortonworks.com:8020 On Wed, Aug 14, 2019 at 2:37 PM 邵志鹏 wrote: 2019-08-14T14:31:09,188 INFO [http-bio-7070-exec-7] org.apache.kylin.cube.CubeManager - Updating cube instance 'kylin_sales_cube_clone_spark' 2019-08-14T14:31:09,188 INFO [http-bio-7070-exec-7] org.apache.kylin.cube.CubeManager - Remove segment kylin_sales_cube_clone_spark[2012010100_20140201235500] Action - Delete Segement can Remove segment , and then rebuild it. Main problem is Caused by: java.lang.ClassNotFoundException: org.apache.spark.api.java.function.Function。 不确定是否是jar报冲突，spark下面所有的jar都一起打包放到hdfs了，里面有spark-core。 spark版本：2.3.2 spark-core_2.11-2.3.2 在 2019-08-14 14:08:03，"邵志鹏" 写道： Hi kylin team: kylin-2.6.3 使用MapReduce已经成功构建smaple cube。 然后clone了一份kylin_sales_cube修改为Spark，build 报错。 Spark是使用脚本下载的，配置参考了http://kylin.apache.org/cn/docs/tutorial/cube_spark.html。 $KYLIN_HOME/bin/download-spark.sh jar cv0f spark-libs.jar -C $KYLIN_HOME/spark/jars/ . hadoop fs -mkdir -p /kylin/spark/ hadoop fs -put spark-libs.jar /kylin/spark/ ... 报错日志： Caused by: java.lang.NoClassDefFoundError: org/apache/spark/api/java/function/Function at org.apache.kylin.engine.spark.SparkBatchCubingJobBuilder2.(SparkBatchCubingJobBuilder2.java:53) ~[kylin-engine-spark-2.6.3.jar:2.6.3] at org.apache.kylin.engine.spark.SparkBatchCubingEngine2.createBatchCubingJob(SparkBatchCubingEngine2.java:44) ~[kylin-engine-spark-2.6.3.jar:2.6.3] at org.apache.kylin.engine.EngineFactory.createBatchCubingJob(EngineFactory.java:60) ~[kylin-core-job-2.6.3.jar:2.6.3] at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:234) ~[kylin-server-base-2.6.3.jar:2.6.3] at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202) ~[kylin-server-base-2.6.3.jar:2.6.3] at org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395) ~[kylin-server-base-2.6.3.jar:2.6.3] ... 77 more Caused by: java.lang.ClassNotFoundException: org.apache.spark.api.java.function.Function at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1928) ~[catalina.jar:7.0.91] at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1771) ~[catalina.jar:7.0.91] at org.apache.kylin.engine.spark.SparkBatchCubingJobBuilder2.(SparkBatchCubingJobBuilder2.java:53) ~[kylin-engine-spark-2.6.3.jar:2.6.3] at org.apache.kylin.engine.spark.SparkBatchCubingEngine2.createBatchCubingJob(SparkBatchCubingEngine2.java:44) ~[kylin-engine-spark-2.6.3.jar:2.6.3] at org.apache.kylin.engine.EngineFactory.createBatchCubingJob(EngineFactory.java:60) ~[kylin-core-job-2.6.3.jar:2.6.3] at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:234) ~[kylin-server-base-2.6.3.jar:2.6.3] at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202) ~[kylin-server-base-2.6.3.jar:2.6.3] at org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395) ~[kylin-server-base-2.6.3.jar:2.6.3] 报错之后，再去提交build，又报新的错： Caused by: org.apache.kylin.rest.exception.BadRequestException: The cube kylin_sales_cube_clone_spark has segments [kylin_sales_cube_clone_spark[2012010100_20140201235500]], but none of them is READY. It's not allowed for parallel building at org.apache.kylin.rest.service.JobService.checkAllowParallelBuilding(JobService.java:422) ~[kylin-server-base-2.6.3.jar:2.6.3] at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:221) ~[kylin-server-base-2.6.3.jar:2.6.3] at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202) ~[kylin-server-base-2.6.3.jar:2.6.3] at org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395) ~[kylin-server-base-2.6.3.jar:2.6.3] ... 77 more Monitor里面是没有新的构建任务的。 Thanks. By 邵志鹏
