[jira] [Updated] (KYLIN-4552) Add a switch to enable/disable flink cubing step: convert to HFile
[ https://issues.apache.org/jira/browse/KYLIN-4552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harvey Yue updated KYLIN-4552: -- Fix Version/s: v3.1.0 > Add a switch to enable/disable flink cubing step: convert to HFile > -- > > Key: KYLIN-4552 > URL: https://issues.apache.org/jira/browse/KYLIN-4552 > Project: Kylin > Issue Type: Sub-task > Components: Flink Engine >Reporter: Harvey Yue >Assignee: Harvey Yue >Priority: Major > Fix For: v3.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4552) Add a switch to enable/disable flink cubing step: convert to HFile
Harvey Yue created KYLIN-4552: - Summary: Add a switch to enable/disable flink cubing step: convert to HFile Key: KYLIN-4552 URL: https://issues.apache.org/jira/browse/KYLIN-4552 Project: Kylin Issue Type: Sub-task Components: Flink Engine Affects Versions: v3.1.1 Reporter: Harvey Yue Assignee: Harvey Yue -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4552) Add a switch to enable/disable flink cubing step: convert to HFile
[ https://issues.apache.org/jira/browse/KYLIN-4552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harvey Yue updated KYLIN-4552: -- Affects Version/s: (was: v3.1.1) > Add a switch to enable/disable flink cubing step: convert to HFile > -- > > Key: KYLIN-4552 > URL: https://issues.apache.org/jira/browse/KYLIN-4552 > Project: Kylin > Issue Type: Sub-task > Components: Flink Engine >Reporter: Harvey Yue >Assignee: Harvey Yue >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4482) Too many logging segment info with CubeBuildJob step
[ https://issues.apache.org/jira/browse/KYLIN-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harvey Yue resolved KYLIN-4482. --- Resolution: Fixed > Too many logging segment info with CubeBuildJob step > > > Key: KYLIN-4482 > URL: https://issues.apache.org/jira/browse/KYLIN-4482 > Project: Kylin > Issue Type: Bug > Components: Spark Engine >Reporter: Harvey Yue >Assignee: Harvey Yue >Priority: Major > Fix For: v4.0.0-beta > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4483) Avoid to build global dictionaries with empty ColumnDesc collection
[ https://issues.apache.org/jira/browse/KYLIN-4483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harvey Yue resolved KYLIN-4483. --- Resolution: Fixed > Avoid to build global dictionaries with empty ColumnDesc collection > --- > > Key: KYLIN-4483 > URL: https://issues.apache.org/jira/browse/KYLIN-4483 > Project: Kylin > Issue Type: Bug > Components: Spark Engine >Reporter: Harvey Yue >Assignee: Harvey Yue >Priority: Major > Fix For: v4.0.0-beta > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4467) Support TopN by kylin on Parquet
[ https://issues.apache.org/jira/browse/KYLIN-4467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harvey Yue resolved KYLIN-4467. --- Resolution: Fixed > Support TopN by kylin on Parquet > > > Key: KYLIN-4467 > URL: https://issues.apache.org/jira/browse/KYLIN-4467 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Reporter: wangrupeng >Assignee: Harvey Yue >Priority: Major > Fix For: v4.0.0-beta > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KYLIN-4501) Support alluxio to adapt more storage like s3, oss ...
[ https://issues.apache.org/jira/browse/KYLIN-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113896#comment-17113896 ] Harvey Yue edited comment on KYLIN-4501 at 5/22/20, 9:56 AM: - *Kylin and Alluxio related configuration changes:* alluxio-site.properties // fix delete cuboid temp directories issue: alluxio.exception.DirectoryNotEmptyException alluxio.user.file.delete.unchecked=true kylin.properties kylin.env.hdfs-working-dir=*alluxio://192.168.xxx.xxx:19998/kylin* kylin.storage.url=*alluxio* kylin.engine.spark-conf.spark.eventLog.dir=alluxio://192.168.xxx.xxx:19998/kylin/spark-history kylin.engine.spark-conf.spark.history.fs.logDirectory=alluxio://192.168.xxx.xxx:19998/kylin/spark-history *Spark jars:* copy ALLUXIO_HOME/client/alluxio-2.2.0-client.jar to SPARK_HOME/jars was (Author: harveyyue): Kylin and Alluxio related configuration changes: alluxio-site.properties // fix delete cuboid temp directories issue: alluxio.exception.DirectoryNotEmptyException alluxio.user.file.delete.unchecked=true kylin.properties kylin.env.hdfs-working-dir=*alluxio://192.168.xxx.xxx:19998/kylin* kylin.storage.url=*alluxio* kylin.engine.spark-conf.spark.eventLog.dir=alluxio://192.168.xxx.xxx:19998/kylin/spark-history kylin.engine.spark-conf.spark.history.fs.logDirectory=alluxio://192.168.xxx.xxx:19998/kylin/spark-history > Support alluxio to adapt more storage like s3, oss ... > -- > > Key: KYLIN-4501 > URL: https://issues.apache.org/jira/browse/KYLIN-4501 > Project: Kylin > Issue Type: New Feature > Components: Spark Engine >Reporter: Harvey Yue >Assignee: Harvey Yue >Priority: Major > Fix For: v4.0.0-beta > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KYLIN-4501) Support alluxio to adapt more storage like s3, oss ...
[ https://issues.apache.org/jira/browse/KYLIN-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113896#comment-17113896 ] Harvey Yue edited comment on KYLIN-4501 at 5/22/20, 9:52 AM: - Kylin and Alluxio related configuration changes: alluxio-site.properties // fix delete cuboid temp directories issue: alluxio.exception.DirectoryNotEmptyException alluxio.user.file.delete.unchecked=true kylin.properties kylin.env.hdfs-working-dir=*alluxio://192.168.xxx.xxx:19998/kylin* kylin.storage.url=*alluxio* kylin.engine.spark-conf.spark.eventLog.dir=alluxio://192.168.xxx.xxx:19998/kylin/spark-history kylin.engine.spark-conf.spark.history.fs.logDirectory=alluxio://192.168.xxx.xxx:19998/kylin/spark-history was (Author: harveyyue): Kylin and Alluxio related configuration changes: alluxio-site.properties ## fix delete cuboid temp directories issue: alluxio.exception.DirectoryNotEmptyException alluxio.user.file.delete.unchecked=true kylin.properties kylin.env.hdfs-working-dir=*alluxio://192.168.xxx.xxx:19998/kylin* kylin.storage.url=*alluxio* kylin.engine.spark-conf.spark.eventLog.dir=alluxio://192.168.xxx.xxx:19998/kylin/spark-history kylin.engine.spark-conf.spark.history.fs.logDirectory=alluxio://192.168.xxx.xxx:19998/kylin/spark-history > Support alluxio to adapt more storage like s3, oss ... > -- > > Key: KYLIN-4501 > URL: https://issues.apache.org/jira/browse/KYLIN-4501 > Project: Kylin > Issue Type: New Feature > Components: Spark Engine >Reporter: Harvey Yue >Assignee: Harvey Yue >Priority: Major > Fix For: v4.0.0-beta > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KYLIN-4501) Support alluxio to adapt more storage like s3, oss ...
[ https://issues.apache.org/jira/browse/KYLIN-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113896#comment-17113896 ] Harvey Yue edited comment on KYLIN-4501 at 5/22/20, 9:52 AM: - Kylin and Alluxio related configuration changes: alluxio-site.properties ## fix delete cuboid temp directories issue: alluxio.exception.DirectoryNotEmptyException alluxio.user.file.delete.unchecked=true kylin.properties kylin.env.hdfs-working-dir=*alluxio://192.168.xxx.xxx:19998/kylin* kylin.storage.url=*alluxio* kylin.engine.spark-conf.spark.eventLog.dir=alluxio://192.168.xxx.xxx:19998/kylin/spark-history kylin.engine.spark-conf.spark.history.fs.logDirectory=alluxio://192.168.xxx.xxx:19998/kylin/spark-history was (Author: harveyyue): Configurations: kylin.properties # kylin.env.hdfs-working-dir=alluxio://192.168.xxx.xxx:19998/kylin # kylin.storage.url=alluxio # kylin.engine.spark-conf.spark.eventLog.dir=alluxio://192.168.xxx.xxx:19998/kylin/spark-history # kylin.engine.spark-conf.spark.history.fs.logDirectory=alluxio://192.168.xxx.xxx:19998/kylin/spark-history > Support alluxio to adapt more storage like s3, oss ... > -- > > Key: KYLIN-4501 > URL: https://issues.apache.org/jira/browse/KYLIN-4501 > Project: Kylin > Issue Type: New Feature > Components: Spark Engine >Reporter: Harvey Yue >Assignee: Harvey Yue >Priority: Major > Fix For: v4.0.0-beta > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4501) Support alluxio to adapt more storage like s3, oss ...
[ https://issues.apache.org/jira/browse/KYLIN-4501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113896#comment-17113896 ] Harvey Yue commented on KYLIN-4501: --- Configurations: kylin.properties # kylin.env.hdfs-working-dir=alluxio://192.168.xxx.xxx:19998/kylin # kylin.storage.url=alluxio # kylin.engine.spark-conf.spark.eventLog.dir=alluxio://192.168.xxx.xxx:19998/kylin/spark-history # kylin.engine.spark-conf.spark.history.fs.logDirectory=alluxio://192.168.xxx.xxx:19998/kylin/spark-history > Support alluxio to adapt more storage like s3, oss ... > -- > > Key: KYLIN-4501 > URL: https://issues.apache.org/jira/browse/KYLIN-4501 > Project: Kylin > Issue Type: New Feature > Components: Spark Engine >Reporter: Harvey Yue >Assignee: Harvey Yue >Priority: Major > Fix For: v4.0.0-beta > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4502) Add alluxio dependence jar
[ https://issues.apache.org/jira/browse/KYLIN-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harvey Yue updated KYLIN-4502: -- Component/s: Spark Engine > Add alluxio dependence jar > -- > > Key: KYLIN-4502 > URL: https://issues.apache.org/jira/browse/KYLIN-4502 > Project: Kylin > Issue Type: New Feature > Components: Spark Engine >Reporter: Harvey Yue >Assignee: Harvey Yue >Priority: Major > Fix For: v4.0.0-beta > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4502) Add alluxio dependence jar
Harvey Yue created KYLIN-4502: - Summary: Add alluxio dependence jar Key: KYLIN-4502 URL: https://issues.apache.org/jira/browse/KYLIN-4502 Project: Kylin Issue Type: New Feature Reporter: Harvey Yue Assignee: Harvey Yue Fix For: v4.0.0-beta -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4501) Support alluxio to adapt more storage like s3, oss ...
Harvey Yue created KYLIN-4501: - Summary: Support alluxio to adapt more storage like s3, oss ... Key: KYLIN-4501 URL: https://issues.apache.org/jira/browse/KYLIN-4501 Project: Kylin Issue Type: New Feature Components: Spark Engine Reporter: Harvey Yue Assignee: Harvey Yue Fix For: v4.0.0-beta -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-3849) Flink cubing step : convert to HFile
[ https://issues.apache.org/jira/browse/KYLIN-3849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harvey Yue updated KYLIN-3849: -- Attachment: flink_cube_build.jpg > Flink cubing step : convert to HFile > > > Key: KYLIN-3849 > URL: https://issues.apache.org/jira/browse/KYLIN-3849 > Project: Kylin > Issue Type: Sub-task > Components: Flink Engine >Reporter: vinoyang >Assignee: Harvey Yue >Priority: Major > Attachments: flink_cube_build.jpg, hfile.jpg > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-3849) Flink cubing step : convert to HFile
[ https://issues.apache.org/jira/browse/KYLIN-3849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harvey Yue updated KYLIN-3849: -- Attachment: hfile.jpg > Flink cubing step : convert to HFile > > > Key: KYLIN-3849 > URL: https://issues.apache.org/jira/browse/KYLIN-3849 > Project: Kylin > Issue Type: Sub-task > Components: Flink Engine >Reporter: vinoyang >Assignee: Harvey Yue >Priority: Major > Attachments: flink_cube_build.jpg, hfile.jpg > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-3849) Flink cubing step : convert to HFile
[ https://issues.apache.org/jira/browse/KYLIN-3849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17107017#comment-17107017 ] Harvey Yue commented on KYLIN-3849: --- Hi ShaoFeng, 3000w+ records mr need about 10+ minutes, but with flink 1.10 version need 25min, I have tried to adjust flink TaskExecutor parameters with task heap, flink managed memory (could help to improve sorting operator), maybe I still missing other things, please pending this pr, I will do more tunning work. Refer XinTong Song's TaskExecutor memory mode sharing [https://www.bilibili.com/s/video/BV1At4y1U7vH] And please refer the snapshot with flink hfile step > Flink cubing step : convert to HFile > > > Key: KYLIN-3849 > URL: https://issues.apache.org/jira/browse/KYLIN-3849 > Project: Kylin > Issue Type: Sub-task > Components: Flink Engine >Reporter: vinoyang >Assignee: Harvey Yue >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4483) Avoid to build global dictionaries with empty ColumnDesc collection
Harvey Yue created KYLIN-4483: - Summary: Avoid to build global dictionaries with empty ColumnDesc collection Key: KYLIN-4483 URL: https://issues.apache.org/jira/browse/KYLIN-4483 Project: Kylin Issue Type: Bug Components: Spark Engine Reporter: Harvey Yue Assignee: Harvey Yue Fix For: v4.0.0-beta -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4482) Too many logging segment info with CubeBuildJob step
[ https://issues.apache.org/jira/browse/KYLIN-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103034#comment-17103034 ] Harvey Yue commented on KYLIN-4482: --- Adding pr #184 > Too many logging segment info with CubeBuildJob step > > > Key: KYLIN-4482 > URL: https://issues.apache.org/jira/browse/KYLIN-4482 > Project: Kylin > Issue Type: Bug > Components: Spark Engine >Reporter: Harvey Yue >Assignee: Harvey Yue >Priority: Major > Fix For: v4.0.0-beta > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4482) Too many logging segment info with CubeBuildJob step
Harvey Yue created KYLIN-4482: - Summary: Too many logging segment info with CubeBuildJob step Key: KYLIN-4482 URL: https://issues.apache.org/jira/browse/KYLIN-4482 Project: Kylin Issue Type: Bug Components: Spark Engine Reporter: Harvey Yue Assignee: Harvey Yue Fix For: v4.0.0-beta -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4455) Query will fail when set calcite.debug=true
[ https://issues.apache.org/jira/browse/KYLIN-4455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094974#comment-17094974 ] Harvey Yue commented on KYLIN-4455: --- In OLAPProjectRel, we added kylin mesure columns to AbstractRelNode#rowType and the RexNodes to OLAPProjectRel#rewriteProjects instead of Project#exps, but the Project#exps is final property, we couldn't to reassign the new RexNode to exps, so when printing plan occur the exception: Caused by: java.lang.IndexOutOfBoundsException: index (3) must be less than size (3) > Query will fail when set calcite.debug=true > --- > > Key: KYLIN-4455 > URL: https://issues.apache.org/jira/browse/KYLIN-4455 > Project: Kylin > Issue Type: Bug > Components: Storage - Parquet >Reporter: xuekaiqi >Assignee: Harvey Yue >Priority: Major > Fix For: v4.0.0-beta > > > Query will failed when set calcite.debug=true -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KYLIN-4455) Query will fail when set calcite.debug=true
[ https://issues.apache.org/jira/browse/KYLIN-4455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harvey Yue reassigned KYLIN-4455: - Assignee: Harvey Yue > Query will fail when set calcite.debug=true > --- > > Key: KYLIN-4455 > URL: https://issues.apache.org/jira/browse/KYLIN-4455 > Project: Kylin > Issue Type: Bug > Components: Storage - Parquet >Reporter: xuekaiqi >Assignee: Harvey Yue >Priority: Major > Fix For: v4.0.0-beta > > > Query will failed when set calcite.debug=true -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (KYLIN-4467) Support TopN by kylin on Parquet
[ https://issues.apache.org/jira/browse/KYLIN-4467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harvey Yue reopened KYLIN-4467: --- Assignee: Harvey Yue (was: wangrupeng) Reopen this issue when querying topn measure sql get ArrayIndexOutOfBoundsException: at org.apache.kylin.engine.spark2.NExecAndComp.sql(NExecAndComp.java:425) at org.apache.kylin.engine.spark2.NExecAndComp.sql(NExecAndComp.java:425) at org.apache.kylin.engine.spark2.NExecAndComp.sql(NExecAndComp.java:408) at org.apache.kylin.engine.spark2.BuildAndQueryEmptySegmentsTest.testEmptySegments(BuildAndQueryEmptySegmentsTest.java:57) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33) at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230) at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58)Caused by: java.sql.SQLException: exception while executing query: java.lang.ArrayIndexOutOfBoundsException: 12 at org.apache.calcite.avatica.Helper.createException(Helper.java:56) at org.apache.calcite.avatica.Helper.createException(Helper.java:41) at org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:577) at org.apache.calcite.avatica.AvaticaPreparedStatement.executeQuery(AvaticaPreparedStatement.java:137) at org.apache.kylin.engine.spark2.NExecAndComp.queryCube(NExecAndComp.java:455) at org.apache.kylin.engine.spark2.NExecAndComp.queryCubeAndSkipCompute(NExecAndComp.java:431) at org.apache.kylin.engine.spark2.NExecAndComp.sql(NExecAndComp.java:418) ... 27 moreCaused by: java.lang.RuntimeException: java.lang.ArrayIndexOutOfBoundsException: 12 at org.apache.kylin.query.exec.SparkExec.collectToEnumerable(SparkExec.java:46) at Baz.bind(Unknown Source) at org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:365) at org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionImpl.java:301) at org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(CalciteMetaImpl.java:559) at org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java:550) at org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:182) at org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:67) at org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:44) at org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:573) ... 31 moreCaused by: java.lang.ArrayIndexOutOfBoundsException: 12 at org.apache.kylin.query.runtime.RuntimeHelper$$anonfun$gtSchemaToCalciteSchema$2.apply(RuntimeHelper.scala:118) at org.apache.kylin.query.runtime.RuntimeHelper$$anonfun$gtSchemaToCalciteSchema$2.apply(RuntimeHelper.scala:97) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.Iterator$class.foreach(Iterator.scala:893) at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at
[jira] [Resolved] (KYLIN-4402) Missing data size in web UI with flink cubing
[ https://issues.apache.org/jira/browse/KYLIN-4402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harvey Yue resolved KYLIN-4402. --- Resolution: Fixed > Missing data size in web UI with flink cubing > - > > Key: KYLIN-4402 > URL: https://issues.apache.org/jira/browse/KYLIN-4402 > Project: Kylin > Issue Type: Sub-task >Reporter: Harvey Yue >Assignee: Harvey Yue >Priority: Major > Fix For: v3.1.0 > > Attachments: image-2020-03-04-10-58-45-832.png > > > # Missing data size label in web UI > # Cuboid file not configured to compress, reduced the replication numbers -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KYLIN-3849) Flink cubing step : convert to HFile
[ https://issues.apache.org/jira/browse/KYLIN-3849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harvey Yue reassigned KYLIN-3849: - Assignee: Harvey Yue (was: vinoyang) > Flink cubing step : convert to HFile > > > Key: KYLIN-3849 > URL: https://issues.apache.org/jira/browse/KYLIN-3849 > Project: Kylin > Issue Type: Sub-task > Components: Flink Engine >Reporter: vinoyang >Assignee: Harvey Yue >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-3904) Support more dependency jars in FlinkExecutable for FlinkCubeHFile
[ https://issues.apache.org/jira/browse/KYLIN-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17055654#comment-17055654 ] Harvey Yue commented on KYLIN-3904: --- Add "-C file://%s" parameter to flink run command. flink run -m yarn-cluster -ytm 4G -yjm 2G -yD taskmanager.memory.preallocate false -ys 1 -C file:///opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/hbase-server-2.1.0-cdh6.2.0.jar -C file:///opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/hbase-shaded-protobuf-2.1.0.jar -C file:///opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/hbase-metrics-2.1.0-cdh6.2.0.jar -C file:///opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/hbase-metrics-api-2.1.0-cdh6.2.0.jar -C file:///opt/kylin/tomcat/webapps/kylin/WEB-INF/lib/metrics-core-2.2.0.jar -C file:///opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/hbase-protocol-2.1.0-cdh6.2.0.jar -C file:///opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/hbase-protocol-shaded-2.1.0-cdh6.2.0.jar -C file:///opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/hbase-mapreduce-2.1.0-cdh6.2.0.jar -C file:///opt/spark-2.4.4-bin-hadoop2.7/jars/htrace-core-3.1.0-incubating.jar -C file:///opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/hbase-common-2.1.0-cdh6.2.0.jar -C file:///opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/hbase-shaded-miscellaneous-2.1.0.jar -C file:///opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/hbase-hadoop-compat-2.1.0-cdh6.2.0.jar -C file:///opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/hbase-hadoop2-compat-2.1.0-cdh6.2.0.jar -C file:///opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/hbase-client-2.1.0-cdh6.2.0.jar -c org.apache.kylin.common.util.FlinkEntry -p 8 /opt/kylin/lib/kylin-job-3.0.1.jar -className org.apache.kylin.storage.hbase.steps.FlinkCubeHFile -output hdfs://ng-dc-master:8020/kylin/kylin_metadata/kylin-51cc6449-15dd-b81f-940f-23a9cd95bd2e/mileage_cube/hfile -partitions hdfs://ng-dc-master:8020/kylin/kylin_metadata/kylin-51cc6449-15dd-b81f-940f-23a9cd95bd2e/mileage_cube/rowkey_stats/part-r-0_hfile -input hdfs://ng-dc-master:8020/kylin/kylin_metadata/kylin-51cc6449-15dd-b81f-940f-23a9cd95bd2e/mileage_cube/cuboid/level_6_cuboid -enableObjectReuse false -segmentId 66eb63d4-b2dd-1d37-62a6-9abc6f6e1e01 -metaUrl kylin_metadata@hdfs,path=hdfs://ng-dc-master:8020/kylin/kylin_metadata/kylin-51cc6449-15dd-b81f-940f-23a9cd95bd2e/mileage_cube/metadata -cubename mileage_cube -hbaseConfPath hdfs://ng-dc-master:8020/kylin/kylin_metadata/kylin-51cc6449-15dd-b81f-940f-23a9cd95bd2e/hbase-conf.xml > Support more dependency jars in FlinkExecutable for FlinkCubeHFile > -- > > Key: KYLIN-3904 > URL: https://issues.apache.org/jira/browse/KYLIN-3904 > Project: Kylin > Issue Type: Sub-task > Components: Flink Engine >Reporter: vinoyang >Assignee: Harvey Yue >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KYLIN-3904) Support more dependency jars in FlinkExecutable for FlinkCubeHFile
[ https://issues.apache.org/jira/browse/KYLIN-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harvey Yue reassigned KYLIN-3904: - Assignee: Harvey Yue (was: vinoyang) > Support more dependency jars in FlinkExecutable for FlinkCubeHFile > -- > > Key: KYLIN-3904 > URL: https://issues.apache.org/jira/browse/KYLIN-3904 > Project: Kylin > Issue Type: Sub-task > Components: Flink Engine >Reporter: vinoyang >Assignee: Harvey Yue >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-3847) Flink cubing step : fact distinct
[ https://issues.apache.org/jira/browse/KYLIN-3847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17055617#comment-17055617 ] Harvey Yue commented on KYLIN-3847: --- Need set following property to enable flink fact distinct step kylin.engine.flink-fact-distinct=true > Flink cubing step : fact distinct > - > > Key: KYLIN-3847 > URL: https://issues.apache.org/jira/browse/KYLIN-3847 > Project: Kylin > Issue Type: Sub-task > Components: Flink Engine >Reporter: vinoyang >Assignee: Harvey Yue >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KYLIN-3847) Flink cubing step : fact distinct
[ https://issues.apache.org/jira/browse/KYLIN-3847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harvey Yue reassigned KYLIN-3847: - Assignee: Harvey Yue (was: vinoyang) > Flink cubing step : fact distinct > - > > Key: KYLIN-3847 > URL: https://issues.apache.org/jira/browse/KYLIN-3847 > Project: Kylin > Issue Type: Sub-task > Components: Flink Engine >Reporter: vinoyang >Assignee: Harvey Yue >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4402) Missing data size in web UI with flink cubing
[ https://issues.apache.org/jira/browse/KYLIN-4402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17050881#comment-17050881 ] Harvey Yue commented on KYLIN-4402: --- About point 1, need to modify following configuration in $FLINK_HOME/conf/log4j-cli.properties, not printing information to console with default log4j/logback configuration file log4j.rootLogger=INFO, file => log4j.rootLogger=INFO, file, *console* > Missing data size in web UI with flink cubing > - > > Key: KYLIN-4402 > URL: https://issues.apache.org/jira/browse/KYLIN-4402 > Project: Kylin > Issue Type: Sub-task >Reporter: Harvey Yue >Assignee: Harvey Yue >Priority: Major > Fix For: v3.1.0 > > Attachments: image-2020-03-04-10-58-45-832.png > > > # Missing data size label in web UI > # Cuboid file not configured to compress, reduced the replication numbers -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KYLIN-4392) Bump Flink version from 1.9.2 to 1.10.0
[ https://issues.apache.org/jira/browse/KYLIN-4392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harvey Yue reassigned KYLIN-4392: - Assignee: Harvey Yue (was: vinoyang) > Bump Flink version from 1.9.2 to 1.10.0 > --- > > Key: KYLIN-4392 > URL: https://issues.apache.org/jira/browse/KYLIN-4392 > Project: Kylin > Issue Type: Sub-task > Components: Flink Engine >Reporter: vinoyang >Assignee: Harvey Yue >Priority: Major > > Recently, Flink has release v1.10.0. It's a major version that contains some > great features and big changes. We can tentatively upgrade to v1.10.0 after > we upgrade to v1.9.2 firstly. If we find some fatal issues, we can fall back > to v1.9.2. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4402) Missing data size in web UI with flink cubing
Harvey Yue created KYLIN-4402: - Summary: Missing data size in web UI with flink cubing Key: KYLIN-4402 URL: https://issues.apache.org/jira/browse/KYLIN-4402 Project: Kylin Issue Type: Sub-task Reporter: Harvey Yue Assignee: Harvey Yue Fix For: v3.1.0 Attachments: image-2020-03-04-10-58-45-832.png # Missing data size label in web UI # Cuboid file not configured to compress, reduced the replication numbers -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4387) Flink cubing merge step failed
[ https://issues.apache.org/jira/browse/KYLIN-4387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harvey Yue resolved KYLIN-4387. --- Resolution: Fixed > Flink cubing merge step failed > -- > > Key: KYLIN-4387 > URL: https://issues.apache.org/jira/browse/KYLIN-4387 > Project: Kylin > Issue Type: Bug > Components: Flink Engine >Reporter: Harvey Yue >Assignee: Harvey Yue >Priority: Major > Fix For: v3.1.0 > > > When job running this step face following issue, I guess which have two class > loader to loading the class *InstantiatorStrategy* when flink execute the > reduceGroup operator > Workaround：using ByteArray instead of Text in reduceGroup operator > The kylin binary including the flink core jar is the root cause, we need to > change the flink related dependence scope to provided. > > The program finished with the following exception: > org.apache.flink.client.program.ProgramInvocationException: The main method > caused an error: error execute > org.apache.kylin.engine.flink.FlinkCubingMerge. Root cause: Job failed. > (JobID: 7a40a92561cfa972e3c5b975db993fce) > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:593) > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:438) > at > org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:274) > at > org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:746) > at > org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:273) > at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:205) > at > org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1010) > at > org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1083) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836) > at > org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) > at > org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1083) > Caused by: java.lang.RuntimeException: error execute > org.apache.kylin.engine.flink.FlinkCubingMerge. Root cause: Job failed. > (JobID: 7a40a92561cfa972e3c5b975db993fce) > at > org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42) > at org.apache.kylin.common.util.FlinkEntry.main(FlinkEntry.java:46) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:576) > ... 12 more > Caused by: org.apache.flink.client.program.ProgramInvocationException: Job > failed. (JobID: 7a40a92561cfa972e3c5b975db993fce) > at > org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:262) > at > org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:338) > at > org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:326) > at > org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:62) > at > org.apache.kylin.engine.flink.FlinkCubingMerge.execute(FlinkCubingMerge.java:235) > at > org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37) > ... 18 more > Caused by: org.apache.flink.runtime.client.JobExecutionException: Job > execution failed. > at > org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146) > at > org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:259) > ... 23 more > Caused by: java.lang.LinkageError: loader constraint violation: when > resolving method > "com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.setFallbackInstantiatorStrategy(*Lorg/objenesis/strategy/InstantiatorStrategy*;)V" > the class loader (instance of org/apache/flink/util/*ChildFirstClassLoader*) > of the current class, > org/apache/flink/api/java/typeutils/runtime/WritableComparator, and the class > loader (instance of sun/misc/Launcher$*AppClassLoader*) for the method's > defining class, com/esotericsoftware/kryo/Kryo$DefaultInstantiatorStrategy, > have
[jira] [Commented] (KYLIN-4391) Bump Flink version from 1.9.0 to 1.9.2
[ https://issues.apache.org/jira/browse/KYLIN-4391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17049966#comment-17049966 ] Harvey Yue commented on KYLIN-4391: --- After discussion with Vino, I will take this task to verify building cube with Flink 1.9.2 and 1.10.0 version > Bump Flink version from 1.9.0 to 1.9.2 > -- > > Key: KYLIN-4391 > URL: https://issues.apache.org/jira/browse/KYLIN-4391 > Project: Kylin > Issue Type: Sub-task > Components: Flink Engine >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > > Flink 1.9.2 is a stable version in 1.9 series. We should upgrade to this > version for flink engine. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KYLIN-4391) Bump Flink version from 1.9.0 to 1.9.2
[ https://issues.apache.org/jira/browse/KYLIN-4391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harvey Yue reassigned KYLIN-4391: - Assignee: Harvey Yue (was: vinoyang) > Bump Flink version from 1.9.0 to 1.9.2 > -- > > Key: KYLIN-4391 > URL: https://issues.apache.org/jira/browse/KYLIN-4391 > Project: Kylin > Issue Type: Sub-task > Components: Flink Engine >Reporter: vinoyang >Assignee: Harvey Yue >Priority: Major > > Flink 1.9.2 is a stable version in 1.9 series. We should upgrade to this > version for flink engine. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4387) Flink cubing merge step failed
[ https://issues.apache.org/jira/browse/KYLIN-4387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048782#comment-17048782 ] Harvey Yue commented on KYLIN-4387: --- Please ignore the pull request #1125, because the branch had the extra commit issue. > Flink cubing merge step failed > -- > > Key: KYLIN-4387 > URL: https://issues.apache.org/jira/browse/KYLIN-4387 > Project: Kylin > Issue Type: Bug > Components: Flink Engine >Reporter: Harvey Yue >Assignee: Harvey Yue >Priority: Major > Fix For: v3.1.0 > > > When job running this step face following issue, I guess which have two class > loader to loading the class *InstantiatorStrategy* when flink execute the > reduceGroup operator > Workaround：using ByteArray instead of Text in reduceGroup operator > The kylin binary including the flink core jar is the root cause, we need to > change the flink related dependence scope to provided. > > The program finished with the following exception: > org.apache.flink.client.program.ProgramInvocationException: The main method > caused an error: error execute > org.apache.kylin.engine.flink.FlinkCubingMerge. Root cause: Job failed. > (JobID: 7a40a92561cfa972e3c5b975db993fce) > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:593) > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:438) > at > org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:274) > at > org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:746) > at > org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:273) > at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:205) > at > org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1010) > at > org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1083) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836) > at > org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) > at > org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1083) > Caused by: java.lang.RuntimeException: error execute > org.apache.kylin.engine.flink.FlinkCubingMerge. Root cause: Job failed. > (JobID: 7a40a92561cfa972e3c5b975db993fce) > at > org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42) > at org.apache.kylin.common.util.FlinkEntry.main(FlinkEntry.java:46) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:576) > ... 12 more > Caused by: org.apache.flink.client.program.ProgramInvocationException: Job > failed. (JobID: 7a40a92561cfa972e3c5b975db993fce) > at > org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:262) > at > org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:338) > at > org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:326) > at > org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:62) > at > org.apache.kylin.engine.flink.FlinkCubingMerge.execute(FlinkCubingMerge.java:235) > at > org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37) > ... 18 more > Caused by: org.apache.flink.runtime.client.JobExecutionException: Job > execution failed. > at > org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146) > at > org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:259) > ... 23 more > Caused by: java.lang.LinkageError: loader constraint violation: when > resolving method > "com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.setFallbackInstantiatorStrategy(*Lorg/objenesis/strategy/InstantiatorStrategy*;)V" > the class loader (instance of org/apache/flink/util/*ChildFirstClassLoader*) > of the current class, > org/apache/flink/api/java/typeutils/runtime/WritableComparator, and the class > loader (instance of sun/misc/Launcher$*AppClassLoader*) for
[jira] [Commented] (KYLIN-4389) Change Flink jar dependence from compile to provided
[ https://issues.apache.org/jira/browse/KYLIN-4389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048775#comment-17048775 ] Harvey Yue commented on KYLIN-4389: --- Please ignore the pull request #1128, because of the branch had two extra commits issue. > Change Flink jar dependence from compile to provided > - > > Key: KYLIN-4389 > URL: https://issues.apache.org/jira/browse/KYLIN-4389 > Project: Kylin > Issue Type: Sub-task > Components: Flink Engine >Reporter: Harvey Yue >Assignee: Harvey Yue >Priority: Major > Fix For: v3.1.0 > > > Currently, build module engine-flink which including flink related core jar > to kylin package, blow up 70M to package, so I suggest that change flink > dependence scope > One more thing, I guess this fat jar(kylin-job-xxx.jar) cause the issue > KYLIN-4387 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4389) Change Flink jar dependence from compile to provided
[ https://issues.apache.org/jira/browse/KYLIN-4389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048530#comment-17048530 ] Harvey Yue commented on KYLIN-4389: --- I have verified these change in my CDH 6.2 environment, attach part of kylin process status info: /opt/spark/jars/jackson-module-scala_2.11-2.6.7.1.jar:/opt/spark/jars/jackson-module-paranamer-2.7.9.jar:/opt/spark/jars/jackson-module-jaxb-annotations-2.6.7.jar:/opt/spark/jars/parquet-avro-1.10.1.jar -Dkylin.flink.dependency=/opt/flink/lib/flink-table_2.11-1.9.0.jar:/opt/flink/lib/flink-dist_2.11-1.9.0.jar:/opt/flink/lib/flink-table-blink_2.11-1.9.0.jar:/opt/flink/lib/flink-metrics-prometheus_2.11-1.9.0.jar -Dkylin.hadoop.conf.dir=/etc/hadoop/conf -Dkylin.server.host-address=ng-dc-node5:7070 -Dspring.profiles.active=testing org.apache.hadoop.util.RunJar /opt/kylin/bin/../tomcat/bin/bootstrap.jar org.apache.catalina.startup.Bootstrap start > Change Flink jar dependence from compile to provided > - > > Key: KYLIN-4389 > URL: https://issues.apache.org/jira/browse/KYLIN-4389 > Project: Kylin > Issue Type: Sub-task > Components: Flink Engine >Reporter: Harvey Yue >Assignee: Harvey Yue >Priority: Major > Fix For: v3.1.0 > > > Currently, build module engine-flink which including flink related core jar > to kylin package, blow up 70M to package, so I suggest that change flink > dependence scope > One more thing, I guess this fat jar(kylin-job-xxx.jar) cause the issue > KYLIN-4387 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4389) Change Flink jar dependence from compile to provided
Harvey Yue created KYLIN-4389: - Summary: Change Flink jar dependence from compile to provided Key: KYLIN-4389 URL: https://issues.apache.org/jira/browse/KYLIN-4389 Project: Kylin Issue Type: Sub-task Components: Flink Engine Reporter: Harvey Yue Assignee: Harvey Yue Fix For: v3.1.0 Currently, build module engine-flink which including flink related core jar to kylin package, blow up 70M to package, so I suggest that change flink dependence scope One more thing, I guess this fat jar(kylin-job-xxx.jar) cause the issue KYLIN-4387 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KYLIN-4387) Flink cubing merge step failed
[ https://issues.apache.org/jira/browse/KYLIN-4387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17046484#comment-17046484 ] Harvey Yue edited comment on KYLIN-4387 at 2/27/20 11:00 AM: - My env is CDH 6.2.0, and I merge related engine-flink module code into 3.0.x-hadoop3.1 branch to build the binary package was (Author: harveyyue): My env is CDH 6.2.0 > Flink cubing merge step failed > -- > > Key: KYLIN-4387 > URL: https://issues.apache.org/jira/browse/KYLIN-4387 > Project: Kylin > Issue Type: Bug > Components: Flink Engine >Reporter: Harvey Yue >Assignee: Harvey Yue >Priority: Major > Fix For: v3.1.0 > > > When job running this step face following issue, I guess which have two class > loader to loading the class *InstantiatorStrategy* when flink execute the > reduceGroup operator > Workaround：using ByteArray instead of Text in reduceGroup operator > The kylin binary including the flink core jar is the root cause, we need to > change the flink related dependence scope to provided. > > The program finished with the following exception: > org.apache.flink.client.program.ProgramInvocationException: The main method > caused an error: error execute > org.apache.kylin.engine.flink.FlinkCubingMerge. Root cause: Job failed. > (JobID: 7a40a92561cfa972e3c5b975db993fce) > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:593) > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:438) > at > org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:274) > at > org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:746) > at > org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:273) > at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:205) > at > org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1010) > at > org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1083) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836) > at > org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) > at > org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1083) > Caused by: java.lang.RuntimeException: error execute > org.apache.kylin.engine.flink.FlinkCubingMerge. Root cause: Job failed. > (JobID: 7a40a92561cfa972e3c5b975db993fce) > at > org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42) > at org.apache.kylin.common.util.FlinkEntry.main(FlinkEntry.java:46) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:576) > ... 12 more > Caused by: org.apache.flink.client.program.ProgramInvocationException: Job > failed. (JobID: 7a40a92561cfa972e3c5b975db993fce) > at > org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:262) > at > org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:338) > at > org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:326) > at > org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:62) > at > org.apache.kylin.engine.flink.FlinkCubingMerge.execute(FlinkCubingMerge.java:235) > at > org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37) > ... 18 more > Caused by: org.apache.flink.runtime.client.JobExecutionException: Job > execution failed. > at > org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146) > at > org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:259) > ... 23 more > Caused by: java.lang.LinkageError: loader constraint violation: when > resolving method > "com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.setFallbackInstantiatorStrategy(*Lorg/objenesis/strategy/InstantiatorStrategy*;)V" > the class loader (instance of org/apache/flink/util/*ChildFirstClassLoader*) > of the current class, >
[jira] [Commented] (KYLIN-4387) Flink cubing merge step failed
[ https://issues.apache.org/jira/browse/KYLIN-4387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17046484#comment-17046484 ] Harvey Yue commented on KYLIN-4387: --- My env is CDH 6.2.0 > Flink cubing merge step failed > -- > > Key: KYLIN-4387 > URL: https://issues.apache.org/jira/browse/KYLIN-4387 > Project: Kylin > Issue Type: Bug > Components: Flink Engine >Reporter: Harvey Yue >Assignee: Harvey Yue >Priority: Major > Fix For: v3.1.0 > > > When job running this step face following issue, I guess which have two class > loader to loading the class *InstantiatorStrategy* when flink execute the > reduceGroup operator > Workaround：using ByteArray instead of Text in reduceGroup operator > The kylin binary including the flink core jar is the root cause, we need to > change the flink related dependence scope to provided. > > The program finished with the following exception: > org.apache.flink.client.program.ProgramInvocationException: The main method > caused an error: error execute > org.apache.kylin.engine.flink.FlinkCubingMerge. Root cause: Job failed. > (JobID: 7a40a92561cfa972e3c5b975db993fce) > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:593) > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:438) > at > org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:274) > at > org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:746) > at > org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:273) > at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:205) > at > org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1010) > at > org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1083) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836) > at > org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) > at > org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1083) > Caused by: java.lang.RuntimeException: error execute > org.apache.kylin.engine.flink.FlinkCubingMerge. Root cause: Job failed. > (JobID: 7a40a92561cfa972e3c5b975db993fce) > at > org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42) > at org.apache.kylin.common.util.FlinkEntry.main(FlinkEntry.java:46) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:576) > ... 12 more > Caused by: org.apache.flink.client.program.ProgramInvocationException: Job > failed. (JobID: 7a40a92561cfa972e3c5b975db993fce) > at > org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:262) > at > org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:338) > at > org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:326) > at > org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:62) > at > org.apache.kylin.engine.flink.FlinkCubingMerge.execute(FlinkCubingMerge.java:235) > at > org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37) > ... 18 more > Caused by: org.apache.flink.runtime.client.JobExecutionException: Job > execution failed. > at > org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146) > at > org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:259) > ... 23 more > Caused by: java.lang.LinkageError: loader constraint violation: when > resolving method > "com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.setFallbackInstantiatorStrategy(*Lorg/objenesis/strategy/InstantiatorStrategy*;)V" > the class loader (instance of org/apache/flink/util/*ChildFirstClassLoader*) > of the current class, > org/apache/flink/api/java/typeutils/runtime/WritableComparator, and the class > loader (instance of sun/misc/Launcher$*AppClassLoader*) for the method's > defining class,
[jira] [Created] (KYLIN-4387) Flink cubing merge step failed
Harvey Yue created KYLIN-4387: - Summary: Flink cubing merge step failed Key: KYLIN-4387 URL: https://issues.apache.org/jira/browse/KYLIN-4387 Project: Kylin Issue Type: Bug Components: Flink Engine Reporter: Harvey Yue When job running this step face following issue, I guess which have two class loader to loading the class *InstantiatorStrategy* when flink execute the reduceGroup operator Workaround：using ByteArray instead of Text in reduceGroup operator The kylin binary including the flink core jar is the root cause, we need to change the flink related dependence scope to provided. The program finished with the following exception: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: error execute org.apache.kylin.engine.flink.FlinkCubingMerge. Root cause: Job failed. (JobID: 7a40a92561cfa972e3c5b975db993fce) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:593) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:438) at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:274) at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:746) at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:273) at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:205) at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1010) at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1083) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836) at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1083) Caused by: java.lang.RuntimeException: error execute org.apache.kylin.engine.flink.FlinkCubingMerge. Root cause: Job failed. (JobID: 7a40a92561cfa972e3c5b975db993fce) at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42) at org.apache.kylin.common.util.FlinkEntry.main(FlinkEntry.java:46) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:576) ... 12 more Caused by: org.apache.flink.client.program.ProgramInvocationException: Job failed. (JobID: 7a40a92561cfa972e3c5b975db993fce) at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:262) at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:338) at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:326) at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:62) at org.apache.kylin.engine.flink.FlinkCubingMerge.execute(FlinkCubingMerge.java:235) at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37) ... 18 more Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed. at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146) at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:259) ... 23 more Caused by: java.lang.LinkageError: loader constraint violation: when resolving method "com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.setFallbackInstantiatorStrategy(*Lorg/objenesis/strategy/InstantiatorStrategy*;)V" the class loader (instance of org/apache/flink/util/*ChildFirstClassLoader*) of the current class, org/apache/flink/api/java/typeutils/runtime/WritableComparator, and the class loader (instance of sun/misc/Launcher$*AppClassLoader*) for the method's defining class, com/esotericsoftware/kryo/Kryo$DefaultInstantiatorStrategy, have different Class objects for the type org/objenesis/strategy/InstantiatorStrategy used in the signature at org.apache.flink.api.java.typeutils.runtime.WritableComparator.checkKryoInitialized(WritableComparator.java:174) at org.apache.flink.api.java.typeutils.runtime.WritableComparator.setReference(WritableComparator.java:67) at
