[jira] [Commented] (KYLIN-2620) Check for "ORDER BY LIMIT" clause when rewrite SUM query as TOPN
[ https://issues.apache.org/jira/browse/KYLIN-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778974#comment-16778974 ] ASF GitHub Bot commented on KYLIN-2620: --- Wayne1c commented on pull request #489: KYLIN-2620 Make the condition stricter to answer query with topN URL: https://github.com/apache/kylin/pull/489 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Check for "ORDER BY LIMIT" clause when rewrite SUM query as TOPN > > > Key: KYLIN-2620 > URL: https://issues.apache.org/jira/browse/KYLIN-2620 > Project: Kylin > Issue Type: Bug > Components: Measure - TopN >Reporter: Lin Tingmao >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.6.1 > > > When running the following query > select sum(measure) from table group by col_id > if there exists TOPN(measure, group by col_id) measure, > TopNMeasureType.isTopNCompatibleSum()will pass, so the SUM is rewritten > to TOPN. This confuses the user since they may expect a accurate result for > every distinct value of group by column(s). > Kylin should check if "ORDER BY col_id LIMIT topncapacity" is present in the > query to determine whether to rewrite. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] Wayne1c opened a new pull request #489: KYLIN-2620 Make the condition stricter to answer query with topN
Wayne1c opened a new pull request #489: KYLIN-2620 Make the condition stricter to answer query with topN URL: https://github.com/apache/kylin/pull/489 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] asfgit commented on issue #489: KYLIN-2620 Make the condition stricter to answer query with topN
asfgit commented on issue #489: KYLIN-2620 Make the condition stricter to answer query with topN URL: https://github.com/apache/kylin/pull/489#issuecomment-467745333 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] coveralls edited a comment on issue #484: KYLIN-3820 Add a curator-based scheduler
coveralls edited a comment on issue #484: KYLIN-3820 Add a curator-based scheduler URL: https://github.com/apache/kylin/pull/484#issuecomment-467030756 ## Pull Request Test Coverage Report for [Build 4169](https://coveralls.io/builds/21871175) * **114** of **177** **(64.41%)** changed or added relevant lines in **4** files are covered. * **6** unchanged lines in **2** files lost coverage. * Overall coverage remained the same at **27.061%** --- | Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | | :-|--||---: | | [core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java](https://coveralls.io/builds/21871175/source?filename=core-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcommon%2FKylinConfigBase.java#L366) | 0 | 4 | 0.0% | [core-job/src/main/java/org/apache/kylin/job/impl/curator/CuratorLeaderSelector.java](https://coveralls.io/builds/21871175/source?filename=core-job%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fjob%2Fimpl%2Fcurator%2FCuratorLeaderSelector.java#L56) | 30 | 43 | 69.77% | [core-common/src/main/java/org/apache/kylin/common/ServerMode.java](https://coveralls.io/builds/21871175/source?filename=core-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcommon%2FServerMode.java#L21) | 0 | 17 | 0.0% | [core-job/src/main/java/org/apache/kylin/job/impl/curator/CuratorScheduler.java](https://coveralls.io/builds/21871175/source?filename=core-job%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fjob%2Fimpl%2Fcurator%2FCuratorScheduler.java#L87) | 84 | 113 | 74.34% | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [server-base/src/main/java/org/apache/kylin/rest/util/QueryRequestLimits.java](https://coveralls.io/builds/21871175/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Futil%2FQueryRequestLimits.java#L72) | 1 | 47.62% | | [core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://coveralls.io/builds/21871175/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Finmemcubing%2FMemDiskStore.java#L439) | 5 | 77.81% | | Totals | [](https://coveralls.io/builds/21871175) | | :-- | --: | | Change from base [Build 4170](https://coveralls.io/builds/21871775): | 0.0% | | Covered Lines: | 18837 | | Relevant Lines: | 69609 | --- # - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] coveralls edited a comment on issue #484: KYLIN-3820 Add a curator-based scheduler
coveralls edited a comment on issue #484: KYLIN-3820 Add a curator-based scheduler URL: https://github.com/apache/kylin/pull/484#issuecomment-467030756 ## Pull Request Test Coverage Report for [Build 4168](https://coveralls.io/builds/21871044) * **114** of **177** **(64.41%)** changed or added relevant lines in **4** files are covered. * **2** unchanged lines in **2** files lost coverage. * Overall coverage remained the same at **27.057%** --- | Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | | :-|--||---: | | [core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java](https://coveralls.io/builds/21871044/source?filename=core-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcommon%2FKylinConfigBase.java#L366) | 0 | 4 | 0.0% | [core-job/src/main/java/org/apache/kylin/job/impl/curator/CuratorLeaderSelector.java](https://coveralls.io/builds/21871044/source?filename=core-job%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fjob%2Fimpl%2Fcurator%2FCuratorLeaderSelector.java#L56) | 30 | 43 | 69.77% | [core-common/src/main/java/org/apache/kylin/common/ServerMode.java](https://coveralls.io/builds/21871044/source?filename=core-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcommon%2FServerMode.java#L21) | 0 | 17 | 0.0% | [core-job/src/main/java/org/apache/kylin/job/impl/curator/CuratorScheduler.java](https://coveralls.io/builds/21871044/source?filename=core-job%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fjob%2Fimpl%2Fcurator%2FCuratorScheduler.java#L87) | 84 | 113 | 74.34% | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [server-base/src/main/java/org/apache/kylin/rest/util/QueryRequestLimits.java](https://coveralls.io/builds/21871044/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Futil%2FQueryRequestLimits.java#L72) | 1 | 47.62% | | [core-metadata/src/main/java/org/apache/kylin/source/datagen/ColumnGenerator.java](https://coveralls.io/builds/21871044/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fsource%2Fdatagen%2FColumnGenerator.java#L319) | 1 | 81.08% | | Totals | [](https://coveralls.io/builds/21871044) | | :-- | --: | | Change from base [Build 4170](https://coveralls.io/builds/21871775): | 0.0% | | Covered Lines: | 18834 | | Relevant Lines: | 69609 | --- # - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Aaaaaaron commented on issue #488: KYLIN-3820 Add a curator-based scheduler
Aaron commented on issue #488: KYLIN-3820 Add a curator-based scheduler URL: https://github.com/apache/kylin/pull/488#issuecomment-467715454 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3820) Add a curator-based scheduler
[ https://issues.apache.org/jira/browse/KYLIN-3820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778840#comment-16778840 ] ASF GitHub Bot commented on KYLIN-3820: --- Wayne1c commented on pull request #488: KYLIN-3820 Add a curator-based scheduler URL: https://github.com/apache/kylin/pull/488 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add a curator-based scheduler > - > > Key: KYLIN-3820 > URL: https://issues.apache.org/jira/browse/KYLIN-3820 > Project: Kylin > Issue Type: Improvement >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v3.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3820) Add a curator-based scheduler
[ https://issues.apache.org/jira/browse/KYLIN-3820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778838#comment-16778838 ] ASF GitHub Bot commented on KYLIN-3820: --- Wayne1c commented on pull request #484: KYLIN-3820 Add a curator-based scheduler URL: https://github.com/apache/kylin/pull/484 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add a curator-based scheduler > - > > Key: KYLIN-3820 > URL: https://issues.apache.org/jira/browse/KYLIN-3820 > Project: Kylin > Issue Type: Improvement >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v3.0.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] asfgit commented on issue #488: KYLIN-3820 Add a curator-based scheduler
asfgit commented on issue #488: KYLIN-3820 Add a curator-based scheduler URL: https://github.com/apache/kylin/pull/488#issuecomment-467711197 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Wayne1c closed pull request #484: KYLIN-3820 Add a curator-based scheduler
Wayne1c closed pull request #484: KYLIN-3820 Add a curator-based scheduler URL: https://github.com/apache/kylin/pull/484 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Wayne1c opened a new pull request #488: KYLIN-3820 Add a curator-based scheduler
Wayne1c opened a new pull request #488: KYLIN-3820 Add a curator-based scheduler URL: https://github.com/apache/kylin/pull/488 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3831) 唯独超过62生成cuboid 错误
[ https://issues.apache.org/jira/browse/KYLIN-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778830#comment-16778830 ] zhangwei commented on KYLIN-3831: - I agree~, if keep the original design, the value of 'kylin.cube.rowkey.max-size' shouldn't great than 63, I think this should be highlight in below guild book other user my change it to higher value and see wired exception like "Cuboid " + cuboid + " is out of scope 0-" + max) http://kylin.apache.org/docs23/install/configuration.html > 唯独超过62生成cuboid 错误 > - > > Key: KYLIN-3831 > URL: https://issues.apache.org/jira/browse/KYLIN-3831 > Project: Kylin > Issue Type: Wish > Components: Others >Affects Versions: v2.3.1 >Reporter: zhangwei >Assignee: zhangwei >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3834) Add monitor for curator-based scheduler
Jiatao Tao created KYLIN-3834: - Summary: Add monitor for curator-based scheduler Key: KYLIN-3834 URL: https://issues.apache.org/jira/browse/KYLIN-3834 Project: Kylin Issue Type: Improvement Components: Job Engine Reporter: Jiatao Tao Assignee: Jiatao Tao Fix For: v3.0.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] javalife0312 commented on issue #475: kylin - livy
javalife0312 commented on issue #475: kylin - livy URL: https://github.com/apache/kylin/pull/475#issuecomment-467702542 fix when submit job fast and livy return state is running(starting -> running) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] javalife0312 commented on issue #475: kylin - livy
javalife0312 commented on issue #475: kylin - livy URL: https://github.com/apache/kylin/pull/475#issuecomment-467702486 fix state This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3833) Potential OOM in Spark Extract Fact Table Distinct Columns step
[ https://issues.apache.org/jira/browse/KYLIN-3833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778752#comment-16778752 ] ASF subversion and git services commented on KYLIN-3833: Commit a0745710759ed43e87033224831ff3b576088c2b in kylin's branch refs/heads/2.5.x from alexandr.sidorchuk [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=a074571 ] KYLIN-3833 Avoid OOM. > Potential OOM in Spark Extract Fact Table Distinct Columns step > --- > > Key: KYLIN-3833 > URL: https://issues.apache.org/jira/browse/KYLIN-3833 > Project: Kylin > Issue Type: Bug > Components: Spark Engine >Affects Versions: v2.5.1 > Environment: CentOS 7 > 3 workers and 1 master. > 4 cpu, 16GB RAM each >Reporter: Alexander >Assignee: Alexander >Priority: Minor > Fix For: v2.6.1 > > Attachments: KYLIN-3833.master.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3714) com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: scala.reflect.ManifestFactory$$anon$2
[ https://issues.apache.org/jira/browse/KYLIN-3714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778753#comment-16778753 ] ASF subversion and git services commented on KYLIN-3714: Commit a8fe2de6373ae5a32dfb28e5bb4b4be32e90111d in kylin's branch refs/heads/2.5.x from alexandr.sidorchuk [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=a8fe2de ] KYLIN-3714 Register kryo for spark spilling process. > com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: > Class is not registered: scala.reflect.ManifestFactory$$anon$2 > --- > > Key: KYLIN-3714 > URL: https://issues.apache.org/jira/browse/KYLIN-3714 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.5.1 > Environment: kylin2.5.1 > spark2.1.2 >Reporter: wangxianbin >Assignee: Alexander >Priority: Major > Fix For: v2.6.1 > > Attachments: Class is not registered.png, KYLIN-3714.master.001.patch > > > Class is not registered in step "Extract Fact Table Distinct Columns", when > use spark as build engine, error as follow: > com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: > Class is not registered: scala.reflect.ManifestFactory$$anon$2 Note: To > register this class use: > kryo.register(scala.reflect.ManifestFactory$$anon$2.class); Serialization > trace: evidence$1 (org.apache.spark.util.collection.CompactBuffer) at > com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:101) > at > com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:518) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) at > org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:207) > at > org.apache.spark.serializer.SerializationStream.writeValue(Serializer.scala:135) > at > org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:239) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.org$apache$spark$util$collection$ExternalAppendOnlyMap$$spillMemoryIteratorToDisk(ExternalAppendOnlyMap.scala:227) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.spill(ExternalAppendOnlyMap.scala:186) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.spill(ExternalAppendOnlyMap.scala:56) > at org.apache.spark.util.collection.Spillable.maybeSpill(Spillable.scala:97) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.insertAll(ExternalAppendOnlyMap.scala:160) > at org.apache.spark.Aggregator.combineValuesByKey(Aggregator.scala:41) at > org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:91) > at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:109) at > org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at > org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at > org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at > org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at > org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at > org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at > org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at > org.apache.spark.scheduler.Task.run(Task.scala:99) at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325) at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) Caused by: > java.lang.IllegalArgumentException: Class is not registered: > scala.reflect.ManifestFactory$$anon$2 Note: To register this class use: > kryo.register(scala.reflect.ManifestFactory$$anon$2.class); at > com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:488) at > com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:97) > at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:517) at > com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:76) > ... 27 more -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3714) com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: scala.reflect.ManifestFactory$$anon$2
[ https://issues.apache.org/jira/browse/KYLIN-3714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778740#comment-16778740 ] ASF subversion and git services commented on KYLIN-3714: Commit e5c7c0de0a93a89b97d9427f95e3bbf6bada61dc in kylin's branch refs/heads/master from alexandr.sidorchuk [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=e5c7c0d ] KYLIN-3714 Register kryo for spark spilling process. > com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: > Class is not registered: scala.reflect.ManifestFactory$$anon$2 > --- > > Key: KYLIN-3714 > URL: https://issues.apache.org/jira/browse/KYLIN-3714 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.5.1 > Environment: kylin2.5.1 > spark2.1.2 >Reporter: wangxianbin >Assignee: Alexander >Priority: Major > Fix For: v2.6.1 > > Attachments: Class is not registered.png, KYLIN-3714.master.001.patch > > > Class is not registered in step "Extract Fact Table Distinct Columns", when > use spark as build engine, error as follow: > com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: > Class is not registered: scala.reflect.ManifestFactory$$anon$2 Note: To > register this class use: > kryo.register(scala.reflect.ManifestFactory$$anon$2.class); Serialization > trace: evidence$1 (org.apache.spark.util.collection.CompactBuffer) at > com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:101) > at > com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:518) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) at > org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:207) > at > org.apache.spark.serializer.SerializationStream.writeValue(Serializer.scala:135) > at > org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:239) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.org$apache$spark$util$collection$ExternalAppendOnlyMap$$spillMemoryIteratorToDisk(ExternalAppendOnlyMap.scala:227) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.spill(ExternalAppendOnlyMap.scala:186) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.spill(ExternalAppendOnlyMap.scala:56) > at org.apache.spark.util.collection.Spillable.maybeSpill(Spillable.scala:97) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.insertAll(ExternalAppendOnlyMap.scala:160) > at org.apache.spark.Aggregator.combineValuesByKey(Aggregator.scala:41) at > org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:91) > at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:109) at > org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at > org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at > org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at > org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at > org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at > org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at > org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at > org.apache.spark.scheduler.Task.run(Task.scala:99) at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325) at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) Caused by: > java.lang.IllegalArgumentException: Class is not registered: > scala.reflect.ManifestFactory$$anon$2 Note: To register this class use: > kryo.register(scala.reflect.ManifestFactory$$anon$2.class); at > com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:488) at > com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:97) > at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:517) at > com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:76) > ... 27 more -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3833) Potential OOM in Spark Extract Fact Table Distinct Columns step
[ https://issues.apache.org/jira/browse/KYLIN-3833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778746#comment-16778746 ] ASF subversion and git services commented on KYLIN-3833: Commit 0acfa323836778f6d5683b43fa1f4b29123be9c6 in kylin's branch refs/heads/2.6.x from alexandr.sidorchuk [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=0acfa32 ] KYLIN-3833 Avoid OOM. > Potential OOM in Spark Extract Fact Table Distinct Columns step > --- > > Key: KYLIN-3833 > URL: https://issues.apache.org/jira/browse/KYLIN-3833 > Project: Kylin > Issue Type: Bug > Components: Spark Engine >Affects Versions: v2.5.1 > Environment: CentOS 7 > 3 workers and 1 master. > 4 cpu, 16GB RAM each >Reporter: Alexander >Assignee: Alexander >Priority: Minor > Fix For: v2.6.1 > > Attachments: KYLIN-3833.master.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3714) com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: scala.reflect.ManifestFactory$$anon$2
[ https://issues.apache.org/jira/browse/KYLIN-3714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778747#comment-16778747 ] ASF subversion and git services commented on KYLIN-3714: Commit 84b15482715e877c33daadfdc9e0f860433b0904 in kylin's branch refs/heads/2.6.x from alexandr.sidorchuk [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=84b1548 ] KYLIN-3714 Register kryo for spark spilling process. > com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: > Class is not registered: scala.reflect.ManifestFactory$$anon$2 > --- > > Key: KYLIN-3714 > URL: https://issues.apache.org/jira/browse/KYLIN-3714 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.5.1 > Environment: kylin2.5.1 > spark2.1.2 >Reporter: wangxianbin >Assignee: Alexander >Priority: Major > Fix For: v2.6.1 > > Attachments: Class is not registered.png, KYLIN-3714.master.001.patch > > > Class is not registered in step "Extract Fact Table Distinct Columns", when > use spark as build engine, error as follow: > com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: > Class is not registered: scala.reflect.ManifestFactory$$anon$2 Note: To > register this class use: > kryo.register(scala.reflect.ManifestFactory$$anon$2.class); Serialization > trace: evidence$1 (org.apache.spark.util.collection.CompactBuffer) at > com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:101) > at > com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:518) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) at > org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:207) > at > org.apache.spark.serializer.SerializationStream.writeValue(Serializer.scala:135) > at > org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:239) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.org$apache$spark$util$collection$ExternalAppendOnlyMap$$spillMemoryIteratorToDisk(ExternalAppendOnlyMap.scala:227) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.spill(ExternalAppendOnlyMap.scala:186) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.spill(ExternalAppendOnlyMap.scala:56) > at org.apache.spark.util.collection.Spillable.maybeSpill(Spillable.scala:97) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.insertAll(ExternalAppendOnlyMap.scala:160) > at org.apache.spark.Aggregator.combineValuesByKey(Aggregator.scala:41) at > org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:91) > at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:109) at > org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at > org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at > org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at > org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at > org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at > org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at > org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at > org.apache.spark.scheduler.Task.run(Task.scala:99) at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325) at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) Caused by: > java.lang.IllegalArgumentException: Class is not registered: > scala.reflect.ManifestFactory$$anon$2 Note: To register this class use: > kryo.register(scala.reflect.ManifestFactory$$anon$2.class); at > com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:488) at > com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:97) > at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:517) at > com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:76) > ... 27 more -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3833) Potential OOM in Spark Extract Fact Table Distinct Columns step
[ https://issues.apache.org/jira/browse/KYLIN-3833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778736#comment-16778736 ] ASF GitHub Bot commented on KYLIN-3833: --- shaofengshi commented on pull request #486: KYLIN-3833 Avoid OOM. URL: https://github.com/apache/kylin/pull/486 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Potential OOM in Spark Extract Fact Table Distinct Columns step > --- > > Key: KYLIN-3833 > URL: https://issues.apache.org/jira/browse/KYLIN-3833 > Project: Kylin > Issue Type: Bug > Components: Spark Engine >Affects Versions: v2.5.1 > Environment: CentOS 7 > 3 workers and 1 master. > 4 cpu, 16GB RAM each >Reporter: Alexander >Assignee: Alexander >Priority: Minor > Attachments: KYLIN-3833.master.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3714) com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: scala.reflect.ManifestFactory$$anon$2
[ https://issues.apache.org/jira/browse/KYLIN-3714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778739#comment-16778739 ] ASF GitHub Bot commented on KYLIN-3714: --- shaofengshi commented on pull request #487: KYLIN-3714 Register kryo for spark spilling process. URL: https://github.com/apache/kylin/pull/487 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: > Class is not registered: scala.reflect.ManifestFactory$$anon$2 > --- > > Key: KYLIN-3714 > URL: https://issues.apache.org/jira/browse/KYLIN-3714 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.5.1 > Environment: kylin2.5.1 > spark2.1.2 >Reporter: wangxianbin >Assignee: Alexander >Priority: Major > Fix For: v2.6.1 > > Attachments: Class is not registered.png, KYLIN-3714.master.001.patch > > > Class is not registered in step "Extract Fact Table Distinct Columns", when > use spark as build engine, error as follow: > com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: > Class is not registered: scala.reflect.ManifestFactory$$anon$2 Note: To > register this class use: > kryo.register(scala.reflect.ManifestFactory$$anon$2.class); Serialization > trace: evidence$1 (org.apache.spark.util.collection.CompactBuffer) at > com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:101) > at > com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:518) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) at > org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:207) > at > org.apache.spark.serializer.SerializationStream.writeValue(Serializer.scala:135) > at > org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:239) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.org$apache$spark$util$collection$ExternalAppendOnlyMap$$spillMemoryIteratorToDisk(ExternalAppendOnlyMap.scala:227) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.spill(ExternalAppendOnlyMap.scala:186) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.spill(ExternalAppendOnlyMap.scala:56) > at org.apache.spark.util.collection.Spillable.maybeSpill(Spillable.scala:97) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.insertAll(ExternalAppendOnlyMap.scala:160) > at org.apache.spark.Aggregator.combineValuesByKey(Aggregator.scala:41) at > org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:91) > at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:109) at > org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at > org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at > org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at > org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at > org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at > org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at > org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at > org.apache.spark.scheduler.Task.run(Task.scala:99) at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325) at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) Caused by: > java.lang.IllegalArgumentException: Class is not registered: > scala.reflect.ManifestFactory$$anon$2 Note: To register this class use: > kryo.register(scala.reflect.ManifestFactory$$anon$2.class); at > com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:488) at > com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:97) > at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:517) at > com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:76) > ... 27 more -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] shaofengshi merged pull request #487: KYLIN-3714 Register kryo for spark spilling process.
shaofengshi merged pull request #487: KYLIN-3714 Register kryo for spark spilling process. URL: https://github.com/apache/kylin/pull/487 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (KYLIN-3714) com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: scala.reflect.ManifestFactory$$anon$2
[ https://issues.apache.org/jira/browse/KYLIN-3714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-3714: Fix Version/s: v2.6.1 > com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: > Class is not registered: scala.reflect.ManifestFactory$$anon$2 > --- > > Key: KYLIN-3714 > URL: https://issues.apache.org/jira/browse/KYLIN-3714 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.5.1 > Environment: kylin2.5.1 > spark2.1.2 >Reporter: wangxianbin >Assignee: Alexander >Priority: Major > Fix For: v2.6.1 > > Attachments: Class is not registered.png, KYLIN-3714.master.001.patch > > > Class is not registered in step "Extract Fact Table Distinct Columns", when > use spark as build engine, error as follow: > com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: > Class is not registered: scala.reflect.ManifestFactory$$anon$2 Note: To > register this class use: > kryo.register(scala.reflect.ManifestFactory$$anon$2.class); Serialization > trace: evidence$1 (org.apache.spark.util.collection.CompactBuffer) at > com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:101) > at > com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:518) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) at > org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:207) > at > org.apache.spark.serializer.SerializationStream.writeValue(Serializer.scala:135) > at > org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:239) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.org$apache$spark$util$collection$ExternalAppendOnlyMap$$spillMemoryIteratorToDisk(ExternalAppendOnlyMap.scala:227) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.spill(ExternalAppendOnlyMap.scala:186) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.spill(ExternalAppendOnlyMap.scala:56) > at org.apache.spark.util.collection.Spillable.maybeSpill(Spillable.scala:97) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.insertAll(ExternalAppendOnlyMap.scala:160) > at org.apache.spark.Aggregator.combineValuesByKey(Aggregator.scala:41) at > org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:91) > at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:109) at > org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at > org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at > org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at > org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at > org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at > org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at > org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at > org.apache.spark.scheduler.Task.run(Task.scala:99) at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325) at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) Caused by: > java.lang.IllegalArgumentException: Class is not registered: > scala.reflect.ManifestFactory$$anon$2 Note: To register this class use: > kryo.register(scala.reflect.ManifestFactory$$anon$2.class); at > com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:488) at > com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:97) > at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:517) at > com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:76) > ... 27 more -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3833) Potential OOM in Spark Extract Fact Table Distinct Columns step
[ https://issues.apache.org/jira/browse/KYLIN-3833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778737#comment-16778737 ] ASF subversion and git services commented on KYLIN-3833: Commit c152807ab200cbfd2253df94a8a371b8ecccad98 in kylin's branch refs/heads/master from alexandr.sidorchuk [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=c152807 ] KYLIN-3833 Avoid OOM. > Potential OOM in Spark Extract Fact Table Distinct Columns step > --- > > Key: KYLIN-3833 > URL: https://issues.apache.org/jira/browse/KYLIN-3833 > Project: Kylin > Issue Type: Bug > Components: Spark Engine >Affects Versions: v2.5.1 > Environment: CentOS 7 > 3 workers and 1 master. > 4 cpu, 16GB RAM each >Reporter: Alexander >Assignee: Alexander >Priority: Minor > Attachments: KYLIN-3833.master.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3833) Potential OOM in Spark Extract Fact Table Distinct Columns step
[ https://issues.apache.org/jira/browse/KYLIN-3833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-3833: Fix Version/s: v2.6.1 > Potential OOM in Spark Extract Fact Table Distinct Columns step > --- > > Key: KYLIN-3833 > URL: https://issues.apache.org/jira/browse/KYLIN-3833 > Project: Kylin > Issue Type: Bug > Components: Spark Engine >Affects Versions: v2.5.1 > Environment: CentOS 7 > 3 workers and 1 master. > 4 cpu, 16GB RAM each >Reporter: Alexander >Assignee: Alexander >Priority: Minor > Fix For: v2.6.1 > > Attachments: KYLIN-3833.master.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] shaofengshi merged pull request #486: KYLIN-3833 Avoid OOM.
shaofengshi merged pull request #486: KYLIN-3833 Avoid OOM. URL: https://github.com/apache/kylin/pull/486 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3714) com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: scala.reflect.ManifestFactory$$anon$2
[ https://issues.apache.org/jira/browse/KYLIN-3714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778110#comment-16778110 ] ASF GitHub Bot commented on KYLIN-3714: --- Sidonet commented on pull request #487: KYLIN-3714 Register kryo for spark spilling process. URL: https://github.com/apache/kylin/pull/487 After patch was applied in my env, i test it. In my test case, i set properties kylin.engine.spark-conf.spark.driver.memory=2G kylin.engine.spark-conf.spark.executor.memory=512M Launch build a huge cube, with no mapreduce.input.fileinputformat.split at all. The most huge data goes to executor 20 (4.6G) and spark spilling process goes here: 2019-02-26 17:55:13 INFO ShuffleBlockFetcherIterator:54 - Getting 3898 non-empty blocks out of 3898 blocks 2019-02-26 17:55:13 INFO ShuffleBlockFetcherIterator:54 - Started 6 remote fetches in 39 ms 2019-02-26 17:55:15 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 97.5 MB to disk (1 time so far) 2019-02-26 17:55:40 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 97.5 MB to disk (2 times so far) 2019-02-26 17:56:02 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 97.6 MB to disk (3 times so far) 2019-02-26 17:56:25 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 97.5 MB to disk (4 times so far) 2019-02-26 17:56:54 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 99.2 MB to disk (5 times so far) 2019-02-26 17:57:18 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 97.5 MB to disk (6 times so far) 2019-02-26 17:57:42 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 98.4 MB to disk (7 times so far) 2019-02-26 17:58:09 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 99.2 MB to disk (8 times so far) 2019-02-26 17:58:34 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 97.5 MB to disk (9 times so far) 2019-02-26 17:58:54 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 97.5 MB to disk (10 times so far) 2019-02-26 17:59:17 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 99.2 MB to disk (11 times so far) 2019-02-26 17:59:46 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 97.5 MB to disk (12 times so far) 2019-02-26 18:00:13 INFO AbstractHadoopJob:511 - KylinConfig cached for : kylin_metadata@hdfs,path=hdfs://apachai1.apm.local:8020/kylin/kylin_metadata/kylin-5774d00a-bd56-ac28-e867-f9f5cb5d24f3/Test_Cube_2_clone/metadata 2019-02-26 18:00:13 INFO SparkFactDistinct:707 - Partition 19 handling column DEFAULT.SIDA_CASHIER_1.CASHIER_NAME, buildDictInReducer=true 2019-02-26 18:00:13 INFO SparkFactDistinct:716 - Received value: Николенко Наталья Николае Step finished succesfully.  This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: > Class is not registered: scala.reflect.ManifestFactory$$anon$2 > --- > > Key: KYLIN-3714 > URL: https://issues.apache.org/jira/browse/KYLIN-3714 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.5.1 > Environment: kylin2.5.1 > spark2.1.2 >Reporter: wangxianbin >Assignee: Alexander >Priority: Major > Attachments: Class is not registered.png, KYLIN-3714.master.001.patch > > > Class is not registered in step "Extract Fact Table Distinct Columns", when > use spark as build engine, error as follow: > com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: > Class is not registered: scala.reflect.ManifestFactory$$anon$2 Note: To > register this class use: > kryo.register(scala.reflect.ManifestFactory$$anon$2.class); Serialization > trace: evidence$1 (org.apache.spark.util.collection.CompactBuffer) at > com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:101) > at > com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:518) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) at > org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:207) > at >
[jira] [Created] (KYLIN-3833) Potential OOM in Spark Extract Fact Table Distinct Columns step
Alexander created KYLIN-3833: Summary: Potential OOM in Spark Extract Fact Table Distinct Columns step Key: KYLIN-3833 URL: https://issues.apache.org/jira/browse/KYLIN-3833 Project: Kylin Issue Type: Bug Components: Spark Engine Affects Versions: v2.5.1 Reporter: Alexander Assignee: Alexander -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] asfgit commented on issue #487: KYLIN-3714 Register kryo for spark spilling process.
asfgit commented on issue #487: KYLIN-3714 Register kryo for spark spilling process. URL: https://github.com/apache/kylin/pull/487#issuecomment-467496008 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Sidonet opened a new pull request #487: KYLIN-3714 Register kryo for spark spilling process.
Sidonet opened a new pull request #487: KYLIN-3714 Register kryo for spark spilling process. URL: https://github.com/apache/kylin/pull/487 After patch was applied in my env, i test it. In my test case, i set properties kylin.engine.spark-conf.spark.driver.memory=2G kylin.engine.spark-conf.spark.executor.memory=512M Launch build a huge cube, with no mapreduce.input.fileinputformat.split at all. The most huge data goes to executor 20 (4.6G) and spark spilling process goes here: 2019-02-26 17:55:13 INFO ShuffleBlockFetcherIterator:54 - Getting 3898 non-empty blocks out of 3898 blocks 2019-02-26 17:55:13 INFO ShuffleBlockFetcherIterator:54 - Started 6 remote fetches in 39 ms 2019-02-26 17:55:15 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 97.5 MB to disk (1 time so far) 2019-02-26 17:55:40 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 97.5 MB to disk (2 times so far) 2019-02-26 17:56:02 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 97.6 MB to disk (3 times so far) 2019-02-26 17:56:25 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 97.5 MB to disk (4 times so far) 2019-02-26 17:56:54 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 99.2 MB to disk (5 times so far) 2019-02-26 17:57:18 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 97.5 MB to disk (6 times so far) 2019-02-26 17:57:42 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 98.4 MB to disk (7 times so far) 2019-02-26 17:58:09 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 99.2 MB to disk (8 times so far) 2019-02-26 17:58:34 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 97.5 MB to disk (9 times so far) 2019-02-26 17:58:54 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 97.5 MB to disk (10 times so far) 2019-02-26 17:59:17 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 99.2 MB to disk (11 times so far) 2019-02-26 17:59:46 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 97.5 MB to disk (12 times so far) 2019-02-26 18:00:13 INFO AbstractHadoopJob:511 - KylinConfig cached for : kylin_metadata@hdfs,path=hdfs://apachai1.apm.local:8020/kylin/kylin_metadata/kylin-5774d00a-bd56-ac28-e867-f9f5cb5d24f3/Test_Cube_2_clone/metadata 2019-02-26 18:00:13 INFO SparkFactDistinct:707 - Partition 19 handling column DEFAULT.SIDA_CASHIER_1.CASHIER_NAME, buildDictInReducer=true 2019-02-26 18:00:13 INFO SparkFactDistinct:716 - Received value: Николенко Наталья Николае Step finished succesfully.  This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (KYLIN-3714) com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: scala.reflect.ManifestFactory$$anon$2
[ https://issues.apache.org/jira/browse/KYLIN-3714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander updated KYLIN-3714: - Attachment: KYLIN-3714.master.001.patch > com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: > Class is not registered: scala.reflect.ManifestFactory$$anon$2 > --- > > Key: KYLIN-3714 > URL: https://issues.apache.org/jira/browse/KYLIN-3714 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.5.1 > Environment: kylin2.5.1 > spark2.1.2 >Reporter: wangxianbin >Assignee: Alexander >Priority: Major > Attachments: Class is not registered.png, KYLIN-3714.master.001.patch > > > Class is not registered in step "Extract Fact Table Distinct Columns", when > use spark as build engine, error as follow: > com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: > Class is not registered: scala.reflect.ManifestFactory$$anon$2 Note: To > register this class use: > kryo.register(scala.reflect.ManifestFactory$$anon$2.class); Serialization > trace: evidence$1 (org.apache.spark.util.collection.CompactBuffer) at > com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:101) > at > com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:518) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) at > org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:207) > at > org.apache.spark.serializer.SerializationStream.writeValue(Serializer.scala:135) > at > org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:239) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.org$apache$spark$util$collection$ExternalAppendOnlyMap$$spillMemoryIteratorToDisk(ExternalAppendOnlyMap.scala:227) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.spill(ExternalAppendOnlyMap.scala:186) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.spill(ExternalAppendOnlyMap.scala:56) > at org.apache.spark.util.collection.Spillable.maybeSpill(Spillable.scala:97) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.insertAll(ExternalAppendOnlyMap.scala:160) > at org.apache.spark.Aggregator.combineValuesByKey(Aggregator.scala:41) at > org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:91) > at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:109) at > org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at > org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at > org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at > org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at > org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at > org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at > org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at > org.apache.spark.scheduler.Task.run(Task.scala:99) at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325) at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) Caused by: > java.lang.IllegalArgumentException: Class is not registered: > scala.reflect.ManifestFactory$$anon$2 Note: To register this class use: > kryo.register(scala.reflect.ManifestFactory$$anon$2.class); at > com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:488) at > com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:97) > at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:517) at > com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:76) > ... 27 more -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3714) com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: scala.reflect.ManifestFactory$$anon$2
[ https://issues.apache.org/jira/browse/KYLIN-3714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778105#comment-16778105 ] Alexander commented on KYLIN-3714: -- After patch was applied in my env, i test it. In my test case, i set properties kylin.engine.spark-conf.spark.driver.memory=2G kylin.engine.spark-conf.spark.executor.memory=512M Launch build a huge cube, with no mapreduce.input.fileinputformat.split at all. The most huge data goes to executor 20 (4.6G) and spark spilling process goes here: 2019-02-26 17:55:13 INFO ShuffleBlockFetcherIterator:54 - Getting 3898 non-empty blocks out of 3898 blocks 2019-02-26 17:55:13 INFO ShuffleBlockFetcherIterator:54 - Started 6 remote fetches in 39 ms 2019-02-26 17:55:15 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 97.5 MB to disk (1 time so far) 2019-02-26 17:55:40 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 97.5 MB to disk (2 times so far) 2019-02-26 17:56:02 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 97.6 MB to disk (3 times so far) 2019-02-26 17:56:25 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 97.5 MB to disk (4 times so far) 2019-02-26 17:56:54 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 99.2 MB to disk (5 times so far) 2019-02-26 17:57:18 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 97.5 MB to disk (6 times so far) 2019-02-26 17:57:42 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 98.4 MB to disk (7 times so far) 2019-02-26 17:58:09 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 99.2 MB to disk (8 times so far) 2019-02-26 17:58:34 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 97.5 MB to disk (9 times so far) 2019-02-26 17:58:54 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 97.5 MB to disk (10 times so far) 2019-02-26 17:59:17 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 99.2 MB to disk (11 times so far) 2019-02-26 17:59:46 INFO ExternalAppendOnlyMap:54 - Thread 34 spilling in-memory map of 97.5 MB to disk (12 times so far) 2019-02-26 18:00:13 INFO AbstractHadoopJob:511 - KylinConfig cached for : kylin_metadata@hdfs,path=hdfs://apachai1.apm.local:8020/kylin/kylin_metadata/kylin-5774d00a-bd56-ac28-e867-f9f5cb5d24f3/Test_Cube_2_clone/metadata 2019-02-26 18:00:13 INFO SparkFactDistinct:707 - Partition 19 handling column DEFAULT.SIDA_CASHIER_1.CASHIER_NAME, buildDictInReducer=true 2019-02-26 18:00:13 INFO SparkFactDistinct:716 - Received value: Николенко Наталья Николае Step finished succesfully. > com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: > Class is not registered: scala.reflect.ManifestFactory$$anon$2 > --- > > Key: KYLIN-3714 > URL: https://issues.apache.org/jira/browse/KYLIN-3714 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.5.1 > Environment: kylin2.5.1 > spark2.1.2 >Reporter: wangxianbin >Assignee: Alexander >Priority: Major > Attachments: Class is not registered.png > > > Class is not registered in step "Extract Fact Table Distinct Columns", when > use spark as build engine, error as follow: > com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: > Class is not registered: scala.reflect.ManifestFactory$$anon$2 Note: To > register this class use: > kryo.register(scala.reflect.ManifestFactory$$anon$2.class); Serialization > trace: evidence$1 (org.apache.spark.util.collection.CompactBuffer) at > com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:101) > at > com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:518) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) at > org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:207) > at > org.apache.spark.serializer.SerializationStream.writeValue(Serializer.scala:135) > at > org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:239) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.org$apache$spark$util$collection$ExternalAppendOnlyMap$$spillMemoryIteratorToDisk(ExternalAppendOnlyMap.scala:227) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.spill(ExternalAppendOnlyMap.scala:186) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.spill(ExternalAppendOnlyMap.scala:56) > at org.apache.spark.util.collection.Spillable.maybeSpill(Spillable.scala:97) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.insertAll(ExternalAppendOnlyMap.scala:160)
[jira] [Commented] (KYLIN-3833) Potential OOM in Spark Extract Fact Table Distinct Columns step
[ https://issues.apache.org/jira/browse/KYLIN-3833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778103#comment-16778103 ] ASF GitHub Bot commented on KYLIN-3833: --- Sidonet commented on pull request #486: KYLIN-3833 Avoid OOM. URL: https://github.com/apache/kylin/pull/486 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Potential OOM in Spark Extract Fact Table Distinct Columns step > --- > > Key: KYLIN-3833 > URL: https://issues.apache.org/jira/browse/KYLIN-3833 > Project: Kylin > Issue Type: Bug > Components: Spark Engine >Affects Versions: v2.5.1 > Environment: CentOS 7 > 3 workers and 1 master. > 4 cpu, 16GB RAM each >Reporter: Alexander >Assignee: Alexander >Priority: Minor > Attachments: KYLIN-3833.master.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] Sidonet opened a new pull request #486: KYLIN-3833 Avoid OOM.
Sidonet opened a new pull request #486: KYLIN-3833 Avoid OOM. URL: https://github.com/apache/kylin/pull/486 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] asfgit commented on issue #486: KYLIN-3833 Avoid OOM.
asfgit commented on issue #486: KYLIN-3833 Avoid OOM. URL: https://github.com/apache/kylin/pull/486#issuecomment-467494240 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (KYLIN-3833) Potential OOM in Spark Extract Fact Table Distinct Columns step
[ https://issues.apache.org/jira/browse/KYLIN-3833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander updated KYLIN-3833: - Environment: CentOS 7 3 workers and 1 master. 4 cpu, 16GB RAM each > Potential OOM in Spark Extract Fact Table Distinct Columns step > --- > > Key: KYLIN-3833 > URL: https://issues.apache.org/jira/browse/KYLIN-3833 > Project: Kylin > Issue Type: Bug > Components: Spark Engine >Affects Versions: v2.5.1 > Environment: CentOS 7 > 3 workers and 1 master. > 4 cpu, 16GB RAM each >Reporter: Alexander >Assignee: Alexander >Priority: Minor > Attachments: KYLIN-3833.master.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3833) Potential OOM in Spark Extract Fact Table Distinct Columns step
[ https://issues.apache.org/jira/browse/KYLIN-3833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander updated KYLIN-3833: - Attachment: KYLIN-3833.master.001.patch > Potential OOM in Spark Extract Fact Table Distinct Columns step > --- > > Key: KYLIN-3833 > URL: https://issues.apache.org/jira/browse/KYLIN-3833 > Project: Kylin > Issue Type: Bug > Components: Spark Engine >Affects Versions: v2.5.1 >Reporter: Alexander >Assignee: Alexander >Priority: Minor > Attachments: KYLIN-3833.master.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3833) Potential OOM in Spark Extract Fact Table Distinct Columns step
[ https://issues.apache.org/jira/browse/KYLIN-3833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778100#comment-16778100 ] Alexander commented on KYLIN-3833: -- Tested on my Env. Ok. > Potential OOM in Spark Extract Fact Table Distinct Columns step > --- > > Key: KYLIN-3833 > URL: https://issues.apache.org/jira/browse/KYLIN-3833 > Project: Kylin > Issue Type: Bug > Components: Spark Engine >Affects Versions: v2.5.1 >Reporter: Alexander >Assignee: Alexander >Priority: Minor > Attachments: KYLIN-3833.master.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] Aaaaaaron edited a comment on issue #484: KYLIN-3820 Add a curator-based scheduler
Aaron edited a comment on issue #484: KYLIN-3820 Add a curator-based scheduler URL: https://github.com/apache/kylin/pull/484#issuecomment-467484764 Please wait for a moment. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Aaaaaaron edited a comment on issue #484: KYLIN-3820 Add a curator-based scheduler
Aaron edited a comment on issue #484: KYLIN-3820 Add a curator-based scheduler URL: https://github.com/apache/kylin/pull/484#issuecomment-467484764 Please wait for a moment. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Aaaaaaron commented on issue #484: KYLIN-3820 Add a curator-based scheduler
Aaron commented on issue #484: KYLIN-3820 Add a curator-based scheduler URL: https://github.com/apache/kylin/pull/484#issuecomment-467484764 Please hold on a moment. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] Aaaaaaron commented on issue #484: KYLIN-3820 Add a curator-based scheduler
Aaron commented on issue #484: KYLIN-3820 Add a curator-based scheduler URL: https://github.com/apache/kylin/pull/484#issuecomment-467484764 Please hold on a moment. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3832) Kylin Pushdown query not support postgresql
[ https://issues.apache.org/jira/browse/KYLIN-3832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778049#comment-16778049 ] Billy Liu commented on KYLIN-3832: -- Postgresql as pushdown engine is not supported by default. The contribution on postgresql data source adapter is welcomed. To learn more about data source sdk: http://kylin.apache.org/development/datasource_sdk.html > Kylin Pushdown query not support postgresql > --- > > Key: KYLIN-3832 > URL: https://issues.apache.org/jira/browse/KYLIN-3832 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.5.2 >Reporter: hailin.huang >Priority: Major > Fix For: Future > > > when I run pushdown to postgresql in my env, I encount the below exception. > it seems that kylin need support more JDBC Driver, > PushDownRunnerJdbcImpl.class should be more general. > 2019-02-26 16:12:53,168 ERROR [Query 207dcf77-7c14-8078-ea8b-79644a0c576d-48] > service.QueryService:989 : pushdown engine failed current query too > java.sql.SQLException: Unrecognized column type: int8 > at > org.apache.kylin.query.adhoc.PushDownRunnerJdbcImpl.toSqlType(PushDownRunnerJdbcImpl.java:260) > at > org.apache.kylin.query.adhoc.PushDownRunnerJdbcImpl.extractColumnMeta(PushDownRunnerJdbcImpl.java:192) > at > org.apache.kylin.query.adhoc.PushDownRunnerJdbcImpl.executeQuery(PushDownRunnerJdbcImpl.java:68) > at > org.apache.kylin.query.util.PushDownUtil.tryPushDownQuery(PushDownUtil.java:122) > at > org.apache.kylin.query.util.PushDownUtil.tryPushDownSelectQuery(PushDownUtil.java:69) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3831) 唯独超过62生成cuboid 错误
[ https://issues.apache.org/jira/browse/KYLIN-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778043#comment-16778043 ] Billy Liu commented on KYLIN-3831: -- [~photogamrun] The limitation is by design. Kylin use the _long-length_ bits to define the dimensions. > 唯独超过62生成cuboid 错误 > - > > Key: KYLIN-3831 > URL: https://issues.apache.org/jira/browse/KYLIN-3831 > Project: Kylin > Issue Type: Wish > Components: Others >Affects Versions: v2.3.1 >Reporter: zhangwei >Assignee: zhangwei >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3816) Current CI doesn't cover the case of streaming table join lookup table
[ https://issues.apache.org/jira/browse/KYLIN-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777990#comment-16777990 ] ASF GitHub Bot commented on KYLIN-3816: --- shaofengshi commented on pull request #483: KYLIN-3816 Make CI cover streaming table join case URL: https://github.com/apache/kylin/pull/483 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Current CI doesn't cover the case of streaming table join lookup table > --- > > Key: KYLIN-3816 > URL: https://issues.apache.org/jira/browse/KYLIN-3816 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v2.6.1 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3816) Current CI doesn't cover the case of streaming table join lookup table
[ https://issues.apache.org/jira/browse/KYLIN-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777995#comment-16777995 ] ASF subversion and git services commented on KYLIN-3816: Commit bab003c57a68c5e242ca02b1281cdf19ce29d34e in kylin's branch refs/heads/master from chao long [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=bab003c ] KYLIN-3816 Make CI cover streaming table join case > Current CI doesn't cover the case of streaming table join lookup table > --- > > Key: KYLIN-3816 > URL: https://issues.apache.org/jira/browse/KYLIN-3816 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v2.6.1 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3816) Current CI doesn't cover the case of streaming table join lookup table
[ https://issues.apache.org/jira/browse/KYLIN-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777988#comment-16777988 ] ASF GitHub Bot commented on KYLIN-3816: --- shaofengshi commented on pull request #483: KYLIN-3816 Make CI cover streaming table join case URL: https://github.com/apache/kylin/pull/483 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Current CI doesn't cover the case of streaming table join lookup table > --- > > Key: KYLIN-3816 > URL: https://issues.apache.org/jira/browse/KYLIN-3816 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v2.6.1 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] shaofengshi merged pull request #483: KYLIN-3816 Make CI cover streaming table join case
shaofengshi merged pull request #483: KYLIN-3816 Make CI cover streaming table join case URL: https://github.com/apache/kylin/pull/483 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3829) Chinese document on the kylin official site about sample cube got the sql wrong
[ https://issues.apache.org/jira/browse/KYLIN-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777986#comment-16777986 ] ASF subversion and git services commented on KYLIN-3829: Commit e6b4f8430edff235f5d8d7954fbd3097c0541fa7 in kylin's branch refs/heads/document from GinaZhai [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=e6b4f84 ] KYLIN-3829 Chinese document on the kylin official site about sample cube got the sql wrong > Chinese document on the kylin official site about sample cube got the sql > wrong > --- > > Key: KYLIN-3829 > URL: https://issues.apache.org/jira/browse/KYLIN-3829 > Project: Kylin > Issue Type: Improvement > Components: Documentation, Website >Affects Versions: v2.6.0 >Reporter: 王汝鹏 >Priority: Trivial > Labels: beginner, easyfix > Attachments: kylin.jpg > > > As we can see, the sql in the [Chinese document about the quick start with > sample cube|[http://kylin.apache.org/cn/docs/tutorial/kylin_sample.html]] > misses a piece which should be "select part_dt, sum(price) as total_selled, > count(distinct seller_id) as sellers from kylin_sales group by part_dt order > by part_dt". But the English version is correct. With the helping of the > front end code checking tool, you can see the right sql, so it's just a front > end display issue and I think it's easy to fix. :) > I've tested by chrome, firefox and edge and got the same problem, so it's not > the problem of brower . > It could be confusing for the beginner though it's easy to resolve. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3829) Chinese document on the kylin official site about sample cube got the sql wrong
[ https://issues.apache.org/jira/browse/KYLIN-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777987#comment-16777987 ] ASF subversion and git services commented on KYLIN-3829: Commit e6b4f8430edff235f5d8d7954fbd3097c0541fa7 in kylin's branch refs/heads/document from GinaZhai [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=e6b4f84 ] KYLIN-3829 Chinese document on the kylin official site about sample cube got the sql wrong > Chinese document on the kylin official site about sample cube got the sql > wrong > --- > > Key: KYLIN-3829 > URL: https://issues.apache.org/jira/browse/KYLIN-3829 > Project: Kylin > Issue Type: Improvement > Components: Documentation, Website >Affects Versions: v2.6.0 >Reporter: 王汝鹏 >Priority: Trivial > Labels: beginner, easyfix > Attachments: kylin.jpg > > > As we can see, the sql in the [Chinese document about the quick start with > sample cube|[http://kylin.apache.org/cn/docs/tutorial/kylin_sample.html]] > misses a piece which should be "select part_dt, sum(price) as total_selled, > count(distinct seller_id) as sellers from kylin_sales group by part_dt order > by part_dt". But the English version is correct. With the helping of the > front end code checking tool, you can see the right sql, so it's just a front > end display issue and I think it's easy to fix. :) > I've tested by chrome, firefox and edge and got the same problem, so it's not > the problem of brower . > It could be confusing for the beginner though it's easy to resolve. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] shaofengshi merged pull request #483: KYLIN-3816 Make CI cover streaming table join case
shaofengshi merged pull request #483: KYLIN-3816 Make CI cover streaming table join case URL: https://github.com/apache/kylin/pull/483 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3829) Chinese document on the kylin official site about sample cube got the sql wrong
[ https://issues.apache.org/jira/browse/KYLIN-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777984#comment-16777984 ] ASF GitHub Bot commented on KYLIN-3829: --- shaofengshi commented on pull request #485: KYLIN-3829 Chinese document on the kylin official site about sample c… URL: https://github.com/apache/kylin/pull/485 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Chinese document on the kylin official site about sample cube got the sql > wrong > --- > > Key: KYLIN-3829 > URL: https://issues.apache.org/jira/browse/KYLIN-3829 > Project: Kylin > Issue Type: Improvement > Components: Documentation, Website >Affects Versions: v2.6.0 >Reporter: 王汝鹏 >Priority: Trivial > Labels: beginner, easyfix > Attachments: kylin.jpg > > > As we can see, the sql in the [Chinese document about the quick start with > sample cube|[http://kylin.apache.org/cn/docs/tutorial/kylin_sample.html]] > misses a piece which should be "select part_dt, sum(price) as total_selled, > count(distinct seller_id) as sellers from kylin_sales group by part_dt order > by part_dt". But the English version is correct. With the helping of the > front end code checking tool, you can see the right sql, so it's just a front > end display issue and I think it's easy to fix. :) > I've tested by chrome, firefox and edge and got the same problem, so it's not > the problem of brower . > It could be confusing for the beginner though it's easy to resolve. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3829) Chinese document on the kylin official site about sample cube got the sql wrong
[ https://issues.apache.org/jira/browse/KYLIN-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777983#comment-16777983 ] ASF GitHub Bot commented on KYLIN-3829: --- shaofengshi commented on pull request #485: KYLIN-3829 Chinese document on the kylin official site about sample c… URL: https://github.com/apache/kylin/pull/485 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Chinese document on the kylin official site about sample cube got the sql > wrong > --- > > Key: KYLIN-3829 > URL: https://issues.apache.org/jira/browse/KYLIN-3829 > Project: Kylin > Issue Type: Improvement > Components: Documentation, Website >Affects Versions: v2.6.0 >Reporter: 王汝鹏 >Priority: Trivial > Labels: beginner, easyfix > Attachments: kylin.jpg > > > As we can see, the sql in the [Chinese document about the quick start with > sample cube|[http://kylin.apache.org/cn/docs/tutorial/kylin_sample.html]] > misses a piece which should be "select part_dt, sum(price) as total_selled, > count(distinct seller_id) as sellers from kylin_sales group by part_dt order > by part_dt". But the English version is correct. With the helping of the > front end code checking tool, you can see the right sql, so it's just a front > end display issue and I think it's easy to fix. :) > I've tested by chrome, firefox and edge and got the same problem, so it's not > the problem of brower . > It could be confusing for the beginner though it's easy to resolve. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] shaofengshi merged pull request #485: KYLIN-3829 Chinese document on the kylin official site about sample c…
shaofengshi merged pull request #485: KYLIN-3829 Chinese document on the kylin official site about sample c… URL: https://github.com/apache/kylin/pull/485 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] shaofengshi merged pull request #485: KYLIN-3829 Chinese document on the kylin official site about sample c…
shaofengshi merged pull request #485: KYLIN-3829 Chinese document on the kylin official site about sample c… URL: https://github.com/apache/kylin/pull/485 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Reopened] (KYLIN-3714) com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: scala.reflect.ManifestFactory$$anon$2
[ https://issues.apache.org/jira/browse/KYLIN-3714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander reopened KYLIN-3714: -- Assignee: Alexander (was: Chao Long) > com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: > Class is not registered: scala.reflect.ManifestFactory$$anon$2 > --- > > Key: KYLIN-3714 > URL: https://issues.apache.org/jira/browse/KYLIN-3714 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.5.1 > Environment: kylin2.5.1 > spark2.1.2 >Reporter: wangxianbin >Assignee: Alexander >Priority: Major > Attachments: Class is not registered.png > > > Class is not registered in step "Extract Fact Table Distinct Columns", when > use spark as build engine, error as follow: > com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: > Class is not registered: scala.reflect.ManifestFactory$$anon$2 Note: To > register this class use: > kryo.register(scala.reflect.ManifestFactory$$anon$2.class); Serialization > trace: evidence$1 (org.apache.spark.util.collection.CompactBuffer) at > com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:101) > at > com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:518) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) at > org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:207) > at > org.apache.spark.serializer.SerializationStream.writeValue(Serializer.scala:135) > at > org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:239) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.org$apache$spark$util$collection$ExternalAppendOnlyMap$$spillMemoryIteratorToDisk(ExternalAppendOnlyMap.scala:227) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.spill(ExternalAppendOnlyMap.scala:186) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.spill(ExternalAppendOnlyMap.scala:56) > at org.apache.spark.util.collection.Spillable.maybeSpill(Spillable.scala:97) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.insertAll(ExternalAppendOnlyMap.scala:160) > at org.apache.spark.Aggregator.combineValuesByKey(Aggregator.scala:41) at > org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:91) > at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:109) at > org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at > org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at > org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at > org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at > org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at > org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at > org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at > org.apache.spark.scheduler.Task.run(Task.scala:99) at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325) at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) Caused by: > java.lang.IllegalArgumentException: Class is not registered: > scala.reflect.ManifestFactory$$anon$2 Note: To register this class use: > kryo.register(scala.reflect.ManifestFactory$$anon$2.class); at > com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:488) at > com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:97) > at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:517) at > com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:76) > ... 27 more -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3714) com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: scala.reflect.ManifestFactory$$anon$2
[ https://issues.apache.org/jira/browse/KYLIN-3714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16777940#comment-16777940 ] Alexander commented on KYLIN-3714: -- In my Env: CentOS 7 3 workers and 1 master. 4 cpu, 16GB RAM each and huge cube, with next settings: mapreduce.input.fileinputformat.split.maxsize 41943040 Hive concurrency lock kylin.engine.spark.rdd-partition-cut-mb=50 kylin.engine.spark.min-partition=1 kylin.engine.spark.max-partition=100 kylin.engine.mr.uhc-reducer-count=1 Still got this exception. I think that spark spilling process must be registered and working when dimensions weight is different. in my case, i have one dic 4.6 GB and other very small... with this settings i've files bigger than executor RAM avaliable. > com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: > Class is not registered: scala.reflect.ManifestFactory$$anon$2 > --- > > Key: KYLIN-3714 > URL: https://issues.apache.org/jira/browse/KYLIN-3714 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.5.1 > Environment: kylin2.5.1 > spark2.1.2 >Reporter: wangxianbin >Assignee: Chao Long >Priority: Major > Attachments: Class is not registered.png > > > Class is not registered in step "Extract Fact Table Distinct Columns", when > use spark as build engine, error as follow: > com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: > Class is not registered: scala.reflect.ManifestFactory$$anon$2 Note: To > register this class use: > kryo.register(scala.reflect.ManifestFactory$$anon$2.class); Serialization > trace: evidence$1 (org.apache.spark.util.collection.CompactBuffer) at > com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:101) > at > com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:518) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) at > org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:207) > at > org.apache.spark.serializer.SerializationStream.writeValue(Serializer.scala:135) > at > org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:239) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.org$apache$spark$util$collection$ExternalAppendOnlyMap$$spillMemoryIteratorToDisk(ExternalAppendOnlyMap.scala:227) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.spill(ExternalAppendOnlyMap.scala:186) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.spill(ExternalAppendOnlyMap.scala:56) > at org.apache.spark.util.collection.Spillable.maybeSpill(Spillable.scala:97) > at > org.apache.spark.util.collection.ExternalAppendOnlyMap.insertAll(ExternalAppendOnlyMap.scala:160) > at org.apache.spark.Aggregator.combineValuesByKey(Aggregator.scala:41) at > org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:91) > at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:109) at > org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at > org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at > org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at > org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at > org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at > org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at > org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at > org.apache.spark.scheduler.Task.run(Task.scala:99) at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325) at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) Caused by: > java.lang.IllegalArgumentException: Class is not registered: > scala.reflect.ManifestFactory$$anon$2 Note: To register this class use: > kryo.register(scala.reflect.ManifestFactory$$anon$2.class); at > com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:488) at > com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:97) > at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:517) at > com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:76) > ... 27 more -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3832) Kylin Pushdown query not support postgresql
hailin.huang created KYLIN-3832: --- Summary: Kylin Pushdown query not support postgresql Key: KYLIN-3832 URL: https://issues.apache.org/jira/browse/KYLIN-3832 Project: Kylin Issue Type: Bug Components: Query Engine Affects Versions: v2.5.2 Reporter: hailin.huang Fix For: Future when I run pushdown to postgresql in my env, I encount the below exception. it seems that kylin need support more JDBC Driver, PushDownRunnerJdbcImpl.class should be more general. 2019-02-26 16:12:53,168 ERROR [Query 207dcf77-7c14-8078-ea8b-79644a0c576d-48] service.QueryService:989 : pushdown engine failed current query too java.sql.SQLException: Unrecognized column type: int8 at org.apache.kylin.query.adhoc.PushDownRunnerJdbcImpl.toSqlType(PushDownRunnerJdbcImpl.java:260) at org.apache.kylin.query.adhoc.PushDownRunnerJdbcImpl.extractColumnMeta(PushDownRunnerJdbcImpl.java:192) at org.apache.kylin.query.adhoc.PushDownRunnerJdbcImpl.executeQuery(PushDownRunnerJdbcImpl.java:68) at org.apache.kylin.query.util.PushDownUtil.tryPushDownQuery(PushDownUtil.java:122) at org.apache.kylin.query.util.PushDownUtil.tryPushDownSelectQuery(PushDownUtil.java:69) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] asfgit commented on issue #485: KYLIN-3829 Chinese document on the kylin official site about sample c…
asfgit commented on issue #485: KYLIN-3829 Chinese document on the kylin official site about sample c… URL: https://github.com/apache/kylin/pull/485#issuecomment-467341431 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3829) Chinese document on the kylin official site about sample cube got the sql wrong
[ https://issues.apache.org/jira/browse/KYLIN-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1612#comment-1612 ] ASF GitHub Bot commented on KYLIN-3829: --- GinaZhai commented on pull request #485: KYLIN-3829 Chinese document on the kylin official site about sample c… URL: https://github.com/apache/kylin/pull/485 …ube got the sql wrong This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Chinese document on the kylin official site about sample cube got the sql > wrong > --- > > Key: KYLIN-3829 > URL: https://issues.apache.org/jira/browse/KYLIN-3829 > Project: Kylin > Issue Type: Improvement > Components: Documentation, Website >Affects Versions: v2.6.0 >Reporter: 王汝鹏 >Priority: Trivial > Labels: beginner, easyfix > Attachments: kylin.jpg > > > As we can see, the sql in the [Chinese document about the quick start with > sample cube|[http://kylin.apache.org/cn/docs/tutorial/kylin_sample.html]] > misses a piece which should be "select part_dt, sum(price) as total_selled, > count(distinct seller_id) as sellers from kylin_sales group by part_dt order > by part_dt". But the English version is correct. With the helping of the > front end code checking tool, you can see the right sql, so it's just a front > end display issue and I think it's easy to fix. :) > I've tested by chrome, firefox and edge and got the same problem, so it's not > the problem of brower . > It could be confusing for the beginner though it's easy to resolve. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] GinaZhai opened a new pull request #485: KYLIN-3829 Chinese document on the kylin official site about sample c…
GinaZhai opened a new pull request #485: KYLIN-3829 Chinese document on the kylin official site about sample c… URL: https://github.com/apache/kylin/pull/485 …ube got the sql wrong This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
