[jira] [Commented] (KYLIN-2654) Kylin failed to build cube at 3rd step on Hadoop 2.6.0-cdh5.4.4 + jdk 1.8
[
https://issues.apache.org/jira/browse/KYLIN-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17004079#comment-17004079
]
zhaoguoxian commented on KYLIN-2654:
I encountered the same problem. Then your yarn environment is cdh and your
'mapreduce.job.map.output.collector.class' param is
org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator. you can
not use your Custom writable type or comparator，example，kylin`s
org.apache.kylin.engine.mr.steps.SelfDefineSortableKey. my solution was by
setting the param
mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.MapTask$MapOutputBuffer.
btw. my environment was cdh5.5.2
> Kylin failed to build cube at 3rd step on Hadoop 2.6.0-cdh5.4.4 + jdk 1.8
> -
>
> Key: KYLIN-2654
> URL: https://issues.apache.org/jira/browse/KYLIN-2654
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.0.0
>Reporter: Shao Feng Shi
>Assignee: Dong Li
>Priority: Major
>
> From community user qiumingming:
> When I use kylin 2.0 to build cube, I got a error at Extract Fact Table
> Distinct Columns step, mr job failed. My Hadoop version is 2.6.0-cdh5.4.4.
> Below is my error info:
> {code}
> Error: java.io.IOException: Initialization of all the collectors failed.
> Error in last collector was :Cannot find serializer for
> org.apache.kylin.engine.mr.steps.SelfDefineSortableKey at
> org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:423) at
> org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:82) at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:718)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:790) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:350) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by:
> java.io.IOException: Cannot find serializer for
> org.apache.kylin.engine.mr.steps.SelfDefineSortableKey at
> org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator.init(NativeMapOutputCollectorDelegator.java:147)
> at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:411)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-2654) Kylin failed to build cube at 3rd step on Hadoop 2.6.0-cdh5.4.4 + jdk 1.8
[
https://issues.apache.org/jira/browse/KYLIN-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045925#comment-16045925
]
liyang commented on KYLIN-2654:
---
This is where others can help little. Only you know your env the best.
> Kylin failed to build cube at 3rd step on Hadoop 2.6.0-cdh5.4.4 + jdk 1.8
> -
>
> Key: KYLIN-2654
> URL: https://issues.apache.org/jira/browse/KYLIN-2654
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.0.0
>Reporter: Shaofeng SHI
>Assignee: Dong Li
>
> From community user qiumingming:
> When I use kylin 2.0 to build cube, I got a error at Extract Fact Table
> Distinct Columns step, mr job failed. My Hadoop version is 2.6.0-cdh5.4.4.
> Below is my error info:
> {code}
> Error: java.io.IOException: Initialization of all the collectors failed.
> Error in last collector was :Cannot find serializer for
> org.apache.kylin.engine.mr.steps.SelfDefineSortableKey at
> org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:423) at
> org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:82) at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:718)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:790) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:350) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by:
> java.io.IOException: Cannot find serializer for
> org.apache.kylin.engine.mr.steps.SelfDefineSortableKey at
> org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator.init(NativeMapOutputCollectorDelegator.java:147)
> at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:411)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (KYLIN-2654) Kylin failed to build cube at 3rd step on Hadoop 2.6.0-cdh5.4.4 + jdk 1.8
[
https://issues.apache.org/jira/browse/KYLIN-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16035029#comment-16035029
]
qiumingming commented on KYLIN-2654:
By the way, is there any specific configuration to make SelfDefineSortableKey
work, like register serializer for SelfDefineSortableKey explicitly. And Have
you modified the pom.xml in your test, like change some versions ?
> Kylin failed to build cube at 3rd step on Hadoop 2.6.0-cdh5.4.4 + jdk 1.8
> -
>
> Key: KYLIN-2654
> URL: https://issues.apache.org/jira/browse/KYLIN-2654
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.0.0
>Reporter: Shaofeng SHI
>Assignee: Dong Li
>
> From community user qiumingming:
> When I use kylin 2.0 to build cube, I got a error at Extract Fact Table
> Distinct Columns step, mr job failed. My Hadoop version is 2.6.0-cdh5.4.4.
> Below is my error info:
> {code}
> Error: java.io.IOException: Initialization of all the collectors failed.
> Error in last collector was :Cannot find serializer for
> org.apache.kylin.engine.mr.steps.SelfDefineSortableKey at
> org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:423) at
> org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:82) at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:718)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:790) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:350) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by:
> java.io.IOException: Cannot find serializer for
> org.apache.kylin.engine.mr.steps.SelfDefineSortableKey at
> org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator.init(NativeMapOutputCollectorDelegator.java:147)
> at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:411)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (KYLIN-2654) Kylin failed to build cube at 3rd step on Hadoop 2.6.0-cdh5.4.4 + jdk 1.8
[
https://issues.apache.org/jira/browse/KYLIN-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16035017#comment-16035017
]
qiumingming commented on KYLIN-2654:
Yeah, I think so. Maybe my environment is so specific. And I solved this
problem by backing SelfDefineSortableKey to Text as old version done. I want to
know is this will involve any bug ? I am not very sure about that.
> Kylin failed to build cube at 3rd step on Hadoop 2.6.0-cdh5.4.4 + jdk 1.8
> -
>
> Key: KYLIN-2654
> URL: https://issues.apache.org/jira/browse/KYLIN-2654
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.0.0
>Reporter: Shaofeng SHI
>Assignee: Dong Li
>
> From community user qiumingming:
> When I use kylin 2.0 to build cube, I got a error at Extract Fact Table
> Distinct Columns step, mr job failed. My Hadoop version is 2.6.0-cdh5.4.4.
> Below is my error info:
> {code}
> Error: java.io.IOException: Initialization of all the collectors failed.
> Error in last collector was :Cannot find serializer for
> org.apache.kylin.engine.mr.steps.SelfDefineSortableKey at
> org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:423) at
> org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:82) at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:718)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:790) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:350) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by:
> java.io.IOException: Cannot find serializer for
> org.apache.kylin.engine.mr.steps.SelfDefineSortableKey at
> org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator.init(NativeMapOutputCollectorDelegator.java:147)
> at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:411)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (KYLIN-2654) Kylin failed to build cube at 3rd step on Hadoop 2.6.0-cdh5.4.4 + jdk 1.8
[
https://issues.apache.org/jira/browse/KYLIN-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16034860#comment-16034860
]
liyang commented on KYLIN-2654:
---
>From source code, the cause should be {{SelfDefineSortableKey}} is not a
>{{Writeable}}. However {{SelfDefineSortableKey}} does implements
>{{WritableComparable}}. Doubt this issue is very specific to the environment
>under test.
> Kylin failed to build cube at 3rd step on Hadoop 2.6.0-cdh5.4.4 + jdk 1.8
> -
>
> Key: KYLIN-2654
> URL: https://issues.apache.org/jira/browse/KYLIN-2654
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.0.0
>Reporter: Shaofeng SHI
>Assignee: Dong Li
>
> From community user qiumingming:
> When I use kylin 2.0 to build cube, I got a error at Extract Fact Table
> Distinct Columns step, mr job failed. My Hadoop version is 2.6.0-cdh5.4.4.
> Below is my error info:
> {code}
> Error: java.io.IOException: Initialization of all the collectors failed.
> Error in last collector was :Cannot find serializer for
> org.apache.kylin.engine.mr.steps.SelfDefineSortableKey at
> org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:423) at
> org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:82) at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:718)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:790) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:350) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by:
> java.io.IOException: Cannot find serializer for
> org.apache.kylin.engine.mr.steps.SelfDefineSortableKey at
> org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator.init(NativeMapOutputCollectorDelegator.java:147)
> at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:411)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
