[jira] [Commented] (KYLIN-2654) Kylin failed to build cube at 3rd step on Hadoop 2.6

2017-06-02 Thread liyang (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034854#comment-16034854
 ] 

liyang commented on KYLIN-2654:
---

I've verified Kylin 2.0 worked well on a Hadoop 2.6 sandbox (HDP 2.2). Confirm 
this is not a common problem on Hadoop 2.6.

> Kylin failed to build cube at 3rd step on Hadoop 2.6
> 
>
> Key: KYLIN-2654
> URL: https://issues.apache.org/jira/browse/KYLIN-2654
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine
>Affects Versions: v2.0.0
>Reporter: Shaofeng SHI
>Assignee: Dong Li
>
> From community user qiumingming:
> When I use kylin 2.0 to build cube, I got a error at Extract Fact Table
> Distinct Columns step, mr job failed. My Hadoop version is 2.6.0-cdh5.4.4.
> Below is my error info:
> {code}
> Error: java.io.IOException: Initialization of all the collectors failed.
> Error in last collector was :Cannot find serializer for
> org.apache.kylin.engine.mr.steps.SelfDefineSortableKey at
> org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:423) at
> org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:82) at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:718)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:790) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:350) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by:
> java.io.IOException: Cannot find serializer for
> org.apache.kylin.engine.mr.steps.SelfDefineSortableKey at
> org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator.init(NativeMapOutputCollectorDelegator.java:147)
> at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:411)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KYLIN-2654) Kylin failed to build cube at 3rd step on Hadoop 2.6

2017-06-02 Thread liyang (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034440#comment-16034440
 ] 

liyang commented on KYLIN-2654:
---

My Hadoop version is 2.6.0-cdh5.4.4 and jdk is 1.8.

> Kylin failed to build cube at 3rd step on Hadoop 2.6
> 
>
> Key: KYLIN-2654
> URL: https://issues.apache.org/jira/browse/KYLIN-2654
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine
>Affects Versions: v2.0.0
>Reporter: Shaofeng SHI
>Assignee: Dong Li
>
> From community user qiumingming:
> When I use kylin 2.0 to build cube, I got a error at Extract Fact Table
> Distinct Columns step, mr job failed. My Hadoop version is 2.6.0-cdh5.4.4.
> Below is my error info:
> {code}
> Error: java.io.IOException: Initialization of all the collectors failed.
> Error in last collector was :Cannot find serializer for
> org.apache.kylin.engine.mr.steps.SelfDefineSortableKey at
> org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:423) at
> org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:82) at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:718)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:790) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:350) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by:
> java.io.IOException: Cannot find serializer for
> org.apache.kylin.engine.mr.steps.SelfDefineSortableKey at
> org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator.init(NativeMapOutputCollectorDelegator.java:147)
> at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:411)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KYLIN-2654) Kylin failed to build cube at 3rd step on Hadoop 2.6

2017-06-02 Thread liyang (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034423#comment-16034423
 ] 

liyang commented on KYLIN-2654:
---

I will verify if kylin 2.0 runs on Hadoop 2.6...

> Kylin failed to build cube at 3rd step on Hadoop 2.6
> 
>
> Key: KYLIN-2654
> URL: https://issues.apache.org/jira/browse/KYLIN-2654
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine
>Affects Versions: v2.0.0
>Reporter: Shaofeng SHI
>Assignee: Dong Li
>
> From community user qiumingming:
> When I use kylin 2.0 to build cube, I got a error at Extract Fact Table
> Distinct Columns step, mr job failed. My Hadoop version is 2.6.0-cdh5.4.4.
> Below is my error info:
> {code}
> Error: java.io.IOException: Initialization of all the collectors failed.
> Error in last collector was :Cannot find serializer for
> org.apache.kylin.engine.mr.steps.SelfDefineSortableKey at
> org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:423) at
> org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:82) at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:718)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:790) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:350) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by:
> java.io.IOException: Cannot find serializer for
> org.apache.kylin.engine.mr.steps.SelfDefineSortableKey at
> org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator.init(NativeMapOutputCollectorDelegator.java:147)
> at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:411)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KYLIN-2654) Kylin failed to build cube at 3rd step on Hadoop 2.6

2017-06-02 Thread qiumingming (JIRA) 

[ 
https://issues.apache.org/jira/browse/KYLIN-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034354#comment-16034354
 ] 

qiumingming commented on KYLIN-2654:


I modified the code of Extract Fact Table Distinct Columns step, just back 
SelfDefineSortableKey to Text as old version done. Finally, it works. In my 
understanding, using Text as the Map Output Key Class won't involve any error. 
However, SelfDefineSortableKey can handle numeric type better and have a good 
performance for it can sort digits in a right way.  So, I am looking forward to 
this problem can be solved in a elegant way!

> Kylin failed to build cube at 3rd step on Hadoop 2.6
> 
>
> Key: KYLIN-2654
> URL: https://issues.apache.org/jira/browse/KYLIN-2654
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine
>Affects Versions: v2.0.0
>Reporter: Shaofeng SHI
>Assignee: Dong Li
>
> From community user qiumingming:
> When I use kylin 2.0 to build cube, I got a error at Extract Fact Table
> Distinct Columns step, mr job failed. My Hadoop version is 2.6.0-cdh5.4.4.
> Below is my error info:
> {code}
> Error: java.io.IOException: Initialization of all the collectors failed.
> Error in last collector was :Cannot find serializer for
> org.apache.kylin.engine.mr.steps.SelfDefineSortableKey at
> org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:423) at
> org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:82) at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:718)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:790) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:350) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by:
> java.io.IOException: Cannot find serializer for
> org.apache.kylin.engine.mr.steps.SelfDefineSortableKey at
> org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator.init(NativeMapOutputCollectorDelegator.java:147)
> at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:411)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)