[jira] [Commented] (KYLIN-3547) DimensionRangeInfo: Unsupported data type boolean

2018-09-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610129#comment-16610129
 ] 

ASF GitHub Bot commented on KYLIN-3547:
---

asfgit commented on issue #226: KYLIN-3547 add boolean DataTypeOrder
URL: https://github.com/apache/kylin/pull/226#issuecomment-420154825
 
 
   Can one of the admins verify this patch?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> DimensionRangeInfo: Unsupported data type boolean
> -
>
> Key: KYLIN-3547
> URL: https://issues.apache.org/jira/browse/KYLIN-3547
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine, Metadata
>Affects Versions: v2.5.0
>Reporter: Yichen Zhou
>Assignee: Yichen Zhou
>Priority: Blocker
> Fix For: v2.5.0
>
>
> In Extract Fact Table Distinct Columns Step, Kylin can not get dimension 
> range information from data of boolean type because DataType.getOrder() does 
> not support boolean.
> {quote}java.lang.RuntimeException: error execute 
> org.apache.kylin.engine.spark.SparkFactDistinct at 
> org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
>  at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44) at 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498) at 
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:636)
>  Caused by: org.apache.spark.SparkException: Job aborted due to stage 
> failure: Task 5 in stage 1.0 failed 4 times, most recent failure: Lost task 
> 5.3 in stage 1.0 (TID 17, slave1.kcluster, executor 5): 
> java.lang.IllegalArgumentException: Unsupported data type boolean at 
> org.apache.kylin.metadata.datatype.DataTypeOrder.getInstance(DataTypeOrder.java:53)
>  at org.apache.kylin.metadata.datatype.DataType.getOrder(DataType.java:230) 
> at org.apache.kylin.metadata.datatype.DataType.compare(DataType.java:236) at 
> org.apache.kylin.engine.spark.SparkFactDistinct$MultiOutputFunction.call(SparkFactDistinct.java:739)
>  at 
> org.apache.kylin.engine.spark.SparkFactDistinct$MultiOutputFunction.call(SparkFactDistinct.java:612)
>  at 
> org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:186)
>  at 
> org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:186)
>  at 
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:797)
>  at 
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:797)
>  at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) 
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at 
> org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at 
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at 
> org.apache.spark.scheduler.Task.run(Task.scala:99) at 
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325) at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  at java.lang.Thread.run(Thread.java:748)
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-3547) DimensionRangeInfo: Unsupported data type boolean

2018-09-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610128#comment-16610128
 ] 

ASF GitHub Bot commented on KYLIN-3547:
---

ZhouYC627 opened a new pull request #226: KYLIN-3547 add boolean DataTypeOrder
URL: https://github.com/apache/kylin/pull/226
 
 
   https://issues.apache.org/jira/browse/KYLIN-3547


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> DimensionRangeInfo: Unsupported data type boolean
> -
>
> Key: KYLIN-3547
> URL: https://issues.apache.org/jira/browse/KYLIN-3547
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine, Metadata
>Affects Versions: v2.5.0
>Reporter: Yichen Zhou
>Assignee: Yichen Zhou
>Priority: Blocker
> Fix For: v2.5.0
>
>
> In Extract Fact Table Distinct Columns Step, Kylin can not get dimension 
> range information from data of boolean type because DataType.getOrder() does 
> not support boolean.
> {quote}java.lang.RuntimeException: error execute 
> org.apache.kylin.engine.spark.SparkFactDistinct at 
> org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
>  at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44) at 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498) at 
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:636)
>  Caused by: org.apache.spark.SparkException: Job aborted due to stage 
> failure: Task 5 in stage 1.0 failed 4 times, most recent failure: Lost task 
> 5.3 in stage 1.0 (TID 17, slave1.kcluster, executor 5): 
> java.lang.IllegalArgumentException: Unsupported data type boolean at 
> org.apache.kylin.metadata.datatype.DataTypeOrder.getInstance(DataTypeOrder.java:53)
>  at org.apache.kylin.metadata.datatype.DataType.getOrder(DataType.java:230) 
> at org.apache.kylin.metadata.datatype.DataType.compare(DataType.java:236) at 
> org.apache.kylin.engine.spark.SparkFactDistinct$MultiOutputFunction.call(SparkFactDistinct.java:739)
>  at 
> org.apache.kylin.engine.spark.SparkFactDistinct$MultiOutputFunction.call(SparkFactDistinct.java:612)
>  at 
> org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:186)
>  at 
> org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:186)
>  at 
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:797)
>  at 
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:797)
>  at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) 
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at 
> org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at 
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at 
> org.apache.spark.scheduler.Task.run(Task.scala:99) at 
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325) at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  at java.lang.Thread.run(Thread.java:748)
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (KYLIN-3525) kylin.source.hive.keep-flat-table参数问题

2018-09-10 Thread Shaofeng SHI (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-3525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI reassigned KYLIN-3525:
---

Assignee: XiaoXiang Yu  (was: Shaofeng SHI)

> kylin.source.hive.keep-flat-table参数问题
> -
>
> Key: KYLIN-3525
> URL: https://issues.apache.org/jira/browse/KYLIN-3525
> Project: Kylin
>  Issue Type: Improvement
>  Components: Streaming
>Affects Versions: v2.4.0
>Reporter: wlxie
>Assignee: XiaoXiang Yu
>Priority: Major
> Fix For: v2.6.0
>
> Attachments: 1535534470(1).png, HiveMRInput.java
>
>
> kylin.source.hive.keep-flat-table这个参数设置为true后,只是保留了hive表结构,但是数据还是被清除了,查看了源码,这个参数确实只是对表结构进行控制。如果希望数据也保存下来,并且是一个cube下面所有作业的数据都保存到一个表里面(目前是一个作业会产生一个表),是否有什么好的解决方案。
> 附件为v2.4.0源码
> 谢谢。



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (KYLIN-3525) kylin.source.hive.keep-flat-table参数问题

2018-09-10 Thread Shaofeng SHI (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-3525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI reassigned KYLIN-3525:
---

Assignee: Shaofeng SHI

> kylin.source.hive.keep-flat-table参数问题
> -
>
> Key: KYLIN-3525
> URL: https://issues.apache.org/jira/browse/KYLIN-3525
> Project: Kylin
>  Issue Type: Improvement
>  Components: Streaming
>Affects Versions: v2.4.0
>Reporter: wlxie
>Assignee: Shaofeng SHI
>Priority: Major
> Fix For: v2.6.0
>
> Attachments: 1535534470(1).png, HiveMRInput.java
>
>
> kylin.source.hive.keep-flat-table这个参数设置为true后,只是保留了hive表结构,但是数据还是被清除了,查看了源码,这个参数确实只是对表结构进行控制。如果希望数据也保存下来,并且是一个cube下面所有作业的数据都保存到一个表里面(目前是一个作业会产生一个表),是否有什么好的解决方案。
> 附件为v2.4.0源码
> 谢谢。



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Resolved] (KYLIN-3533) Can not save hybrid

2018-09-10 Thread Chao Long (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-3533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chao Long resolved KYLIN-3533.
--
Resolution: Fixed

> Can not save hybrid
> ---
>
> Key: KYLIN-3533
> URL: https://issues.apache.org/jira/browse/KYLIN-3533
> Project: Kylin
>  Issue Type: Bug
>Reporter: Na Zhai
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.5.0
>
>
> Steps:
>  # create hybrid with 3 cube, cube A's status is disable(not build), cube B 
> and C are ready(build successful without segment overlap)
>  # click submit
>  # build cube A, the partition date is early than cube B and C
>  # add cube D(not build) to hybrid, click submit
> hybrid can not save and error message will pop up



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Resolved] (KYLIN-3527) Hybrid couldn't save when there is only 1 cube

2018-09-10 Thread Shaofeng SHI (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-3527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3527.
-
Resolution: Fixed

> Hybrid couldn't save when there is only 1 cube
> --
>
> Key: KYLIN-3527
> URL: https://issues.apache.org/jira/browse/KYLIN-3527
> Project: Kylin
>  Issue Type: Bug
>  Components: Web 
>Affects Versions: v2.5.0
>Reporter: Shaofeng SHI
>Assignee: Roger
>Priority: Minor
> Fix For: v2.5.0
>
> Attachments: hybrid-not-saved.png, move-in-out-icons.png
>
>
> I created a hybrid with two cubes; Later I want to remove one cube from it, 
> but I found when there is only 1 cube, the "Submit" button was disabled.
>  
> The hybrid should allows 0, 1 or more cubes.
>  
> Besides, I found the move in and out icons were not properly displayed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Resolved] (KYLIN-3524) Clone cube reports "Overwriting conflict /project/learn_kylin.json" error

2018-09-10 Thread Shaofeng SHI (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-3524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3524.
-
   Resolution: Not A Problem
Fix Version/s: (was: v2.5.0)

> Clone cube reports "Overwriting conflict /project/learn_kylin.json" error
> -
>
> Key: KYLIN-3524
> URL: https://issues.apache.org/jira/browse/KYLIN-3524
> Project: Kylin
>  Issue Type: Bug
>  Components: Metadata
>Affects Versions: v2.5.0
>Reporter: Shaofeng SHI
>Priority: Major
>
> {code:java}
> 53956 2018-09-02 20:25:51,458 INFO [http-bio-7080-exec-4] 
> service.CubeService:218 : New cube kylin_sales_cube_shao2 has 161 cuboids
> 53957 2018-09-02 20:25:51,458 INFO [http-bio-7080-exec-4] 
> cube.CubeManager:225 : Creating cube 'learn_kylin-->kylin_sales_cube_shao2' 
> from desc 'kylin_sales_cube_shao2'
> 53958 2018-09-02 20:25:51,458 INFO [http-bio-7080-exec-4] 
> cube.CubeManager:316 : Updating cube instance 'kylin_sales_cube_shao2'
> 53959 2018-09-02 20:25:51,459 DEBUG [http-bio-7080-exec-4] 
> cachesync.CachedCrudAssist:190 : Saving CubeInstance at 
> /cube/kylin_sales_cube_shao2.json
> 53960 2018-09-02 20:25:51,460 DEBUG [pool-16-thread-1] 
> cachesync.Broadcaster:113 : Servers in the cluster: [localhost:7070]
> 53961 2018-09-02 20:25:51,460 DEBUG [pool-16-thread-1] 
> cachesync.Broadcaster:123 : Announcing new broadcast to all: 
> BroadcastEvent{entity=cube, event=create, cacheKey=kylin_sales_cube_shao2}
> 53962 2018-09-02 20:25:51,461 DEBUG [http-bio-7080-exec-4] 
> cachesync.CachedCrudAssist:190 : Saving ProjectInstance at 
> /project/learn_kylin.json
> 53963 2018-09-02 20:25:51,462 ERROR [http-bio-7080-exec-4] 
> controller.BasicController:62 :
> 53964 org.apache.kylin.common.persistence.WriteConflictException: Overwriting 
> conflict /project/learn_kylin.json, expect old TS 1535685985266, but it is 
> 1535695655000
> 53965 at 
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:325)
> 53966 at 
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:318)
> 53967 at 
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:303)
> 53968 at 
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:282)
> 53969 at 
> org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192)
> 53970 at 
> org.apache.kylin.metadata.project.ProjectManager.save(ProjectManager.java:371)
> 53971 at 
> org.apache.kylin.metadata.project.ProjectManager.addRealizationToProject(ProjectManager.java:283)
> 53972 at 
> org.apache.kylin.metadata.project.ProjectManager.moveRealizationToProject(ProjectManager.java:266)
> 53973 at org.apache.kylin.cube.CubeManager.createCube(CubeManager.java:232)
> 53974 at 
> org.apache.kylin.rest.service.CubeService.createCubeAndDesc(CubeService.java:220)
> 53975 at 
> org.apache.kylin.rest.service.CubeService$$FastClassBySpringCGLIB$$17a07c0e.invoke()
> 53976 at 
> org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
> 53977 at 
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Resolved] (KYLIN-3534) Failed at update cube info step

2018-09-10 Thread Chao Long (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chao Long resolved KYLIN-3534.
--
Resolution: Fixed

> Failed at update cube info step
> ---
>
> Key: KYLIN-3534
> URL: https://issues.apache.org/jira/browse/KYLIN-3534
> Project: Kylin
>  Issue Type: Bug
>Reporter: Yichen Zhou
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.5.0
>
>
> When building cube with spark(full build), I got the following error in the 
> step of "Update Cube Info":
> {quote}2018-09-04 14:49:49,007 INFO [Scheduler 158588 Job 
> d05156bb-2635-cb2f-ac19-b38f6147fcb2-699] 
> steps.UpdateCubeInfoAfterBuildStep:158 : updateSegment step. TRANS_ID 
> minValue:x�3�.�) maxValue:null
>  2018-09-04 14:49:49,008 ERROR [Scheduler 158588 Job 
> d05156bb-2635-cb2f-ac19-b38f6147fcb2-699] execution.AbstractExecutable:165 : 
> error running Executable: 
> UpdateCubeInfoAfterBuildStep\{id=d05156bb-2635-cb2f-ac19-b38f6147fcb2-09, 
> name=Update Cube Info, state=RUNNING}
>  2018-09-04 14:49:49,011 INFO [Scheduler 158588 Job 
> d05156bb-2635-cb2f-ac19-b38f6147fcb2-699] execution.ExecutableManager:434 : 
> job id:d05156bb-2635-cb2f-ac19-b38f6147fcb2-09 from RUNNING to ERROR
>  2018-09-04 14:49:49,012 ERROR [Scheduler 158588 Job 
> d05156bb-2635-cb2f-ac19-b38f6147fcb2-699] execution.AbstractExecutable:165 : 
> error running Executable: CubingJob\{id=d05156bb-2635-cb2f-ac19-b38f6147fcb2, 
> name=BUILD CUBE - cube_global_dictionary_spark - FULL_BUILD - GMT+08:00 
> 2018-09-04 11:51:58, state=RUNNING}
>  2018-09-04 14:49:49,014 DEBUG [pool-6-thread-1] cachesync.Broadcaster:113 : 
> Servers in the cluster: [localhost:7070]
>  2018-09-04 14:49:49,014 DEBUG [pool-6-thread-1] cachesync.Broadcaster:123 : 
> Announcing new broadcast to all: BroadcastEvent\{entity=execute_output, 
> event=update, cacheKey=d05156bb-2635-cb2f-ac19-b38f6147fcb2}
>  2018-09-04 14:49:49,015 INFO [Scheduler 158588 Job 
> d05156bb-2635-cb2f-ac19-b38f6147fcb2-699] execution.ExecutableManager:434 : 
> job id:d05156bb-2635-cb2f-ac19-b38f6147fcb2 from RUNNING to ERROR
>  2018-09-04 14:49:49,015 DEBUG [pool-6-thread-1] cachesync.Broadcaster:113 : 
> Servers in the cluster: [localhost:7070]
>  2018-09-04 14:49:49,015 DEBUG [pool-6-thread-1] cachesync.Broadcaster:123 : 
> Announcing new broadcast to all: BroadcastEvent\{entity=execute_output, 
> event=update, cacheKey=d05156bb-2635-cb2f-ac19-b38f6147fcb2}
>  2018-09-04 14:49:49,015 DEBUG [Scheduler 158588 Job 
> d05156bb-2635-cb2f-ac19-b38f6147fcb2-699] execution.AbstractExecutable:316 : 
> no need to send email, user list is empty
>  2018-09-04 14:49:49,017 ERROR [pool-10-thread-10] 
> threadpool.DefaultScheduler:115 : ExecuteException 
> job:d05156bb-2635-cb2f-ac19-b38f6147fcb2
> org.apache.kylin.job.exception.ExecuteException: 
> org.apache.kylin.job.exception.ExecuteException:java.lang.IllegalStateException
>  at 
> org.apache.kylin.cube.DimensionRangeInfo.(DimensionRangeInfo.java:86)
>  at 
> org.apache.kylin.engine.mr.steps.UpdateCubeInfoAfterBuildStep.updateSegment(UpdateCubeInfoAfterBuildStep.java:166)
>  at 
> org.apache.kylin.engine.mr.steps.UpdateCubeInfoAfterBuildStep.doWork(UpdateCubeInfoAfterBuildStep.java:85)
>  at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163)
>  at 
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:69)
>  at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163)
>  at 
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:113)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  at java.lang.Thread.run(Thread.java:748)
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Closed] (KYLIN-3524) Clone cube reports "Overwriting conflict /project/learn_kylin.json" error

2018-09-10 Thread Shaofeng SHI (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-3524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI closed KYLIN-3524.
---

> Clone cube reports "Overwriting conflict /project/learn_kylin.json" error
> -
>
> Key: KYLIN-3524
> URL: https://issues.apache.org/jira/browse/KYLIN-3524
> Project: Kylin
>  Issue Type: Bug
>  Components: Metadata
>Affects Versions: v2.5.0
>Reporter: Shaofeng SHI
>Priority: Major
>
> {code:java}
> 53956 2018-09-02 20:25:51,458 INFO [http-bio-7080-exec-4] 
> service.CubeService:218 : New cube kylin_sales_cube_shao2 has 161 cuboids
> 53957 2018-09-02 20:25:51,458 INFO [http-bio-7080-exec-4] 
> cube.CubeManager:225 : Creating cube 'learn_kylin-->kylin_sales_cube_shao2' 
> from desc 'kylin_sales_cube_shao2'
> 53958 2018-09-02 20:25:51,458 INFO [http-bio-7080-exec-4] 
> cube.CubeManager:316 : Updating cube instance 'kylin_sales_cube_shao2'
> 53959 2018-09-02 20:25:51,459 DEBUG [http-bio-7080-exec-4] 
> cachesync.CachedCrudAssist:190 : Saving CubeInstance at 
> /cube/kylin_sales_cube_shao2.json
> 53960 2018-09-02 20:25:51,460 DEBUG [pool-16-thread-1] 
> cachesync.Broadcaster:113 : Servers in the cluster: [localhost:7070]
> 53961 2018-09-02 20:25:51,460 DEBUG [pool-16-thread-1] 
> cachesync.Broadcaster:123 : Announcing new broadcast to all: 
> BroadcastEvent{entity=cube, event=create, cacheKey=kylin_sales_cube_shao2}
> 53962 2018-09-02 20:25:51,461 DEBUG [http-bio-7080-exec-4] 
> cachesync.CachedCrudAssist:190 : Saving ProjectInstance at 
> /project/learn_kylin.json
> 53963 2018-09-02 20:25:51,462 ERROR [http-bio-7080-exec-4] 
> controller.BasicController:62 :
> 53964 org.apache.kylin.common.persistence.WriteConflictException: Overwriting 
> conflict /project/learn_kylin.json, expect old TS 1535685985266, but it is 
> 1535695655000
> 53965 at 
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:325)
> 53966 at 
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:318)
> 53967 at 
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:303)
> 53968 at 
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:282)
> 53969 at 
> org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192)
> 53970 at 
> org.apache.kylin.metadata.project.ProjectManager.save(ProjectManager.java:371)
> 53971 at 
> org.apache.kylin.metadata.project.ProjectManager.addRealizationToProject(ProjectManager.java:283)
> 53972 at 
> org.apache.kylin.metadata.project.ProjectManager.moveRealizationToProject(ProjectManager.java:266)
> 53973 at org.apache.kylin.cube.CubeManager.createCube(CubeManager.java:232)
> 53974 at 
> org.apache.kylin.rest.service.CubeService.createCubeAndDesc(CubeService.java:220)
> 53975 at 
> org.apache.kylin.rest.service.CubeService$$FastClassBySpringCGLIB$$17a07c0e.invoke()
> 53976 at 
> org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
> 53977 at 
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Resolved] (KYLIN-3539) Hybrid segment overlap not cover some case

2018-09-10 Thread Chao Long (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-3539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chao Long resolved KYLIN-3539.
--
Resolution: Fixed

> Hybrid segment overlap not cover some case
> --
>
> Key: KYLIN-3539
> URL: https://issues.apache.org/jira/browse/KYLIN-3539
> Project: Kylin
>  Issue Type: Improvement
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.5.0
>
>
> now Hybrid uses every cube's latest segment to check whether overlap exists, 
> but it can't cover all cases.
> example:
> hybrid:
>  cube1:
>  segment1: range(2012-01-01, 2012-01-02)
> cube2:
>  segment1: range(2012-01-01, 2012-01-02)
>  segment2: range(2012-01-03, 2012-01-04)
> this hybrid has overlap segment range(2012-01-01, 2012-01-02), but the 
> program can not find out.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-3527) Hybrid couldn't save when there is only 1 cube

2018-09-10 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610068#comment-16610068
 ] 

ASF subversion and git services commented on KYLIN-3527:


Commit 5ce6588eda365cf4aca5f53ab6cb1adaa4ffe9e8 in kylin's branch 
refs/heads/2.5.x-hadoop3.1 from chao long
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=5ce6588 ]

KYLIN-3527 Hybrid allows 0,1 cube


> Hybrid couldn't save when there is only 1 cube
> --
>
> Key: KYLIN-3527
> URL: https://issues.apache.org/jira/browse/KYLIN-3527
> Project: Kylin
>  Issue Type: Bug
>  Components: Web 
>Affects Versions: v2.5.0
>Reporter: Shaofeng SHI
>Assignee: Roger
>Priority: Minor
> Fix For: v2.5.0
>
> Attachments: hybrid-not-saved.png, move-in-out-icons.png
>
>
> I created a hybrid with two cubes; Later I want to remove one cube from it, 
> but I found when there is only 1 cube, the "Submit" button was disabled.
>  
> The hybrid should allows 0, 1 or more cubes.
>  
> Besides, I found the move in and out icons were not properly displayed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-2565) Upgrade Kylin to Hadoop3.0

2018-09-10 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610071#comment-16610071
 ] 

ASF subversion and git services commented on KYLIN-2565:


Commit a220ff6d289796bd5ec4800cecd5f5eb7d05b527 in kylin's branch 
refs/heads/2.5.x-hadoop3.1 from shaofengshi
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=a220ff6 ]

KYLIN-2565 upgrade to hadoop 3.0 hbase 2.0, pass UT


> Upgrade Kylin to Hadoop3.0
> --
>
> Key: KYLIN-2565
> URL: https://issues.apache.org/jira/browse/KYLIN-2565
> Project: Kylin
>  Issue Type: New Feature
>Reporter: Wang Cheng
>Assignee: Shaofeng SHI
>Priority: Major
>
> Hadoop3.0-alpha is released, Kylin should also keep compatible with it. Below 
> is the Hadoop3.0 components requirements:
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+release 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-3529) Prompt not friendly

2018-09-10 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610066#comment-16610066
 ] 

ASF subversion and git services commented on KYLIN-3529:


Commit 3b3717f11fb7e7f4ed70ace943a064bf704a42a7 in kylin's branch 
refs/heads/2.5.x-hadoop3.1 from chao long
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=3b3717f ]

KYLIN-3529 Show more error message


> Prompt not friendly
> ---
>
> Key: KYLIN-3529
> URL: https://issues.apache.org/jira/browse/KYLIN-3529
> Project: Kylin
>  Issue Type: Bug
>  Components: Web 
>Reporter: Na Zhai
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.5.0
>
>
>  *Scene 1:*When I create a hybrid with duplicate name, then I click submit, 
> the error message "error execute org.apache.kylin.rest.job.HybridCubeCLI" 
> will pop up.
>  
> *Scene 2:*When I create a hybrid with duplicate date/time cube, then I click 
> submit, the error message "error execute 
> org.apache.kylin.rest.job.HybridCubeCLI" will pop up.
>  
> I think the error message should be more friendly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-3539) Hybrid segment overlap not cover some case

2018-09-10 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610069#comment-16610069
 ] 

ASF subversion and git services commented on KYLIN-3539:


Commit f15ae54b10ef7b9f22f605d61618d16038512113 in kylin's branch 
refs/heads/2.5.x-hadoop3.1 from chao long
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=f15ae54 ]

KYLIN-3539 Hybrid segment overlap check


> Hybrid segment overlap not cover some case
> --
>
> Key: KYLIN-3539
> URL: https://issues.apache.org/jira/browse/KYLIN-3539
> Project: Kylin
>  Issue Type: Improvement
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.5.0
>
>
> now Hybrid uses every cube's latest segment to check whether overlap exists, 
> but it can't cover all cases.
> example:
> hybrid:
>  cube1:
>  segment1: range(2012-01-01, 2012-01-02)
> cube2:
>  segment1: range(2012-01-01, 2012-01-02)
>  segment2: range(2012-01-03, 2012-01-04)
> this hybrid has overlap segment range(2012-01-01, 2012-01-02), but the 
> program can not find out.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-2565) Upgrade Kylin to Hadoop3.0

2018-09-10 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610070#comment-16610070
 ] 

ASF subversion and git services commented on KYLIN-2565:


Commit 2cf62401ae7e2d48e9d8bb086558dadf7d63fa56 in kylin's branch 
refs/heads/2.5.x-hadoop3.1 from [~blue20080]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=2cf6240 ]

KYLIN-2565, upgrade to Hadoop3.0


> Upgrade Kylin to Hadoop3.0
> --
>
> Key: KYLIN-2565
> URL: https://issues.apache.org/jira/browse/KYLIN-2565
> Project: Kylin
>  Issue Type: New Feature
>Reporter: Wang Cheng
>Assignee: Shaofeng SHI
>Priority: Major
>
> Hadoop3.0-alpha is released, Kylin should also keep compatible with it. Below 
> is the Hadoop3.0 components requirements:
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+release 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-3534) Failed at update cube info step

2018-09-10 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610067#comment-16610067
 ] 

ASF subversion and git services commented on KYLIN-3534:


Commit d90bb046d191dd06f194e7380da13698772e001f in kylin's branch 
refs/heads/2.5.x-hadoop3.1 from chao long
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=d90bb04 ]

KYLIN-3534 Don't compress fact distinct output file


> Failed at update cube info step
> ---
>
> Key: KYLIN-3534
> URL: https://issues.apache.org/jira/browse/KYLIN-3534
> Project: Kylin
>  Issue Type: Bug
>Reporter: Yichen Zhou
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.5.0
>
>
> When building cube with spark(full build), I got the following error in the 
> step of "Update Cube Info":
> {quote}2018-09-04 14:49:49,007 INFO [Scheduler 158588 Job 
> d05156bb-2635-cb2f-ac19-b38f6147fcb2-699] 
> steps.UpdateCubeInfoAfterBuildStep:158 : updateSegment step. TRANS_ID 
> minValue:x�3�.�) maxValue:null
>  2018-09-04 14:49:49,008 ERROR [Scheduler 158588 Job 
> d05156bb-2635-cb2f-ac19-b38f6147fcb2-699] execution.AbstractExecutable:165 : 
> error running Executable: 
> UpdateCubeInfoAfterBuildStep\{id=d05156bb-2635-cb2f-ac19-b38f6147fcb2-09, 
> name=Update Cube Info, state=RUNNING}
>  2018-09-04 14:49:49,011 INFO [Scheduler 158588 Job 
> d05156bb-2635-cb2f-ac19-b38f6147fcb2-699] execution.ExecutableManager:434 : 
> job id:d05156bb-2635-cb2f-ac19-b38f6147fcb2-09 from RUNNING to ERROR
>  2018-09-04 14:49:49,012 ERROR [Scheduler 158588 Job 
> d05156bb-2635-cb2f-ac19-b38f6147fcb2-699] execution.AbstractExecutable:165 : 
> error running Executable: CubingJob\{id=d05156bb-2635-cb2f-ac19-b38f6147fcb2, 
> name=BUILD CUBE - cube_global_dictionary_spark - FULL_BUILD - GMT+08:00 
> 2018-09-04 11:51:58, state=RUNNING}
>  2018-09-04 14:49:49,014 DEBUG [pool-6-thread-1] cachesync.Broadcaster:113 : 
> Servers in the cluster: [localhost:7070]
>  2018-09-04 14:49:49,014 DEBUG [pool-6-thread-1] cachesync.Broadcaster:123 : 
> Announcing new broadcast to all: BroadcastEvent\{entity=execute_output, 
> event=update, cacheKey=d05156bb-2635-cb2f-ac19-b38f6147fcb2}
>  2018-09-04 14:49:49,015 INFO [Scheduler 158588 Job 
> d05156bb-2635-cb2f-ac19-b38f6147fcb2-699] execution.ExecutableManager:434 : 
> job id:d05156bb-2635-cb2f-ac19-b38f6147fcb2 from RUNNING to ERROR
>  2018-09-04 14:49:49,015 DEBUG [pool-6-thread-1] cachesync.Broadcaster:113 : 
> Servers in the cluster: [localhost:7070]
>  2018-09-04 14:49:49,015 DEBUG [pool-6-thread-1] cachesync.Broadcaster:123 : 
> Announcing new broadcast to all: BroadcastEvent\{entity=execute_output, 
> event=update, cacheKey=d05156bb-2635-cb2f-ac19-b38f6147fcb2}
>  2018-09-04 14:49:49,015 DEBUG [Scheduler 158588 Job 
> d05156bb-2635-cb2f-ac19-b38f6147fcb2-699] execution.AbstractExecutable:316 : 
> no need to send email, user list is empty
>  2018-09-04 14:49:49,017 ERROR [pool-10-thread-10] 
> threadpool.DefaultScheduler:115 : ExecuteException 
> job:d05156bb-2635-cb2f-ac19-b38f6147fcb2
> org.apache.kylin.job.exception.ExecuteException: 
> org.apache.kylin.job.exception.ExecuteException:java.lang.IllegalStateException
>  at 
> org.apache.kylin.cube.DimensionRangeInfo.(DimensionRangeInfo.java:86)
>  at 
> org.apache.kylin.engine.mr.steps.UpdateCubeInfoAfterBuildStep.updateSegment(UpdateCubeInfoAfterBuildStep.java:166)
>  at 
> org.apache.kylin.engine.mr.steps.UpdateCubeInfoAfterBuildStep.doWork(UpdateCubeInfoAfterBuildStep.java:85)
>  at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163)
>  at 
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:69)
>  at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163)
>  at 
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:113)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  at java.lang.Thread.run(Thread.java:748)
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-3517) Couldn't update coprocessor on HBase 2.0

2018-09-10 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610074#comment-16610074
 ] 

ASF subversion and git services commented on KYLIN-3517:


Commit 7dbd6281913c7c9701a76fe4e84a4da54d7f5ffe in kylin's branch 
refs/heads/2.5.x-hadoop3.1 from [~caolijun1166]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=7dbd628 ]

KYLIN-3517 Upadate coprocessor on HBase2.0 is avaliable.

Signed-off-by: shaofengshi 


> Couldn't update coprocessor on HBase 2.0
> 
>
> Key: KYLIN-3517
> URL: https://issues.apache.org/jira/browse/KYLIN-3517
> Project: Kylin
>  Issue Type: Bug
>  Components: Storage - HBase
>Reporter: Shaofeng SHI
>Assignee: Lijun Cao
>Priority: Major
>
> On HDP 3.0, run update coprocessor, got this error:
>  
> {code:java}
> 2018-08-28 00:24:26,683 ERROR [pool-7-thread-1] util.DeployCoprocessorCLI:383 
> : Error processing KYLIN_O9JRT8XOQ9
> java.lang.UnsupportedOperationException: HTableDescriptor is read-only
> at 
> org.apache.hadoop.hbase.client.ImmutableHTableDescriptor.getDelegateeForModification(ImmutableHTableDescriptor.java:59)
> at 
> org.apache.hadoop.hbase.HTableDescriptor.removeCoprocessor(HTableDescriptor.java:768)
> at 
> org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI.resetCoprocessor(DeployCoprocessorCLI.java:300)
> at 
> org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI$ResetCoprocessorWorker.run(DeployCoprocessorCLI.java:375)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-3258) No check for duplicate cube name when creating a hybrid cube

2018-09-10 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610065#comment-16610065
 ] 

ASF subversion and git services commented on KYLIN-3258:


Commit 3e6307473d60036feb8c8497379eb4841d82d093 in kylin's branch 
refs/heads/2.5.x-hadoop3.1 from GinaZhai
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=3e63074 ]

KYLIN-3258 No check for duplicate cube name when creating a hybrid cube


> No check for duplicate cube name when creating a hybrid cube
> 
>
> Key: KYLIN-3258
> URL: https://issues.apache.org/jira/browse/KYLIN-3258
> Project: Kylin
>  Issue Type: Bug
>  Components: Metadata
>Affects Versions: v2.2.0
> Environment: HDP 2.5.6, Kylin 2.2
>Reporter: Vsevolod Ostapenko
>Assignee: Shaofeng SHI
>Priority: Minor
> Fix For: v2.5.0
>
>
> When loading hybrid cube definitions via REST API, there is no check for 
> duplicate cube names is the list. If due to a user error or incorrectly 
> generated list of cubes by an external application/script the same cube name 
> is listed more than once, new or updated hybrid cube will contain the same 
> cube listed multiple times.
> It does not seem to cause any immediate issues with querying, but it's just 
> not right. REST API should throw and exception, when the same cube name is 
> listed multiple times.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-3518) Coprocessor reports NPE when execute a query on HBase 2.0

2018-09-10 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610072#comment-16610072
 ] 

ASF subversion and git services commented on KYLIN-3518:


Commit 45b20ca17a6070c35d3a8967f595a06cbae3a108 in kylin's branch 
refs/heads/2.5.x-hadoop3.1 from [~caolijun1166]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=45b20ca ]

KYLIN-3518 Fix Coprocessor NPE problem on hbase 2


> Coprocessor reports NPE when execute a query on HBase 2.0
> -
>
> Key: KYLIN-3518
> URL: https://issues.apache.org/jira/browse/KYLIN-3518
> Project: Kylin
>  Issue Type: Bug
>  Components: Storage - HBase
>Reporter: Shaofeng SHI
>Assignee: Lijun Cao
>Priority: Major
> Fix For: v2.5.0
>
>
> On HDP 3.0, build a cube and then run a simple count query, NPE occurred:
>  
> {code:java}
> 2018-08-28 01:30:16,969 DEBUG [Query f7bf8004-b516-e372-18df-0d507075d471-71] 
> v2.CubeHBaseRPC:315 : hbase.rpc.timeout = 9 ms, use 81000 ms as timeout 
> for coprocessor
> 2018-08-28 01:30:16,983 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71] 
> v2.CubeHBaseEndpointRPC:141 : Serialized scanRequestBytes 522 bytes, 
> rawScanBytesString 44 bytes
> 2018-08-28 01:30:16,984 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71] 
> v2.CubeHBaseEndpointRPC:143 : The scan 67b41fc6 for segment 
> kylin_sales_cube_clone[2012010100_2013010100] is as below with 1 
> separate raw scans, shard part of start/end key is set to 0
> 2018-08-28 01:30:16,991 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71] 
> v2.CubeHBaseRPC:288 : Visiting hbase table KYLIN_5Q088VO5I0: cuboid require 
> post aggregation, from 0 to 16384 Start: 
> \x00\x00\x00\x00\x00\x00\x00\x00\x40\x00\x00\x00\x00 
> (\x00\x00\x00\x00\x00\x00\x00\x00@\x00\x00\x00\x00) Stop: 
> \x00\x00\x00\x00\x00\x00\x00\x00\x40\x00\xFF\xFF\xFF\x00 
> (\x00\x00\x00\x00\x00\x00\x00\x00@\x00\xFF\xFF\xFF\x00), No Fuzzy Key
> 2018-08-28 01:30:16,991 DEBUG [Query f7bf8004-b516-e372-18df-0d507075d471-71] 
> v2.CubeHBaseEndpointRPC:148 : Submitting rpc to 1 shards starting from shard 
> 0, scan range count 1
> 2018-08-28 01:30:17,010 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71] 
> common.KylinConfig:332 : Loading kylin-defaults.properties from 
> file:/root/shaofengshi/apache-kylin-2.5.0-SNAPSHOT-bin/tomcat/webapps/kylin/WEB-INF/lib/kylin-core-common-2.5.0-SNAPSHOT.jar!/kylin-defaults.properties
> 2018-08-28 01:30:17,033 DEBUG [Query f7bf8004-b516-e372-18df-0d507075d471-71] 
> common.KylinConfig:291 : KYLIN_CONF property was not set, will seek 
> KYLIN_HOME env variable
> 2018-08-28 01:30:17,051 INFO [pool-14-thread-1] hbase.HBaseConnection:110 : 
> Creating coprocessor thread pool with max of 2048, core of 2048
> 2018-08-28 01:30:17,094 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71] 
> gtrecord.SequentialCubeTupleIterator:73 : Using SortedIteratorMergerWithLimit 
> to merge segment results
> 2018-08-28 01:30:17,097 DEBUG [Query f7bf8004-b516-e372-18df-0d507075d471-71] 
> enumerator.OLAPEnumerator:117 : return TupleIterator...
> 2018-08-28 01:30:21,607 INFO [kylin-coproc--pool9-t1] 
> client.RpcRetryingCallerImpl:134 : Call exception, tries=6, retries=6, 
> started=4410 ms ago, cancelled=false, msg=java.io.IOException
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:468)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> Caused by: java.lang.NullPointerException
> at 
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService.visitCube(CubeVisitService.java:253)
> at 
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService.callMethod(CubeVisitProtos.java:)
> at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8032)
> at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2426)
> at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2408)
> at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42010)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
> ... 3 more
> , details=row '' on table 'KYLIN_5Q088VO5I0' at 
> region=KYLIN_5Q088VO5I0,,1535417272444.27b82cb4702db4557a98b9a7e60b7692., 
> hostname=ignite03.com,16020,1534313612401, seqNum=2
> 2018-08-28 01:30:25,633 INFO [kylin-coproc--pool9-t1] 
> client.RpcRetryingCallerImpl:134 : Call exception, tries=7, retries=7, 
> started=8436 ms ago, cancelled=false, msg=java.io.IOException
> at 

[jira] [Commented] (KYLIN-3539) Hybrid segment overlap not cover some case

2018-09-10 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610064#comment-16610064
 ] 

ASF subversion and git services commented on KYLIN-3539:


Commit f15ae54b10ef7b9f22f605d61618d16038512113 in kylin's branch 
refs/heads/2.5.x from chao long
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=f15ae54 ]

KYLIN-3539 Hybrid segment overlap check


> Hybrid segment overlap not cover some case
> --
>
> Key: KYLIN-3539
> URL: https://issues.apache.org/jira/browse/KYLIN-3539
> Project: Kylin
>  Issue Type: Improvement
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.5.0
>
>
> now Hybrid uses every cube's latest segment to check whether overlap exists, 
> but it can't cover all cases.
> example:
> hybrid:
>  cube1:
>  segment1: range(2012-01-01, 2012-01-02)
> cube2:
>  segment1: range(2012-01-01, 2012-01-02)
>  segment2: range(2012-01-03, 2012-01-04)
> this hybrid has overlap segment range(2012-01-01, 2012-01-02), but the 
> program can not find out.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-3258) No check for duplicate cube name when creating a hybrid cube

2018-09-10 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610060#comment-16610060
 ] 

ASF subversion and git services commented on KYLIN-3258:


Commit 3e6307473d60036feb8c8497379eb4841d82d093 in kylin's branch 
refs/heads/2.5.x from GinaZhai
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=3e63074 ]

KYLIN-3258 No check for duplicate cube name when creating a hybrid cube


> No check for duplicate cube name when creating a hybrid cube
> 
>
> Key: KYLIN-3258
> URL: https://issues.apache.org/jira/browse/KYLIN-3258
> Project: Kylin
>  Issue Type: Bug
>  Components: Metadata
>Affects Versions: v2.2.0
> Environment: HDP 2.5.6, Kylin 2.2
>Reporter: Vsevolod Ostapenko
>Assignee: Shaofeng SHI
>Priority: Minor
> Fix For: v2.5.0
>
>
> When loading hybrid cube definitions via REST API, there is no check for 
> duplicate cube names is the list. If due to a user error or incorrectly 
> generated list of cubes by an external application/script the same cube name 
> is listed more than once, new or updated hybrid cube will contain the same 
> cube listed multiple times.
> It does not seem to cause any immediate issues with querying, but it's just 
> not right. REST API should throw and exception, when the same cube name is 
> listed multiple times.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-3529) Prompt not friendly

2018-09-10 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610061#comment-16610061
 ] 

ASF subversion and git services commented on KYLIN-3529:


Commit 3b3717f11fb7e7f4ed70ace943a064bf704a42a7 in kylin's branch 
refs/heads/2.5.x from chao long
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=3b3717f ]

KYLIN-3529 Show more error message


> Prompt not friendly
> ---
>
> Key: KYLIN-3529
> URL: https://issues.apache.org/jira/browse/KYLIN-3529
> Project: Kylin
>  Issue Type: Bug
>  Components: Web 
>Reporter: Na Zhai
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.5.0
>
>
>  *Scene 1:*When I create a hybrid with duplicate name, then I click submit, 
> the error message "error execute org.apache.kylin.rest.job.HybridCubeCLI" 
> will pop up.
>  
> *Scene 2:*When I create a hybrid with duplicate date/time cube, then I click 
> submit, the error message "error execute 
> org.apache.kylin.rest.job.HybridCubeCLI" will pop up.
>  
> I think the error message should be more friendly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-3527) Hybrid couldn't save when there is only 1 cube

2018-09-10 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610063#comment-16610063
 ] 

ASF subversion and git services commented on KYLIN-3527:


Commit 5ce6588eda365cf4aca5f53ab6cb1adaa4ffe9e8 in kylin's branch 
refs/heads/2.5.x from chao long
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=5ce6588 ]

KYLIN-3527 Hybrid allows 0,1 cube


> Hybrid couldn't save when there is only 1 cube
> --
>
> Key: KYLIN-3527
> URL: https://issues.apache.org/jira/browse/KYLIN-3527
> Project: Kylin
>  Issue Type: Bug
>  Components: Web 
>Affects Versions: v2.5.0
>Reporter: Shaofeng SHI
>Assignee: Roger
>Priority: Minor
> Fix For: v2.5.0
>
> Attachments: hybrid-not-saved.png, move-in-out-icons.png
>
>
> I created a hybrid with two cubes; Later I want to remove one cube from it, 
> but I found when there is only 1 cube, the "Submit" button was disabled.
>  
> The hybrid should allows 0, 1 or more cubes.
>  
> Besides, I found the move in and out icons were not properly displayed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-3534) Failed at update cube info step

2018-09-10 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610062#comment-16610062
 ] 

ASF subversion and git services commented on KYLIN-3534:


Commit d90bb046d191dd06f194e7380da13698772e001f in kylin's branch 
refs/heads/2.5.x from chao long
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=d90bb04 ]

KYLIN-3534 Don't compress fact distinct output file


> Failed at update cube info step
> ---
>
> Key: KYLIN-3534
> URL: https://issues.apache.org/jira/browse/KYLIN-3534
> Project: Kylin
>  Issue Type: Bug
>Reporter: Yichen Zhou
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.5.0
>
>
> When building cube with spark(full build), I got the following error in the 
> step of "Update Cube Info":
> {quote}2018-09-04 14:49:49,007 INFO [Scheduler 158588 Job 
> d05156bb-2635-cb2f-ac19-b38f6147fcb2-699] 
> steps.UpdateCubeInfoAfterBuildStep:158 : updateSegment step. TRANS_ID 
> minValue:x�3�.�) maxValue:null
>  2018-09-04 14:49:49,008 ERROR [Scheduler 158588 Job 
> d05156bb-2635-cb2f-ac19-b38f6147fcb2-699] execution.AbstractExecutable:165 : 
> error running Executable: 
> UpdateCubeInfoAfterBuildStep\{id=d05156bb-2635-cb2f-ac19-b38f6147fcb2-09, 
> name=Update Cube Info, state=RUNNING}
>  2018-09-04 14:49:49,011 INFO [Scheduler 158588 Job 
> d05156bb-2635-cb2f-ac19-b38f6147fcb2-699] execution.ExecutableManager:434 : 
> job id:d05156bb-2635-cb2f-ac19-b38f6147fcb2-09 from RUNNING to ERROR
>  2018-09-04 14:49:49,012 ERROR [Scheduler 158588 Job 
> d05156bb-2635-cb2f-ac19-b38f6147fcb2-699] execution.AbstractExecutable:165 : 
> error running Executable: CubingJob\{id=d05156bb-2635-cb2f-ac19-b38f6147fcb2, 
> name=BUILD CUBE - cube_global_dictionary_spark - FULL_BUILD - GMT+08:00 
> 2018-09-04 11:51:58, state=RUNNING}
>  2018-09-04 14:49:49,014 DEBUG [pool-6-thread-1] cachesync.Broadcaster:113 : 
> Servers in the cluster: [localhost:7070]
>  2018-09-04 14:49:49,014 DEBUG [pool-6-thread-1] cachesync.Broadcaster:123 : 
> Announcing new broadcast to all: BroadcastEvent\{entity=execute_output, 
> event=update, cacheKey=d05156bb-2635-cb2f-ac19-b38f6147fcb2}
>  2018-09-04 14:49:49,015 INFO [Scheduler 158588 Job 
> d05156bb-2635-cb2f-ac19-b38f6147fcb2-699] execution.ExecutableManager:434 : 
> job id:d05156bb-2635-cb2f-ac19-b38f6147fcb2 from RUNNING to ERROR
>  2018-09-04 14:49:49,015 DEBUG [pool-6-thread-1] cachesync.Broadcaster:113 : 
> Servers in the cluster: [localhost:7070]
>  2018-09-04 14:49:49,015 DEBUG [pool-6-thread-1] cachesync.Broadcaster:123 : 
> Announcing new broadcast to all: BroadcastEvent\{entity=execute_output, 
> event=update, cacheKey=d05156bb-2635-cb2f-ac19-b38f6147fcb2}
>  2018-09-04 14:49:49,015 DEBUG [Scheduler 158588 Job 
> d05156bb-2635-cb2f-ac19-b38f6147fcb2-699] execution.AbstractExecutable:316 : 
> no need to send email, user list is empty
>  2018-09-04 14:49:49,017 ERROR [pool-10-thread-10] 
> threadpool.DefaultScheduler:115 : ExecuteException 
> job:d05156bb-2635-cb2f-ac19-b38f6147fcb2
> org.apache.kylin.job.exception.ExecuteException: 
> org.apache.kylin.job.exception.ExecuteException:java.lang.IllegalStateException
>  at 
> org.apache.kylin.cube.DimensionRangeInfo.(DimensionRangeInfo.java:86)
>  at 
> org.apache.kylin.engine.mr.steps.UpdateCubeInfoAfterBuildStep.updateSegment(UpdateCubeInfoAfterBuildStep.java:166)
>  at 
> org.apache.kylin.engine.mr.steps.UpdateCubeInfoAfterBuildStep.doWork(UpdateCubeInfoAfterBuildStep.java:85)
>  at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163)
>  at 
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:69)
>  at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163)
>  at 
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:113)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  at java.lang.Thread.run(Thread.java:748)
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Resolved] (KYLIN-3538) Automatic cube enabled functionality is not merged into 2.4.0

2018-09-10 Thread Shaofeng SHI (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3538.
-
Resolution: Fixed

Merged in 2.4.x, 2.5.x and master branch.

> Automatic cube enabled functionality is not merged into 2.4.0
> -
>
> Key: KYLIN-3538
> URL: https://issues.apache.org/jira/browse/KYLIN-3538
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine
>Affects Versions: v2.4.0
>Reporter: Iñigo Martinez
>Assignee: Shaofeng SHI
>Priority: Minor
> Fix For: v2.5.0
>
>
> This functionality that was scheduled for version 2.4.0 finally was not 
> included in proper release. 
> [https://jira.apache.org/jira/browse/KYLIN-3366|https://jira.apache.org/jira/browse/KYLIN-3366]
> Checking source code show us that is missing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-3366) Configure automatic enabling of cubes after a build process

2018-09-10 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610052#comment-16610052
 ] 

ASF subversion and git services commented on KYLIN-3366:


Commit 165378bb13a374fad80830ab9a5c56a45420aa7c in kylin's branch 
refs/heads/master from Zhong
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=165378b ]

KYLIN-3366 Configure automatic enabling of cubes after a build process

Signed-off-by: shaofengshi 


> Configure automatic enabling of cubes after a build process
> ---
>
> Key: KYLIN-3366
> URL: https://issues.apache.org/jira/browse/KYLIN-3366
> Project: Kylin
>  Issue Type: New Feature
>Affects Versions: v2.3.1
> Environment: Kylin 2.3.1 and Hadoop EMR 5.7
>Reporter: Roberto Tardío Olmos
>Assignee: Pan, Julian
>Priority: Minor
>  Labels: features
> Fix For: v2.4.0
>
> Attachments: KYLIN-3366.patch
>
>
> Kylin automatically enables the disabled cubes after a construction process. 
> This behavior forces us to constantly disable a new cube that is under 
> development to replace an existing and enabled cube. If we do not disable it, 
> we could have problems with the routing of the queries.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (KYLIN-3538) Automatic cube enabled functionality is not merged into 2.4.0

2018-09-10 Thread Shaofeng SHI (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI reassigned KYLIN-3538:
---

Assignee: Shaofeng SHI

> Automatic cube enabled functionality is not merged into 2.4.0
> -
>
> Key: KYLIN-3538
> URL: https://issues.apache.org/jira/browse/KYLIN-3538
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine
>Affects Versions: v2.4.0
>Reporter: Iñigo Martinez
>Assignee: Shaofeng SHI
>Priority: Minor
> Fix For: v2.5.0
>
>
> This functionality that was scheduled for version 2.4.0 finally was not 
> included in proper release. 
> [https://jira.apache.org/jira/browse/KYLIN-3366|https://jira.apache.org/jira/browse/KYLIN-3366]
> Checking source code show us that is missing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-3538) Automatic cube enabled functionality is not merged into 2.4.0

2018-09-10 Thread Shaofeng SHI (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610050#comment-16610050
 ] 

Shaofeng SHI commented on KYLIN-3538:
-

The patch should be missed; I just merged it in 2.4.x branch; You can manually 
make a binary build from it. If there will be a v2.4.2 version, it will be 
included. 

I'm adding it to 2.5.0 now, so it will be released in 2.5.0.

> Automatic cube enabled functionality is not merged into 2.4.0
> -
>
> Key: KYLIN-3538
> URL: https://issues.apache.org/jira/browse/KYLIN-3538
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine
>Affects Versions: v2.4.0
>Reporter: Iñigo Martinez
>Priority: Minor
> Fix For: v2.5.0
>
>
> This functionality that was scheduled for version 2.4.0 finally was not 
> included in proper release. 
> [https://jira.apache.org/jira/browse/KYLIN-3366|https://jira.apache.org/jira/browse/KYLIN-3366]
> Checking source code show us that is missing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-3366) Configure automatic enabling of cubes after a build process

2018-09-10 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610051#comment-16610051
 ] 

ASF subversion and git services commented on KYLIN-3366:


Commit ded181304846a52452ce1bc3c597611d167a7eb0 in kylin's branch 
refs/heads/2.5.x from Zhong
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=ded1813 ]

KYLIN-3366 Configure automatic enabling of cubes after a build process

Signed-off-by: shaofengshi 


> Configure automatic enabling of cubes after a build process
> ---
>
> Key: KYLIN-3366
> URL: https://issues.apache.org/jira/browse/KYLIN-3366
> Project: Kylin
>  Issue Type: New Feature
>Affects Versions: v2.3.1
> Environment: Kylin 2.3.1 and Hadoop EMR 5.7
>Reporter: Roberto Tardío Olmos
>Assignee: Pan, Julian
>Priority: Minor
>  Labels: features
> Fix For: v2.4.0
>
> Attachments: KYLIN-3366.patch
>
>
> Kylin automatically enables the disabled cubes after a construction process. 
> This behavior forces us to constantly disable a new cube that is under 
> development to replace an existing and enabled cube. If we do not disable it, 
> we could have problems with the routing of the queries.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-3366) Configure automatic enabling of cubes after a build process

2018-09-10 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610048#comment-16610048
 ] 

ASF subversion and git services commented on KYLIN-3366:


Commit 723cc7a6b6ac7d260511a45f0188746a59d87c3c in kylin's branch 
refs/heads/2.4.x from Zhong
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=723cc7a ]

KYLIN-3366 Configure automatic enabling of cubes after a build process

Signed-off-by: shaofengshi 


> Configure automatic enabling of cubes after a build process
> ---
>
> Key: KYLIN-3366
> URL: https://issues.apache.org/jira/browse/KYLIN-3366
> Project: Kylin
>  Issue Type: New Feature
>Affects Versions: v2.3.1
> Environment: Kylin 2.3.1 and Hadoop EMR 5.7
>Reporter: Roberto Tardío Olmos
>Assignee: Pan, Julian
>Priority: Minor
>  Labels: features
> Fix For: v2.4.0
>
> Attachments: KYLIN-3366.patch
>
>
> Kylin automatically enables the disabled cubes after a construction process. 
> This behavior forces us to constantly disable a new cube that is under 
> development to replace an existing and enabled cube. If we do not disable it, 
> we could have problems with the routing of the queries.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (KYLIN-3550) "kylin.source.hive.flat-table-field-delimiter" has extra "\" when create intermediate flat table

2018-09-10 Thread Shaofeng SHI (JIRA)
Shaofeng SHI created KYLIN-3550:
---

 Summary: "kylin.source.hive.flat-table-field-delimiter" has extra 
"\" when create intermediate flat table
 Key: KYLIN-3550
 URL: https://issues.apache.org/jira/browse/KYLIN-3550
 Project: Kylin
  Issue Type: Bug
  Components: Job Engine
Affects Versions: v2.4.0, v2.4.1
Reporter: Shaofeng SHI






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (KYLIN-3550) "kylin.source.hive.flat-table-field-delimiter" has extra "\" when create intermediate flat table

2018-09-10 Thread Shaofeng SHI (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-3550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI reassigned KYLIN-3550:
---

Assignee: Shaofeng SHI

> "kylin.source.hive.flat-table-field-delimiter" has extra "\" when create 
> intermediate flat table
> 
>
> Key: KYLIN-3550
> URL: https://issues.apache.org/jira/browse/KYLIN-3550
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine
>Affects Versions: v2.4.0, v2.4.1
>Reporter: Shaofeng SHI
>Assignee: Shaofeng SHI
>Priority: Minor
> Fix For: v2.5.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (KYLIN-3538) Automatic cube enabled functionality is not merged into 2.4.0

2018-09-10 Thread Shaofeng SHI (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3538:

Fix Version/s: v2.5.0

> Automatic cube enabled functionality is not merged into 2.4.0
> -
>
> Key: KYLIN-3538
> URL: https://issues.apache.org/jira/browse/KYLIN-3538
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine
>Affects Versions: v2.4.0
>Reporter: Iñigo Martinez
>Priority: Minor
> Fix For: v2.5.0
>
>
> This functionality that was scheduled for version 2.4.0 finally was not 
> included in proper release. 
> [https://jira.apache.org/jira/browse/KYLIN-3366|https://jira.apache.org/jira/browse/KYLIN-3366]
> Checking source code show us that is missing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (KYLIN-3550) "kylin.source.hive.flat-table-field-delimiter" has extra "\" when create intermediate flat table

2018-09-10 Thread Shaofeng SHI (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-3550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3550:

Fix Version/s: v2.5.0

> "kylin.source.hive.flat-table-field-delimiter" has extra "\" when create 
> intermediate flat table
> 
>
> Key: KYLIN-3550
> URL: https://issues.apache.org/jira/browse/KYLIN-3550
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine
>Affects Versions: v2.4.0, v2.4.1
>Reporter: Shaofeng SHI
>Priority: Minor
> Fix For: v2.5.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-3538) Automatic cube enabled functionality is not merged into 2.4.0

2018-09-10 Thread Shaofeng SHI (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1661#comment-1661
 ] 

Shaofeng SHI commented on KYLIN-3538:
-

Thanks for the reporting!

> Automatic cube enabled functionality is not merged into 2.4.0
> -
>
> Key: KYLIN-3538
> URL: https://issues.apache.org/jira/browse/KYLIN-3538
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine
>Affects Versions: v2.4.0
>Reporter: Iñigo Martinez
>Priority: Minor
>
> This functionality that was scheduled for version 2.4.0 finally was not 
> included in proper release. 
> [https://jira.apache.org/jira/browse/KYLIN-3366|https://jira.apache.org/jira/browse/KYLIN-3366]
> Checking source code show us that is missing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-2565) Upgrade Kylin to Hadoop3.0

2018-09-10 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609980#comment-16609980
 ] 

ASF subversion and git services commented on KYLIN-2565:


Commit a83133ad565a98cf90ce8d2cf603ecbf8e96a3b0 in kylin's branch 
refs/heads/master-hadoop3.1-2.5.0 from shaofengshi
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=a83133a ]

KYLIN-2565 upgrade to hadoop 3.0 hbase 2.0, pass UT


> Upgrade Kylin to Hadoop3.0
> --
>
> Key: KYLIN-2565
> URL: https://issues.apache.org/jira/browse/KYLIN-2565
> Project: Kylin
>  Issue Type: New Feature
>Reporter: Wang Cheng
>Assignee: Shaofeng SHI
>Priority: Major
>
> Hadoop3.0-alpha is released, Kylin should also keep compatible with it. Below 
> is the Hadoop3.0 components requirements:
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+release 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-3517) Couldn't update coprocessor on HBase 2.0

2018-09-10 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609982#comment-16609982
 ] 

ASF subversion and git services commented on KYLIN-3517:


Commit 96026886774aff16f03e6d5424e9c0ccdfca1ca0 in kylin's branch 
refs/heads/master-hadoop3.1-2.5.0 from [~caolijun1166]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=9602688 ]

KYLIN-3517 Upadate coprocessor on HBase2.0 is avaliable.

Signed-off-by: shaofengshi 


> Couldn't update coprocessor on HBase 2.0
> 
>
> Key: KYLIN-3517
> URL: https://issues.apache.org/jira/browse/KYLIN-3517
> Project: Kylin
>  Issue Type: Bug
>  Components: Storage - HBase
>Reporter: Shaofeng SHI
>Assignee: Lijun Cao
>Priority: Major
>
> On HDP 3.0, run update coprocessor, got this error:
>  
> {code:java}
> 2018-08-28 00:24:26,683 ERROR [pool-7-thread-1] util.DeployCoprocessorCLI:383 
> : Error processing KYLIN_O9JRT8XOQ9
> java.lang.UnsupportedOperationException: HTableDescriptor is read-only
> at 
> org.apache.hadoop.hbase.client.ImmutableHTableDescriptor.getDelegateeForModification(ImmutableHTableDescriptor.java:59)
> at 
> org.apache.hadoop.hbase.HTableDescriptor.removeCoprocessor(HTableDescriptor.java:768)
> at 
> org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI.resetCoprocessor(DeployCoprocessorCLI.java:300)
> at 
> org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI$ResetCoprocessorWorker.run(DeployCoprocessorCLI.java:375)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-3518) Coprocessor reports NPE when execute a query on HBase 2.0

2018-09-10 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609981#comment-16609981
 ] 

ASF subversion and git services commented on KYLIN-3518:


Commit 11c2ab2d6f6fd422402345b1d920a55a353a9124 in kylin's branch 
refs/heads/master-hadoop3.1-2.5.0 from [~caolijun1166]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=11c2ab2 ]

KYLIN-3518 Fix Coprocessor NPE problem on hbase 2


> Coprocessor reports NPE when execute a query on HBase 2.0
> -
>
> Key: KYLIN-3518
> URL: https://issues.apache.org/jira/browse/KYLIN-3518
> Project: Kylin
>  Issue Type: Bug
>  Components: Storage - HBase
>Reporter: Shaofeng SHI
>Assignee: Lijun Cao
>Priority: Major
> Fix For: v2.5.0
>
>
> On HDP 3.0, build a cube and then run a simple count query, NPE occurred:
>  
> {code:java}
> 2018-08-28 01:30:16,969 DEBUG [Query f7bf8004-b516-e372-18df-0d507075d471-71] 
> v2.CubeHBaseRPC:315 : hbase.rpc.timeout = 9 ms, use 81000 ms as timeout 
> for coprocessor
> 2018-08-28 01:30:16,983 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71] 
> v2.CubeHBaseEndpointRPC:141 : Serialized scanRequestBytes 522 bytes, 
> rawScanBytesString 44 bytes
> 2018-08-28 01:30:16,984 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71] 
> v2.CubeHBaseEndpointRPC:143 : The scan 67b41fc6 for segment 
> kylin_sales_cube_clone[2012010100_2013010100] is as below with 1 
> separate raw scans, shard part of start/end key is set to 0
> 2018-08-28 01:30:16,991 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71] 
> v2.CubeHBaseRPC:288 : Visiting hbase table KYLIN_5Q088VO5I0: cuboid require 
> post aggregation, from 0 to 16384 Start: 
> \x00\x00\x00\x00\x00\x00\x00\x00\x40\x00\x00\x00\x00 
> (\x00\x00\x00\x00\x00\x00\x00\x00@\x00\x00\x00\x00) Stop: 
> \x00\x00\x00\x00\x00\x00\x00\x00\x40\x00\xFF\xFF\xFF\x00 
> (\x00\x00\x00\x00\x00\x00\x00\x00@\x00\xFF\xFF\xFF\x00), No Fuzzy Key
> 2018-08-28 01:30:16,991 DEBUG [Query f7bf8004-b516-e372-18df-0d507075d471-71] 
> v2.CubeHBaseEndpointRPC:148 : Submitting rpc to 1 shards starting from shard 
> 0, scan range count 1
> 2018-08-28 01:30:17,010 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71] 
> common.KylinConfig:332 : Loading kylin-defaults.properties from 
> file:/root/shaofengshi/apache-kylin-2.5.0-SNAPSHOT-bin/tomcat/webapps/kylin/WEB-INF/lib/kylin-core-common-2.5.0-SNAPSHOT.jar!/kylin-defaults.properties
> 2018-08-28 01:30:17,033 DEBUG [Query f7bf8004-b516-e372-18df-0d507075d471-71] 
> common.KylinConfig:291 : KYLIN_CONF property was not set, will seek 
> KYLIN_HOME env variable
> 2018-08-28 01:30:17,051 INFO [pool-14-thread-1] hbase.HBaseConnection:110 : 
> Creating coprocessor thread pool with max of 2048, core of 2048
> 2018-08-28 01:30:17,094 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71] 
> gtrecord.SequentialCubeTupleIterator:73 : Using SortedIteratorMergerWithLimit 
> to merge segment results
> 2018-08-28 01:30:17,097 DEBUG [Query f7bf8004-b516-e372-18df-0d507075d471-71] 
> enumerator.OLAPEnumerator:117 : return TupleIterator...
> 2018-08-28 01:30:21,607 INFO [kylin-coproc--pool9-t1] 
> client.RpcRetryingCallerImpl:134 : Call exception, tries=6, retries=6, 
> started=4410 ms ago, cancelled=false, msg=java.io.IOException
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:468)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> Caused by: java.lang.NullPointerException
> at 
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService.visitCube(CubeVisitService.java:253)
> at 
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService.callMethod(CubeVisitProtos.java:)
> at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8032)
> at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2426)
> at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2408)
> at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42010)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
> ... 3 more
> , details=row '' on table 'KYLIN_5Q088VO5I0' at 
> region=KYLIN_5Q088VO5I0,,1535417272444.27b82cb4702db4557a98b9a7e60b7692., 
> hostname=ignite03.com,16020,1534313612401, seqNum=2
> 2018-08-28 01:30:25,633 INFO [kylin-coproc--pool9-t1] 
> client.RpcRetryingCallerImpl:134 : Call exception, tries=7, retries=7, 
> started=8436 ms ago, cancelled=false, msg=java.io.IOException
> at 

[jira] [Commented] (KYLIN-3517) Couldn't update coprocessor on HBase 2.0

2018-09-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609978#comment-16609978
 ] 

ASF GitHub Bot commented on KYLIN-3517:
---

shaofengshi closed pull request #225: KYLIN-3517 Upadate coprocessor on 
HBase2.0 is avaliable.
URL: https://github.com/apache/kylin/pull/225
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/lookup/LookupTableToHFileJob.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/lookup/LookupTableToHFileJob.java
index 054e1466b6..27894010f7 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/lookup/LookupTableToHFileJob.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/lookup/LookupTableToHFileJob.java
@@ -26,12 +26,12 @@
 import org.apache.commons.lang.StringUtils;
 import org.apache.hadoop.fs.Path;
 import org.apache.hadoop.hbase.HBaseConfiguration;
-import org.apache.hadoop.hbase.HColumnDescriptor;
-import org.apache.hadoop.hbase.HTableDescriptor;
 import org.apache.hadoop.hbase.TableName;
 import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptor;
 import org.apache.hadoop.hbase.client.Connection;
 import org.apache.hadoop.hbase.client.HTable;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
 import org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2;
 import org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy;
 import org.apache.hadoop.hdfs.DFSConfigKeys;
@@ -207,24 +207,24 @@ private void 
removeSnapshotIfExist(ExtTableSnapshotInfoManager extSnapshotInfoMa
 String hTableName = genHTableName(kylinConfig, admin, sourceTableName);
 
 TableName tableName = TableName.valueOf(hTableName);
-HTableDescriptor hTableDesc = new HTableDescriptor(tableName);
-hTableDesc.setCompactionEnabled(false);
-hTableDesc.setValue(HTableDescriptor.SPLIT_POLICY, 
DisabledRegionSplitPolicy.class.getName());
-hTableDesc.setValue(IRealizationConstants.HTableTag, 
kylinConfig.getMetadataUrlPrefix());
-hTableDesc.setValue(IRealizationConstants.HTableCreationTime, 
String.valueOf(System.currentTimeMillis()));
+TableDescriptorBuilder descBuilder = 
TableDescriptorBuilder.newBuilder(tableName);
+descBuilder.setCompactionEnabled(false);
+descBuilder.setValue(TableDescriptorBuilder.SPLIT_POLICY, 
DisabledRegionSplitPolicy.class.getName());
+descBuilder.setValue(IRealizationConstants.HTableTag, 
kylinConfig.getMetadataUrlPrefix());
+descBuilder.setValue(IRealizationConstants.HTableCreationTime, 
String.valueOf(System.currentTimeMillis()));
 String commitInfo = KylinVersion.getGitCommitInfo();
 if (!StringUtils.isEmpty(commitInfo)) {
-hTableDesc.setValue(IRealizationConstants.HTableGitTag, 
commitInfo);
+descBuilder.setValue(IRealizationConstants.HTableGitTag, 
commitInfo);
 }
 
-HColumnDescriptor cf = CubeHTableUtil.createColumnFamily(kylinConfig, 
HBaseLookupRowEncoder.CF_STRING, false);
-hTableDesc.addFamily(cf);
+ColumnFamilyDescriptor cf = 
CubeHTableUtil.createColumnFamily(kylinConfig, HBaseLookupRowEncoder.CF_STRING, 
false);
+descBuilder.modifyColumnFamily(cf);
 
 try {
 if (shardNum > 1) {
-admin.createTable(hTableDesc, getSplitsByShardNum(shardNum));
+admin.createTable(descBuilder.build(), 
getSplitsByShardNum(shardNum));
 } else {
-admin.createTable(hTableDesc);
+admin.createTable(descBuilder.build());
 }
 } finally {
 IOUtils.closeQuietly(admin);
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CubeHTableUtil.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CubeHTableUtil.java
index f006adb748..9e3703c409 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CubeHTableUtil.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CubeHTableUtil.java
@@ -23,11 +23,12 @@
 import org.apache.commons.io.IOUtils;
 import org.apache.commons.lang.StringUtils;
 import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.hbase.HColumnDescriptor;
-import org.apache.hadoop.hbase.HTableDescriptor;
 import org.apache.hadoop.hbase.TableName;
 import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptor;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
 import org.apache.hadoop.hbase.client.Connection;
+import 

[jira] [Closed] (KYLIN-3548) Typo on Community page

2018-09-10 Thread Shaofeng SHI (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-3548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI closed KYLIN-3548.
---

> Typo on Community page
> --
>
> Key: KYLIN-3548
> URL: https://issues.apache.org/jira/browse/KYLIN-3548
> Project: Kylin
>  Issue Type: Bug
>Reporter: Sebb
>Assignee: Shaofeng SHI
>Priority: Major
>
> "artical" should probably be "article" on page
> https://kylin.apache.org/community/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-3549) Download page must link to KEYS at https://www.apache.org/dist/kylin/KEYS

2018-09-10 Thread Shaofeng SHI (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609970#comment-16609970
 ] 

Shaofeng SHI commented on KYLIN-3549:
-

Kylin's download page has been updated, using 
[https://www.apache.org/dist/kylin/KEYS]. 

> Download page must link to KEYS at https://www.apache.org/dist/kylin/KEYS
> -
>
> Key: KYLIN-3549
> URL: https://issues.apache.org/jira/browse/KYLIN-3549
> Project: Kylin
>  Issue Type: Bug
>Reporter: Sebb
>Assignee: Zhong Yanghong
>Priority: Major
>
> As the subject says.
> The KEYS file must exist at
> https://www.apache.org/dist/kylin/KEYS
> Please maintain the file at
> https://dist.apache.org/repos/dist/release/kylin/KEYS



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-3517) Couldn't update coprocessor on HBase 2.0

2018-09-10 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609962#comment-16609962
 ] 

ASF subversion and git services commented on KYLIN-3517:


Commit cd0f7cbc9473c3b0bfabc076154d85f679c1a41b in kylin's branch 
refs/heads/master-hadoop3.1 from [~caolijun1166]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=cd0f7cb ]

KYLIN-3517 Upadate coprocessor on HBase2.0 is avaliable.

Signed-off-by: shaofengshi 


> Couldn't update coprocessor on HBase 2.0
> 
>
> Key: KYLIN-3517
> URL: https://issues.apache.org/jira/browse/KYLIN-3517
> Project: Kylin
>  Issue Type: Bug
>  Components: Storage - HBase
>Reporter: Shaofeng SHI
>Assignee: Lijun Cao
>Priority: Major
>
> On HDP 3.0, run update coprocessor, got this error:
>  
> {code:java}
> 2018-08-28 00:24:26,683 ERROR [pool-7-thread-1] util.DeployCoprocessorCLI:383 
> : Error processing KYLIN_O9JRT8XOQ9
> java.lang.UnsupportedOperationException: HTableDescriptor is read-only
> at 
> org.apache.hadoop.hbase.client.ImmutableHTableDescriptor.getDelegateeForModification(ImmutableHTableDescriptor.java:59)
> at 
> org.apache.hadoop.hbase.HTableDescriptor.removeCoprocessor(HTableDescriptor.java:768)
> at 
> org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI.resetCoprocessor(DeployCoprocessorCLI.java:300)
> at 
> org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI$ResetCoprocessorWorker.run(DeployCoprocessorCLI.java:375)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-2565) Upgrade Kylin to Hadoop3.0

2018-09-10 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609959#comment-16609959
 ] 

ASF subversion and git services commented on KYLIN-2565:


Commit 616bb38b41880136f3b649730ba910fa74a64216 in kylin's branch 
refs/heads/master-hadoop3.1 from [~blue20080]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=616bb38 ]

KYLIN-2565, upgrade to Hadoop3.0


> Upgrade Kylin to Hadoop3.0
> --
>
> Key: KYLIN-2565
> URL: https://issues.apache.org/jira/browse/KYLIN-2565
> Project: Kylin
>  Issue Type: New Feature
>Reporter: Wang Cheng
>Assignee: Shaofeng SHI
>Priority: Major
>
> Hadoop3.0-alpha is released, Kylin should also keep compatible with it. Below 
> is the Hadoop3.0 components requirements:
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+release 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-3534) Failed at update cube info step

2018-09-10 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609956#comment-16609956
 ] 

ASF subversion and git services commented on KYLIN-3534:


Commit d164225e91fa59aa1dd06d2ddf64e6acc1035a3f in kylin's branch 
refs/heads/master-hadoop3.1 from chao long
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=d164225 ]

KYLIN-3534 Don't compress fact distinct output file


> Failed at update cube info step
> ---
>
> Key: KYLIN-3534
> URL: https://issues.apache.org/jira/browse/KYLIN-3534
> Project: Kylin
>  Issue Type: Bug
>Reporter: Yichen Zhou
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.5.0
>
>
> When building cube with spark(full build), I got the following error in the 
> step of "Update Cube Info":
> {quote}2018-09-04 14:49:49,007 INFO [Scheduler 158588 Job 
> d05156bb-2635-cb2f-ac19-b38f6147fcb2-699] 
> steps.UpdateCubeInfoAfterBuildStep:158 : updateSegment step. TRANS_ID 
> minValue:x�3�.�) maxValue:null
>  2018-09-04 14:49:49,008 ERROR [Scheduler 158588 Job 
> d05156bb-2635-cb2f-ac19-b38f6147fcb2-699] execution.AbstractExecutable:165 : 
> error running Executable: 
> UpdateCubeInfoAfterBuildStep\{id=d05156bb-2635-cb2f-ac19-b38f6147fcb2-09, 
> name=Update Cube Info, state=RUNNING}
>  2018-09-04 14:49:49,011 INFO [Scheduler 158588 Job 
> d05156bb-2635-cb2f-ac19-b38f6147fcb2-699] execution.ExecutableManager:434 : 
> job id:d05156bb-2635-cb2f-ac19-b38f6147fcb2-09 from RUNNING to ERROR
>  2018-09-04 14:49:49,012 ERROR [Scheduler 158588 Job 
> d05156bb-2635-cb2f-ac19-b38f6147fcb2-699] execution.AbstractExecutable:165 : 
> error running Executable: CubingJob\{id=d05156bb-2635-cb2f-ac19-b38f6147fcb2, 
> name=BUILD CUBE - cube_global_dictionary_spark - FULL_BUILD - GMT+08:00 
> 2018-09-04 11:51:58, state=RUNNING}
>  2018-09-04 14:49:49,014 DEBUG [pool-6-thread-1] cachesync.Broadcaster:113 : 
> Servers in the cluster: [localhost:7070]
>  2018-09-04 14:49:49,014 DEBUG [pool-6-thread-1] cachesync.Broadcaster:123 : 
> Announcing new broadcast to all: BroadcastEvent\{entity=execute_output, 
> event=update, cacheKey=d05156bb-2635-cb2f-ac19-b38f6147fcb2}
>  2018-09-04 14:49:49,015 INFO [Scheduler 158588 Job 
> d05156bb-2635-cb2f-ac19-b38f6147fcb2-699] execution.ExecutableManager:434 : 
> job id:d05156bb-2635-cb2f-ac19-b38f6147fcb2 from RUNNING to ERROR
>  2018-09-04 14:49:49,015 DEBUG [pool-6-thread-1] cachesync.Broadcaster:113 : 
> Servers in the cluster: [localhost:7070]
>  2018-09-04 14:49:49,015 DEBUG [pool-6-thread-1] cachesync.Broadcaster:123 : 
> Announcing new broadcast to all: BroadcastEvent\{entity=execute_output, 
> event=update, cacheKey=d05156bb-2635-cb2f-ac19-b38f6147fcb2}
>  2018-09-04 14:49:49,015 DEBUG [Scheduler 158588 Job 
> d05156bb-2635-cb2f-ac19-b38f6147fcb2-699] execution.AbstractExecutable:316 : 
> no need to send email, user list is empty
>  2018-09-04 14:49:49,017 ERROR [pool-10-thread-10] 
> threadpool.DefaultScheduler:115 : ExecuteException 
> job:d05156bb-2635-cb2f-ac19-b38f6147fcb2
> org.apache.kylin.job.exception.ExecuteException: 
> org.apache.kylin.job.exception.ExecuteException:java.lang.IllegalStateException
>  at 
> org.apache.kylin.cube.DimensionRangeInfo.(DimensionRangeInfo.java:86)
>  at 
> org.apache.kylin.engine.mr.steps.UpdateCubeInfoAfterBuildStep.updateSegment(UpdateCubeInfoAfterBuildStep.java:166)
>  at 
> org.apache.kylin.engine.mr.steps.UpdateCubeInfoAfterBuildStep.doWork(UpdateCubeInfoAfterBuildStep.java:85)
>  at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163)
>  at 
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:69)
>  at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163)
>  at 
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:113)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  at java.lang.Thread.run(Thread.java:748)
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-3527) Hybrid couldn't save when there is only 1 cube

2018-09-10 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609957#comment-16609957
 ] 

ASF subversion and git services commented on KYLIN-3527:


Commit 5faddf6eb5ba7e8bb682e1ee6fded4a7e89a2d92 in kylin's branch 
refs/heads/master-hadoop3.1 from chao long
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=5faddf6 ]

KYLIN-3527 Hybrid allows 0,1 cube


> Hybrid couldn't save when there is only 1 cube
> --
>
> Key: KYLIN-3527
> URL: https://issues.apache.org/jira/browse/KYLIN-3527
> Project: Kylin
>  Issue Type: Bug
>  Components: Web 
>Affects Versions: v2.5.0
>Reporter: Shaofeng SHI
>Assignee: Roger
>Priority: Minor
> Fix For: v2.5.0
>
> Attachments: hybrid-not-saved.png, move-in-out-icons.png
>
>
> I created a hybrid with two cubes; Later I want to remove one cube from it, 
> but I found when there is only 1 cube, the "Submit" button was disabled.
>  
> The hybrid should allows 0, 1 or more cubes.
>  
> Besides, I found the move in and out icons were not properly displayed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-3518) Coprocessor reports NPE when execute a query on HBase 2.0

2018-09-10 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609961#comment-16609961
 ] 

ASF subversion and git services commented on KYLIN-3518:


Commit 00e8bd7be00296533414e2a869b18ee855fa89da in kylin's branch 
refs/heads/master-hadoop3.1 from [~caolijun1166]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=00e8bd7 ]

KYLIN-3518 Fix Coprocessor NPE problem on hbase 2


> Coprocessor reports NPE when execute a query on HBase 2.0
> -
>
> Key: KYLIN-3518
> URL: https://issues.apache.org/jira/browse/KYLIN-3518
> Project: Kylin
>  Issue Type: Bug
>  Components: Storage - HBase
>Reporter: Shaofeng SHI
>Assignee: Lijun Cao
>Priority: Major
> Fix For: v2.5.0
>
>
> On HDP 3.0, build a cube and then run a simple count query, NPE occurred:
>  
> {code:java}
> 2018-08-28 01:30:16,969 DEBUG [Query f7bf8004-b516-e372-18df-0d507075d471-71] 
> v2.CubeHBaseRPC:315 : hbase.rpc.timeout = 9 ms, use 81000 ms as timeout 
> for coprocessor
> 2018-08-28 01:30:16,983 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71] 
> v2.CubeHBaseEndpointRPC:141 : Serialized scanRequestBytes 522 bytes, 
> rawScanBytesString 44 bytes
> 2018-08-28 01:30:16,984 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71] 
> v2.CubeHBaseEndpointRPC:143 : The scan 67b41fc6 for segment 
> kylin_sales_cube_clone[2012010100_2013010100] is as below with 1 
> separate raw scans, shard part of start/end key is set to 0
> 2018-08-28 01:30:16,991 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71] 
> v2.CubeHBaseRPC:288 : Visiting hbase table KYLIN_5Q088VO5I0: cuboid require 
> post aggregation, from 0 to 16384 Start: 
> \x00\x00\x00\x00\x00\x00\x00\x00\x40\x00\x00\x00\x00 
> (\x00\x00\x00\x00\x00\x00\x00\x00@\x00\x00\x00\x00) Stop: 
> \x00\x00\x00\x00\x00\x00\x00\x00\x40\x00\xFF\xFF\xFF\x00 
> (\x00\x00\x00\x00\x00\x00\x00\x00@\x00\xFF\xFF\xFF\x00), No Fuzzy Key
> 2018-08-28 01:30:16,991 DEBUG [Query f7bf8004-b516-e372-18df-0d507075d471-71] 
> v2.CubeHBaseEndpointRPC:148 : Submitting rpc to 1 shards starting from shard 
> 0, scan range count 1
> 2018-08-28 01:30:17,010 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71] 
> common.KylinConfig:332 : Loading kylin-defaults.properties from 
> file:/root/shaofengshi/apache-kylin-2.5.0-SNAPSHOT-bin/tomcat/webapps/kylin/WEB-INF/lib/kylin-core-common-2.5.0-SNAPSHOT.jar!/kylin-defaults.properties
> 2018-08-28 01:30:17,033 DEBUG [Query f7bf8004-b516-e372-18df-0d507075d471-71] 
> common.KylinConfig:291 : KYLIN_CONF property was not set, will seek 
> KYLIN_HOME env variable
> 2018-08-28 01:30:17,051 INFO [pool-14-thread-1] hbase.HBaseConnection:110 : 
> Creating coprocessor thread pool with max of 2048, core of 2048
> 2018-08-28 01:30:17,094 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71] 
> gtrecord.SequentialCubeTupleIterator:73 : Using SortedIteratorMergerWithLimit 
> to merge segment results
> 2018-08-28 01:30:17,097 DEBUG [Query f7bf8004-b516-e372-18df-0d507075d471-71] 
> enumerator.OLAPEnumerator:117 : return TupleIterator...
> 2018-08-28 01:30:21,607 INFO [kylin-coproc--pool9-t1] 
> client.RpcRetryingCallerImpl:134 : Call exception, tries=6, retries=6, 
> started=4410 ms ago, cancelled=false, msg=java.io.IOException
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:468)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> Caused by: java.lang.NullPointerException
> at 
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService.visitCube(CubeVisitService.java:253)
> at 
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService.callMethod(CubeVisitProtos.java:)
> at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8032)
> at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2426)
> at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2408)
> at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42010)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
> ... 3 more
> , details=row '' on table 'KYLIN_5Q088VO5I0' at 
> region=KYLIN_5Q088VO5I0,,1535417272444.27b82cb4702db4557a98b9a7e60b7692., 
> hostname=ignite03.com,16020,1534313612401, seqNum=2
> 2018-08-28 01:30:25,633 INFO [kylin-coproc--pool9-t1] 
> client.RpcRetryingCallerImpl:134 : Call exception, tries=7, retries=7, 
> started=8436 ms ago, cancelled=false, msg=java.io.IOException
> at 

[jira] [Commented] (KYLIN-2565) Upgrade Kylin to Hadoop3.0

2018-09-10 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609960#comment-16609960
 ] 

ASF subversion and git services commented on KYLIN-2565:


Commit a09485037906d8105cc38d28b898c85d3c4b7920 in kylin's branch 
refs/heads/master-hadoop3.1 from shaofengshi
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=a094850 ]

KYLIN-2565 upgrade to hadoop 3.0 hbase 2.0, pass UT


> Upgrade Kylin to Hadoop3.0
> --
>
> Key: KYLIN-2565
> URL: https://issues.apache.org/jira/browse/KYLIN-2565
> Project: Kylin
>  Issue Type: New Feature
>Reporter: Wang Cheng
>Assignee: Shaofeng SHI
>Priority: Major
>
> Hadoop3.0-alpha is released, Kylin should also keep compatible with it. Below 
> is the Hadoop3.0 components requirements:
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+release 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-3539) Hybrid segment overlap not cover some case

2018-09-10 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609958#comment-16609958
 ] 

ASF subversion and git services commented on KYLIN-3539:


Commit bffaa8a26a2d93b3aa62326484bc31a8717a7c0b in kylin's branch 
refs/heads/master-hadoop3.1 from chao long
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=bffaa8a ]

KYLIN-3539 Hybrid segment overlap check


> Hybrid segment overlap not cover some case
> --
>
> Key: KYLIN-3539
> URL: https://issues.apache.org/jira/browse/KYLIN-3539
> Project: Kylin
>  Issue Type: Improvement
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.5.0
>
>
> now Hybrid uses every cube's latest segment to check whether overlap exists, 
> but it can't cover all cases.
> example:
> hybrid:
>  cube1:
>  segment1: range(2012-01-01, 2012-01-02)
> cube2:
>  segment1: range(2012-01-01, 2012-01-02)
>  segment2: range(2012-01-03, 2012-01-04)
> this hybrid has overlap segment range(2012-01-01, 2012-01-02), but the 
> program can not find out.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-2565) Upgrade Kylin to Hadoop3.0

2018-09-10 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609963#comment-16609963
 ] 

ASF subversion and git services commented on KYLIN-2565:


Commit 766a472bc294e06ac2f13c5437bda349ff3aa480 in kylin's branch 
refs/heads/master-hadoop3.1 from shaofengshi
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=766a472 ]

KYLIN-2565 Resolve conflict


> Upgrade Kylin to Hadoop3.0
> --
>
> Key: KYLIN-2565
> URL: https://issues.apache.org/jira/browse/KYLIN-2565
> Project: Kylin
>  Issue Type: New Feature
>Reporter: Wang Cheng
>Assignee: Shaofeng SHI
>Priority: Major
>
> Hadoop3.0-alpha is released, Kylin should also keep compatible with it. Below 
> is the Hadoop3.0 components requirements:
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+release 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-3529) Prompt not friendly

2018-09-10 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609955#comment-16609955
 ] 

ASF subversion and git services commented on KYLIN-3529:


Commit 37a4966db41c775dfc48cdd4a4a244f03a18a96c in kylin's branch 
refs/heads/master-hadoop3.1 from chao long
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=37a4966 ]

KYLIN-3529 Show more error message


> Prompt not friendly
> ---
>
> Key: KYLIN-3529
> URL: https://issues.apache.org/jira/browse/KYLIN-3529
> Project: Kylin
>  Issue Type: Bug
>  Components: Web 
>Reporter: Na Zhai
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.5.0
>
>
>  *Scene 1:*When I create a hybrid with duplicate name, then I click submit, 
> the error message "error execute org.apache.kylin.rest.job.HybridCubeCLI" 
> will pop up.
>  
> *Scene 2:*When I create a hybrid with duplicate date/time cube, then I click 
> submit, the error message "error execute 
> org.apache.kylin.rest.job.HybridCubeCLI" will pop up.
>  
> I think the error message should be more friendly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Resolved] (KYLIN-3548) Typo on Community page

2018-09-10 Thread Shaofeng SHI (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-3548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3548.
-
Resolution: Fixed

> Typo on Community page
> --
>
> Key: KYLIN-3548
> URL: https://issues.apache.org/jira/browse/KYLIN-3548
> Project: Kylin
>  Issue Type: Bug
>Reporter: Sebb
>Assignee: Shaofeng SHI
>Priority: Major
>
> "artical" should probably be "article" on page
> https://kylin.apache.org/community/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-3548) Typo on Community page

2018-09-10 Thread Shaofeng SHI (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609936#comment-16609936
 ] 

Shaofeng SHI commented on KYLIN-3548:
-

 The typo has been fixed and published. Refresh the web to see the change. 
Thanks for the reporting!

> Typo on Community page
> --
>
> Key: KYLIN-3548
> URL: https://issues.apache.org/jira/browse/KYLIN-3548
> Project: Kylin
>  Issue Type: Bug
>Reporter: Sebb
>Assignee: Shaofeng SHI
>Priority: Major
>
> "artical" should probably be "article" on page
> https://kylin.apache.org/community/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (KYLIN-3394) Prepare for Kafka 2.0

2018-09-10 Thread Ted Yu (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-3394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated KYLIN-3394:
--
Description: 
Kafka 2.0 is around the corner.

I got the following when compiling against Kafka 2.0.0:

{code}
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:266: 
error: type mismatch;
[ERROR]  found   : Option[org.apache.kafka.common.Node]
[ERROR]  required: org.apache.kafka.common.Node
[ERROR] getBrokerInfoFromCache(zkUtils, cachedBrokerInfo, 
List(l)).head.getNode(listenerName)
[ERROR] 
   ^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:273: 
error: type mismatch;
[ERROR]  found   : Seq[Option[org.apache.kafka.common.Node]]
[ERROR]  required: Seq[org.apache.kafka.common.Node]
[ERROR] replicaInfo = getBrokerInfoFromCache(zkUtils, 
cachedBrokerInfo, replicas).map(_.getNode(listenerName))
[ERROR] 
 ^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:274: 
error: type mismatch;
[ERROR]  found   : Seq[Option[org.apache.kafka.common.Node]]
[ERROR]  required: Seq[org.apache.kafka.common.Node]
[ERROR] isrInfo = getBrokerInfoFromCache(zkUtils, cachedBrokerInfo, 
inSyncReplicas).map(_.getNode(listenerName))
[ERROR] 
   ^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:476: 
error: value getConsumersInGroup is not a member of kafka.utils.ZkUtils
[ERROR] zkUtils.getConsumersInGroup(group).nonEmpty
[ERROR] ^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:489: 
error: not found: type ZKGroupDirs
[ERROR]   val dir = new ZKGroupDirs(group)
[ERROR] ^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:507: 
error: value getTopicsByConsumerGroup is not a member of kafka.utils.ZkUtils
[ERROR] val topics = zkUtils.getTopicsByConsumerGroup(group)
[ERROR]  ^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:512: 
error: not found: type ZKGroupTopicDirs
[ERROR]   val dir = new ZKGroupTopicDirs(group, topic)
[ERROR] ^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:528: 
error: value getAllConsumerGroupsForTopic is not a member of kafka.utils.ZkUtils
[ERROR] val groups = zkUtils.getAllConsumerGroupsForTopic(topic)
[ERROR]  ^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:720: 
error: value encode is not a member of object kafka.utils.Json
[ERROR] val content = 
Json.encode(getConfigChangeZnodeData(sanitizedEntityPath))
[ERROR]^
[ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:733: 
error: value encode is not a member of object kafka.utils.Json
[ERROR] zkUtils.updatePersistentPath(entityPath, Json.encode(map))
{code}

  was:
Kafka 2.0 is around the corner.

I got the following when compiling against Kafka 2.0.0-SNAPSHOT :

{code}
[ERROR] 
/a/kylin/kylin-it/src/test/java/org/apache/kylin/provision/MockKafka.java:[79,74]
 error: cannot find symbol
[ERROR]   symbol:   method serverConfig()
[ERROR]   location: variable kafkaServer of type KafkaServerStartable
[ERROR] 
/a/kylin/kylin-it/src/test/java/org/apache/kylin/provision/MockKafka.java:[79,113]
 error: cannot find symbol
[ERROR]   symbol:   method serverConfig()
[ERROR]   location: variable kafkaServer of type KafkaServerStartable
[ERROR] 
/a/kylin/kylin-it/src/test/java/org/apache/kylin/provision/MockKafka.java:[79,148]
 error: cannot find symbol
[ERROR]   symbol:   method serverConfig()
[ERROR]   location: variable kafkaServer of type KafkaServerStartable
[ERROR] 
/a/kylin/kylin-it/src/test/java/org/apache/kylin/provision/MockKafka.java:[98,65]
 error: cannot find symbol
{code}


> Prepare for Kafka 2.0
> -
>
> Key: KYLIN-3394
> URL: https://issues.apache.org/jira/browse/KYLIN-3394
> Project: Kylin
>  Issue Type: Task
>Reporter: Ted Yu
>Priority: Major
>
> Kafka 2.0 is around the corner.
> I got the following when compiling against Kafka 2.0.0:
> {code}
> [ERROR] /a/kylin/kylin-it/src/test/scala/kafka/admin/AdminUtils.scala:266: 
> error: type mismatch;
> [ERROR]  found   : Option[org.apache.kafka.common.Node]
> [ERROR]  required: org.apache.kafka.common.Node
> [ERROR] getBrokerInfoFromCache(zkUtils, cachedBrokerInfo, 
> List(l)).head.getNode(listenerName)
> [ERROR]   
>  ^
> [ERROR] 

[jira] [Assigned] (KYLIN-3547) DimensionRangeInfo: Unsupported data type boolean

2018-09-10 Thread Shaofeng SHI (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-3547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI reassigned KYLIN-3547:
---

 Assignee: Yichen Zhou
Affects Version/s: v2.5.0
Fix Version/s: v2.5.0
  Component/s: Metadata

> DimensionRangeInfo: Unsupported data type boolean
> -
>
> Key: KYLIN-3547
> URL: https://issues.apache.org/jira/browse/KYLIN-3547
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine, Metadata
>Affects Versions: v2.5.0
>Reporter: Yichen Zhou
>Assignee: Yichen Zhou
>Priority: Blocker
> Fix For: v2.5.0
>
>
> In Extract Fact Table Distinct Columns Step, Kylin can not get dimension 
> range information from data of boolean type because DataType.getOrder() does 
> not support boolean.
> {quote}java.lang.RuntimeException: error execute 
> org.apache.kylin.engine.spark.SparkFactDistinct at 
> org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
>  at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44) at 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498) at 
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:636)
>  Caused by: org.apache.spark.SparkException: Job aborted due to stage 
> failure: Task 5 in stage 1.0 failed 4 times, most recent failure: Lost task 
> 5.3 in stage 1.0 (TID 17, slave1.kcluster, executor 5): 
> java.lang.IllegalArgumentException: Unsupported data type boolean at 
> org.apache.kylin.metadata.datatype.DataTypeOrder.getInstance(DataTypeOrder.java:53)
>  at org.apache.kylin.metadata.datatype.DataType.getOrder(DataType.java:230) 
> at org.apache.kylin.metadata.datatype.DataType.compare(DataType.java:236) at 
> org.apache.kylin.engine.spark.SparkFactDistinct$MultiOutputFunction.call(SparkFactDistinct.java:739)
>  at 
> org.apache.kylin.engine.spark.SparkFactDistinct$MultiOutputFunction.call(SparkFactDistinct.java:612)
>  at 
> org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:186)
>  at 
> org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:186)
>  at 
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:797)
>  at 
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:797)
>  at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) 
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at 
> org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at 
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at 
> org.apache.spark.scheduler.Task.run(Task.scala:99) at 
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325) at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  at java.lang.Thread.run(Thread.java:748)
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Resolved] (KYLIN-3549) Download page must link to KEYS at https://www.apache.org/dist/kylin/KEYS

2018-09-10 Thread Zhong Yanghong (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-3549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhong Yanghong resolved KYLIN-3549.
---
Resolution: Fixed

> Download page must link to KEYS at https://www.apache.org/dist/kylin/KEYS
> -
>
> Key: KYLIN-3549
> URL: https://issues.apache.org/jira/browse/KYLIN-3549
> Project: Kylin
>  Issue Type: Bug
>Reporter: Sebb
>Assignee: Zhong Yanghong
>Priority: Major
>
> As the subject says.
> The KEYS file must exist at
> https://www.apache.org/dist/kylin/KEYS
> Please maintain the file at
> https://dist.apache.org/repos/dist/release/kylin/KEYS



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (KYLIN-3549) Download page must link to KEYS at https://www.apache.org/dist/kylin/KEYS

2018-09-10 Thread Zhong Yanghong (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-3549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhong Yanghong reassigned KYLIN-3549:
-

Assignee: Zhong Yanghong

> Download page must link to KEYS at https://www.apache.org/dist/kylin/KEYS
> -
>
> Key: KYLIN-3549
> URL: https://issues.apache.org/jira/browse/KYLIN-3549
> Project: Kylin
>  Issue Type: Bug
>Reporter: Sebb
>Assignee: Zhong Yanghong
>Priority: Major
>
> As the subject says.
> The KEYS file must exist at
> https://www.apache.org/dist/kylin/KEYS
> Please maintain the file at
> https://dist.apache.org/repos/dist/release/kylin/KEYS



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-3548) Typo on Community page

2018-09-10 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608990#comment-16608990
 ] 

ASF subversion and git services commented on KYLIN-3548:


Commit 94927aa9a74486aa3a5b83ca417bc12bcd6ff646 in kylin's branch 
refs/heads/document from shaofengshi
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=94927aa ]

KYLIN-3548 Typo on Community page


> Typo on Community page
> --
>
> Key: KYLIN-3548
> URL: https://issues.apache.org/jira/browse/KYLIN-3548
> Project: Kylin
>  Issue Type: Bug
>Reporter: Sebb
>Assignee: Shaofeng SHI
>Priority: Major
>
> "artical" should probably be "article" on page
> https://kylin.apache.org/community/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (KYLIN-3549) Download page must link to KEYS at https://www.apache.org/dist/kylin/KEYS

2018-09-10 Thread Sebb (JIRA)
Sebb created KYLIN-3549:
---

 Summary: Download page must link to KEYS at 
https://www.apache.org/dist/kylin/KEYS
 Key: KYLIN-3549
 URL: https://issues.apache.org/jira/browse/KYLIN-3549
 Project: Kylin
  Issue Type: Bug
Reporter: Sebb


As the subject says.

The KEYS file must exist at

https://www.apache.org/dist/kylin/KEYS

Please maintain the file at

https://dist.apache.org/repos/dist/release/kylin/KEYS



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-3517) Couldn't update coprocessor on HBase 2.0

2018-09-10 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608908#comment-16608908
 ] 

ASF subversion and git services commented on KYLIN-3517:


Commit 96026886774aff16f03e6d5424e9c0ccdfca1ca0 in kylin's branch 
refs/heads/master-hadoop3.1-2.5.0 from [~caolijun1166]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=9602688 ]

KYLIN-3517 Upadate coprocessor on HBase2.0 is avaliable.

Signed-off-by: shaofengshi 


> Couldn't update coprocessor on HBase 2.0
> 
>
> Key: KYLIN-3517
> URL: https://issues.apache.org/jira/browse/KYLIN-3517
> Project: Kylin
>  Issue Type: Bug
>  Components: Storage - HBase
>Reporter: Shaofeng SHI
>Assignee: Lijun Cao
>Priority: Major
>
> On HDP 3.0, run update coprocessor, got this error:
>  
> {code:java}
> 2018-08-28 00:24:26,683 ERROR [pool-7-thread-1] util.DeployCoprocessorCLI:383 
> : Error processing KYLIN_O9JRT8XOQ9
> java.lang.UnsupportedOperationException: HTableDescriptor is read-only
> at 
> org.apache.hadoop.hbase.client.ImmutableHTableDescriptor.getDelegateeForModification(ImmutableHTableDescriptor.java:59)
> at 
> org.apache.hadoop.hbase.HTableDescriptor.removeCoprocessor(HTableDescriptor.java:768)
> at 
> org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI.resetCoprocessor(DeployCoprocessorCLI.java:300)
> at 
> org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI$ResetCoprocessorWorker.run(DeployCoprocessorCLI.java:375)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-3529) Prompt not friendly

2018-09-10 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608890#comment-16608890
 ] 

ASF subversion and git services commented on KYLIN-3529:


Commit 37a4966db41c775dfc48cdd4a4a244f03a18a96c in kylin's branch 
refs/heads/master from chao long
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=37a4966 ]

KYLIN-3529 Show more error message


> Prompt not friendly
> ---
>
> Key: KYLIN-3529
> URL: https://issues.apache.org/jira/browse/KYLIN-3529
> Project: Kylin
>  Issue Type: Bug
>  Components: Web 
>Reporter: Na Zhai
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.5.0
>
>
>  *Scene 1:*When I create a hybrid with duplicate name, then I click submit, 
> the error message "error execute org.apache.kylin.rest.job.HybridCubeCLI" 
> will pop up.
>  
> *Scene 2:*When I create a hybrid with duplicate date/time cube, then I click 
> submit, the error message "error execute 
> org.apache.kylin.rest.job.HybridCubeCLI" will pop up.
>  
> I think the error message should be more friendly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-3517) Couldn't update coprocessor on HBase 2.0

2018-09-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608881#comment-16608881
 ] 

ASF GitHub Bot commented on KYLIN-3517:
---

shaofengshi commented on issue #224: KYLIN-3517 Update Coprocessor is avaliable 
for HDP3.0
URL: https://github.com/apache/kylin/pull/224#issuecomment-419836254
 
 
   This PR has duplicated commit, so close it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Couldn't update coprocessor on HBase 2.0
> 
>
> Key: KYLIN-3517
> URL: https://issues.apache.org/jira/browse/KYLIN-3517
> Project: Kylin
>  Issue Type: Bug
>  Components: Storage - HBase
>Reporter: Shaofeng SHI
>Assignee: Lijun Cao
>Priority: Major
>
> On HDP 3.0, run update coprocessor, got this error:
>  
> {code:java}
> 2018-08-28 00:24:26,683 ERROR [pool-7-thread-1] util.DeployCoprocessorCLI:383 
> : Error processing KYLIN_O9JRT8XOQ9
> java.lang.UnsupportedOperationException: HTableDescriptor is read-only
> at 
> org.apache.hadoop.hbase.client.ImmutableHTableDescriptor.getDelegateeForModification(ImmutableHTableDescriptor.java:59)
> at 
> org.apache.hadoop.hbase.HTableDescriptor.removeCoprocessor(HTableDescriptor.java:768)
> at 
> org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI.resetCoprocessor(DeployCoprocessorCLI.java:300)
> at 
> org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI$ResetCoprocessorWorker.run(DeployCoprocessorCLI.java:375)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-3517) Couldn't update coprocessor on HBase 2.0

2018-09-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608882#comment-16608882
 ] 

ASF GitHub Bot commented on KYLIN-3517:
---

shaofengshi closed pull request #224: KYLIN-3517 Update Coprocessor is 
avaliable for HDP3.0
URL: https://github.com/apache/kylin/pull/224
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/build/deploy/server.xml b/build/deploy/server.xml
index 96f329b8fe..920be253e4 100644
--- a/build/deploy/server.xml
+++ b/build/deploy/server.xml
@@ -26,7 +26,7 @@
 
 
 
-
+
 
 
 
diff --git a/build/script/elimate-jar-conflict.sh 
b/build/script/elimate-jar-conflict.sh
new file mode 100644
index 00..d02a87416e
--- /dev/null
+++ b/build/script/elimate-jar-conflict.sh
@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+
+current_dir=`pwd`
+cd ${current_dir}/build/tomcat/webapps
+unzip kylin.war && rm -f kylin.war
+cd WEB-INF/lib
+#remove slf4j-api-1.7.21.jar to solve slf4j conflict
+rm -f slf4j-api-1.7.21.jar
+mkdir modify_avatica_jar && mv avatica-1.10.0.jar modify_avatica_jar
+cd modify_avatica_jar
+#remove org/slf4j in avatica-1.10.0.jar and repackage it to solve slf4j 
conflict
+unzip avatica-1.10.0.jar && rm -f avatica-1.10.0.jar
+rm -rf org/slf4j && jar -cf avatica-1.10.0.jar ./
+rm -rf `ls | egrep -v avatica-1.10.0.jar`
+mv avatica-1.10.0.jar ..
+cd .. && rm -rf modify_avatica_jar
+cd ${current_dir}/build/tomcat/webapps
+#repackage kylin.war
+jar -cf kylin.war ./ && rm -rf `ls | egrep -v kylin.war`
+cd ${current_dir}
\ No newline at end of file
diff --git a/build/script/prepare.sh b/build/script/prepare.sh
index deaf58de91..be9dd9d0a8 100755
--- a/build/script/prepare.sh
+++ b/build/script/prepare.sh
@@ -31,6 +31,9 @@ export version
 sh build/script/prepare-libs.sh || { exit 1; }
 
 cp server/target/kylin-server-${version}.war build/tomcat/webapps/kylin.war
+
+sh build/script/elimate-jar-conflict.sh
+
 chmod 644 build/tomcat/webapps/kylin.war
 
 echo "add js css to war"
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v2/coprocessor/endpoint/CubeVisitService.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v2/coprocessor/endpoint/CubeVisitService.java
index ded35009da..2beddc7439 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v2/coprocessor/endpoint/CubeVisitService.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/cube/v2/coprocessor/endpoint/CubeVisitService.java
@@ -24,6 +24,7 @@
 import java.net.InetAddress;
 import java.nio.BufferOverflowException;
 import java.nio.ByteBuffer;
+import java.util.Collections;
 import java.util.Iterator;
 import java.util.List;
 
@@ -31,12 +32,11 @@
 import org.apache.commons.lang.ArrayUtils;
 import org.apache.hadoop.hbase.Cell;
 import org.apache.hadoop.hbase.CellUtil;
-import org.apache.hadoop.hbase.Coprocessor;
 import org.apache.hadoop.hbase.CoprocessorEnvironment;
 import org.apache.hadoop.hbase.DoNotRetryIOException;
 import org.apache.hadoop.hbase.client.Scan;
 import org.apache.hadoop.hbase.coprocessor.CoprocessorException;
-import org.apache.hadoop.hbase.coprocessor.CoprocessorService;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessor;
 import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
 import org.apache.hadoop.hbase.regionserver.HRegion;
 import org.apache.hadoop.hbase.regionserver.RegionScanner;
@@ -78,7 +78,7 @@
 
 @SuppressWarnings("unused")
 //used in hbase endpoint
-public class CubeVisitService extends CubeVisitProtos.CubeVisitService 
implements Coprocessor, CoprocessorService {
+public class CubeVisitService extends CubeVisitProtos.CubeVisitService 
implements RegionCoprocessor {
 
 private static final Logger logger = 
LoggerFactory.getLogger(CubeVisitService.class);
 //TODO limit memory footprint
@@ -448,7 +448,7 @@ public void stop(CoprocessorEnvironment env) throws 
IOException {
 }
 
 @Override
-public Service getService() {
-return this;
+public Iterable getServices() {
+return Collections.singleton(this);
 }
 }
diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/lookup/LookupTableToHFileJob.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/lookup/LookupTableToHFileJob.java
index 054e1466b6..27894010f7 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/lookup/LookupTableToHFileJob.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/lookup/LookupTableToHFileJob.java
@@ -26,12 +26,12 @@
 import org.apache.commons.lang.StringUtils;
 import org.apache.hadoop.fs.Path;
 import 

[jira] [Comment Edited] (KYLIN-3543) Unclosed Job instance in CreateHTableJob#exportHBaseConfiguration

2018-09-10 Thread Yichen Zhou (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606656#comment-16606656
 ] 

Yichen Zhou edited comment on KYLIN-3543 at 9/10/18 7:45 AM:
-

Thank you Ted. The job configuration is saved in HDFS for later use. Please 
refer to this: https://issues.apache.org/jira/browse/KYLIN-3446

Since we could not connect HBase in spark, this job should remain open until 
spark use it in "convert to HFlie" step. However, when building cube with MR, 
this job does become useless. But I think it's ok for the little cost.


was (Author: yichen.zhou):
Thank you Ted. The job configuration is saved in HDFS for later use. Please 
refer to this: https://issues.apache.org/jira/browse/KYLIN-3446

Since we could not connect HBase in spark, this job should remain open until 
spark use it in "convert to HFlie" step. However, when building cube with MR, 
this job does become unless. But I think it's ok for the little cost.

> Unclosed Job instance in CreateHTableJob#exportHBaseConfiguration
> -
>
> Key: KYLIN-3543
> URL: https://issues.apache.org/jira/browse/KYLIN-3543
> Project: Kylin
>  Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
>
> {code}
> out = fs.create(new Path(hbaseConfPath));
> job.getConfiguration().writeXml(out);
> {code}
> The job instance should be closed upon return from the method.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (KYLIN-3547) DimensionRangeInfo: Unsupported data type boolean

2018-09-10 Thread Yichen Zhou (JIRA)
Yichen Zhou created KYLIN-3547:
--

 Summary: DimensionRangeInfo: Unsupported data type boolean
 Key: KYLIN-3547
 URL: https://issues.apache.org/jira/browse/KYLIN-3547
 Project: Kylin
  Issue Type: Bug
  Components: Job Engine
Reporter: Yichen Zhou


In Extract Fact Table Distinct Columns Step, Kylin can not get dimension range 
information from data of boolean type because DataType.getOrder() does not 
support boolean.
{quote}java.lang.RuntimeException: error execute 
org.apache.kylin.engine.spark.SparkFactDistinct at 
org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
 at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44) at 
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498) at 
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:636)
 Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: 
Task 5 in stage 1.0 failed 4 times, most recent failure: Lost task 5.3 in stage 
1.0 (TID 17, slave1.kcluster, executor 5): java.lang.IllegalArgumentException: 
Unsupported data type boolean at 
org.apache.kylin.metadata.datatype.DataTypeOrder.getInstance(DataTypeOrder.java:53)
 at org.apache.kylin.metadata.datatype.DataType.getOrder(DataType.java:230) at 
org.apache.kylin.metadata.datatype.DataType.compare(DataType.java:236) at 
org.apache.kylin.engine.spark.SparkFactDistinct$MultiOutputFunction.call(SparkFactDistinct.java:739)
 at 
org.apache.kylin.engine.spark.SparkFactDistinct$MultiOutputFunction.call(SparkFactDistinct.java:612)
 at 
org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:186)
 at 
org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:186)
 at 
org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:797)
 at 
org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:797)
 at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at 
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at 
org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at 
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at 
org.apache.spark.scheduler.Task.run(Task.scala:99) at 
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325) at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
at java.lang.Thread.run(Thread.java:748)
{quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-3517) Couldn't update coprocessor on HBase 2.0

2018-09-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608778#comment-16608778
 ] 

ASF GitHub Bot commented on KYLIN-3517:
---

asfgit commented on issue #225: KYLIN-3517 Upadate coprocessor on HBase2.0 is 
avaliable.
URL: https://github.com/apache/kylin/pull/225#issuecomment-419802107
 
 
   Can one of the admins verify this patch?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Couldn't update coprocessor on HBase 2.0
> 
>
> Key: KYLIN-3517
> URL: https://issues.apache.org/jira/browse/KYLIN-3517
> Project: Kylin
>  Issue Type: Bug
>  Components: Storage - HBase
>Reporter: Shaofeng SHI
>Assignee: Lijun Cao
>Priority: Major
>
> On HDP 3.0, run update coprocessor, got this error:
>  
> {code:java}
> 2018-08-28 00:24:26,683 ERROR [pool-7-thread-1] util.DeployCoprocessorCLI:383 
> : Error processing KYLIN_O9JRT8XOQ9
> java.lang.UnsupportedOperationException: HTableDescriptor is read-only
> at 
> org.apache.hadoop.hbase.client.ImmutableHTableDescriptor.getDelegateeForModification(ImmutableHTableDescriptor.java:59)
> at 
> org.apache.hadoop.hbase.HTableDescriptor.removeCoprocessor(HTableDescriptor.java:768)
> at 
> org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI.resetCoprocessor(DeployCoprocessorCLI.java:300)
> at 
> org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI$ResetCoprocessorWorker.run(DeployCoprocessorCLI.java:375)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (KYLIN-3517) Couldn't update coprocessor on HBase 2.0

2018-09-10 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608777#comment-16608777
 ] 

ASF GitHub Bot commented on KYLIN-3517:
---

caolijun1166 opened a new pull request #225: KYLIN-3517 Upadate coprocessor on 
HBase2.0 is avaliable.
URL: https://github.com/apache/kylin/pull/225
 
 
   KYLIN-3517 Upadate coprocessor on HBase2.0 is avaliable.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Couldn't update coprocessor on HBase 2.0
> 
>
> Key: KYLIN-3517
> URL: https://issues.apache.org/jira/browse/KYLIN-3517
> Project: Kylin
>  Issue Type: Bug
>  Components: Storage - HBase
>Reporter: Shaofeng SHI
>Assignee: Lijun Cao
>Priority: Major
>
> On HDP 3.0, run update coprocessor, got this error:
>  
> {code:java}
> 2018-08-28 00:24:26,683 ERROR [pool-7-thread-1] util.DeployCoprocessorCLI:383 
> : Error processing KYLIN_O9JRT8XOQ9
> java.lang.UnsupportedOperationException: HTableDescriptor is read-only
> at 
> org.apache.hadoop.hbase.client.ImmutableHTableDescriptor.getDelegateeForModification(ImmutableHTableDescriptor.java:59)
> at 
> org.apache.hadoop.hbase.HTableDescriptor.removeCoprocessor(HTableDescriptor.java:768)
> at 
> org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI.resetCoprocessor(DeployCoprocessorCLI.java:300)
> at 
> org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI$ResetCoprocessorWorker.run(DeployCoprocessorCLI.java:375)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)