[jira] [Comment Edited] (KYLIN-2547) Fix the bug of multi-process concurrence in mergeCubeSegment
[ https://issues.apache.org/jira/browse/KYLIN-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969033#comment-15969033 ] kangkaisen edited comment on KYLIN-2547 at 4/14/17 12:49 PM: - This is the patch. This patch has passed the UT and IT was (Author: kangkaisen): This is the patch. > Fix the bug of multi-process concurrence in mergeCubeSegment > > > Key: KYLIN-2547 > URL: https://issues.apache.org/jira/browse/KYLIN-2547 > Project: Kylin > Issue Type: Bug > Components: Metadata >Affects Versions: v2.0.0 >Reporter: kangkaisen >Assignee: kangkaisen >Priority: Minor > Fix For: v2.0.0 > > Attachments: KYLIN-2547.patch > > > There is a minor bug in "Update Cube Info" step when build a cube in > distributed env, which will make this job failed, but the next auto merge > job won't be affected and will succeed. So the cube data still is consistent > and correct. > {code:java} >Caused by: java.lang.IllegalStateException: Segments overlap: > waimai_dolphin_topic_flow_activity_expose_food_d_cube[2017040500_2017041200] > and > waimai_dolphin_topic_flow_activity_expose_food_d_cube[2017040500_2017041200] > at org.apache.kylin.cube.CubeValidator.validate(CubeValidator.java:85) > at > org.apache.kylin.cube.CubeManager.updateCubeWithRetry(CubeManager.java:359) > at > org.apache.kylin.cube.CubeManager.updateCubeWithRetry(CubeManager.java:386) > at org.apache.kylin.cube.CubeManager.updateCube(CubeManager.java:302) > at > org.apache.kylin.cube.CubeManager.mergeSegments(CubeManager.java:533) > at > org.apache.kylin.rest.service.CubeService.mergeCubeSegment(CubeService.java:635) > at > org.apache.kylin.rest.service.CubeService.updateOnNewSegmentReady(CubeService.java:587) > at > org.apache.kylin.rest.service.CubeServiceFastClassBySpringCGLIB17a07c0e.invoke() > at > org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) > at > org.springframework.aop.framework.CglibAopProxyDynamicAdvisedInterceptor.intercept(CglibAopProxy.java:629) > at > org.apache.kylin.rest.service.CubeServiceEnhancerBySpringCGLIB$$c6fabb3f.updateOnNewSegmentReady() > at > org.apache.kylin.rest.service.CacheService.rebuildCubeCache(CacheService.java:237) > at > org.apache.kylin.rest.service.CacheService.access$000(CacheService.java:62) > at > org.apache.kylin.rest.service.CacheService$1.afterCubeUpdate(CacheService.java:86) > at org.apache.kylin.cube.CubeManager.updateCube(CubeManager.java:305) > at > org.apache.kylin.cube.CubeManager.promoteNewlyBuiltSegments(CubeManager.java:735) > at > org.apache.kylin.engine.mr.steps.UpdateCubeInfoAfterBuildStep.doWork(UpdateCubeInfoAfterBuildStep.java:62) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113) > ... 6 more > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KYLIN-2547) Fix the bug of multi-process concurrence in mergeCubeSegment
[ https://issues.apache.org/jira/browse/KYLIN-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kangkaisen updated KYLIN-2547: -- Attachment: KYLIN-2547.patch This is the patch. > Fix the bug of multi-process concurrence in mergeCubeSegment > > > Key: KYLIN-2547 > URL: https://issues.apache.org/jira/browse/KYLIN-2547 > Project: Kylin > Issue Type: Bug > Components: Metadata >Affects Versions: v2.0.0 >Reporter: kangkaisen >Assignee: kangkaisen >Priority: Minor > Fix For: v2.0.0 > > Attachments: KYLIN-2547.patch > > > There is a minor bug in "Update Cube Info" step when build a cube in > distributed env, which will make this job failed, but the next auto merge > job won't be affected and will succeed. So the cube data still is consistent > and correct. > {code:java} >Caused by: java.lang.IllegalStateException: Segments overlap: > waimai_dolphin_topic_flow_activity_expose_food_d_cube[2017040500_2017041200] > and > waimai_dolphin_topic_flow_activity_expose_food_d_cube[2017040500_2017041200] > at org.apache.kylin.cube.CubeValidator.validate(CubeValidator.java:85) > at > org.apache.kylin.cube.CubeManager.updateCubeWithRetry(CubeManager.java:359) > at > org.apache.kylin.cube.CubeManager.updateCubeWithRetry(CubeManager.java:386) > at org.apache.kylin.cube.CubeManager.updateCube(CubeManager.java:302) > at > org.apache.kylin.cube.CubeManager.mergeSegments(CubeManager.java:533) > at > org.apache.kylin.rest.service.CubeService.mergeCubeSegment(CubeService.java:635) > at > org.apache.kylin.rest.service.CubeService.updateOnNewSegmentReady(CubeService.java:587) > at > org.apache.kylin.rest.service.CubeServiceFastClassBySpringCGLIB17a07c0e.invoke() > at > org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) > at > org.springframework.aop.framework.CglibAopProxyDynamicAdvisedInterceptor.intercept(CglibAopProxy.java:629) > at > org.apache.kylin.rest.service.CubeServiceEnhancerBySpringCGLIB$$c6fabb3f.updateOnNewSegmentReady() > at > org.apache.kylin.rest.service.CacheService.rebuildCubeCache(CacheService.java:237) > at > org.apache.kylin.rest.service.CacheService.access$000(CacheService.java:62) > at > org.apache.kylin.rest.service.CacheService$1.afterCubeUpdate(CacheService.java:86) > at org.apache.kylin.cube.CubeManager.updateCube(CubeManager.java:305) > at > org.apache.kylin.cube.CubeManager.promoteNewlyBuiltSegments(CubeManager.java:735) > at > org.apache.kylin.engine.mr.steps.UpdateCubeInfoAfterBuildStep.doWork(UpdateCubeInfoAfterBuildStep.java:62) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113) > ... 6 more > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (KYLIN-2543) Still build dictionary for TopN group by column even using non-dict encoding
[ https://issues.apache.org/jira/browse/KYLIN-2543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-2543. - Resolution: Fixed bug fixed, and passed integration test. > Still build dictionary for TopN group by column even using non-dict encoding > > > Key: KYLIN-2543 > URL: https://issues.apache.org/jira/browse/KYLIN-2543 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.0.0 >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI > Fix For: v2.0.0 > > > Using latest 2.0 beta, create a cube with TopN, but specifying using non-dict > encoding, the JSON is like: > {code} > { > "name": "TOP_CUSTOMER", > "function": { > "expression": "TOP_N", > "parameter": { > "type": "column", > "value": "P_LINEORDER.V_REVENUE", > "next_parameter": { > "type": "column", > "value": "P_LINEORDER.LO_CUSTKEY" > } > }, > "returntype": "topn(100,4)", > "configuration": { > "topn.encoding.P_LINEORDER.LO_CUSTKEY": "integer:4", > "topn.encoding_version.P_LINEORDER.LO_CUSTKEY": "1" > } > } > {code} > But, in the third step "Extract Fact Table Distinct Columns", there is still > a reducer running for "P_LINEORDER.LO_CUSTKEY"; as this column is a UHC, it > takes a long time and finally got failed. > Expected behavior: no reducer for this column. 1.6.0 doesn't have this issue; > this is a new issue in 2.0 during the upgrade to snowflake. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (KYLIN-2549) Modify tools that related to Acl
XIE FAN created KYLIN-2549: -- Summary: Modify tools that related to Acl Key: KYLIN-2549 URL: https://issues.apache.org/jira/browse/KYLIN-2549 Project: Kylin Issue Type: Sub-task Reporter: XIE FAN Assignee: XIE FAN Many tools, such as MigrationTooll, StorageCleanUpJob need to read acl records, and they need to be modified. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (KYLIN-2548) Forwards compatibility
XIE FAN created KYLIN-2548: -- Summary: Forwards compatibility Key: KYLIN-2548 URL: https://issues.apache.org/jira/browse/KYLIN-2548 Project: Kylin Issue Type: Sub-task Reporter: XIE FAN Assignee: XIE FAN -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KYLIN-2547) Fix the bug of multi-process concurrence in mergeCubeSegment
[ https://issues.apache.org/jira/browse/KYLIN-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kangkaisen updated KYLIN-2547: -- Description: There is a minor bug in "Update Cube Info" step when build a cube in distributed env, which will make this job failed, but the next auto merge job won't be affected and will succeed. So the cube data still is consistent and correct. {code:java} Caused by: java.lang.IllegalStateException: Segments overlap: waimai_dolphin_topic_flow_activity_expose_food_d_cube[2017040500_2017041200] and waimai_dolphin_topic_flow_activity_expose_food_d_cube[2017040500_2017041200] at org.apache.kylin.cube.CubeValidator.validate(CubeValidator.java:85) at org.apache.kylin.cube.CubeManager.updateCubeWithRetry(CubeManager.java:359) at org.apache.kylin.cube.CubeManager.updateCubeWithRetry(CubeManager.java:386) at org.apache.kylin.cube.CubeManager.updateCube(CubeManager.java:302) at org.apache.kylin.cube.CubeManager.mergeSegments(CubeManager.java:533) at org.apache.kylin.rest.service.CubeService.mergeCubeSegment(CubeService.java:635) at org.apache.kylin.rest.service.CubeService.updateOnNewSegmentReady(CubeService.java:587) at org.apache.kylin.rest.service.CubeServiceFastClassBySpringCGLIB17a07c0e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxyDynamicAdvisedInterceptor.intercept(CglibAopProxy.java:629) at org.apache.kylin.rest.service.CubeServiceEnhancerBySpringCGLIB$$c6fabb3f.updateOnNewSegmentReady() at org.apache.kylin.rest.service.CacheService.rebuildCubeCache(CacheService.java:237) at org.apache.kylin.rest.service.CacheService.access$000(CacheService.java:62) at org.apache.kylin.rest.service.CacheService$1.afterCubeUpdate(CacheService.java:86) at org.apache.kylin.cube.CubeManager.updateCube(CubeManager.java:305) at org.apache.kylin.cube.CubeManager.promoteNewlyBuiltSegments(CubeManager.java:735) at org.apache.kylin.engine.mr.steps.UpdateCubeInfoAfterBuildStep.doWork(UpdateCubeInfoAfterBuildStep.java:62) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113) ... 6 more {code} was: There is a minor bug in "Update Cube Info" when build a cube and in distributed env. {code:java} Caused by: java.lang.IllegalStateException: Segments overlap: waimai_dolphin_topic_flow_activity_expose_food_d_cube[2017040500_2017041200] and waimai_dolphin_topic_flow_activity_expose_food_d_cube[2017040500_2017041200] at org.apache.kylin.cube.CubeValidator.validate(CubeValidator.java:85) at org.apache.kylin.cube.CubeManager.updateCubeWithRetry(CubeManager.java:359) at org.apache.kylin.cube.CubeManager.updateCubeWithRetry(CubeManager.java:386) at org.apache.kylin.cube.CubeManager.updateCube(CubeManager.java:302) at org.apache.kylin.cube.CubeManager.mergeSegments(CubeManager.java:533) at org.apache.kylin.rest.service.CubeService.mergeCubeSegment(CubeService.java:635) at org.apache.kylin.rest.service.CubeService.updateOnNewSegmentReady(CubeService.java:587) at org.apache.kylin.rest.service.CubeServiceFastClassBySpringCGLIB17a07c0e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxyDynamicAdvisedInterceptor.intercept(CglibAopProxy.java:629) at org.apache.kylin.rest.service.CubeServiceEnhancerBySpringCGLIB$$c6fabb3f.updateOnNewSegmentReady() at org.apache.kylin.rest.service.CacheService.rebuildCubeCache(CacheService.java:237) at org.apache.kylin.rest.service.CacheService.access$000(CacheService.java:62) at org.apache.kylin.rest.service.CacheService$1.afterCubeUpdate(CacheService.java:86) at org.apache.kylin.cube.CubeManager.updateCube(CubeManager.java:305) at org.apache.kylin.cube.CubeManager.promoteNewlyBuiltSegments(CubeManager.java:735) at org.apache.kylin.engine.mr.steps.UpdateCubeInfoAfterBuildStep.doWork(UpdateCubeInfoAfterBuildStep.java:62) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113) ... 6 more {code} > Fix the bug of multi-process concurrence in mergeCubeSegment > > > Key: KYLIN-2547 > URL: https://issues.apache.org/jira/browse/KYLIN-2547 > Project: Kylin > Issue Type: Bug > Components: Metadata >Affects Versions: v2.0.0 >Reporter: kangkaisen >Assignee: kangkaisen >Priority: Minor > Fix For: v2.0.0 > > > There is a minor bug in "Update Cube
[jira] [Commented] (KYLIN-2547) Fix the bug of multi-process concurrence in mergeCubeSegment
[ https://issues.apache.org/jira/browse/KYLIN-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968799#comment-15968799 ] kangkaisen commented on KYLIN-2547: --- The issue reason is keepCubeRetention() and mergeCubeSegment() *only supports multithreading concurrence* and doesn't support multi-process concurrence. So, we should only allow the job server keeping the segment lock to handle the retention check and segment auto merge. > Fix the bug of multi-process concurrence in mergeCubeSegment > > > Key: KYLIN-2547 > URL: https://issues.apache.org/jira/browse/KYLIN-2547 > Project: Kylin > Issue Type: Bug > Components: Metadata >Affects Versions: v2.0.0 >Reporter: kangkaisen >Assignee: kangkaisen >Priority: Minor > Fix For: v2.0.0 > > > There is a minor bug in "Update Cube Info" when build a cube and in > distributed env. > {code:java} >Caused by: java.lang.IllegalStateException: Segments overlap: > waimai_dolphin_topic_flow_activity_expose_food_d_cube[2017040500_2017041200] > and > waimai_dolphin_topic_flow_activity_expose_food_d_cube[2017040500_2017041200] > at org.apache.kylin.cube.CubeValidator.validate(CubeValidator.java:85) > at > org.apache.kylin.cube.CubeManager.updateCubeWithRetry(CubeManager.java:359) > at > org.apache.kylin.cube.CubeManager.updateCubeWithRetry(CubeManager.java:386) > at org.apache.kylin.cube.CubeManager.updateCube(CubeManager.java:302) > at > org.apache.kylin.cube.CubeManager.mergeSegments(CubeManager.java:533) > at > org.apache.kylin.rest.service.CubeService.mergeCubeSegment(CubeService.java:635) > at > org.apache.kylin.rest.service.CubeService.updateOnNewSegmentReady(CubeService.java:587) > at > org.apache.kylin.rest.service.CubeServiceFastClassBySpringCGLIB17a07c0e.invoke() > at > org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) > at > org.springframework.aop.framework.CglibAopProxyDynamicAdvisedInterceptor.intercept(CglibAopProxy.java:629) > at > org.apache.kylin.rest.service.CubeServiceEnhancerBySpringCGLIB$$c6fabb3f.updateOnNewSegmentReady() > at > org.apache.kylin.rest.service.CacheService.rebuildCubeCache(CacheService.java:237) > at > org.apache.kylin.rest.service.CacheService.access$000(CacheService.java:62) > at > org.apache.kylin.rest.service.CacheService$1.afterCubeUpdate(CacheService.java:86) > at org.apache.kylin.cube.CubeManager.updateCube(CubeManager.java:305) > at > org.apache.kylin.cube.CubeManager.promoteNewlyBuiltSegments(CubeManager.java:735) > at > org.apache.kylin.engine.mr.steps.UpdateCubeInfoAfterBuildStep.doWork(UpdateCubeInfoAfterBuildStep.java:62) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113) > ... 6 more > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KYLIN-2544) lookup table show wrong join type
[ https://issues.apache.org/jira/browse/KYLIN-2544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhixiong Chen updated KYLIN-2544: - Component/s: Web > lookup table show wrong join type > - > > Key: KYLIN-2544 > URL: https://issues.apache.org/jira/browse/KYLIN-2544 > Project: Kylin > Issue Type: Bug > Components: Web >Reporter: Zhixiong Chen >Assignee: Zhixiong Chen > > lookup table show wrong join type,when edit a left join lookup table -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (KYLIN-2547) Fix the bug of multi-process concurrence in mergeCubeSegment
kangkaisen created KYLIN-2547: - Summary: Fix the bug of multi-process concurrence in mergeCubeSegment Key: KYLIN-2547 URL: https://issues.apache.org/jira/browse/KYLIN-2547 Project: Kylin Issue Type: Bug Components: Metadata Affects Versions: v2.0.0 Reporter: kangkaisen Assignee: kangkaisen Priority: Minor Fix For: v2.0.0 There is a minor bug in "Update Cube Info" when build a cube and in distributed env. {code:java} Caused by: java.lang.IllegalStateException: Segments overlap: waimai_dolphin_topic_flow_activity_expose_food_d_cube[2017040500_2017041200] and waimai_dolphin_topic_flow_activity_expose_food_d_cube[2017040500_2017041200] at org.apache.kylin.cube.CubeValidator.validate(CubeValidator.java:85) at org.apache.kylin.cube.CubeManager.updateCubeWithRetry(CubeManager.java:359) at org.apache.kylin.cube.CubeManager.updateCubeWithRetry(CubeManager.java:386) at org.apache.kylin.cube.CubeManager.updateCube(CubeManager.java:302) at org.apache.kylin.cube.CubeManager.mergeSegments(CubeManager.java:533) at org.apache.kylin.rest.service.CubeService.mergeCubeSegment(CubeService.java:635) at org.apache.kylin.rest.service.CubeService.updateOnNewSegmentReady(CubeService.java:587) at org.apache.kylin.rest.service.CubeServiceFastClassBySpringCGLIB17a07c0e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxyDynamicAdvisedInterceptor.intercept(CglibAopProxy.java:629) at org.apache.kylin.rest.service.CubeServiceEnhancerBySpringCGLIB$$c6fabb3f.updateOnNewSegmentReady() at org.apache.kylin.rest.service.CacheService.rebuildCubeCache(CacheService.java:237) at org.apache.kylin.rest.service.CacheService.access$000(CacheService.java:62) at org.apache.kylin.rest.service.CacheService$1.afterCubeUpdate(CacheService.java:86) at org.apache.kylin.cube.CubeManager.updateCube(CubeManager.java:305) at org.apache.kylin.cube.CubeManager.promoteNewlyBuiltSegments(CubeManager.java:735) at org.apache.kylin.engine.mr.steps.UpdateCubeInfoAfterBuildStep.doWork(UpdateCubeInfoAfterBuildStep.java:62) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113) ... 6 more {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KYLIN-2506) Refactor Global Dictionary
[ https://issues.apache.org/jira/browse/KYLIN-2506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968750#comment-15968750 ] kangkaisen commented on KYLIN-2506: --- Yes, [~xiefan46]. I agree with you. > Refactor Global Dictionary > -- > > Key: KYLIN-2506 > URL: https://issues.apache.org/jira/browse/KYLIN-2506 > Project: Kylin > Issue Type: Improvement > Components: General >Affects Versions: v2.0.0 >Reporter: kangkaisen >Assignee: kangkaisen > Fix For: v2.0.0 > > > The main points of this refactor: > 1 Fix the bug that the RemoveListener of LoadingCache swallowed any > exceptions when building the GlobalDict. > 2 Fix the bug that the HDFS filename of DictSliceKey had Illegal characters. > 3 Fix the bug that the HDFS filename of DictSliceKey maybe longer than 255. > 4 Fix the bug that DictNode split failed if value length greater than 255 > bytes. > 5 Decouple the build and query of GlobalDict: > Abstract the builder of AppendTrieDictionary to AppendTrieDictionaryBuilder; > Add LoadingCache to AppendTrieDictionary and make AppendTrieDictionary is > only readable. > 6 Remove dependence of LoadingCache when building the GlobalDict. > 7 Abstract the HDFS operations to GlobalDictStore. > 8 Abstract the metadata of GlobalDict to GlobalDictMetadata. > 9 Delete CachedTreeMap. > 10 Add distributed lock for GlobalDict. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (KYLIN-2546) support config key and value deserializer for Kafka client
Xing Huang created KYLIN-2546: - Summary: support config key and value deserializer for Kafka client Key: KYLIN-2546 URL: https://issues.apache.org/jira/browse/KYLIN-2546 Project: Kylin Issue Type: Improvement Components: streaming Affects Versions: v1.6.0 Reporter: Xing Huang So custom deserializer can be used to convert value of Kafka message to json which is required by Kylin. For example, from Avro to Json. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (KYLIN-2545) Number2BytesConverter could tolerate malformed numbers
liyang created KYLIN-2545: - Summary: Number2BytesConverter could tolerate malformed numbers Key: KYLIN-2545 URL: https://issues.apache.org/jira/browse/KYLIN-2545 Project: Kylin Issue Type: Improvement Reporter: liyang Assignee: liyang Some malformed numbers like "0100", "100.1200" currently does not work with Number Dictionary. And could be improved. -- This message was sent by Atlassian JIRA (v6.3.15#6346)