[
https://issues.apache.org/jira/browse/KYLIN-5646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17746346#comment-17746346
]
ASF subversion and git services commented on KYLIN-5646:
--------------------------------------------------------
Commit 6dc5c992b65fdfac107db5ab970260dc06603c91 in kylin's branch
refs/heads/kylin5 from sibingzhang
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=6dc5c992b6 ]
KYLIN-5646 Fix in yarn cluster mode, an error is reported for incrementally
building the partition table job
Co-authored-by: sibing.zhang <[email protected]>
> The build job reports an error at the step of detecting time partition
> columns in the Yarn Cluster mode
> -------------------------------------------------------------------------------------------------------
>
> Key: KYLIN-5646
> URL: https://issues.apache.org/jira/browse/KYLIN-5646
> Project: Kylin
> Issue Type: Bug
> Components: Tools, Build and Test
> Affects Versions: 5.0-alpha
> Reporter: Zhiting Guo
> Assignee: Zhiting Guo
> Priority: Major
> Fix For: 5.0-beta
>
>
> When building Spark YARN-Cluster mode, when detecting incremental time
> partition columns, initializing KylinConfig reports an error Didn't find
> KYLIN_HOME or KYLIN_HOME
> *Reproduce method*
> Build the partition table model incrementally using Spark YARN_Cluster mode,
> and set kylin.engine.check-partition-col-enabled=true (the default value is
> true)
> *Root Cause*
> Modified the autoSetShufflePartitions of the pushdown query in [KYLIN-5571],
> no need to execute when the pre-modification build task detects the delta
> time column format (only the pushdown query is executed)
> After modification, autoSetShufflePartitions is executed asynchronously, the
> following two methods will get KylinConfig through
> KylinConfig.getInstanceFromEnv,
> At this time, the asynchronous execution of the new thread cannot use the
> built KylinConfig, so the KylinConfig will be initialized,
> However, the build task jvm and the KE main process are not the same machine,
> and KYLIN_CONF and KYLIN_HOME cannot be obtained, so the build task fails to
> run
> * ResourceDetectUtils.getResourceSizeWithTimeoutByConcurrency
> * ResourceDetectUtils.getResourceSizBySerial
> *fix design*
> In all the logic of newly opened threads, if KylinConfig is used, this method
> KylinConfig.getInstanceFromEnv() is not used. Unified is obtained by an
> external thread and passed to the place where it needs to be used
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
