[
https://issues.apache.org/jira/browse/KYLIN-4967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17318993#comment-17318993
]
ASF subversion and git services commented on KYLIN-4967:
--------------------------------------------------------
Commit 6153889ce1b1fad2c409364e77ec471166654551 in kylin's branch
refs/heads/kylin-on-parquet-v2 from Zhichao Zhang
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=6153889 ]
KYLIN-4967 Forbid to set 'spark.sql.adaptive.enabled' to true when building
cube with Spark 2.X
With spark 2.X, when set 'spark.sql.adaptive.enabled' to true, it will impact
the actually partition count when doing repartition with spark, which will lead
to the wrong results for global dict and repartition by shardby column.
For example, after writing a cuboid data, kylin will repartition the cuboid
data with 3 partition if need, but if 'spark.sql.adaptive.enabled' is true,
spark will optimize the partition num to 1, which leads to wrong.
> Forbid to set 'spark.sql.adaptive.enabled' to true when building cube with
> Spark 2.X
> ------------------------------------------------------------------------------------
>
> Key: KYLIN-4967
> URL: https://issues.apache.org/jira/browse/KYLIN-4967
> Project: Kylin
> Issue Type: Bug
> Affects Versions: v4.0.0-beta
> Reporter: Zhichao Zhang
> Assignee: Zhichao Zhang
> Priority: Minor
> Fix For: v4.0.0-GA
>
>
> With spark 2.X, when set 'spark.sql.adaptive.enabled' to true, it will impact
> the actually partition count when doing repartition with spark, which will
> lead to the wrong results for global dict and repartition by shardby column.
> For example, after writing a cuboid data, kylin will repartition the cuboid
> data with 3 partition if need, but if 'spark.sql.adaptive.enabled' is true,
> spark will optimize the partition num to 1, which leads to wrong.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
