[
https://issues.apache.org/jira/browse/KYLIN-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17358973#comment-17358973
]
ASF GitHub Bot commented on KYLIN-4925:
---------------------------------------
zzcclp commented on a change in pull request #1601:
URL: https://github.com/apache/kylin/pull/1601#discussion_r647045104
##########
File path:
kylin-spark-project/kylin-spark-engine/src/test/scala/org/apache/kylin/engine/spark/builder/TestCreateFlatTable.scala
##########
@@ -96,17 +96,32 @@ class TestCreateFlatTable extends SparderBaseFunSuite with
SharedSparkSession wi
val afterJoin1 = generateFlatTable(seg1, cube, true)
afterJoin1.collect()
- val jobs = helper.getJobsByGroupId(groupId)
- Assert.assertEquals(jobs.length, 15)
+ if (SPARK_VERSION.startsWith("2.4")) {
+ val jobs = helper.getJobsByGroupId(groupId)
+ Assert.assertEquals(jobs.length, 15)
+ } else if (SPARK_VERSION.startsWith("3.1")) {
+ val jobs = helper.getJobsByGroupId(null)
+ Assert.assertEquals(jobs.length, 15)
+ Assert.assertEquals(6,
jobs.count(_.jobGroup.exists(_.endsWith(groupId))))
+ Assert.assertEquals(9,
jobs.count(_.description.exists(_.contains("broadcast exchange"))))
Review comment:
Add a comment: Since Spark 3.X, BroadcastExchangeExec overwrites job
group ID
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Use Spark 3 as build and query engine for Kylin 4
> -------------------------------------------------
>
> Key: KYLIN-4925
> URL: https://issues.apache.org/jira/browse/KYLIN-4925
> Project: Kylin
> Issue Type: New Feature
> Components: Job Engine, Query Engine
> Reporter: Congling Xia
> Assignee: Congling Xia
> Priority: Major
> Fix For: v4.0.0
>
>
> Spark 3 provides much more developed AQE framework, DPP and the increased
> support for GPUs and Kubernetes. The perspectives of performance increase are
> very promising.
> Currently, we try to run Kylin 4.0 with Spark 3.0 as build & query engine.
> We'd like to share the code changes here.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
