This is an automated email from the ASF dual-hosted git repository.
chengpan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new 522a28e1d [KYUUBI #6398] Fix lineage plugin UT for Spark 4.0
522a28e1d is described below
commit 522a28e1d56808776d0090108f51fbd52dcdf6cc
Author: Cheng Pan <[email protected]>
AuthorDate: Mon May 20 22:03:48 2024 +0800
[KYUUBI #6398] Fix lineage plugin UT for Spark 4.0
# :mag: Description
```
build/mvn clean test -Pscala-2.13 -Pspark-master -pl
:kyuubi-spark-lineage_2.13
```
```
- test group by *** FAILED ***
org.apache.spark.sql.catalyst.ExtendedAnalysisException:
[DATATYPE_MISMATCH.BINARY_OP_WRONG_TYPE] Cannot resolve "(b + c)" due to data
type mismatch: the binary operator requires the input type ("NUMERIC" or
"INTERVAL DAY TO SECOND" or "INTERVAL YEAR TO MONTH" or "INTERVAL"), not
"STRING". SQLSTATE: 42K09; line 1 pos 59;
'InsertIntoStatement RelationV2[a#546, b#547, c#548] v2_catalog.db.t1
v2_catalog.db.t1, false, false, false
+- 'Aggregate [a#543], [a#543, unresolvedalias('count(distinct (b#544 +
c#545))), (count(distinct b#544) * count(distinct c#545)) AS (count(DISTINCT b)
* count(DISTINCT c))#551L]
+- SubqueryAlias v2_catalog.db.t2
+- RelationV2[a#543, b#544, c#545] v2_catalog.db.t2 v2_catalog.db.t2
at
org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.dataTypeMismatch(package.scala:73)
at
org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis0$7(CheckAnalysis.scala:315)
at
org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis0$7$adapted(CheckAnalysis.scala:302)
at
org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:244)
at
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1(TreeNode.scala:243)
at
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1$adapted(TreeNode.scala:243)
at scala.collection.immutable.Vector.foreach(Vector.scala:1856)
at
org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:243)
at
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1(TreeNode.scala:243)
at
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$foreachUp$1$adapted(TreeNode.scala:243)
...
```
## Types of changes :bookmark:
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
## Test Plan ๐งช
Pass UT.
---
# Checklist ๐
- [x] This patch was not authored or co-authored using [Generative
Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6398 from pan3793/lineage-fix.
Closes #6398
afce6b880 [Cheng Pan] Fix lineage plugin UT for Spark 4.0
Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
---
.../kyuubi/plugin/lineage/helper/SparkSQLLineageParserHelperSuite.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/extensions/spark/kyuubi-spark-lineage/src/test/scala/org/apache/kyuubi/plugin/lineage/helper/SparkSQLLineageParserHelperSuite.scala
b/extensions/spark/kyuubi-spark-lineage/src/test/scala/org/apache/kyuubi/plugin/lineage/helper/SparkSQLLineageParserHelperSuite.scala
index 761274379..b65cc4c5a 100644
---
a/extensions/spark/kyuubi-spark-lineage/src/test/scala/org/apache/kyuubi/plugin/lineage/helper/SparkSQLLineageParserHelperSuite.scala
+++
b/extensions/spark/kyuubi-spark-lineage/src/test/scala/org/apache/kyuubi/plugin/lineage/helper/SparkSQLLineageParserHelperSuite.scala
@@ -1237,7 +1237,7 @@ class SparkSQLLineageParserHelperSuite extends
KyuubiFunSuite
val ret2 =
extractLineage(
s"insert into table v2_catalog.db.t1 select a," +
- s"count(distinct(b+c))," +
+ s"count(distinct(cast(b as int)+cast(c as int)))," +
s"count(distinct(b)) * count(distinct(c))" +
s"from v2_catalog.db.t2 group by a")
assert(ret2 == Lineage(
