This is an automated email from the ASF dual-hosted git repository.
chengpan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new 83b2f8326 [KYUUBI #5257] [LINEAGE] Adapt `CreateTableAsSelect` plan to
Spark 3.5 changes
83b2f8326 is described below
commit 83b2f8326b21561f22464409904bdccb5a4feed1
Author: Fu Chen <[email protected]>
AuthorDate: Wed Sep 6 07:26:30 2023 +0000
[KYUUBI #5257] [LINEAGE] Adapt `CreateTableAsSelect` plan to Spark 3.5
changes
### _Why are the changes needed?_
to fix
```
- columns lineage extract - CreateTableAsSelect *** FAILED ***
java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:529)
at scala.None$.get(Option.scala:527)
at
org.apache.kyuubi.plugin.lineage.helper.SparkSQLLineageParserHelperSuite.extractLineage(SparkSQLLineageParserHelperSuite.scala:1452)
at
org.apache.kyuubi.plugin.lineage.helper.SparkSQLLineageParserHelperSuite.$anonfun$new$24(SparkSQLLineageParserHelperSuite.scala:354)
at
org.apache.kyuubi.plugin.lineage.helper.SparkSQLLineageParserHelperSuite.$anonfun$new$24$adapted(SparkSQLLineageParserHelperSuite.scala:353)
at
org.apache.spark.sql.SparkListenerExtensionTest.withTable(SparkListenerExtensionTest.scala:48)
at
org.apache.spark.sql.SparkListenerExtensionTest.withTable$(SparkListenerExtensionTest.scala:46)
at
org.apache.kyuubi.plugin.lineage.helper.SparkSQLLineageParserHelperSuite.withTable(SparkSQLLineageParserHelperSuite.scala:34)
at
org.apache.kyuubi.plugin.lineage.helper.SparkSQLLineageParserHelperSuite.$anonfun$new$23(SparkSQLLineageParserHelperSuite.scala:353)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
...
```
the method `CreateTableAsSelect#left` was removed since Spark 3.5, for more
details, see https://github.com/apache/spark/pull/40734
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including
negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run
test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests)
locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes #5257 from cfmcgrady/fix-SparkSQLLineageParserHelperSuite.
Closes #5257
aa5e730fc [Fu Chen] fix
Authored-by: Fu Chen <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
---
.../kyuubi/plugin/lineage/helper/SparkSQLLineageParseHelper.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/extensions/spark/kyuubi-spark-lineage/src/main/scala/org/apache/kyuubi/plugin/lineage/helper/SparkSQLLineageParseHelper.scala
b/extensions/spark/kyuubi-spark-lineage/src/main/scala/org/apache/kyuubi/plugin/lineage/helper/SparkSQLLineageParseHelper.scala
index 6beecba36..273111464 100644
---
a/extensions/spark/kyuubi-spark-lineage/src/main/scala/org/apache/kyuubi/plugin/lineage/helper/SparkSQLLineageParseHelper.scala
+++
b/extensions/spark/kyuubi-spark-lineage/src/main/scala/org/apache/kyuubi/plugin/lineage/helper/SparkSQLLineageParseHelper.scala
@@ -251,7 +251,7 @@ trait LineageParser {
invokeAs[Identifier](plan, "tableName").name(),
invokeAs[Identifier](plan,
"tableName").namespace().mkString("."),
getField[CatalogPlugin](
- invokeAs[LogicalPlan](plan, "left"),
+ invokeAs[LogicalPlan](plan, "name"),
"catalog").name())
}
extractColumnsLineage(getQuery(plan), parentColumnsLineage).map { case
(k, v) =>
