liuxiaocs7 commented on issue #6246:
URL: https://github.com/apache/kyuubi/issues/6246#issuecomment-2041777050
According to the Log displayed in
[CI](https://github.com/apache/kyuubi/actions/workflows/nightly.yml)
run locally
```
./build/mvn clean install -Pscala-2.13 -Pspark-master -pl
extensions/spark/kyuubi-spark-lineage -am -Dmaven.javadoc.skip=true -V
-Dtest=none
-DwildcardSuites=org.apache.kyuubi.plugin.lineage.helper.SparkSQLLineageParserHelperSuite
```
got the same result
```
Discovery starting.
Discovery completed in 810 milliseconds.
Run starting. Expected test count is: 33
SparkSQLLineageParserHelperSuite:
*** RUN ABORTED ***
java.lang.NoClassDefFoundError: jakarta/servlet/Servlet
at
org.apache.spark.metrics.sink.MetricsServlet.getHandlers(MetricsServlet.scala:50)
at
org.apache.spark.metrics.MetricsSystem.$anonfun$getServletHandlers$2(MetricsSystem.scala:91)
at scala.Option.map(Option.scala:242)
at
org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:91)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:686)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2937)
at
org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:1117)
at scala.Option.getOrElse(Option.scala:201)
at
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:1111)
at
org.apache.spark.sql.SparkListenerExtensionTest.spark(SparkListenerExtensionTest.scala:42)
...
Cause: java.lang.ClassNotFoundException: jakarta.servlet.Servlet
at
java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
at
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
at
org.apache.spark.metrics.sink.MetricsServlet.getHandlers(MetricsServlet.scala:50)
at
org.apache.spark.metrics.MetricsSystem.$anonfun$getServletHandlers$2(MetricsSystem.scala:91)
at scala.Option.map(Option.scala:242)
at
org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:91)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:686)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2937)
at
org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:1117)
...
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary for Kyuubi Project Parent 1.10.0-SNAPSHOT:
[INFO]
[INFO] Kyuubi Project Parent .............................. SUCCESS [ 5.595
s]
[INFO] Kyuubi Project Util ................................ SUCCESS [ 3.653
s]
[INFO] Kyuubi Project Util Scala .......................... SUCCESS [ 21.428
s]
[INFO] Kyuubi Project Common .............................. SUCCESS [ 50.013
s]
[INFO] Kyuubi Project Events .............................. SUCCESS [ 13.887
s]
[INFO] Kyuubi Dev Spark Lineage Extension ................. FAILURE [ 19.960
s]
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
```
add `<jakarta.servlet-api.version>5.0.0</jakarta.servlet-api.version>` to
`spark-master` profile
(https://github.com/apache/spark/blob/b299b2bc06a91db630ab39b9c35663342931bb56/pom.xml#L147)
find new error:
```
Discovery starting.
Discovery completed in 883 milliseconds.
Run starting. Expected test count is: 33
SparkSQLLineageParserHelperSuite:
ANTLR Tool version 4.13.1 used for code generation does not match the
current runtime version 4.9.3
ANTLR Runtime version 4.13.1 used for parser compilation does not match the
current runtime version 4.9.3
*** RUN ABORTED ***
java.lang.ExceptionInInitializerError:
at
org.apache.spark.sql.catalyst.parser.AbstractParser.parse(parsers.scala:58)
at
org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:55)
at
org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(AbstractSqlParser.scala:68)
at org.apache.spark.sql.SparkSession.$anonfun$sql$5(SparkSession.scala:701)
at
org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:138)
at org.apache.spark.sql.SparkSession.$anonfun$sql$4(SparkSession.scala:700)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:918)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:699)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:730)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:761)
...
Cause: java.lang.UnsupportedOperationException:
java.io.InvalidClassException: org.antlr.v4.runtime.atn.ATN; Could not
deserialize ATN with version 4 (expected 3).
at
org.antlr.v4.runtime.atn.ATNDeserializer.deserialize(ATNDeserializer.java:187)
at
org.apache.spark.sql.catalyst.parser.SqlBaseLexer.<clinit>(SqlBaseLexer.java:2949)
at
org.apache.spark.sql.catalyst.parser.AbstractParser.parse(parsers.scala:58)
at
org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:55)
at
org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(AbstractSqlParser.scala:68)
at org.apache.spark.sql.SparkSession.$anonfun$sql$5(SparkSession.scala:701)
at
org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:138)
at org.apache.spark.sql.SparkSession.$anonfun$sql$4(SparkSession.scala:700)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:918)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:699)
...
Cause: java.io.InvalidClassException: org.antlr.v4.runtime.atn.ATN; Could
not deserialize ATN with version 4 (expected 3).
at
org.antlr.v4.runtime.atn.ATNDeserializer.deserialize(ATNDeserializer.java:187)
at
org.apache.spark.sql.catalyst.parser.SqlBaseLexer.<clinit>(SqlBaseLexer.java:2949)
at
org.apache.spark.sql.catalyst.parser.AbstractParser.parse(parsers.scala:58)
at
org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:55)
at
org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(AbstractSqlParser.scala:68)
at org.apache.spark.sql.SparkSession.$anonfun$sql$5(SparkSession.scala:701)
at
org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:138)
at org.apache.spark.sql.SparkSession.$anonfun$sql$4(SparkSession.scala:700)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:918)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:699)
...
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary for Kyuubi Project Parent 1.10.0-SNAPSHOT:
[INFO]
[INFO] Kyuubi Project Parent .............................. SUCCESS [ 5.822
s]
[INFO] Kyuubi Project Util ................................ SUCCESS [ 7.390
s]
[INFO] Kyuubi Project Util Scala .......................... SUCCESS [ 24.927
s]
[INFO] Kyuubi Project Common .............................. SUCCESS [01:30
min]
[INFO] Kyuubi Project Events .............................. SUCCESS [ 32.613
s]
[INFO] Kyuubi Dev Spark Lineage Extension ................. FAILURE [ 18.767
s]
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
```
continue to add `<antlr4.version>4.13.1</antlr4.version>` to `spark-master`
profile
(https://github.com/apache/spark/blob/b299b2bc06a91db630ab39b9c35663342931bb56/pom.xml#L210)
build and test are ok in this module.
```
[INFO] Reactor Summary for Kyuubi Project Parent 1.10.0-SNAPSHOT:
[INFO]
[INFO] Kyuubi Project Parent .............................. SUCCESS [ 5.369
s]
[INFO] Kyuubi Project Util ................................ SUCCESS [ 3.648
s]
[INFO] Kyuubi Project Util Scala .......................... SUCCESS [ 14.716
s]
[INFO] Kyuubi Project Common .............................. SUCCESS [ 51.174
s]
[INFO] Kyuubi Project Events .............................. SUCCESS [ 14.505
s]
[INFO] Kyuubi Dev Spark Lineage Extension ................. SUCCESS [ 44.672
s]
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
```
but `./build/mvn clean install -Pscala-2.13 -Pspark-master -pl
externals/kyuubi-spark-sql-engine -am -Dmaven.javadoc.skip=true -V -Dtest=none
-DskipTests` build fails
```
[INFO] compiling 59 Scala sources to
/home/liuxiao/Code/JavaProjects/kyuubi/externals/kyuubi-spark-sql-engine/target/scala-2.13/classes
...
[ERROR] [Error]
/home/liuxiao/Code/JavaProjects/kyuubi/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/ExecutePython.scala:27:
object ws is not a member of package javax
[ERROR] [Error]
/home/liuxiao/Code/JavaProjects/kyuubi/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/ExecutePython.scala:307:
not found: value UriBuilder
[ERROR] [Error]
/home/liuxiao/Code/JavaProjects/kyuubi/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/operation/ExecutePython.scala:320:
not found: value UriBuilder
[ERROR] [Error]
/home/liuxiao/Code/JavaProjects/kyuubi/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/ui/EnginePage.scala:80:
type mismatch;
found : HttpServletRequest (in javax.servlet.http)
required: HttpServletRequest (in jakarta.servlet.http)
[ERROR] [Error]
/home/liuxiao/Code/JavaProjects/kyuubi/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/ui/EnginePage.scala:135:
type mismatch;
found : HttpServletRequest (in javax.servlet.http)
required: HttpServletRequest (in jakarta.servlet.http)
[ERROR] [Error]
/home/liuxiao/Code/JavaProjects/kyuubi/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/ui/EnginePage.scala:258:
type mismatch;
found : HttpServletRequest (in javax.servlet.http)
required: HttpServletRequest (in jakarta.servlet.http)
[ERROR] [Error]
/home/liuxiao/Code/JavaProjects/kyuubi/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/ui/EnginePage.scala:341:
type mismatch;
found : HttpServletRequest (in javax.servlet.http)
required: HttpServletRequest (in jakarta.servlet.http)
[ERROR] [Error]
/home/liuxiao/Code/JavaProjects/kyuubi/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/ui/EnginePage.scala:421:
type mismatch;
found : HttpServletRequest (in javax.servlet.http)
required: HttpServletRequest (in jakarta.servlet.http)
[ERROR] [Error]
/home/liuxiao/Code/JavaProjects/kyuubi/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/ui/EnginePage.scala:510:
type mismatch;
found : HttpServletRequest (in javax.servlet.http)
required: HttpServletRequest (in jakarta.servlet.http)
[ERROR] [Error]
/home/liuxiao/Code/JavaProjects/kyuubi/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/ui/EnginePage.scala:547:
type mismatch;
found : HttpServletRequest (in javax.servlet.http)
required: HttpServletRequest (in jakarta.servlet.http)
[ERROR] [Error]
/home/liuxiao/Code/JavaProjects/kyuubi/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/ui/EngineSessionPage.scala:101:
type mismatch;
found : HttpServletRequest (in javax.servlet.http)
required: HttpServletRequest (in jakarta.servlet.http)
[ERROR] [Error]
/home/liuxiao/Code/JavaProjects/kyuubi/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/ui/EngineSessionPage.scala:234:
type mismatch;
found : HttpServletRequest (in javax.servlet.http)
required: HttpServletRequest (in jakarta.servlet.http)
[ERROR] 12 errors found
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary for Kyuubi Project Parent 1.10.0-SNAPSHOT:
[INFO]
[INFO] Kyuubi Project Parent .............................. SUCCESS [ 4.661
s]
[INFO] Kyuubi Project Util ................................ SUCCESS [ 3.557
s]
[INFO] Kyuubi Project Util Scala .......................... SUCCESS [ 21.494
s]
[INFO] Kyuubi Project Common .............................. SUCCESS [ 52.340
s]
[INFO] Kyuubi Project Embedded Zookeeper .................. SUCCESS [ 9.851
s]
[INFO] Kyuubi Project High Availability ................... SUCCESS [ 25.424
s]
[INFO] Kyuubi Project Events .............................. SUCCESS [ 14.723
s]
[INFO] Kyuubi Dev Spark Lineage Extension ................. SUCCESS [ 27.378
s]
[INFO] Kyuubi Project Hive JDBC Client .................... SUCCESS [ 13.132
s]
[INFO] Kyuubi Project Engine Spark SQL .................... FAILURE [ 8.426
s]
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
```
seems `javax` and `jakarta` namespace conflicts in spark 4.x and kyuubi
codebase now? @pan3793 any suggestions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
