parisni opened a new issue, #10402:
URL: https://github.com/apache/hudi/issues/10402
```python
# spark-3.2.4-bin-hadoop3.2/bin/pyspark --jars
/projects/hudi/packaging/hudi-spark-bundle/target/hudi-spark3.2-bundle_2.12-0.14.1-rc1.jar
--conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' --conf
'spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog'
--conf
'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension'
--conf 'spark.kryo.registrator=org.apache.spark.HoodieSparkKryoRegistrar'
spark.sql("select 1").show()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File
"/home/nparis/Soft/spark-3.2.4-bin-hadoop3.2/python/pyspark/sql/session.py",
line 723, in sql
return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/nparis/Soft/spark-3.2.4-bin-hadoop3.2/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py",
line 1321, in __call__
File
"/home/nparis/Soft/spark-3.2.4-bin-hadoop3.2/python/pyspark/sql/utils.py", line
111, in deco
return f(*a, **kw)
^^^^^^^^^^^
File
"/home/nparis/Soft/spark-3.2.4-bin-hadoop3.2/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py",
line 326, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o31.sql.
: java.lang.NoSuchMethodError: 'scala.Option
org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin$default$2()'
at
org.apache.spark.sql.parser.HoodieSqlCommonAstBuilder.visitSingleStatement(HoodieSqlCommonAstBuilder.scala:54)
at
org.apache.spark.sql.parser.HoodieSqlCommonAstBuilder.visitSingleStatement(HoodieSqlCommonAstBuilder.scala:36)
at
org.apache.hudi.spark.sql.parser.HoodieSqlCommonParser$SingleStatementContext.accept(HoodieSqlCommonParser.java:141)
at
org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:18)
at
org.apache.spark.sql.parser.HoodieCommonSqlParser.$anonfun$parsePlan$1(HoodieCommonSqlParser.scala:41)
at
org.apache.spark.sql.parser.HoodieCommonSqlParser.parse(HoodieCommonSqlParser.scala:91)
at
org.apache.spark.sql.parser.HoodieCommonSqlParser.parsePlan(HoodieCommonSqlParser.scala:40)
at
org.apache.spark.sql.SparkSession.$anonfun$sql$2(SparkSession.scala:616)
at
org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
at
org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:616)
at
org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at
py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at
py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.base/java.lang.Thread.run(Thread.java:829)
```
While spark 3.3 works fine:
```python
# /spark-3.3.4-bin-hadoop3/bin/pyspark --jars
/projects/hudi/packaging/hudi-spark-bundle/target/hudi-spark3.3-bundle_2.12-0.14.1-rc1.jar
--conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' --conf
'spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog'
--conf
'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension'
--conf 'spark.kryo.registrator=org.apache.spark.HoodieSparkKryoRegistrar'
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 3.3.4
/_/
Using Python version 3.11.6 (main, Oct 8 2023 05:06:43)
Spark context Web UI available at
http://172-254-000-001.biz.spectrum.com:4040
Spark context available as 'sc' (master = local[*], app id =
local-1703250073652).
SparkSession available as 'spark'.
>>> spark.sql("select 1").show()
+---+
| 1|
+---+
| 1|
+---+
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
