cfmcgrady commented on PR #4381: URL: https://github.com/apache/kyuubi/pull/4381#issuecomment-1442871106
> Moved `PySparkTests` from the module `kyuubi-spark-sql-engine` to `kyuubi-server` to ensure that the python progress loading library PYSPARK has the same version as the launched Spark engine. FYI: the signature of the method`pyspark.sql.SparkSession.sql()` is changed after https://github.com/apache/spark/pull/39971, there is a compatibility issue if the loaded pyspark library is version 3.4.0 but the launched spark engine is version 3.3.2. ``` org.apache.kyuubi.jdbc.hive.KyuubiSQLException: org.apache.kyuubi.KyuubiSQLException: Error operating ExecutePython: org.apache.kyuubi.KyuubiSQLException: Interpret error: spark.sql("select 1").show() Some(PythonResponse(execute_reply,PythonResponseContent(null,Py4JError,An error occurred while calling o79.sql. Trace: py4j.Py4JException: Method sql([class java.lang.String, class java.util.HashMap]) does not exist at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318) at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326) at py4j.Gateway.invoke(Gateway.java:274) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182) at py4j.ClientServerConnection.run(ClientServerConnection.java:106) at java.lang.Thread.run(Thread.java:750) ,List(Traceback (most recent call last): , File "/home/runner/work/kyuubi/kyuubi/externals/kyuubi-download/target/spark-3.4.0-bin-hadoop3/python/pyspark/sql/session.py", line 1427, in sql return DataFrame(self._jsparkSession.sql(sqlQuery, args or {}), self) , File "/home/runner/work/kyuubi/kyuubi/externals/kyuubi-download/target/spark-3.4.0-bin-hadoop3/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in __call__ return_value = get_return_value( , File "/home/runner/work/kyuubi/kyuubi/externals/kyuubi-download/target/spark-3.4.0-bin-hadoop3/python/pyspark/errors/exceptions/captured.py", line 153, in deco return f(*a, **kw) , File "/home/runner/work/kyuubi/kyuubi/externals/kyuubi-download/target/spark-3.4.0-bin-hadoop3/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py", line 330, in get_return_value raise Py4JError( , py4j.protocol.Py4JError: An error occurred while calling o79.sql. Trace: py4j.Py4JException: Method sql([class java.lang.String, class java.util.HashMap]) does not exist at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318) at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326) at py4j.Gateway.invoke(Gateway.java:274) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182) at py4j.ClientServerConnection.run(ClientServerConnection.java:106) at java.lang.Thread.run(Thread.java:750) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
