Moritz created LIVY-863:
---------------------------
Summary: Missing JVM class imports for Spark3
Key: LIVY-863
URL: https://issues.apache.org/jira/browse/LIVY-863
Project: Livy
Issue Type: Bug
Components: REPL
Reporter: Moritz
Livy creates a fake spark shell initiating its own java gateway. This gateway
imports some java classes for the user which have diverged from the ones in
Spark 3. Some imports are missing which lead to methods like
`dataframe.explain()` to be broken:
{code:java}
result.explain(True)
An error was encountered:
'JavaPackage' object is not callable
Traceback (most recent call last):
File "/srv/hops/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line
386, in explain
print(self._sc._jvm.PythonSQLUtils.explainString(self._jdf.queryExecution(),
explain_mode))
TypeError: 'JavaPackage' object is not callable
{code}
Spark is importing:
[https://github.com/apache/spark/blob/87bf6b0ea4ca0618c8604895d05037edce8b7cb0/python/pyspark/java_gateway.py#L153]
Livy is currently importing:
[https://github.com/apache/incubator-livy/blob/4d8a912699683b973eee76d4e91447d769a0cb0d/repl/src/main/resources/fake_shell.py#L581]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
