[
https://issues.apache.org/jira/browse/LIVY-863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17366469#comment-17366469
]
Moritz commented on LIVY-863:
-----------------------------
I proposed a fix here, it's missing a test and potentially needs some code to
account for different spark versions.
If you can help me with this and give some guidance, that would be great and I
can add the additional things :)
https://github.com/apache/incubator-livy/pull/322
> Missing JVM class imports for Spark3
> ------------------------------------
>
> Key: LIVY-863
> URL: https://issues.apache.org/jira/browse/LIVY-863
> Project: Livy
> Issue Type: Bug
> Components: REPL
> Reporter: Moritz
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Livy creates a fake spark shell initiating its own java gateway. This gateway
> imports some java classes for the user which have diverged from the ones in
> Spark 3. Some imports are missing which lead to methods like
> `dataframe.explain()` to be broken:
> {code:java}
> result.explain(True)
> An error was encountered:
> 'JavaPackage' object is not callable
> Traceback (most recent call last):
> File "/srv/hops/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py",
> line 386, in explain
>
> print(self._sc._jvm.PythonSQLUtils.explainString(self._jdf.queryExecution(),
> explain_mode))
> TypeError: 'JavaPackage' object is not callable
> {code}
> Spark is importing:
> [https://github.com/apache/spark/blob/87bf6b0ea4ca0618c8604895d05037edce8b7cb0/python/pyspark/java_gateway.py#L153]
> Livy is currently importing:
> [https://github.com/apache/incubator-livy/blob/4d8a912699683b973eee76d4e91447d769a0cb0d/repl/src/main/resources/fake_shell.py#L581]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
