[
https://issues.apache.org/jira/browse/LIVY-863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17818208#comment-17818208
]
Larry McCay commented on LIVY-863:
----------------------------------
[~eyal] - I just spent a little time looking at this and I'm just not familiar
enough to provide guidance to [~Meister] in terms of how to test this in a
meaningful way that would ensure multiple spark versions are good with the
change.
Do you have any suggestions?
Anyone else?
> Missing JVM class imports for Spark3
> ------------------------------------
>
> Key: LIVY-863
> URL: https://issues.apache.org/jira/browse/LIVY-863
> Project: Livy
> Issue Type: Bug
> Components: REPL
> Reporter: Moritz
> Priority: Major
> Fix For: 0.9.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Livy creates a fake spark shell initiating its own java gateway. This gateway
> imports some java classes for the user which have diverged from the ones in
> Spark 3. Some imports are missing which lead to methods like
> `dataframe.explain()` to be broken:
> {code:java}
> result.explain(True)
> An error was encountered:
> 'JavaPackage' object is not callable
> Traceback (most recent call last):
> File "/srv/hops/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py",
> line 386, in explain
>
> print(self._sc._jvm.PythonSQLUtils.explainString(self._jdf.queryExecution(),
> explain_mode))
> TypeError: 'JavaPackage' object is not callable
> {code}
> Spark is importing:
> [https://github.com/apache/spark/blob/87bf6b0ea4ca0618c8604895d05037edce8b7cb0/python/pyspark/java_gateway.py#L153]
> Livy is currently importing:
> [https://github.com/apache/incubator-livy/blob/4d8a912699683b973eee76d4e91447d769a0cb0d/repl/src/main/resources/fake_shell.py#L581]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
