[
https://issues.apache.org/jira/browse/LIVY-863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17816914#comment-17816914
]
Eyal Allweil commented on LIVY-863:
-----------------------------------
Hi [~lmccay] [~Meister] - is there any update with this? Any chance of
pushing it to a "small" 0.8.1 release? Or is 0.9.0 expected to come out soon?
(we've also encountered the problem it solves)
> Missing JVM class imports for Spark3
> ------------------------------------
>
> Key: LIVY-863
> URL: https://issues.apache.org/jira/browse/LIVY-863
> Project: Livy
> Issue Type: Bug
> Components: REPL
> Reporter: Moritz
> Priority: Major
> Fix For: 0.9.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Livy creates a fake spark shell initiating its own java gateway. This gateway
> imports some java classes for the user which have diverged from the ones in
> Spark 3. Some imports are missing which lead to methods like
> `dataframe.explain()` to be broken:
> {code:java}
> result.explain(True)
> An error was encountered:
> 'JavaPackage' object is not callable
> Traceback (most recent call last):
> File "/srv/hops/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py",
> line 386, in explain
>
> print(self._sc._jvm.PythonSQLUtils.explainString(self._jdf.queryExecution(),
> explain_mode))
> TypeError: 'JavaPackage' object is not callable
> {code}
> Spark is importing:
> [https://github.com/apache/spark/blob/87bf6b0ea4ca0618c8604895d05037edce8b7cb0/python/pyspark/java_gateway.py#L153]
> Livy is currently importing:
> [https://github.com/apache/incubator-livy/blob/4d8a912699683b973eee76d4e91447d769a0cb0d/repl/src/main/resources/fake_shell.py#L581]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
