[
https://issues.apache.org/jira/browse/LIVY-1010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mnr Bsf updated LIVY-1010:
--------------------------
Description:
It will be good to keep the Apache Livy project up to date with the latest
Spark version (3.5.4).
Tried Apache Livy 0.8 for Spark 3.5.4 and did not work as Spark version 3.0.0
with Java 11, 17.
Java 17, I was not able to start an Apache Livy session
{code:java}
Exception in thread "main" java.util.concurrent.ExecutionException:
javax.security.sasl.SaslException: Client closed before SASL negotiation
finished {code}
Java 11, I was able to start an Apache Livy session but running any sample code
was throwing this exception
{code:java}
'JavaPackage' object is not callable
Traceback (most recent call last):
File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/session.py", line 1443,
in createDataFrame
return self._create_dataframe(
File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/session.py", line 1485,
in _create_dataframe
rdd, struct = self._createFromLocal(map(prepare, data), schema)
File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/session.py", line 1093,
in _createFromLocal
struct = self._inferSchemaFromList(data, names=schema)
File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/session.py", line 954, in
_inferSchemaFromList
prefer_timestamp_ntz = is_timestamp_ntz_preferred()
File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 153, in
is_timestamp_ntz_preferred
return jvm is not None and jvm.PythonSQLUtils.isTimestampNTZPreferred()
TypeError: 'JavaPackage' object is not callable {code}
The only way to make Spark 3.5.4 working with Java 11 is to issue some
java_imports when a session starts
{code:java}
from py4j.java_gateway import java_import
java_import(spark._sc._jvm, "org.apache.spark.SparkConf")
java_import(spark._sc._jvm, "org.apache.spark.api.java.*")
java_import(spark._sc._jvm, "org.apache.spark.api.python.*")
java_import(spark._sc._jvm, "org.apache.spark.ml.python.*")
java_import(spark._sc._jvm, "org.apache.spark.mllib.api.python.*")
java_import(spark._sc._jvm, "org.apache.spark.resource.*")
java_import(spark._sc._jvm, "org.apache.spark.sql.*")
java_import(spark._sc._jvm, "org.apache.spark.sql.api.python.*")
java_import(spark._sc._jvm, "org.apache.spark.sql.hive.*") {code}
was:
It will be good to keep the Apache Livy project up to date with the latest
Spark version (3.5.4).
Tried Apache Livy 0.8 for Spark 3.5.4 and did not work as Spark version 3.0.0
with Java 11, 17.
Java 17, I was not able to start an Apache Livy session
{code:java}
Exception in thread "main" java.util.concurrent.ExecutionException:
javax.security.sasl.SaslException: Client closed before SASL negotiation
finished {code}
Java 11, I was able to start an Apache Livy session but running alivy-server |
25/02/05 17:08:27 INFO LineBufferedStream: Type --help for more information.
livy-server | 25/02/05 17:08:27 WARN LivySparkUtils$: Current Spark (3,5) is
not verified in Livy, please use it carefully livy-server | 25/02/05 17:08:27
WARN LivySparkUtils$: Spark version (3,5) is greater then the maximum version
(3,0) supported by Livy, will choose Scala version 2.12 instead, please specify
manually if it is the expected Scala version you want livy-server | Exception
in thread "main" java.lang.ExceptionInInitializerError livy-server | at
org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80) livy-server |
at
org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
livy-server | at
org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
livy-server | at
org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261)
livy-server | at
org.apache.hadoop.security.UserGroupInformation.isAuthenticationMethodEnabled(UserGroupInformation.java:338)
livy-server | at
org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:332)
livy-server | at
org.apache.livy.server.LivyServer.start(LivyServer.scala:106) livy-server |
at org.apache.livy.server.LivyServer$.main(LivyServer.scala:431) livy-server |
at org.apache.livy.server.LivyServer.main(LivyServer.scala) livy-server |
Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length 2
livy-server | at
java.base/java.lang.String.checkBoundsBeginEnd(String.java:3319) livy-server |
at java.base/java.lang.String.substring(String.java:1874) livy-server | at
org.apache.hadoop.util.Shell.<clinit>(Shell.java:52) livy-server | ... 9 more
> Add support for Spark 3.5.4
> ---------------------------
>
> Key: LIVY-1010
> URL: https://issues.apache.org/jira/browse/LIVY-1010
> Project: Livy
> Issue Type: Improvement
> Affects Versions: 0.8.0
> Reporter: Mnr Bsf
> Priority: Major
> Fix For: 0.8.0
>
>
> It will be good to keep the Apache Livy project up to date with the latest
> Spark version (3.5.4).
> Tried Apache Livy 0.8 for Spark 3.5.4 and did not work as Spark version 3.0.0
> with Java 11, 17.
> Java 17, I was not able to start an Apache Livy session
> {code:java}
> Exception in thread "main" java.util.concurrent.ExecutionException:
> javax.security.sasl.SaslException: Client closed before SASL negotiation
> finished {code}
> Java 11, I was able to start an Apache Livy session but running any sample
> code was throwing this exception
> {code:java}
> 'JavaPackage' object is not callable
> Traceback (most recent call last):
> File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/session.py", line 1443,
> in createDataFrame
> return self._create_dataframe(
> File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/session.py", line 1485,
> in _create_dataframe
> rdd, struct = self._createFromLocal(map(prepare, data), schema)
> File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/session.py", line 1093,
> in _createFromLocal
> struct = self._inferSchemaFromList(data, names=schema)
> File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/session.py", line 954,
> in _inferSchemaFromList
> prefer_timestamp_ntz = is_timestamp_ntz_preferred()
> File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 153, in
> is_timestamp_ntz_preferred
> return jvm is not None and jvm.PythonSQLUtils.isTimestampNTZPreferred()
> TypeError: 'JavaPackage' object is not callable {code}
> The only way to make Spark 3.5.4 working with Java 11 is to issue some
> java_imports when a session starts
> {code:java}
> from py4j.java_gateway import java_import
> java_import(spark._sc._jvm, "org.apache.spark.SparkConf")
> java_import(spark._sc._jvm, "org.apache.spark.api.java.*")
> java_import(spark._sc._jvm, "org.apache.spark.api.python.*")
> java_import(spark._sc._jvm, "org.apache.spark.ml.python.*")
> java_import(spark._sc._jvm, "org.apache.spark.mllib.api.python.*")
> java_import(spark._sc._jvm, "org.apache.spark.resource.*")
> java_import(spark._sc._jvm, "org.apache.spark.sql.*")
> java_import(spark._sc._jvm, "org.apache.spark.sql.api.python.*")
> java_import(spark._sc._jvm, "org.apache.spark.sql.hive.*") {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
