RussellSpitzer commented on issue #3018:
URL: https://github.com/apache/iceberg/issues/3018#issuecomment-904708016
Looks like a version mismatch with the Pyspark Spark version and the version
of the runtime extension
Sent from my iPhone
> On Aug 24, 2021, at 9:21 AM, massquantity ***@***.***> wrote:
>
>
> However, there is one more little issue. Although now I can use MERGE INTO
in spark shell, spark SQL shell and pyspark shell, it doesn't work in normal
python shell. After installing pyspark from pip(pip install pyspark) and
starting a SparkSession with sql extensions, it has a problem with normal
queries.
>
> spark = SparkSession
> .builder
> .config("spark.sql.extensions",
"org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
> .getOrCreate()
>
> df = spark.sql("SELECT * FROM lake_catalog.database.table")
> ---------------------------------------------------------------------------
> Py4JJavaError Traceback (most recent call last)
> <timed exec> in <module>
>
> /usr/local/lib/python3.7/dist-packages/pyspark/sql/session.py in sql(self,
sqlQuery)
> 721 [Row(f1=1, f2='row1'), Row(f1=2, f2='row2'), Row(f1=3,
f2='row3')]
> 722 """
> --> 723 return DataFrame(self._jsparkSession.sql(sqlQuery),
self._wrapped)
> 724
> 725 def table(self, tableName):
>
> /usr/local/lib/python3.7/dist-packages/py4j/java_gateway.py in
__call__(self, *args)
> 1303 answer = self.gateway_client.send_command(command)
> 1304 return_value = get_return_value(
> -> 1305 answer, self.gateway_client, self.target_id, self.name)
> 1306
> 1307 for temp_arg in temp_args:
>
> /usr/local/lib/python3.7/dist-packages/pyspark/sql/utils.py in deco(*a,
**kw)
> 109 def deco(*a, **kw):
> 110 try:
> --> 111 return f(*a, **kw)
> 112 except py4j.protocol.Py4JJavaError as e:
> 113 converted = convert_exception(e.java_exception)
>
> /usr/local/lib/python3.7/dist-packages/py4j/protocol.py in
get_return_value(answer, gateway_client, target_id, name)
> 326 raise Py4JJavaError(
> 327 "An error occurred while calling {0}{1}{2}.\n".
> --> 328 format(target_id, ".", name), value)
> 329 else:
> 330 raise Py4JError(
>
> Py4JJavaError: An error occurred while calling o79.sql.
> : java.lang.NoSuchMethodError: 'void
org.apache.spark.sql.internal.VariableSubstitution.<init>(org.apache.spark.sql.internal.SQLConf)'
> at
org.apache.spark.sql.catalyst.parser.extensions.IcebergSparkSqlExtensionsParser.substitutor$lzycompute(IcebergSparkSqlExtensionsParser.scala:39)
> at
org.apache.spark.sql.catalyst.parser.extensions.IcebergSparkSqlExtensionsParser.substitutor(IcebergSparkSqlExtensionsParser.scala:39)
> at
org.apache.spark.sql.catalyst.parser.extensions.IcebergSparkSqlExtensionsParser.parsePlan(IcebergSparkSqlExtensionsParser.scala:96)
> at
org.apache.spark.sql.SparkSession.$anonfun$sql$2(SparkSession.scala:616)
> at
org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
> at
org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:616)
> at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
> at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613)
> at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
Source)
> at java.base/java.lang.reflect.Method.invoke(Unknown Source)
> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
> at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
> at py4j.Gateway.invoke(Gateway.java:282)
> at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
> at py4j.commands.CallCommand.execute(CallCommand.java:79)
> at py4j.GatewayConnection.run(GatewayConnection.java:238)
> at java.base/java.lang.Thread.run(Unknown Source)
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
