massquantity edited a comment on issue #3018:
URL: https://github.com/apache/iceberg/issues/3018#issuecomment-904686910
However, there is one more little issue. Although now I can use `MERGE INTO`
in spark shell, spark SQL shell and pyspark shell, it doesn't work in normal
python shell. After installing pyspark from pip(`pip install pyspark`) and
starting a SparkSession with sql extensions, it raises errors on normal queries.
```python
spark = SparkSession
.builder
.config("spark.sql.extensions",
"org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
.getOrCreate()
df = spark.sql("SELECT * FROM lake_catalog.database.table")
```
```shell
---------------------------------------------------------------------------
Py4JJavaError Traceback (most recent call last)
<timed exec> in <module>
/usr/local/lib/python3.7/dist-packages/pyspark/sql/session.py in sql(self,
sqlQuery)
721 [Row(f1=1, f2='row1'), Row(f1=2, f2='row2'), Row(f1=3,
f2='row3')]
722 """
--> 723 return DataFrame(self._jsparkSession.sql(sqlQuery),
self._wrapped)
724
725 def table(self, tableName):
/usr/local/lib/python3.7/dist-packages/py4j/java_gateway.py in
__call__(self, *args)
1303 answer = self.gateway_client.send_command(command)
1304 return_value = get_return_value(
-> 1305 answer, self.gateway_client, self.target_id, self.name)
1306
1307 for temp_arg in temp_args:
/usr/local/lib/python3.7/dist-packages/pyspark/sql/utils.py in deco(*a, **kw)
109 def deco(*a, **kw):
110 try:
--> 111 return f(*a, **kw)
112 except py4j.protocol.Py4JJavaError as e:
113 converted = convert_exception(e.java_exception)
/usr/local/lib/python3.7/dist-packages/py4j/protocol.py in
get_return_value(answer, gateway_client, target_id, name)
326 raise Py4JJavaError(
327 "An error occurred while calling {0}{1}{2}.\n".
--> 328 format(target_id, ".", name), value)
329 else:
330 raise Py4JError(
Py4JJavaError: An error occurred while calling o79.sql.
: java.lang.NoSuchMethodError: 'void
org.apache.spark.sql.internal.VariableSubstitution.<init>(org.apache.spark.sql.internal.SQLConf)'
at
org.apache.spark.sql.catalyst.parser.extensions.IcebergSparkSqlExtensionsParser.substitutor$lzycompute(IcebergSparkSqlExtensionsParser.scala:39)
at
org.apache.spark.sql.catalyst.parser.extensions.IcebergSparkSqlExtensionsParser.substitutor(IcebergSparkSqlExtensionsParser.scala:39)
at
org.apache.spark.sql.catalyst.parser.extensions.IcebergSparkSqlExtensionsParser.parsePlan(IcebergSparkSqlExtensionsParser.scala:96)
at
org.apache.spark.sql.SparkSession.$anonfun$sql$2(SparkSession.scala:616)
at
org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
at
org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:616)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.base/java.lang.Thread.run(Unknown Source)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
