cccs-dm opened a new issue #1560:
URL: https://github.com/apache/iceberg/issues/1560
Hello!
It seems there's a bug trying to insert data into a table containing a Spark
StructField.
I can create a table without any issues:
```
sc = spark.sparkContext
df = SQLContext(sc).range(0, 1000)
df = df.withColumn("MARK", F.struct(F.lit(7)))
df.write.option("path", outputPath).format("iceberg").saveAsTable(tableName)
```
But when I try to insert more data in this table, I get an exception:
`df.write.mode("overwrite").insertInto(tableName)`
---------------------------------------------------------------------------
AnalysisException Traceback (most recent call last)
<ipython-input-16-03570806093c> in <module>
----> 1 df.write.mode("overwrite").insertInto(tableName)
/usr/local/spark/python/pyspark/sql/readwriter.py in insertInto(self,
tableName, overwrite)
838 if overwrite is not None:
839 self.mode("overwrite" if overwrite else "append")
--> 840 self._jwrite.insertInto(tableName)
841
842 @since(1.4)
/usr/local/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py in
__call__(self, *args)
1302
1303 answer = self.gateway_client.send_command(command)
-> 1304 return_value = get_return_value(
1305 answer, self.gateway_client, self.target_id, self.name)
1306
/usr/local/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
135 # Hide where the exception came from that shows a
non-Pythonic
136 # JVM exception message.
--> 137 raise_from(converted)
138 else:
139 raise
/usr/local/spark/python/pyspark/sql/utils.py in raise_from(e)
AnalysisException: unresolved operator 'OverwriteByExpression
RelationV2[id#42L, MARK#43] spark_catalog.default.iceberg_spark_bug, true,
false;;
'OverwriteByExpression RelationV2[id#42L, MARK#43]
spark_catalog.default.iceberg_spark_bug, true, false
+- Project [id#31L, struct(col1, 7) AS MARK#33]
+- Range (0, 1000, step=1, splits=Some(12))
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
