vikrambohra commented on issue #2456:
URL: https://github.com/apache/iceberg/issues/2456#issuecomment-1049231490
@rdblue
I am using spark version: 2.3.0 and iceberg-spark: 0.11.0.14
The code for nullability check during write is in IcebergSource
specifically here:
public Optional<DataSourceWriter> createWriter(String jobId, StructType
dsStruct, SaveMode mode, DataSourceOptions options) {
...
Schema writeSchema = SparkSchemaUtil.convert(table.schema(), dsStruct);
TypeUtil.validateWriteSchema(table.schema(), writeSchema,
this.checkNullability(options), this.checkOrdering(options));
I tried disabling the nullability check in the dataframewriter options but
that only worked for non-list and non-map columns of the schema. This is
described in https://github.com/apache/iceberg/issues/510
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
