abjain123 opened a new issue, #18386:
URL: https://github.com/apache/iceberg/issues/18386
### Apache Iceberg version
1.11.0
### Query engine
Spark
### Please describe the bug 🐞
In Apache Iceberg v3, creating a table with a VARIANT column using ORC as
the default write format ('write.format.default' = 'orc') succeeds during DDL
creation without validation warnings. However, executing an INSERT INTO
statement on the table aborts the Spark job due to an unhandled
NullPointerException on executor workers.
While VARIANT data type writes function as expected for Parquet-backed
Iceberg tables, Iceberg's SparkOrcWriter lacks type mapping logic for VARIANT.
During ORC schema visitor traversal (OrcSchemaWithTypeVisitor.visitVariant),
primitive type resolution returns null, causing a task crash.
**Environment**
```
Engine: Apache Spark 4.1
Catalog / Format: Apache Iceberg v3 (REST Catalog / Polaris)
Write Format: ORC ('write.format.default' = 'orc')
```
**Steps to reproduce:**
```
-- 1. Create an Iceberg v3 table with ORC write format
CREATE TABLE ds2_standalone.iceberg_v3_qe.test_variant_orc (
id INT,
payload VARIANT
)
USING iceberg
TBLPROPERTIES (
'format-version' = '3',
'write.format.default' = 'orc'
);
-- 2. Attempt to insert VARIANT data into the ORC table
INSERT INTO ds2_standalone.iceberg_v3_qe.test_variant_orc
VALUES (1, parse_json('{"key": "value"}'));
```
**Actual Behavior**
CREATE TABLE succeeds, but INSERT INTO fails with a stage failure due to a
repeating NullPointerException on all task retries:
```
26/10/05 16:48:15 INFO TaskSchedulerImpl: Killing all running tasks in stage
0: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most
recent failure: Lost task 0.3 in stage 0.0 (TID 3) (10.233.121.170 executor 2):
java.lang.NullPointerException: Cannot invoke
"org.apache.iceberg.types.Type$PrimitiveType.typeId()" because "iPrimitive" is
null
at
org.apache.iceberg.spark.data.SparkOrcWriter$WriteBuilder.primitive(SparkOrcWriter.java:116)
at
org.apache.iceberg.spark.data.SparkOrcWriter$WriteBuilder.primitive(SparkOrcWriter.java:73)
at
org.apache.iceberg.orc.OrcSchemaWithTypeVisitor.visit(OrcSchemaWithTypeVisitor.java:66)
at
org.apache.iceberg.orc.OrcSchemaWithTypeVisitor.visitVariant(OrcSchemaWithTypeVisitor.java:94)
at
org.apache.iceberg.orc.OrcSchemaWithTypeVisitor.visit(OrcSchemaWithTypeVisitor.java:42)
at
org.apache.iceberg.orc.OrcSchemaWithTypeVisitor.visitRecord(OrcSchemaWithTypeVisitor.java:78)
at
org.apache.iceberg.orc.OrcSchemaWithTypeVisitor.visit(OrcSchemaWithTypeVisitor.java:44)
at
org.apache.iceberg.orc.OrcSchemaWithTypeVisitor.visit(OrcSchemaWithTypeVisitor.java:33)
at
org.apache.iceberg.spark.data.SparkOrcWriter.<init>(SparkOrcWriter.java:54)
at
org.apache.iceberg.spark.source.SparkFormatModels.lambda$register$4(SparkFormatModels.java:71)
at
org.apache.iceberg.orc.ORCFormatModel$WriteBuilderWrapper.lambda$build$0(ORCFormatModel.java:175)
at
org.apache.iceberg.orc.OrcFileAppender.newOrcRowWriter(OrcFileAppender.java:178)
at
org.apache.iceberg.orc.OrcFileAppender.<init>(OrcFileAppender.java:93)
at org.apache.iceberg.orc.ORC$WriteBuilder.build(ORC.java:222)
at
org.apache.iceberg.orc.ORCFormatModel$WriteBuilderWrapper.build(ORCFormatModel.java:205)
at
org.apache.iceberg.formats.FileWriterBuilderImpl$DataFileWriterBuilder.build(FileWriterBuilderImpl.java:231)
at
org.apache.iceberg.formats.FileWriterBuilderImpl$DataFileWriterBuilder.build(FileWriterBuilderImpl.java:219)
at
org.apache.iceberg.data.RegistryBasedFileWriterFactory.newDataWriter(RegistryBasedFileWriterFactory.java:121)
at
org.apache.iceberg.io.RollingDataWriter.newWriter(RollingDataWriter.java:52)
at
org.apache.iceberg.io.RollingDataWriter.newWriter(RollingDataWriter.java:32)
at
org.apache.iceberg.io.RollingFileWriter.openCurrentWriter(RollingFileWriter.java:112)
at
org.apache.iceberg.io.RollingDataWriter.<init>(RollingDataWriter.java:47)
at
org.apache.iceberg.spark.source.SparkWrite$UnpartitionedDataWriter.<init>(SparkWrite.java:796)
at
org.apache.iceberg.spark.source.SparkWrite$WriterFactory.createWriter(SparkWrite.java:766)
at
org.apache.iceberg.spark.source.SparkWrite$WriterFactory.createWriter(SparkWrite.java:740)
at
org.apache.spark.sql.execution.datasources.v2.WritingSparkTask.run(WriteToDataSourceV2Exec.scala:524)
at
org.apache.spark.sql.execution.datasources.v2.WritingSparkTask.run$(WriteToDataSourceV2Exec.scala:513)
at
org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:631)
at
org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:454)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
at
org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:180)
at org.apache.spark.scheduler.Task.run(Task.scala:147)
at
org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$5(Executor.scala:716)
at
org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:86)
at
org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:83)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:97)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:719)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Driver stacktrace:
26/10/05 16:48:15 INFO DAGScheduler: ResultStage 0 (sql at
NativeMethodAccessorImpl.java:0) failed in 4405 ms due to Job aborted due to
stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost
task 0.3 in stage 0.0 (TID 3) (10.233.121.170 executor 2):
java.lang.NullPointerException: Cannot invoke
"org.apache.iceberg.types.Type$PrimitiveType.typeId()" because "iPrimitive" is
null
at
org.apache.iceberg.spark.data.SparkOrcWriter$WriteBuilder.primitive(SparkOrcWriter.java:116)
at
org.apache.iceberg.spark.data.SparkOrcWriter$WriteBuilder.primitive(SparkOrcWriter.java:73)
at
org.apache.iceberg.orc.OrcSchemaWithTypeVisitor.visit(OrcSchemaWithTypeVisitor.java:66)
at
org.apache.iceberg.orc.OrcSchemaWithTypeVisitor.visitVariant(OrcSchemaWithTypeVisitor.java:94)
at
org.apache.iceberg.orc.OrcSchemaWithTypeVisitor.visit(OrcSchemaWithTypeVisitor.java:42)
at
org.apache.iceberg.orc.OrcSchemaWithTypeVisitor.visitRecord(OrcSchemaWithTypeVisitor.java:78)
at
org.apache.iceberg.orc.OrcSchemaWithTypeVisitor.visit(OrcSchemaWithTypeVisitor.java:44)
at
org.apache.iceberg.orc.OrcSchemaWithTypeVisitor.visit(OrcSchemaWithTypeVisitor.java:33)
at
org.apache.iceberg.spark.data.SparkOrcWriter.<init>(SparkOrcWriter.java:54)
at
org.apache.iceberg.spark.source.SparkFormatModels.lambda$register$4(SparkFormatModels.java:71)
at
org.apache.iceberg.orc.ORCFormatModel$WriteBuilderWrapper.lambda$build$0(ORCFormatModel.java:175)
at
org.apache.iceberg.orc.OrcFileAppender.newOrcRowWriter(OrcFileAppender.java:178)
at
org.apache.iceberg.orc.OrcFileAppender.<init>(OrcFileAppender.java:93)
at org.apache.iceberg.orc.ORC$WriteBuilder.build(ORC.java:222)
at
org.apache.iceberg.orc.ORCFormatModel$WriteBuilderWrapper.build(ORCFormatModel.java:205)
at
org.apache.iceberg.formats.FileWriterBuilderImpl$DataFileWriterBuilder.build(FileWriterBuilderImpl.java:231)
at
org.apache.iceberg.formats.FileWriterBuilderImpl$DataFileWriterBuilder.build(FileWriterBuilderImpl.java:219)
at
org.apache.iceberg.data.RegistryBasedFileWriterFactory.newDataWriter(RegistryBasedFileWriterFactory.java:121)
at
org.apache.iceberg.io.RollingDataWriter.newWriter(RollingDataWriter.java:52)
at
org.apache.iceberg.io.RollingDataWriter.newWriter(RollingDataWriter.java:32)
at
org.apache.iceberg.io.RollingFileWriter.openCurrentWriter(RollingFileWriter.java:112)
at
org.apache.iceberg.io.RollingDataWriter.<init>(RollingDataWriter.java:47)
at
org.apache.iceberg.spark.source.SparkWrite$UnpartitionedDataWriter.<init>(SparkWrite.java:796)
at
org.apache.iceberg.spark.source.SparkWrite$WriterFactory.createWriter(SparkWrite.java:766)
at
org.apache.iceberg.spark.source.SparkWrite$WriterFactory.createWriter(SparkWrite.java:740)
at
org.apache.spark.sql.execution.datasources.v2.WritingSparkTask.run(WriteToDataSourceV2Exec.scala:524)
at
org.apache.spark.sql.execution.datasources.v2.WritingSparkTask.run$(WriteToDataSourceV2Exec.scala:513)
at
org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:631)
at
org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:454)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
at
org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:180)
at org.apache.spark.scheduler.Task.run(Task.scala:147)
at
org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$5(Executor.scala:716)
at
org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:86)
at
org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:83)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:97)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:719)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
```
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [x] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
