sam-1112 opened a new issue, #3611: URL: https://github.com/apache/iceberg-rust/issues/3611
### Apache Iceberg Rust version v0.10.1 ### Describe the bug ArrowReader silently returns NULL for a non-null INT96 timestamp column when the Iceberg schema declares it as `timestamptz` and the projection includes other columns. Selecting only the timestamp column returns the correct values. The problem reproduces directly through iceberg-rust without Spark or Comet in the execution path. It was originally observed downstream in [apache/datafusion-comet#7212](https://github.com/apache/datafusion-comet/issues/7212). The tested Parquet file contains three columns: `id`, `ts`, and `s`. The timestamp column is physically encoded as INT96, the file has no embedded field IDs, and all physical values are non-null. A name mapping assigns field IDs 1, 2, and 3 respectively. The Iceberg schema declares ts as an optional `timestamptz` field. ### To Reproduce 1. Create a five-row Parquet file containing a 64-bit integer column named `id`, an INT96 timestamp column named `ts`, and a string column named s. Do not embed Iceberg field IDs. 2. Populate `id` with values 0 through 4 and s with their string representations. Populate `ts` with epoch microsecond values 0, 1000003, 2000006, 3000009, and 4000012. All values should be non-null. 3. Define an Iceberg schema with field IDs 1, 2, and 3 for these columns, declaring `ts` as optional `timestamptz`. 4. Configure a name mapping from the physical column names to those field IDs. 5. Read the same file through ArrowReader using three projections: timestamp only, all three columns, and all three columns with the timestamp first. 6. Compare the timestamp values across the three results. This behavior was reproduced with both a Spark-written INT96 file and a standalone parquet-rs-generated INT96 file. <html><head></head><body><p><strong>Actual behavior</strong></p> Projected field IDs | Selected columns | Timestamp epoch microseconds -- | -- | -- [2] | ts | [0, 1000003, 2000006, 3000009, 4000012] [1, 2, 3] | id, ts, s | Five NULLs [2, 1, 3] | ts, id, s | Five NULLs <p>In both multi-column results, <code>id</code> and <code>s</code> remain correct and have zero NULLs. The output timestamp column has Arrow type <code>Timestamp(Microsecond, Some("+00:00"))</code>, but its null count is five.</p> <p>No error is reported.</p> </body></html> ### Expected behavior All three projections should return the same five non-null timestamp values: epoch microseconds 0, 1000003, 2000006, 3000009, and 4000012. Selecting additional columns or changing their order should preserve the timestamp values. <html><head></head><body><p><strong>Diagnosis</strong></p> Tracing a logging-only copy of the tested reader revision showed the following: 1. The raw INT96 Arrow field is inferred as Timestamp(Nanosecond, None). INT96 coercion changes its resolution to microseconds while preserving timezone None and the mapped field ID. 2. Arrow-to-Iceberg schema conversion identifies this field as Timestamp, while the task schema requests Timestamptz. 3. The projection type compatibility check rejects this combination and excludes the timestamp column. 4. For the multi-column projection, the physical projection mask selects id and s while excluding ts. The parquet-rs batch therefore contains only id and s. 5. The batch transformer treats the timestamp as a missing optional field and fills it with NULL because it has no initial default. 6. When only ts is requested, no field passes the compatibility check. The empty-selection fallback returns a read-all projection mask, allowing the timestamp values to be read and subsequently cast successfully. This masks the problem in the timestamp-only case. <html><head></head><body><p><strong>Additional verification</strong></p> Reading the original Spark-written file directly with parquet-rs, using a microsecond timestamp schema hint without a timezone and selecting all columns, returns all five expected timestamp values. Earlier Spark integration comparisons also showed identical scan read schemas with Comet Native Iceberg Scan enabled and disabled. The standalone ArrowReader reproduction confirms that the NULLs can arise entirely within iceberg-rust. These observations point to the interaction between INT96 coercion and projection type reconciliation. ### Willingness to contribute None -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
