mfellner opened a new issue, #3404: URL: https://github.com/apache/iceberg-rust/issues/3404
Related to #2411 (v3 tracking). Related: #3387 (same date-to-timestamp promotion, manifest bounds only), #3378 (same projection path, different trigger), #3374 (same projection-mask mechanism, name-mapping trigger). ## Apache Iceberg Rust version main at 9092073f9 (crate `iceberg` 0.10.1; arrow/parquet 59.3.0 per Cargo.lock). The code links below are permalinks at a76678363, where the same behaviour was first reproduced. ## Describe the bug Format v3 allows promoting `date` to `timestamp` and `timestamp_ns` (Iceberg spec, Schema Evolution, valid primitive type promotions: https://iceberg.apache.org/spec/). When a data file stores a column as `date` and the table schema declares it `timestamp`, the read returns NULL for every row if another column is projected in the same read. Projecting only the promoted column returns the correct values. The cause is in `get_arrow_projection_mask_with_field_ids`. `type_promotion_is_valid` ([projection.rs#L99-L121](https://github.com/apache/iceberg-rust/blob/a76678363835a3035c38b9ab53785e9d9b762d24/crates/iceberg/src/arrow/reader/projection.rs#L99-L121)) has no arm for `Date` to `Timestamp`/`TimestampNs`, so the Parquet leaf is left out of the projection mask ([projection.rs#L208](https://github.com/apache/iceberg-rust/blob/a76678363835a3035c38b9ab53785e9d9b762d24/crates/iceberg/src/arrow/reader/projection.rs#L208)). The reader then treats the column as absent from the file, and `RecordBatchTransformer` fills it with NULL, or with `initial-default` if one is set ([record_batch_transformer.rs#L818-L870](https://github.com/apache/iceberg-rust/blob/a76678363835a3035c38b9ab53785e9d9b762d24/crates/iceberg/src/arrow/record_batch_transformer.rs#L818-L870)). For a required column the read fails with `Missing required field`. Projecting only the promoted column works because no leaf matches , so the mask falls back to `ProjectionMask::all()` ([projection.rs#L231-L234](https://github.com/apache/iceberg-rust/blob/a76678363835a3035c38b9ab53785e9d9b762d24/crates/iceberg/src/arrow/reader/projection.rs#L231-L234)) and the cast in `Promote` runs. Verified locally on a76678363: adding `(Some(PrimitiveType::Date), Some(PrimitiveType::Timestamp | PrimitiveType::TimestampNs)) => true` to `type_promotion_is_valid` makes the test below pass, and `cargo test -p iceberg --lib` passes with that change. Tests in other crates were not run. ## To Reproduce Add this test at the end of `mod tests` in `crates/iceberg/src/arrow/reader/projection.rs`, then run: ``` cargo test -p iceberg --lib test_date_column_promoted_to_timestamp_with_other_column_projected ``` ```rust /// A Date32 column written to Parquet must read back as the promoted timestamp type /// when another column is projected alongside it. #[tokio::test] async fn test_date_column_promoted_to_timestamp_with_other_column_projected() { use arrow_array::types::TimestampMicrosecondType; use arrow_array::{Date32Array, Int32Array, TimestampMicrosecondArray}; let table_schema = Arc::new( Schema::builder() .with_schema_id(2) .with_fields(vec![ NestedField::optional(1, "d", Type::Primitive(PrimitiveType::Timestamp)).into(), NestedField::optional(2, "a", Type::Primitive(PrimitiveType::Int)).into(), ]) .build() .unwrap(), ); let file_schema = Arc::new(ArrowSchema::new(vec![ Field::new("d", DataType::Date32, true).with_metadata(HashMap::from([( PARQUET_FIELD_ID_META_KEY.to_string(), "1".to_string(), )])), Field::new("a", DataType::Int32, true).with_metadata(HashMap::from([( PARQUET_FIELD_ID_META_KEY.to_string(), "2".to_string(), )])), ])); let days = vec![18000, 18001, 18002]; let to_write = RecordBatch::try_new(file_schema.clone(), vec![ Arc::new(Date32Array::from(days.clone())) as ArrayRef, Arc::new(Int32Array::from(vec![1, 2, 3])) as ArrayRef, ]) .unwrap(); let tmp_dir = TempDir::new().unwrap(); let path = format!("{}/date_file.parquet", tmp_dir.path().to_str().unwrap()); let file = File::create(&path).unwrap(); let mut writer = ArrowWriter::try_new(file, file_schema, None).unwrap(); writer.write(&to_write).unwrap(); writer.close().unwrap(); let reader = ArrowReaderBuilder::new(FileIO::new_with_fs(), Runtime::current()).build(); let tasks = Box::pin(futures::stream::iter( vec![Ok(FileScanTask::builder() .with_file_size_in_bytes(std::fs::metadata(&path).unwrap().len()) .with_start(0) .with_length(0) .with_data_file_path(path.clone()) .with_data_file_format(DataFileFormat::Parquet) .with_schema(table_schema) .with_project_field_ids(vec![1, 2]) .with_case_sensitive(false) .build() .unwrap())] .into_iter(), )) as FileScanTaskStream; let batches = reader .read(tasks) .unwrap() .stream() .try_collect::<Vec<RecordBatch>>() .await .unwrap(); let expected = TimestampMicrosecondArray::from( days.iter() .map(|d| i64::from(*d) * 86_400_000_000) .collect::<Vec<_>>(), ); assert_eq!(batches.len(), 1); assert_eq!( batches[0].column(0).as_primitive::<TimestampMicrosecondType>(), &expected ); } ``` Observed output on main 9092073f9 (macOS arm64): ``` thread 'arrow::reader::projection::tests::test_date_column_promoted_to_timestamp_with_other_column_projected' panicked at crates/iceberg/src/arrow/reader/projection.rs:2246:9: assertion `left == right` failed left: PrimitiveArray<Timestamp(µs)> [ null, null, null, ] right: PrimitiveArray<Timestamp(µs)> [ 2019-04-14T00:00:00, 2019-04-15T00:00:00, 2019-04-16T00:00:00, ] ``` Probe results on a76678363 (3-row file, `d` is `date`, `a` is `int`; these probes are not part of the repro): | Table schema for `d` | Projection | Result | |---|---|---| | optional `timestamp` | `d` only | correct values | | optional `timestamp` | `d`, `a` | NULL for every row (above) | | required `timestamp` | `d`, `a` | `DataInvalid => Missing required field: d` | | optional `timestamp`, `initial-default` = 7 µs | `d`, `a` | `1970-01-01T00:00:00.000007` for every row; the file values are discarded | ## Out-of-range values (separate defect) The spec says values outside the promoted type's range "must result in a runtime failure". The read currently returns NULL instead. This is a separate defect in the `Promote` cast, and the mask fix above does not change it. With a 2-row file holding `date` values 18000 and 200000 days since epoch, `d` as optional `timestamp_ns`, and only `d` projected: ``` [ 2019-04-14T00:00:00, null, ] ``` With `d` and `a` projected, both rows are NULL. A required `timestamp_ns` column with the value 200000 days fails with `Arrow Schema Error ... Column 'd' is declared as non-nullable but contains null values`. The same value in an optional `timestamp` (µs) column reads correctly as `2517-08-01T00:00:00`. The NULL comes from the `Promote` cast at [record_batch_transformer.rs#L936](https://github.com/apache/iceberg-rust/blob/a76678363835a3035c38b9ab53785e9d9b762d24/crates/iceberg/src/arrow/record_batch_transformer.rs#L936), `cast(&*columns[*source_index], target_type)`. That call uses arrow's default cast options, which are safe (`safe: true` in arrow-cast 59.3.0), so an overflowing value becomes NULL. 200000 days in nanoseconds is about 1.7e19, which is larger than `i64::MAX`. ## Expected behavior - Promoted values read back as the target type (days × 86,400,000,000 µs for `timestamp`, days × 86,400,000,000,000 ns for `timestamp_ns`), whether or not other columns are projected, as the single-column case already does. - A value outside the target type's range fails the read, as the spec requires. - A required promoted column is read, not reported as missing. - `initial-default` is not used when the file contains the column. ## Willingness to contribute I would be willing to contribute a fix with guidance from the Iceberg community (any contribution would be agent-assisted and I would disclose that). Found during an agent-assisted audit of v3 support (see #2411). The reproduction above was run against main at 9092073f9. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
