mbutrovich commented on code in PR #3632:
URL: https://github.com/apache/iceberg-rust/pull/3632#discussion_r4515807780
##########
crates/iceberg/src/arrow/int96.rs:
##########
@@ -44,66 +47,67 @@ use crate::{Error, ErrorKind};
pub(crate) fn coerce_int96_timestamps(
arrow_schema: &ArrowSchemaRef,
iceberg_schema: &Schema,
+ parquet_schema: &SchemaDescriptor,
) -> Option<Arc<ArrowSchema>> {
- let mut visitor = Int96CoercionVisitor::new(iceberg_schema);
+ if !parquet_schema
+ .columns()
+ .iter()
+ .any(|column| column.physical_type() == PhysicalType::INT96)
+ {
+ return None;
+ }
+ let mut visitor = Int96CoercionVisitor::new(iceberg_schema,
parquet_schema);
let coerced = visit_schema(arrow_schema, &mut visitor).ok()?;
- if visitor.changed {
+ if visitor.changed && visitor.columns.len() == 0 {
Some(Arc::new(coerced))
} else {
None
}
}
Review Comment:
If the visitor fails, or doesn't consume every Parquet column, this returns
`None` and the file is read without coercion. That brings back both bugs this
function exists to fix. Dates outside ~1677-2262 overflow, and a `timestamptz`
column fails the projection type check and reads as NULL (#3611).
One way to get here today is a file whose embedded `ARROW:schema` declares a
`ListView` column. arrow-rs [reads a Parquet list as
`ListView`](https://github.com/apache/arrow-rs/blob/9ea30dfacb7003de9ffc87ef50231f18229fcdb5/parquet/src/arrow/schema/complex.rs#L646-L647)
when the schema hint asks for it, and `visit_schema` [rejects
`ListView`](https://github.com/apache/iceberg-rust/blob/b63e83487d65581675a801139ec130060ea31580/crates/iceberg/src/arrow/schema.rs#L314).
I called `coerce_int96_timestamps` with a `ListView<Int32>` column ahead of an
INT96 column declared `timestamptz`, and it returned `None`.
Could this return a `Result` and fail when the Arrow leaves and Parquet
columns don't line up? `configure_arrow_reader_metadata` already returns
`Result`, so both callers only need a `?`. A file like the one above would then
fail to read instead of returning NULLs or overflowed values.
```suggestion
) -> Result<Option<Arc<ArrowSchema>>> {
if !parquet_schema
.columns()
.iter()
.any(|column| column.physical_type() == PhysicalType::INT96)
{
return Ok(None);
}
let mut visitor = Int96CoercionVisitor::new(iceberg_schema,
parquet_schema);
let coerced = visit_schema(arrow_schema, &mut visitor)?;
if visitor.columns.len() != 0 {
return Err(Error::new(
ErrorKind::Unexpected,
"Arrow schema has fewer primitive leaves than the Parquet schema
has columns",
));
}
Ok(visitor.changed.then(|| Arc::new(coerced)))
}
```
##########
crates/iceberg/src/arrow/reader/pipeline.rs:
##########
@@ -937,33 +943,99 @@ mod tests {
file_path: &str,
schema: SchemaRef,
project_field_ids: Vec<i32>,
+ ) -> Vec<RecordBatch> {
+ read_int96_batches_with_name_mapping(file_path, schema,
project_field_ids, None).await
+ }
+
+ async fn read_int96_batches_with_name_mapping(
+ file_path: &str,
+ schema: SchemaRef,
+ project_field_ids: Vec<i32>,
+ name_mapping: Option<Arc<NameMapping>>,
) -> Vec<RecordBatch> {
let file_io = FileIO::new_with_fs();
let reader = ArrowReaderBuilder::new(file_io,
Runtime::current()).build();
+ let task = int96_scan_task(file_path, schema, project_field_ids,
name_mapping);
+ let tasks = Box::pin(futures::stream::iter(vec![Ok(task)])) as
FileScanTaskStream;
+ reader
+ .read(tasks)
+ .unwrap()
+ .stream()
+ .try_collect()
+ .await
+ .unwrap()
+ }
+
+ fn int96_scan_task(
+ file_path: &str,
+ schema: SchemaRef,
+ project_field_ids: Vec<i32>,
+ name_mapping: Option<Arc<NameMapping>>,
+ ) -> FileScanTask {
let file_size = std::fs::metadata(file_path).unwrap().len();
- let task = FileScanTask::builder()
+ FileScanTask::builder()
.with_file_size_in_bytes(file_size)
.with_start(0)
.with_length(file_size)
.with_data_file_path(file_path.to_string())
.with_data_file_format(DataFileFormat::Parquet)
.with_schema(schema)
.with_project_field_ids(project_field_ids)
+ .with_name_mapping(name_mapping)
.with_case_sensitive(false)
.build()
- .unwrap();
-
- let tasks = Box::pin(futures::stream::iter(vec![Ok(task)])) as
FileScanTaskStream;
- reader
- .read(tasks)
- .unwrap()
- .stream()
- .try_collect()
- .await
.unwrap()
}
+ fn assert_int96_projection_mask(task: &FileScanTask, expected_indices:
&[usize]) {
+ use parquet::arrow::arrow_reader::{ArrowReaderMetadata,
ArrowReaderOptions};
+ use parquet::file::reader::{FileReader, SerializedFileReader};
+
+ use super::{ArrowReader, FileScanTaskReader};
+
+ let reader =
SerializedFileReader::new(File::open(task.data_file_path()).unwrap()).unwrap();
+ let metadata = ArrowReaderMetadata::try_new(
+ Arc::new(reader.metadata().clone()),
+ ArrowReaderOptions::default(),
+ )
+ .unwrap();
+ let missing_field_ids = metadata
+ .schema()
+ .fields()
+ .iter()
+ .next()
+ .is_some_and(|field|
!field.metadata().contains_key(PARQUET_FIELD_ID_META_KEY));
+ let use_position_fallback = missing_field_ids &&
task.name_mapping().is_none();
+ let metadata = FileScanTaskReader::configure_arrow_reader_metadata(
+ metadata,
+ task,
+ missing_field_ids,
+ false,
+ )
+ .unwrap();
+ let mask = ArrowReader::get_arrow_projection_mask(
+ task.project_field_ids(),
+ task.schema(),
+ metadata.parquet_schema(),
+ metadata.schema(),
+ use_position_fallback,
+ )
+ .unwrap();
+ let actual: Vec<bool> = (0..metadata.parquet_schema().num_columns())
+ .map(|index| mask.leaf_included(index))
+ .collect();
+ let expected: Vec<bool> = (0..metadata.parquet_schema().num_columns())
+ .map(|index| expected_indices.contains(&index))
+ .collect();
+ assert_eq!(
+ actual,
+ expected,
+ "projection {:?}",
+ task.project_field_ids()
+ );
+ }
Review Comment:
This helper copies the reader's [field-ID
detection](https://github.com/apache/iceberg-rust/blob/b63e83487d65581675a801139ec130060ea31580/crates/iceberg/src/arrow/reader/pipeline.rs#L192-L203)
and calls `get_arrow_projection_mask` directly. #3624 changes both: it detects
field IDs from the Parquet schema, and it changes the signature of
`get_arrow_projection_mask`. If the copy drifts from the reader, these
assertions check a mask the reader no longer builds.
The batch comparisons already catch #3611, because the NULLs show up in the
output. Could we drop the mask assertions and keep the batch comparisons?
##########
crates/iceberg/src/arrow/reader/pipeline.rs:
##########
@@ -2871,6 +2943,437 @@ mod tests {
assert_int96_read_matches(&file_path, schema, vec![1, 2],
&expected_micros).await;
}
+ #[tokio::test]
+ async fn test_read_int96_timestamptz_with_name_mapping() {
+ assert_int96_timestamp_projections(
+ PrimitiveType::Timestamptz,
+ Int96FieldIds::NameMapping,
+ false,
+ )
+ .await;
+ }
+
+ #[tokio::test]
+ async fn test_read_int96_timestamptz_ns_with_name_mapping() {
+ assert_int96_timestamp_projections(
+ PrimitiveType::TimestamptzNs,
+ Int96FieldIds::NameMapping,
+ false,
+ )
+ .await;
+ }
+
+ #[tokio::test]
+ async fn test_read_int96_timestamptz_with_field_ids_and_nulls() {
+ assert_int96_timestamp_projections(
+ PrimitiveType::Timestamptz,
+ Int96FieldIds::Embedded,
+ true,
+ )
+ .await;
+ }
+
+ #[tokio::test]
+ async fn test_read_int96_timestamptz_with_positional_fallback() {
+ assert_int96_timestamp_projections(
+ PrimitiveType::Timestamptz,
+ Int96FieldIds::Position,
+ false,
+ )
+ .await;
+ }
+
+ #[tokio::test]
+ async fn test_read_int96_timestamp_without_timezone_with_name_mapping() {
+ assert_int96_timestamp_projections(
+ PrimitiveType::Timestamp,
+ Int96FieldIds::NameMapping,
+ true,
+ )
+ .await;
+ }
+
+ #[tokio::test]
+ async fn test_read_int96_timestamp_ns_with_field_ids_and_nulls() {
+ assert_int96_timestamp_projections(
+ PrimitiveType::TimestampNs,
+ Int96FieldIds::Embedded,
+ true,
+ )
+ .await;
+ }
+
+ enum Int96FieldIds {
+ Embedded,
+ NameMapping,
+ Position,
+ }
+
+ async fn assert_int96_timestamp_projections(
+ timestamp_type: PrimitiveType,
+ field_ids: Int96FieldIds,
+ with_nulls: bool,
+ ) {
+ use arrow_array::{TimestampMicrosecondArray, TimestampNanosecondArray,
new_null_array};
+ use parquet::basic::{LogicalType, Repetition, Type as PhysicalType};
+ use parquet::data_type::{ByteArray, ByteArrayType, Int64Type, Int96,
Int96Type};
+ use parquet::file::writer::SerializedFileWriter;
+ use parquet::schema::types::Type as SchemaType;
+
+ use crate::arrow::schema::UTC_TIME_ZONE;
+ use crate::arrow::schema_to_arrow_schema;
+ use crate::spec::Literal;
+
+ let schema = Arc::new(
+ Schema::builder()
+ .with_fields(vec![
+ NestedField::required(1, "id",
Type::Primitive(PrimitiveType::Long)).into(),
+ NestedField::optional(2, "ts",
Type::Primitive(timestamp_type.clone())).into(),
+ NestedField::required(3, "s",
Type::Primitive(PrimitiveType::String)).into(),
+ NestedField::optional(4, "missing_ts",
Type::Primitive(timestamp_type.clone()))
+ .into(),
+ NestedField::optional(5, "default_s",
Type::Primitive(PrimitiveType::String))
+ .with_initial_default(Literal::string("default"))
+ .into(),
+ ])
+ .build()
+ .unwrap(),
+ );
+ let name_mapping = matches!(field_ids,
Int96FieldIds::NameMapping).then(|| {
+ Arc::new(NameMapping::new(vec![
+ MappedField::new(Some(1), vec!["id".to_string()], vec![]),
+ MappedField::new(Some(2), vec!["ts".to_string()], vec![]),
+ MappedField::new(Some(3), vec!["s".to_string()], vec![]),
+ ]))
+ });
+ let with_field_ids = matches!(field_ids, Int96FieldIds::Embedded);
+ let parquet_schema = SchemaType::group_type_builder("schema")
+ .with_fields(vec![
+ Arc::new(
+ SchemaType::primitive_type_builder("id",
PhysicalType::INT64)
+ .with_repetition(Repetition::REQUIRED)
+ .with_id(with_field_ids.then_some(1))
+ .build()
+ .unwrap(),
+ ),
+ Arc::new(
+ SchemaType::primitive_type_builder("ts",
PhysicalType::INT96)
+ .with_repetition(Repetition::OPTIONAL)
+ .with_id(with_field_ids.then_some(2))
+ .build()
+ .unwrap(),
+ ),
+ Arc::new(
+ SchemaType::primitive_type_builder("s",
PhysicalType::BYTE_ARRAY)
+ .with_repetition(Repetition::REQUIRED)
+ .with_logical_type(Some(LogicalType::String))
+ .with_id(with_field_ids.then_some(3))
+ .build()
+ .unwrap(),
+ ),
+ ])
+ .build()
+ .unwrap();
+ let micros = vec![0_i64, 1_000_003, 2_000_006, 3_000_009, 4_000_012];
+ let nanos: Vec<i64> = micros
+ .iter()
+ .map(|micros| {
+ micros * 1_000
+ + i64::from(matches!(
+ timestamp_type,
+ PrimitiveType::TimestampNs |
PrimitiveType::TimestamptzNs
+ ))
+ })
+ .collect();
+ let ids = vec![0_i64, 1, 2, 3, 4];
+ let strings = vec!["0", "1", "2", "3", "4"];
+ let definition_levels: Vec<i16> = (0..5)
+ .map(|index| i16::from(!with_nulls || index % 2 == 0))
+ .collect();
+ let int96_values: Vec<Int96> = nanos
+ .iter()
+ .zip(&definition_levels)
+ .filter(|(_, level)| **level == 1)
+ .map(|(nanos, _)| {
+ let mut value = Int96::new();
+ value.set_data(
+ *nanos as u32,
+ (nanos >> 32) as u32,
+ UNIX_EPOCH_JULIAN as u32,
+ );
+ value
+ })
+ .collect();
+ let string_values: Vec<ByteArray> = strings
+ .iter()
+ .map(|value| ByteArray::from(*value))
+ .collect();
+
+ let tmp_dir = TempDir::new().unwrap();
+ let file_path =
tmp_dir.path().join("timestamptz_name_mapping.parquet");
+ let mut writer = SerializedFileWriter::new(
+ File::create(&file_path).unwrap(),
+ Arc::new(parquet_schema),
+ Default::default(),
+ )
+ .unwrap();
+ let mut row_group = writer.next_row_group().unwrap();
+ let mut column = row_group.next_column().unwrap().unwrap();
+ column
+ .typed::<Int64Type>()
+ .write_batch(&ids, None, None)
+ .unwrap();
+ column.close().unwrap();
+ let mut column = row_group.next_column().unwrap().unwrap();
+ column
+ .typed::<Int96Type>()
+ .write_batch(&int96_values, Some(&definition_levels), None)
+ .unwrap();
+ column.close().unwrap();
+ let mut column = row_group.next_column().unwrap().unwrap();
+ column
+ .typed::<ByteArrayType>()
+ .write_batch(&string_values, None, None)
+ .unwrap();
+ column.close().unwrap();
+ row_group.close().unwrap();
+ writer.close().unwrap();
+
+ let timezone = matches!(
+ timestamp_type,
+ PrimitiveType::Timestamptz | PrimitiveType::TimestamptzNs
+ )
+ .then_some(UTC_TIME_ZONE);
+ let expected_timestamps: ArrayRef = match timestamp_type {
+ PrimitiveType::Timestamp | PrimitiveType::Timestamptz => {
+ let values: Vec<Option<i64>> = micros
+ .into_iter()
+ .zip(&definition_levels)
+ .map(|(value, level)| (*level == 1).then_some(value))
+ .collect();
+
Arc::new(TimestampMicrosecondArray::from(values).with_timezone_opt(timezone))
+ }
+ PrimitiveType::TimestampNs | PrimitiveType::TimestamptzNs => {
+ let values: Vec<Option<i64>> = nanos
+ .into_iter()
+ .zip(&definition_levels)
+ .map(|(value, level)| (*level == 1).then_some(value))
+ .collect();
+
Arc::new(TimestampNanosecondArray::from(values).with_timezone_opt(timezone))
+ }
+ other => panic!("Expected timestamp type, got {other}"),
+ };
+ let missing_timestamps =
new_null_array(expected_timestamps.data_type(), 5);
+ let expected_batch =
+
RecordBatch::try_new(Arc::new(schema_to_arrow_schema(&schema).unwrap()), vec![
+ Arc::new(Int64Array::from(ids)),
+ expected_timestamps,
+ Arc::new(StringArray::from(strings)),
+ missing_timestamps,
+ Arc::new(StringArray::from(vec!["default"; 5])),
+ ])
+ .unwrap();
+ let mut actual = Vec::new();
+ let mut expected = Vec::new();
+ for field_ids in [
+ vec![2],
+ vec![1, 2, 3],
+ vec![2, 1, 3],
+ vec![2, 4, 5, 1],
+ vec![4, 5],
+ ] {
+ let column_indices: Vec<usize> =
+ field_ids.iter().map(|id| (*id - 1) as usize).collect();
+
expected.push(vec![expected_batch.project(&column_indices).unwrap()]);
+ if field_ids.iter().any(|id| *id <= 3) {
+ let task = int96_scan_task(
+ file_path.to_str().unwrap(),
+ schema.clone(),
+ field_ids.clone(),
+ name_mapping.clone(),
+ );
+ let physical_indices: Vec<usize> = column_indices
+ .iter()
+ .copied()
+ .filter(|index| *index < 3)
+ .collect();
+ assert_int96_projection_mask(&task, &physical_indices);
+ }
+ actual.push(
+ read_int96_batches_with_name_mapping(
+ file_path.to_str().unwrap(),
+ schema.clone(),
+ field_ids,
+ name_mapping.clone(),
+ )
+ .await,
+ );
+ }
+ assert_eq!(actual, expected);
+ }
Review Comment:
This helper writes another INT96 file by hand next to
`write_int96_parquet_file`. It also mixes in cases that aren't about INT96: a
missing field, an initial default, column order, and a projection of only
missing fields. The transformer tests already cover defaults and missing
fields, for example
[`test_all_four_spec_rules`](https://github.com/apache/iceberg-rust/blob/b63e83487d65581675a801139ec130060ea31580/crates/iceberg/src/arrow/record_batch_transformer.rs#L3197).
`write_int96_parquet_file` already writes an INT96 `ts` next to an `id`,
with or without field IDs. Its year-3333 values also cover the overflow. I read
that file with `read_int96_batches_with_name_mapping`, with `ts` declared
`timestamptz` and projection `[1, 2]`, once with embedded IDs and once with a
name mapping. On `main` both read `ts` as NULL, and on this branch both match
the expected batch.
Could the regression tests use that file? One way is to take the Iceberg
timestamp type as a parameter in `test_read_int96_timestamps_with_field_ids`
and `test_read_int96_timestamps_without_field_ids`, and add a name-mapping
case. Then this helper only needs the nanosecond cases, which need values
inside the nanosecond range. For that smaller file, `parse_message_type` (as in
`test_read_int96_with_mixed_timestamp_types_in_struct`) is shorter than the
schema builders.
##########
crates/iceberg/src/arrow/reader/pipeline.rs:
##########
@@ -2871,6 +2943,437 @@ mod tests {
assert_int96_read_matches(&file_path, schema, vec![1, 2],
&expected_micros).await;
}
+ #[tokio::test]
+ async fn test_read_int96_timestamptz_with_name_mapping() {
+ assert_int96_timestamp_projections(
+ PrimitiveType::Timestamptz,
+ Int96FieldIds::NameMapping,
+ false,
+ )
+ .await;
+ }
+
+ #[tokio::test]
+ async fn test_read_int96_timestamptz_ns_with_name_mapping() {
+ assert_int96_timestamp_projections(
+ PrimitiveType::TimestamptzNs,
+ Int96FieldIds::NameMapping,
+ false,
+ )
+ .await;
+ }
+
+ #[tokio::test]
+ async fn test_read_int96_timestamptz_with_field_ids_and_nulls() {
+ assert_int96_timestamp_projections(
+ PrimitiveType::Timestamptz,
+ Int96FieldIds::Embedded,
+ true,
+ )
+ .await;
+ }
+
+ #[tokio::test]
+ async fn test_read_int96_timestamptz_with_positional_fallback() {
+ assert_int96_timestamp_projections(
+ PrimitiveType::Timestamptz,
+ Int96FieldIds::Position,
+ false,
+ )
+ .await;
+ }
+
+ #[tokio::test]
+ async fn test_read_int96_timestamp_without_timezone_with_name_mapping() {
+ assert_int96_timestamp_projections(
+ PrimitiveType::Timestamp,
+ Int96FieldIds::NameMapping,
+ true,
+ )
+ .await;
+ }
+
+ #[tokio::test]
+ async fn test_read_int96_timestamp_ns_with_field_ids_and_nulls() {
+ assert_int96_timestamp_projections(
+ PrimitiveType::TimestampNs,
+ Int96FieldIds::Embedded,
+ true,
+ )
+ .await;
+ }
+
+ enum Int96FieldIds {
+ Embedded,
+ NameMapping,
+ Position,
+ }
+
+ async fn assert_int96_timestamp_projections(
+ timestamp_type: PrimitiveType,
+ field_ids: Int96FieldIds,
+ with_nulls: bool,
+ ) {
+ use arrow_array::{TimestampMicrosecondArray, TimestampNanosecondArray,
new_null_array};
+ use parquet::basic::{LogicalType, Repetition, Type as PhysicalType};
+ use parquet::data_type::{ByteArray, ByteArrayType, Int64Type, Int96,
Int96Type};
+ use parquet::file::writer::SerializedFileWriter;
+ use parquet::schema::types::Type as SchemaType;
+
+ use crate::arrow::schema::UTC_TIME_ZONE;
+ use crate::arrow::schema_to_arrow_schema;
+ use crate::spec::Literal;
+
+ let schema = Arc::new(
+ Schema::builder()
+ .with_fields(vec![
+ NestedField::required(1, "id",
Type::Primitive(PrimitiveType::Long)).into(),
+ NestedField::optional(2, "ts",
Type::Primitive(timestamp_type.clone())).into(),
+ NestedField::required(3, "s",
Type::Primitive(PrimitiveType::String)).into(),
+ NestedField::optional(4, "missing_ts",
Type::Primitive(timestamp_type.clone()))
+ .into(),
+ NestedField::optional(5, "default_s",
Type::Primitive(PrimitiveType::String))
+ .with_initial_default(Literal::string("default"))
+ .into(),
+ ])
+ .build()
+ .unwrap(),
+ );
+ let name_mapping = matches!(field_ids,
Int96FieldIds::NameMapping).then(|| {
+ Arc::new(NameMapping::new(vec![
+ MappedField::new(Some(1), vec!["id".to_string()], vec![]),
+ MappedField::new(Some(2), vec!["ts".to_string()], vec![]),
+ MappedField::new(Some(3), vec!["s".to_string()], vec![]),
+ ]))
+ });
+ let with_field_ids = matches!(field_ids, Int96FieldIds::Embedded);
+ let parquet_schema = SchemaType::group_type_builder("schema")
+ .with_fields(vec![
+ Arc::new(
+ SchemaType::primitive_type_builder("id",
PhysicalType::INT64)
+ .with_repetition(Repetition::REQUIRED)
+ .with_id(with_field_ids.then_some(1))
+ .build()
+ .unwrap(),
+ ),
+ Arc::new(
+ SchemaType::primitive_type_builder("ts",
PhysicalType::INT96)
+ .with_repetition(Repetition::OPTIONAL)
+ .with_id(with_field_ids.then_some(2))
+ .build()
+ .unwrap(),
+ ),
+ Arc::new(
+ SchemaType::primitive_type_builder("s",
PhysicalType::BYTE_ARRAY)
+ .with_repetition(Repetition::REQUIRED)
+ .with_logical_type(Some(LogicalType::String))
+ .with_id(with_field_ids.then_some(3))
+ .build()
+ .unwrap(),
+ ),
+ ])
+ .build()
+ .unwrap();
+ let micros = vec![0_i64, 1_000_003, 2_000_006, 3_000_009, 4_000_012];
+ let nanos: Vec<i64> = micros
+ .iter()
+ .map(|micros| {
+ micros * 1_000
+ + i64::from(matches!(
+ timestamp_type,
+ PrimitiveType::TimestampNs |
PrimitiveType::TimestamptzNs
+ ))
+ })
+ .collect();
+ let ids = vec![0_i64, 1, 2, 3, 4];
+ let strings = vec!["0", "1", "2", "3", "4"];
+ let definition_levels: Vec<i16> = (0..5)
+ .map(|index| i16::from(!with_nulls || index % 2 == 0))
+ .collect();
+ let int96_values: Vec<Int96> = nanos
+ .iter()
+ .zip(&definition_levels)
+ .filter(|(_, level)| **level == 1)
+ .map(|(nanos, _)| {
+ let mut value = Int96::new();
+ value.set_data(
+ *nanos as u32,
+ (nanos >> 32) as u32,
+ UNIX_EPOCH_JULIAN as u32,
+ );
+ value
+ })
+ .collect();
+ let string_values: Vec<ByteArray> = strings
+ .iter()
+ .map(|value| ByteArray::from(*value))
+ .collect();
+
+ let tmp_dir = TempDir::new().unwrap();
+ let file_path =
tmp_dir.path().join("timestamptz_name_mapping.parquet");
+ let mut writer = SerializedFileWriter::new(
+ File::create(&file_path).unwrap(),
+ Arc::new(parquet_schema),
+ Default::default(),
+ )
+ .unwrap();
+ let mut row_group = writer.next_row_group().unwrap();
+ let mut column = row_group.next_column().unwrap().unwrap();
+ column
+ .typed::<Int64Type>()
+ .write_batch(&ids, None, None)
+ .unwrap();
+ column.close().unwrap();
+ let mut column = row_group.next_column().unwrap().unwrap();
+ column
+ .typed::<Int96Type>()
+ .write_batch(&int96_values, Some(&definition_levels), None)
+ .unwrap();
+ column.close().unwrap();
+ let mut column = row_group.next_column().unwrap().unwrap();
+ column
+ .typed::<ByteArrayType>()
+ .write_batch(&string_values, None, None)
+ .unwrap();
+ column.close().unwrap();
+ row_group.close().unwrap();
+ writer.close().unwrap();
+
+ let timezone = matches!(
+ timestamp_type,
+ PrimitiveType::Timestamptz | PrimitiveType::TimestamptzNs
+ )
+ .then_some(UTC_TIME_ZONE);
+ let expected_timestamps: ArrayRef = match timestamp_type {
+ PrimitiveType::Timestamp | PrimitiveType::Timestamptz => {
+ let values: Vec<Option<i64>> = micros
+ .into_iter()
+ .zip(&definition_levels)
+ .map(|(value, level)| (*level == 1).then_some(value))
+ .collect();
+
Arc::new(TimestampMicrosecondArray::from(values).with_timezone_opt(timezone))
+ }
+ PrimitiveType::TimestampNs | PrimitiveType::TimestamptzNs => {
+ let values: Vec<Option<i64>> = nanos
+ .into_iter()
+ .zip(&definition_levels)
+ .map(|(value, level)| (*level == 1).then_some(value))
+ .collect();
+
Arc::new(TimestampNanosecondArray::from(values).with_timezone_opt(timezone))
+ }
+ other => panic!("Expected timestamp type, got {other}"),
+ };
+ let missing_timestamps =
new_null_array(expected_timestamps.data_type(), 5);
+ let expected_batch =
+
RecordBatch::try_new(Arc::new(schema_to_arrow_schema(&schema).unwrap()), vec![
+ Arc::new(Int64Array::from(ids)),
+ expected_timestamps,
+ Arc::new(StringArray::from(strings)),
+ missing_timestamps,
+ Arc::new(StringArray::from(vec!["default"; 5])),
+ ])
+ .unwrap();
+ let mut actual = Vec::new();
+ let mut expected = Vec::new();
+ for field_ids in [
+ vec![2],
+ vec![1, 2, 3],
+ vec![2, 1, 3],
+ vec![2, 4, 5, 1],
+ vec![4, 5],
+ ] {
+ let column_indices: Vec<usize> =
+ field_ids.iter().map(|id| (*id - 1) as usize).collect();
+
expected.push(vec![expected_batch.project(&column_indices).unwrap()]);
+ if field_ids.iter().any(|id| *id <= 3) {
+ let task = int96_scan_task(
+ file_path.to_str().unwrap(),
+ schema.clone(),
+ field_ids.clone(),
+ name_mapping.clone(),
+ );
+ let physical_indices: Vec<usize> = column_indices
+ .iter()
+ .copied()
+ .filter(|index| *index < 3)
+ .collect();
+ assert_int96_projection_mask(&task, &physical_indices);
+ }
+ actual.push(
+ read_int96_batches_with_name_mapping(
+ file_path.to_str().unwrap(),
+ schema.clone(),
+ field_ids,
+ name_mapping.clone(),
+ )
+ .await,
+ );
+ }
+ assert_eq!(actual, expected);
+ }
+
+ #[tokio::test]
+ async fn test_read_int96_with_mixed_timestamp_types_in_struct() {
+ use arrow_array::{StructArray, TimestampMicrosecondArray,
TimestampNanosecondArray};
+ use parquet::data_type::{Int64Type, Int96Type};
+ use parquet::file::writer::SerializedFileWriter;
+ use parquet::schema::parser::parse_message_type;
+
+ use crate::arrow::schema::UTC_TIME_ZONE;
+ use crate::arrow::schema_to_arrow_schema;
+ use crate::spec::StructType;
+
+ let parquet_schema = parse_message_type(
+ "message schema {
+ required int64 id = 1;
+ required group data = 2 {
+ optional int64 micros (TIMESTAMP(MICROS,false)) = 3;
+ optional int96 legacy = 4;
+ }
+ optional int64 nanos (TIMESTAMP(NANOS,true)) = 5;
+ optional int96 trailing = 6;
+ }",
+ )
+ .unwrap();
+ let schema_with_micros_type = |micros_type| {
+ Arc::new(
+ Schema::builder()
+ .with_fields(vec![
+ NestedField::required(1, "id",
Type::Primitive(PrimitiveType::Long)).into(),
+ NestedField::required(
+ 2,
+ "data",
+ Type::Struct(StructType::new(vec![
+ NestedField::optional(3, "micros",
Type::Primitive(micros_type))
+ .into(),
+ NestedField::optional(
+ 4,
+ "legacy",
+
Type::Primitive(PrimitiveType::Timestamptz),
+ )
+ .into(),
+ ])),
+ )
+ .into(),
+ NestedField::optional(
+ 5,
+ "nanos",
+ Type::Primitive(PrimitiveType::TimestamptzNs),
+ )
+ .into(),
+ NestedField::optional(
+ 6,
+ "trailing",
+ Type::Primitive(PrimitiveType::Timestamp),
+ )
+ .into(),
+ ])
+ .build()
+ .unwrap(),
+ )
+ };
+ let schema = schema_with_micros_type(PrimitiveType::Timestamp);
+ let (int96_value, expected_micros) = make_int96_test_value();
+ let nanos = vec![1_000_003_001_i64, -1_000_003_001, 0];
+ let tmp_dir = TempDir::new().unwrap();
+ let file_path = tmp_dir.path().join("mixed_timestamps.parquet");
+ let mut writer = SerializedFileWriter::new(
+ File::create(&file_path).unwrap(),
+ Arc::new(parquet_schema),
+ Default::default(),
+ )
+ .unwrap();
+ let mut row_group = writer.next_row_group().unwrap();
+ let mut column = row_group.next_column().unwrap().unwrap();
+ column
+ .typed::<Int64Type>()
+ .write_batch(&[0, 1, 2], None, None)
+ .unwrap();
+ column.close().unwrap();
+ let mut column = row_group.next_column().unwrap().unwrap();
+ column
+ .typed::<Int64Type>()
+ .write_batch(&[17, -17], Some(&[1, 0, 1]), None)
+ .unwrap();
+ column.close().unwrap();
+ let mut column = row_group.next_column().unwrap().unwrap();
+ column
+ .typed::<Int96Type>()
+ .write_batch(&[int96_value, int96_value], Some(&[1, 0, 1]), None)
+ .unwrap();
+ column.close().unwrap();
+ let mut column = row_group.next_column().unwrap().unwrap();
+ column
+ .typed::<Int64Type>()
+ .write_batch(&nanos, Some(&[1; 3]), None)
+ .unwrap();
+ column.close().unwrap();
+ let mut column = row_group.next_column().unwrap().unwrap();
+ column
+ .typed::<Int96Type>()
+ .write_batch(&[int96_value; 3], Some(&[1; 3]), None)
+ .unwrap();
+ column.close().unwrap();
+ row_group.close().unwrap();
+ writer.close().unwrap();
+
+ let arrow_schema = Arc::new(schema_to_arrow_schema(&schema).unwrap());
+ let DataType::Struct(fields) = arrow_schema.field(1).data_type() else {
+ panic!("Expected struct field");
+ };
+ let expected_batch = RecordBatch::try_new(arrow_schema.clone(), vec![
+ Arc::new(Int64Array::from(vec![0, 1, 2])),
+ Arc::new(StructArray::new(
+ fields.clone(),
+ vec![
+ Arc::new(TimestampMicrosecondArray::from(vec![
+ Some(17),
+ None,
+ Some(-17),
+ ])),
+ Arc::new(
+ TimestampMicrosecondArray::from(vec![
+ Some(expected_micros),
+ None,
+ Some(expected_micros),
+ ])
+ .with_timezone(UTC_TIME_ZONE),
+ ),
+ ],
+ None,
+ )),
+
Arc::new(TimestampNanosecondArray::from(nanos).with_timezone(UTC_TIME_ZONE)),
+ Arc::new(TimestampMicrosecondArray::from(vec![expected_micros;
3])),
+ ])
+ .unwrap();
+
+ for (field_ids, column_indices, physical_indices) in [
+ (vec![1, 2, 5, 6], vec![0, 1, 2, 3], vec![0, 1, 2, 3, 4]),
+ (vec![2], vec![1], vec![1, 2]),
+ (vec![6, 2, 5], vec![3, 1, 2], vec![1, 2, 3, 4]),
+ ] {
+ let task = int96_scan_task(
+ file_path.to_str().unwrap(),
+ schema.clone(),
+ field_ids.clone(),
+ None,
+ );
+ assert_int96_projection_mask(&task, &physical_indices);
+ let actual =
+ read_int96_batches(file_path.to_str().unwrap(),
schema.clone(), field_ids).await;
+ let expected =
vec![expected_batch.project(&column_indices).unwrap()];
+ assert_eq!(actual, expected);
+ }
+
+ let incompatible_task = int96_scan_task(
+ file_path.to_str().unwrap(),
+ schema_with_micros_type(PrimitiveType::Timestamptz),
+ vec![1, 2, 6],
+ None,
+ );
+ assert_int96_projection_mask(&incompatible_task, &[0, 2, 4]);
Review Comment:
This assertion checks that an INT64 `TIMESTAMP(MICROS,false)` column is left
out of the read when the table declares it `timestamptz`, so it reads as NULL.
That's the same silent NULL as #3611, for a different encoding.
The spec only says what [writers must
produce](https://github.com/apache/iceberg/blob/031a6b93acf142ba86f5060ded5532f54901dbfc/format/spec.md?plain=1#L1927-L1928).
Iceberg Java returns the values in both Spark readers. The row reader
[dispatches on
`TIMESTAMP_MICROS`](https://github.com/apache/iceberg/blob/031a6b93acf142ba86f5060ded5532f54901dbfc/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/SparkParquetReaders.java#L224-L226)
without checking `isAdjustedToUTC`, and the vectorized reader [takes the zone
from the Iceberg
type](https://github.com/apache/iceberg/blob/031a6b93acf142ba86f5060ded5532f54901dbfc/arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedArrowReader.java#L564-L570).
Could we drop this assertion so the test doesn't lock in that result?
`test_only_int96_timestamps_are_coerced` already shows that non-INT96 columns
keep their types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]