laskoviymishka commented on code in PR #3242:
URL: https://github.com/apache/iceberg-rust/pull/3242#discussion_r4087473631
##########
crates/iceberg/src/arrow/reader/pipeline.rs:
##########
@@ -723,6 +635,65 @@ impl FileScanTaskReader {
Ok(Box::pin(record_batch_stream) as ArrowRecordBatchStream)
}
+ /// Applies all task-specific schema and virtual-column options,
rebuilding the
+ /// Arrow reader metadata at most once.
+ fn configure_arrow_reader_metadata(
+ arrow_metadata: ArrowReaderMetadata,
+ task: &FileScanTask,
+ missing_field_ids: bool,
+ install_row_number: bool,
+ ) -> Result<ArrowReaderMetadata> {
+ // Three-branch schema resolution strategy matching Java's ReadConf
constructor.
+ // When Parquet files lack field IDs, apply a name mapping when
available and use
+ // position-based fallback IDs otherwise. Files with embedded IDs keep
their schema.
+ let mut arrow_schema = if missing_field_ids {
+ if let Some(name_mapping) = task.name_mapping() {
+ apply_name_mapping_to_arrow_schema(
+ Arc::clone(arrow_metadata.schema()),
+ name_mapping,
+ )?
+ } else {
+ add_fallback_field_ids_to_arrow_schema(arrow_metadata.schema())
+ }
+ } else {
+ Arc::clone(arrow_metadata.schema())
+ };
+
+ // Coerce INT96 timestamp columns before building the stream reader to
avoid i64
+ // overflow in arrow-rs. Apply this after assigning any missing field
IDs so the
+ // final schema contains both changes.
+ let mut should_rebuild = missing_field_ids;
+ if let Some(coerced_schema) = coerce_int96_timestamps(&arrow_schema,
task.schema()) {
+ arrow_schema = coerced_schema;
+ should_rebuild = true;
+ }
+
+ if !should_rebuild && !install_row_number {
+ return Ok(arrow_metadata);
+ }
+
+ let mut options = ArrowReaderOptions::new().with_schema(arrow_schema);
+ if install_row_number {
+ let row_number_field = Arc::new(
+ Field::new(RESERVED_COL_NAME_POS, DataType::Int64, false)
+ .with_metadata(HashMap::from([(
+ PARQUET_FIELD_ID_META_KEY.to_string(),
+ RESERVED_FIELD_ID_POS.to_string(),
+ )]))
+ .with_extension_type(RowNumber),
+ );
+ options = options.with_virtual_columns(vec![row_number_field])?;
+ }
+
+ ArrowReaderMetadata::try_new(Arc::clone(arrow_metadata.metadata()),
options).map_err(|e| {
+ Error::new(
+ ErrorKind::Unexpected,
+ "Failed to create ArrowReaderMetadata with the configured
reader options",
Review Comment:
Now that the three `try_new` calls collapse into one, a failure here can't
tell us which transform tripped it — field-ID assignment, INT96 coercion, or
the row-number virtual column — and we've lost the coerced-schema dump the old
INT96 branch printed. These are schema-validation failures, exactly the class
we want to triage fast. `missing_field_ids`, `install_row_number`, and the
resulting `arrow_schema` are all in scope here, so I'd interpolate them into
the message rather than leaving it generic.
##########
crates/iceberg/src/arrow/reader/pipeline.rs:
##########
@@ -723,6 +635,65 @@ impl FileScanTaskReader {
Ok(Box::pin(record_batch_stream) as ArrowRecordBatchStream)
}
+ /// Applies all task-specific schema and virtual-column options,
rebuilding the
+ /// Arrow reader metadata at most once.
+ fn configure_arrow_reader_metadata(
+ arrow_metadata: ArrowReaderMetadata,
+ task: &FileScanTask,
+ missing_field_ids: bool,
+ install_row_number: bool,
+ ) -> Result<ArrowReaderMetadata> {
+ // Three-branch schema resolution strategy matching Java's ReadConf
constructor.
+ // When Parquet files lack field IDs, apply a name mapping when
available and use
+ // position-based fallback IDs otherwise. Files with embedded IDs keep
their schema.
+ let mut arrow_schema = if missing_field_ids {
+ if let Some(name_mapping) = task.name_mapping() {
+ apply_name_mapping_to_arrow_schema(
+ Arc::clone(arrow_metadata.schema()),
+ name_mapping,
+ )?
+ } else {
+ add_fallback_field_ids_to_arrow_schema(arrow_metadata.schema())
+ }
+ } else {
+ Arc::clone(arrow_metadata.schema())
Review Comment:
On the common fast path — embedded field IDs, no INT96, no row-number — we
clone the schema here and then discard it at the early return without ever
using it. Small atomic refcount bump on the per-file hot path, a bit ironic for
a build-once change. I'd defer materializing an owned schema until we know we
need it:
```rust
let mut arrow_schema = if missing_field_ids {
if let Some(name_mapping) = task.name_mapping() {
apply_name_mapping_to_arrow_schema(Arc::clone(arrow_metadata.schema()),
name_mapping)?
} else {
add_fallback_field_ids_to_arrow_schema(arrow_metadata.schema())
}
} else if let Some(coerced) =
coerce_int96_timestamps(arrow_metadata.schema(), task.schema()) {
should_rebuild = true;
coerced
} else if install_row_number {
Arc::clone(arrow_metadata.schema())
} else {
return Ok(arrow_metadata);
};
```
##########
crates/iceberg/src/arrow/reader/pipeline.rs:
##########
@@ -723,6 +635,65 @@ impl FileScanTaskReader {
Ok(Box::pin(record_batch_stream) as ArrowRecordBatchStream)
}
+ /// Applies all task-specific schema and virtual-column options,
rebuilding the
+ /// Arrow reader metadata at most once.
+ fn configure_arrow_reader_metadata(
+ arrow_metadata: ArrowReaderMetadata,
+ task: &FileScanTask,
+ missing_field_ids: bool,
+ install_row_number: bool,
Review Comment:
`missing_field_ids` and `install_row_number` are two independent bools
sitting adjacent, so a transposed call would type-check silently — guarded
today only by the locals happening to share names. Reads fine at the single
call site, so not worth churn now, but if a third flag shows up I'd fold these
into a small config struct or marker types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
