wgtmac commented on code in PR #822:
URL: https://github.com/apache/iceberg-cpp/pull/822#discussion_r3593397522
##########
src/iceberg/data/file_scan_task_reader.cc:
##########
@@ -161,9 +165,9 @@ class FileScanTaskReader::Impl {
data_file->file_size_in_bytes);
if (task.delete_files().empty()) {
- auto options =
- MakeReaderOptions(*data_file, io_, projected_schema_,
task.residual_filter(),
- name_mapping_, properties_);
+ auto options = MakeReaderOptions(
Review Comment:
I opened apache/iceberg#17234 to address this on the Java/REST side. It
preserves `data-sequence-number` in `ContentFile` JSON and updates the
OpenAPI/generated model, so REST-planned scan tasks can carry the
manifest-entry data sequence number needed for `_last_updated_sequence_number`
inheritance.
##########
src/iceberg/parquet/parquet_data_util.cc:
##########
@@ -70,6 +70,53 @@ Result<std::shared_ptr<::arrow::Array>>
ProjectPrimitiveArray(
return cast_result.make_array();
}
+Result<std::shared_ptr<::arrow::Array>> MakeInheritedRowLineageArray(
+ int32_t field_id, int64_t num_rows,
+ const arrow::MetadataColumnContext& metadata_context, ::arrow::MemoryPool*
pool) {
+ if (field_id == MetadataColumns::kRowIdColumnId) {
+ return arrow::MakeRowIdArray(metadata_context.first_row_id,
+ metadata_context.next_file_pos, num_rows,
pool);
+ }
+ if (field_id == MetadataColumns::kLastUpdatedSequenceNumberColumnId) {
+ return arrow::MakeLastUpdatedSequenceNumberArray(
+ metadata_context.first_row_id, metadata_context.data_sequence_number,
num_rows,
+ pool);
+ }
+ return NotSupported("Unsupported row lineage field id: {}", field_id);
+}
+
+Result<std::shared_ptr<::arrow::Array>> ProjectRowLineageArray(
+ const std::shared_ptr<::arrow::Array>& physical_array, int32_t field_id,
+ const arrow::MetadataColumnContext& metadata_context, ::arrow::MemoryPool*
pool) {
+ if (!arrow::HasRowLineageValue(field_id, metadata_context)) {
Review Comment:
I opened apache/iceberg#17235 for the corresponding Java reader behavior. It
updates Avro, Parquet, ORC, and Arrow readers to preserve physical row-lineage
values when inheritance constants are missing, and to inherit
`_last_updated_sequence_number` from the data sequence number without depending
on `first_row_id`. That matches the spec behavior this C++ path is implementing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
