JeonDaehong commented on code in PR #3704:
URL: https://github.com/apache/iceberg-rust/pull/3704#discussion_r4599703762
##########
crates/iceberg/src/scan/context.rs:
##########
@@ -228,6 +228,12 @@ impl ManifestEntryContext {
.with_column_stats(self.column_stats_columns.as_deref().map(|columns| {
FileScanTaskColumnStats::from_data_file(self.manifest_entry.data_file(),
columns)
}))
+ .with_split_offsets(
Review Comment:
Java's `BaseFile.splitOffsets()` returns null when the list is empty or its
last offset is not below `file_size_in_bytes` (`hasWellDefinedOffsets`), since
some writers recorded offsets that don't belong to the file. An engine
splitting on offsets like that would cut past the end of the file. Would it
make sense to apply the same check here, so a task only carries offsets Java
would use?
##########
crates/iceberg/src/scan/mod.rs:
##########
@@ -1561,6 +1561,51 @@ mod tests {
}
}
+ #[tokio::test]
+ async fn test_plan_files_carries_split_offsets_from_the_manifest_entry() {
+ let mut fixture = TableTestFixture::new();
+
+ // Offsets recorded (the usual case: one per Parquet row group), an
empty list,
+ // and a file whose writer did not record them.
+ fixture
+ .setup_manifest_files_with_split_offsets([
+ Some(vec![4, 1_048_580, 2_097_156]),
+ Some(vec![]),
+ None,
+ ])
+ .await;
+
+ let mut tasks: Vec<_> = fixture
+ .table
+ .scan()
+ .build()
+ .unwrap()
+ .plan_files()
+ .await
+ .unwrap()
+ .try_collect()
+ .await
+ .unwrap();
+ tasks.sort_by(|a, b| a.data_file_path().cmp(b.data_file_path()));
+ assert_eq!(tasks.len(), 3);
+
+ assert_eq!(
+ tasks[0].split_offsets(),
+ Some(&[4, 1_048_580, 2_097_156][..]),
+ "offsets recorded on the data file reach the task unchanged"
+ );
+ assert_eq!(
+ tasks[1].split_offsets(),
+ Some(&[][..]),
Review Comment:
With that check, the empty list would come through as `None`, as
`splitOffsets()` returns in Java.
##########
crates/iceberg/src/scan/task.rs:
##########
@@ -515,6 +527,7 @@ mod _serde {
unified_partition_type: Option<&'a Arc<StructType>>,
#[serde(skip_serializing_if = "Option::is_none")]
sort_order_id: Option<i32>,
+ split_offsets: Option<&'a [i64]>,
Review Comment:
The neighbouring optional fields use `skip_serializing_if =
"Option::is_none"` (and `#[serde(default)]` on the read side), but this one
serializes `null`. Worth matching them?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]