JingsongLi commented on code in PR #1422:
URL: https://github.com/apache/paimon-rust/pull/1422#discussion_r4656526914
##########
crates/paimon/src/table/data_evolution_reader.rs:
##########
@@ -551,7 +656,9 @@ impl DataEvolutionReader {
if remaining == Some(0) {
break 'splits;
}
- if is_raw_convertible(&files) {
+ if is_raw_convertible(&files)
+ || (split.is_streaming() && files.len() == 1)
Review Comment:
**[P1] Allow omitted required table fields in physical update events**
In the same two-insert and compaction fixture, I declared `id INT NOT NULL`
and updated `value` and `payload` together. The update committed one normal
slice and one BLOB slice, both covering the four updated row IDs. The full
incremental window fails with `Cannot read non-nullable field 'id' without a
provider`.
This is a physical update event with two contributing files. Its omitted
`id` is expected to be absent, just as it is on the existing single-file
physical update path. Read grouped event slices with the same missing-column
semantics as the raw event path while preserving required-field checks for
reconstructed snapshots.
##########
crates/paimon/src/table/data_evolution_reader.rs:
##########
@@ -91,6 +91,95 @@ fn is_raw_convertible(files: &[DataFileMeta]) -> bool {
true
}
+/// Whether the files of one row-id group are column slices of a single row
range.
+///
+/// Every normal file must cover the same rows, and every dedicated (BLOB or
+/// vector) file must lie inside that range. Only such a group is merged
+/// column-wise; any other group holds files that were not written as slices of
+/// each other.
+pub(super) fn is_data_evolution_column_slice_group(files: &[DataFileMeta]) ->
bool {
+ let mut normal_files = files
+ .iter()
+ .filter(|file| is_data_evolution_normal_file(file));
+ let Some((first, last)) =
normal_files.next().and_then(DataFileMeta::row_id_range) else {
+ return false;
+ };
+ normal_files.all(|file| file.row_id_range() == Some((first, last)))
+ && files.iter().all(|file| {
+ file.row_id_range()
+ .is_some_and(|(from, to)| from >= first && to <= last)
+ })
+}
+
+/// Split overlapping files of a mixed-range window into readable events.
+pub(super) fn incremental_event_groups(
+ files: &[DataFileMeta],
+) -> crate::Result<Option<Vec<Vec<DataFileMeta>>>> {
+ if files.len() <= 1 || is_data_evolution_column_slice_group(files) {
+ return Ok(None);
+ }
+ let (groups, mixed) =
+
split_incremental_event_groups(group_by_overlapping_row_id(files.to_vec()))?;
+ Ok(mixed.then_some(groups))
+}
+
+/// Preserve the event boundaries in a window; report whether any overlapping
+/// row-ID group was split into distinct events.
+pub(super) fn split_incremental_event_groups(
+ groups: Vec<Vec<DataFileMeta>>,
+) -> crate::Result<(Vec<Vec<DataFileMeta>>, bool)> {
+ let mixed =
+ |group: &[DataFileMeta]| group.len() > 1 &&
!is_data_evolution_column_slice_group(group);
+ if !groups.iter().any(|group| mixed(group)) {
+ return Ok((groups, false));
+ }
+ let mut events = Vec::new();
+ for group in groups {
+ if !mixed(&group) {
+ events.push(group);
+ } else if group.iter().all(is_data_evolution_normal_file) {
+ events.extend(group.into_iter().map(|file| vec![file]));
+ } else {
+ // APPEND files in a row-tracked table get the snapshot sequence
+ // at commit. Row IDs alone cannot distinguish a later update from
+ // the normal and independently rolled dedicated files of an
insert.
+ let mut commits = std::collections::BTreeMap::<i64,
Vec<DataFileMeta>>::new();
+ for file in group {
+ if file.file_source != Some(0) || file.max_sequence_number <=
0 {
+ return Err(Error::Unsupported {
+ message: format!("Cannot identify the write of '{}' in
a mixed-range data-evolution window", file.file_name),
+ });
+ }
+ commits
+ .entry(file.max_sequence_number)
+ .or_default()
+ .push(file);
+ }
+ for files in commits.into_values() {
+ for event in group_by_overlapping_row_id(files) {
+ let anchors = event
+ .iter()
+ .filter(|file| is_data_evolution_normal_file(file))
+ .count();
+ if anchors != 1
Review Comment:
**[P1] Accept dedicated-only updates as identifiable incremental events**
I inserted two two-row batches with a dedicated BLOB, compacted their
ordinary files, and updated only the BLOB for row IDs 0 through 3 via the
public row-ID update API. The legal update committed one `.blob` covering all
four rows. This condition rejects that event because it has zero normal
anchors, although its APPEND sequence identifies it independently of the
inserts.
The regression also affects a projection that reads no BLOBs.
`new_incremental_scan(Delta, 1, 4).plan_combined()` with only `id` projected
returned IDs `[3, 4]` on the base. On this revision it fails planning with
`Cannot identify dedicated-file companions`. Please handle standalone dedicated
updates separately from ambiguous insert companions and retain their event
boundaries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]