JingsongLi commented on code in PR #1422:
URL: https://github.com/apache/paimon-rust/pull/1422#discussion_r4643284262
##########
crates/paimon/src/table/data_evolution_reader.rs:
##########
@@ -551,7 +571,12 @@ impl DataEvolutionReader {
if remaining == Some(0) {
break 'splits;
}
- if is_raw_convertible(&files) {
+ // Change events are physical file changes. Files of a
streaming
+ // split that are not column slices of one row range are
+ // independent changes and are read as they are stored.
+ if is_raw_convertible(&files)
+ || (split.is_streaming() &&
!is_data_evolution_column_slice_group(&files))
Review Comment:
**[P1] Keep an insert's dedicated BLOB slice with its data rows**
This branch treats every file in the overlap group as an independent event.
An ordinary Data Evolution insert can write a Parquet file and a dedicated
`.blob` file for the same rows. Reproduced with a table `(id INT, value INT,
payload BLOB)`: insert two batches of two rows, compact the Parquet files,
update `value` for all four row IDs, and read the timestamp window covering
those snapshots. The scan emits 12 rows. Four contain `id` with a NULL payload,
four contain the payload with a NULL `id`, and four are the value-only updates.
Each original insert row has been split in two.
Keep companion files from the same insert together when separating events
with different ranges. If the available metadata cannot identify those
companions reliably, reject the shape explicitly until it can be read without
inventing rows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]