SteNicholas opened a new issue, #11117: URL: https://github.com/apache/paimon/issues/11117
### Search before asking - [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version master (3821db1972) ### Compute Engine PyPaimon (Python write path, `BlobWriter`) ### Minimal reproduce step On a Data Evolution append table (`row-tracking.enabled=true`, `data-evolution.enabled=true`) with a BLOB column, write enough rows that the BLOB files roll by `blob.target-file-size`. Then update the BLOB column by row id and read the table. ```python import pyarrow as pa from pypaimon import Schema pa_schema = pa.schema([('id', pa.int32()), ('blob_data', pa.large_binary())]) schema = Schema.from_pyarrow_schema(pa_schema, options={ 'row-tracking.enabled': 'true', 'data-evolution.enabled': 'true', 'blob.target-file-size': '1 B', # roll after every blob }) catalog.create_table('db.t', schema, False) table = catalog.get_table('db.t') # snapshot 1: 10 rows -> 1 normal file + 10 rolled blob files wb = table.new_batch_write_builder() w = wb.new_write() w.write_arrow(pa.Table.from_pydict( {'id': list(range(10)), 'blob_data': [f'blob-{i}'.encode() for i in range(10)]}, schema=pa_schema)) wb.new_commit().commit(w.prepare_commit()) w.close() # snapshot 2: update rows 2 and 7 ub = table.new_batch_write_builder() update = ub.new_update().with_update_type(['blob_data']) msgs = update.update_by_arrow_with_row_id(pa.Table.from_pydict({ '_ROW_ID': pa.array([2, 7], type=pa.int64()), 'blob_data': pa.array([b'updated-2', b'updated-7'], type=pa.large_binary()), })) ub.new_commit().commit(msgs) rb = table.new_read_builder() rb.new_read().to_arrow(rb.new_scan().plan().splits()) ``` ### What doesn't meet your expectations? The update commits successfully, but the read afterwards either fails with ``` ValueError: Blob files within the same max sequence should not overlap. ``` or returns the old BLOB value for updated rows. Cause: on a Data Evolution table, `RowTrackingCommitUtils` / pypaimon `_assign_snapshot_id` only replace the sequence range with the snapshot id when `min_sequence_number == 0`. PyPaimon's `BlobWriter` keeps one row counter across rolled files, so the first blob file gets `(0, n-1)` and is stamped with the snapshot id, while later rolled files get `(n, 2n-1)`, `(2n, 3n-1)`, ... and keep these row-based numbers. BLOB files written by update-by-row-id use the `0/0` sentinel and get the update's snapshot id. When reading, BLOB files are grouped and ordered by `max_sequence_number`: - if an old rolled file's max sequence equals the update snapshot id, both files are in the same group with overlapping row ids -> `Blob files within the same max sequence should not overlap`; - if it is larger than the update snapshot id, the old file wins -> the stale BLOB is returned. Java is not affected: on Data Evolution tables `AppendOnlyWriter` provides `() -> new LongCounter(0)` and `RowDataFileSequenceNumberTracker` takes a new counter for every rolled file (`MultipleBlobFileWriter` creates a new `RowDataFileWriter` per file), so every Java BLOB file is `(0, n-1)` and gets the snapshot id at commit. ### Anything else? Proposed fix: in pypaimon `BlobWriter._add_file_metadata`, use `_append_file_sequence_range(row_count)` (i.e. `(0, row_count - 1)`) on Data Evolution tables, the same as normal data files and Java, so every rolled BLOB file is stamped with the snapshot id. Note: tables already written by pypaimon with rolled BLOB files keep the row-based sequence numbers in their manifests; fixing the writer does not correct existing data. ### Are you willing to submit a PR? - [X] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
