SteNicholas opened a new issue, #11117:
URL: https://github.com/apache/paimon/issues/11117

   ### Search before asking
   
   - [X] I searched in the [issues](https://github.com/apache/paimon/issues) 
and found nothing similar.
   
   ### Paimon version
   
   master (3821db1972)
   
   ### Compute Engine
   
   PyPaimon (Python write path, `BlobWriter`)
   
   ### Minimal reproduce step
   
   On a Data Evolution append table (`row-tracking.enabled=true`, 
`data-evolution.enabled=true`) with a BLOB column, write enough rows that the 
BLOB files roll by `blob.target-file-size`. Then update the BLOB column by row 
id and read the table.
   
   ```python
   import pyarrow as pa
   from pypaimon import Schema
   
   pa_schema = pa.schema([('id', pa.int32()), ('blob_data', pa.large_binary())])
   schema = Schema.from_pyarrow_schema(pa_schema, options={
       'row-tracking.enabled': 'true',
       'data-evolution.enabled': 'true',
       'blob.target-file-size': '1 B',   # roll after every blob
   })
   catalog.create_table('db.t', schema, False)
   table = catalog.get_table('db.t')
   
   # snapshot 1: 10 rows -> 1 normal file + 10 rolled blob files
   wb = table.new_batch_write_builder()
   w = wb.new_write()
   w.write_arrow(pa.Table.from_pydict(
       {'id': list(range(10)), 'blob_data': [f'blob-{i}'.encode() for i in 
range(10)]},
       schema=pa_schema))
   wb.new_commit().commit(w.prepare_commit())
   w.close()
   
   # snapshot 2: update rows 2 and 7
   ub = table.new_batch_write_builder()
   update = ub.new_update().with_update_type(['blob_data'])
   msgs = update.update_by_arrow_with_row_id(pa.Table.from_pydict({
       '_ROW_ID': pa.array([2, 7], type=pa.int64()),
       'blob_data': pa.array([b'updated-2', b'updated-7'], 
type=pa.large_binary()),
   }))
   ub.new_commit().commit(msgs)
   
   rb = table.new_read_builder()
   rb.new_read().to_arrow(rb.new_scan().plan().splits())
   ```
   
   ### What doesn't meet your expectations?
   
   The update commits successfully, but the read afterwards either fails with
   
   ```
   ValueError: Blob files within the same max sequence should not overlap.
   ```
   
   or returns the old BLOB value for updated rows.
   
   Cause: on a Data Evolution table, `RowTrackingCommitUtils` / pypaimon 
`_assign_snapshot_id` only replace the sequence range with the snapshot id when 
`min_sequence_number == 0`. PyPaimon's `BlobWriter` keeps one row counter 
across rolled files, so the first blob file gets `(0, n-1)` and is stamped with 
the snapshot id, while later rolled files get `(n, 2n-1)`, `(2n, 3n-1)`, ... 
and keep these row-based numbers.
   
   BLOB files written by update-by-row-id use the `0/0` sentinel and get the 
update's snapshot id. When reading, BLOB files are grouped and ordered by 
`max_sequence_number`:
   
   - if an old rolled file's max sequence equals the update snapshot id, both 
files are in the same group with overlapping row ids -> `Blob files within the 
same max sequence should not overlap`;
   - if it is larger than the update snapshot id, the old file wins -> the 
stale BLOB is returned.
   
   Java is not affected: on Data Evolution tables `AppendOnlyWriter` provides 
`() -> new LongCounter(0)` and `RowDataFileSequenceNumberTracker` takes a new 
counter for every rolled file (`MultipleBlobFileWriter` creates a new 
`RowDataFileWriter` per file), so every Java BLOB file is `(0, n-1)` and gets 
the snapshot id at commit.
   
   ### Anything else?
   
   Proposed fix: in pypaimon `BlobWriter._add_file_metadata`, use 
`_append_file_sequence_range(row_count)` (i.e. `(0, row_count - 1)`) on Data 
Evolution tables, the same as normal data files and Java, so every rolled BLOB 
file is stamped with the snapshot id.
   
   Note: tables already written by pypaimon with rolled BLOB files keep the 
row-based sequence numbers in their manifests; fixing the writer does not 
correct existing data.
   
   ### Are you willing to submit a PR?
   
   - [X] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to