LuciferYang opened a new pull request, #10643: URL: https://github.com/apache/paimon/pull/10643
### Purpose close #10406 On a data-evolution table, a commit that writes only blob columns, for example a Spark `MERGE INTO ... WHEN MATCHED THEN UPDATE SET t.picture = s.picture`, adds a snapshot that holds only `.blob` files. The normal file of those rows is in an earlier snapshot. An incremental read of that snapshot (streaming read, `paimon_incremental_query`) planned the blob files alone and failed unless it read only blob columns: reading a normal column, `_ROW_ID` or a partition column threw at read time, and with deletion vectors on it already threw at planning because the group has no anchor file. Such a delta now reads like a delta that updates normal columns: it emits its rows with the row-tracking fields set and the columns it did not write NULL. - `DataEvolutionFileStoreScan.pruneByReadType` keeps a group of only blob files whole, without looking for an anchor. - `DataEvolutionSplitRead` does not open a blob file whose column is not read, and emits placeholder rows for its row range instead. A group without a normal file takes its row range from all of its files and has no deletion vector. A single blob file is read with only the column it wrote mapped. Groups of only vector-store files keep their current behavior. ### Tests - `BlobUpdateTestBase`: after a `MERGE INTO` that updates only the blob column, `paimon_incremental_query` over that snapshot returns the row with NULL normal columns and its `_ROW_ID`. - `MultipleBlobTableTest`: stream scans over a delta of only blob files, with normal, `_ROW_ID`, empty, blob and full read types, with and without deletion vectors; blob columns covering different rows of the delta, including a projection of one of them; a partitioned table. - `DataEvolutionFileStoreScanTest#testReadTypePruningKeepsDedicatedOnlyGroup`: such a group is kept whole for any projection, with and without deletion vectors. All of them fail on master. ### API and Format no ### Documentation no Generated-by: Claude Opus -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
