JingsongLi opened a new pull request, #11262:
URL: https://github.com/apache/paimon/pull/11262

   ### Purpose
   
   Selective queries on a Data Evolution append table can open many data, BLOB 
and vector files when values of the filter column are scattered across file 
groups. This prototype adds `sys.sort_data_evolution` to Spark and Flink batch 
SQL so callers can cluster a chosen part of the table by the columns they query.
   
   ```sql
   CALL sys.sort_data_evolution('default.samples', 0, 99999, 
'tenant_id,event_time', 'p = 7');
   ```
   
   The input row-ID bounds are inclusive. The optional final argument filters 
partitions. The planner checks every intersecting file in the planning snapshot 
and rejects a boundary that cuts a file group before any writes start. Each 
selected partition gets one spillable sort. The sort uses ascending keys with 
nulls first and uses the old row ID to break ties.
   
   The rewrite reads the latest visible values, applies deletion vectors, and 
writes ordinary columns, managed BLOBs and dedicated vectors in the same new 
row order. Surviving rows receive new IDs. Downstream references to those IDs 
need updating, and global indexes in rewritten partitions need rebuilding.
   
   ### Tests
   
   - Spark 3.5.9 and Flink 1.20.1 and 2.2.0 SQL integration tests read back 
scalar, BLOB and vector values by new row ID. They also check physical file 
replacement, preservation of data outside the selection, partial-group 
rejection and repeated calls.
   - The two targeted core suites passed all 100 tests, including missing input 
and an index built after rewrite planning. The Flink 1.20.1 run also passed all 
13 tests in `DataEvolutionDeleteSqlITCase`.
   
   `SortDataEvolutionBenchmark` compared ordinary compaction and sorting on 
branches from the same 32,768-row snapshot. Both layouts had eight main data 
files with matching row counts. The query selected 64 rows by the sort key and 
read scalar, BLOB and vector values. File and byte counts come from an 
instrumented table read. SQL timings use 20 local Spark runs per layout after 
warm-up.
   
   | Local Spark measurement | Ordinary compaction | Sorted rewrite |
   | --- | ---: | ---: |
   | Distinct data files opened | 24 | 3 |
   | Data bytes read | 2,445,853 | 307,224 |
   | Query p50 | 82.1 ms | 42.6 ms |
   | Query p95 | 104.1 ms | 49.9 ms |
   | Maintenance elapsed time | 1.08 s | 2.89 s |
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to