JeonDaehong opened a new pull request, #1419:
URL: https://github.com/apache/paimon-rust/pull/1419

   ### Purpose
   
   Linked issue: none
   
   Every primary-key data file records min/max and null counts for its key 
columns. `ColumnStatsCollector::collect` builds a `Datum` for each row and 
column and compares it with the current bounds, so a string key allocates a 
`String` per row. In a write profile, `collect`, `extract_datum_from_array` and 
`datum_cmp` add up to about 7% of the time.
   
   ### Brief change log
   
   - For integer, date, time and string columns, `collect` finds the rows with 
the smallest and largest value using a typed scan, and converts only those two 
rows to `Datum`. Null counts come from `null_count()`. Other types (float, 
decimal, timestamp, binary) keep the per-row comparison, so their ordering is 
unchanged.
   
   PK table, 2M keys per write, 3 writes, in-memory FileIO, release build. I 
measured in one binary with the per-row path switched back by an env var, over 
3 alternating rounds. Per write, a BIGINT key goes from 420–442 to 369–391 ms, 
and a STRING key from 473–527 to 436–463 ms.
   
   ### Tests
   
   - `typed_bounds_match_per_row_datum_comparison`: string, bigint and double 
columns with nulls over three batches, one of them all null. It also passes on 
main, so the bounds match the per-row path.
   
   Ran the ci.yml steps locally (fmt, clippy, msrv, unit, rest-server, 
integration, datafusion, python, go).
   
   ### API and Format
   
   None.
   
   ### Documentation
   
   None.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to