JeonDaehong opened a new pull request, #1419: URL: https://github.com/apache/paimon-rust/pull/1419
### Purpose Linked issue: none Every primary-key data file records min/max and null counts for its key columns. `ColumnStatsCollector::collect` builds a `Datum` for each row and column and compares it with the current bounds, so a string key allocates a `String` per row. In a write profile, `collect`, `extract_datum_from_array` and `datum_cmp` add up to about 7% of the time. ### Brief change log - For integer, date, time and string columns, `collect` finds the rows with the smallest and largest value using a typed scan, and converts only those two rows to `Datum`. Null counts come from `null_count()`. Other types (float, decimal, timestamp, binary) keep the per-row comparison, so their ordering is unchanged. PK table, 2M keys per write, 3 writes, in-memory FileIO, release build. I measured in one binary with the per-row path switched back by an env var, over 3 alternating rounds. Per write, a BIGINT key goes from 420–442 to 369–391 ms, and a STRING key from 473–527 to 436–463 ms. ### Tests - `typed_bounds_match_per_row_datum_comparison`: string, bigint and double columns with nulls over three batches, one of them all null. It also passes on main, so the bounds match the per-row path. Ran the ci.yml steps locally (fmt, clippy, msrv, unit, rest-server, integration, datafusion, python, go). ### API and Format None. ### Documentation None. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
