alamb commented on PR #24102: URL: https://github.com/apache/datafusion/pull/24102#issuecomment-5410371001
BTW I made a small test that creates a 1.6GB file and then does a query ```sql -- Create a parquet file with 100M random UUIDv4s stored as FixedSizeBinary(16) -- for benchmarking https://github.com/apache/datafusion/pull/24102 -- -- Run with: datafusion-cli -f /tmp/create_uuids.sql -- -- uuid() generates a fresh random v4 UUID string per row; stripping the -- hyphens and hex-decoding yields the 16 raw bytes. COPY ( SELECT arrow_cast( decode(replace(uuid(), '-', ''), 'hex'), 'FixedSizeBinary(16)' ) AS uuid FROM generate_series(1, 100000000) ) TO '/tmp/uuids.parquet'; ``` Then ```sql SELECT count(*) FROM '/tmp/uuids.parquet' WHERE uuid IN ( arrow_cast(decode('d19615973bb644aa8f154cc4affd48d5', 'hex'), 'FixedSizeBinary(16)'), arrow_cast(decode('cdf8a5c807f443d0bcd14e3791be9632', 'hex'), 'FixedSizeBinary(16)'), arrow_cast(decode('6469a75155a9422d8d9e91b80ff26523', 'hex'), 'FixedSizeBinary(16)'), arrow_cast(decode('67b98b1c0cc4411e865c7344b039446d', 'hex'), 'FixedSizeBinary(16)'), arrow_cast(decode('1535a198832948a79b4424485da0c9b5', 'hex'), 'FixedSizeBinary(16)'), arrow_cast(decode('ed8a5d9fff914f59b627924a58fa385d', 'hex'), 'FixedSizeBinary(16)'), -- ... 100 in total ... ``` Scripts: * [create_uuids.sql](https://github.com/user-attachments/files/31419932/create_uuids.sql) * [query_uuids.sql](https://github.com/user-attachments/files/31419933/query_uuids.sql) To run: ```shell datafusion-cli -f create_uuids.sql datafusion-cli -f query_uuids.sql ``` Results are a few percent end to end performance improvement: ┌─────────────────────────────────────────────────────────────┬─────────┬─────────┬──────────┐ │ Binary │ Min (s) │ Max (s) │ Mean (s) │ ├─────────────────────────────────────────────────────────────┼─────────┼─────────┼──────────┤ │ datafusion-cli (main) │ 0.108 │ 0.137 │ 0.117 │ ├─────────────────────────────────────────────────────────────┼─────────┼─────────┼──────────┤ │ datafusion-cli-codex_in-list-fixed-size-binary-hashset (PR) │ 0.095 │ 0.133 │ 0.112 │ └─────────────────────────────────────────────────────────────┴─────────┴─────────┴──────────┘ The PR branch is about 4% faster on the mean (0.117s → 0.112s) and its best runs are notably faster (min 0.108s → 0.095s, ~12%), -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
