koodin9 commented on PR #18003: URL: https://github.com/apache/iceberg/pull/18003#issuecomment-6573941243
I added the benchmark to this PR (2c593a4), since Péter asked on the [dev@ thread](https://lists.apache.org/thread/gmkj55oc0k1k72gobsylgfc9ngk9lyxn) to measure the conversion before anything is merged. Here are the first results. Setup: a 14-core Mac, an unpartitioned v3 table with 1,000 rows per data file, and 1,000 deleted keys per commit. The table is in RustFS behind toxiproxy, which delays every S3 response by a fixed latency. The worker pool has 14 threads unless noted. Each time is one conversion. How to run it, including the toxiproxy setup, is in the PR description. | Table | Data files read | 0 ms | 10 ms | 20 ms | 50 ms | 50 ms, 64 threads | |---|---|---|---|---|---|---| | 10,000 files, random UUID keys | 10,000 | 2.2 s | 9.5 s | 18.5 s | 42.4 s | 11.4 s | | 10,000 files, sorted UUID keys | ~1,000 | 0.38 s | 1.2 s | 2.1 s | 4.6 s | 1.5 s | | 1,000 files, random UUID keys | 1,000 | 0.30 s | 1.0 s | 2.0 s | 4.3 s | 1.4 s | | 10,000 files, increasing IDs, recent rows | 1 | 0.11 s | 0.19 s | 0.24 s | 0.38 s | 0.39 s | | 10,000 files, increasing IDs, new keys | 0 | 0.09 s | 0.11 s | 0.14 s | 0.20 s | 0.20 s | - The time is close to the data files read times the request latency, divided by the threads. The number of deleted keys barely changes it. - When file bounds can prune the keys, as with increasing IDs, a conversion stays under half a second even at 50 ms. - Random UUID keys read every data file, so the time grows with the table, as Péter expected. I expect bloom filters to help little there, because each file still needs a request. That case needs an index, or partitioning that narrows the files. - More threads help almost linearly, but the worker pool is shared by the whole JVM. - These are lower bounds. The latency is fixed with no tail, the files are small, and there is no TLS. Next, I'd like to add partitioned layouts and larger data files, and run compaction alongside repeated conversion commits. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
