zhoujinsong commented on issue #4173: URL: https://github.com/apache/amoro/issues/4173#issuecomment-5022180061
Thanks for the detailed questions! You're right that the blog focused on the principles and results, and some benchmark details were omitted for brevity. Let me fill in the gaps: - **Primary Key**: Single primary key — `order_id`. - **CDC Operations**: The upsert scenario runs with `write.upsert.enabled=true`. The Flink job randomly generates INSERT messages with `order_id` values that may already exist in the table, triggering upsert behavior (equality delete + insert). There are no pure DELETE operations. - **Dataset**: The data is randomly generated by Spark/Flink, with `order_id` values in the range of 1–500 million. The table was pre-loaded with 500 million records before the compaction benchmark. - **Partitioning**: The table is unpartitioned for this benchmark. - **Amoro Version**: 0.8.0 (incubating). - **Parallelism**: Both Spark compaction and Amoro compaction used the same resource allocation as stated in the blog. For the upsert scenario, both were allocated 32 cores and 64 GB memory. Spark used `parallel` parameter matching the core count; Amoro's optimizer was configured with equivalent parallelism. - **Data Sorting**: No, the data files were not sorted by the primary key before compaction. The data was written in the order produced by the Flink streaming job. - **Equality Delete Column**: `order_id` (same as the primary key). - **Data File Sorting on Eq-Delete Column**: No, the data files were not sorted on `order_id`. This is actually one of the key reasons why Equality Delete files are so expensive for queries — without sorting, each Equality Delete must be checked against all data files in the table. - **Data Volume per CDC Event**: Approximately 1,000 rows changed per second, with all fields randomly varying per change. Hope this helps clarify the benchmark setup. We'll also consider adding these details to the blog post or a follow-up document. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
