wankunde commented on PR #2371: URL: https://github.com/apache/orc/pull/2371#issuecomment-3258345388
Hi, @cxzl25 @dongjoon-hyun Add local benchmark result, orc writer needs less compute time and memory than before. <img width="622" height="234" alt="企业微信截图_c9487097-1e43-4890-8dad-243399915d01" src="https://github.com/user-attachments/assets/42dc2c70-8286-4840-835d-8aefb5fd8dab"; /> ```log Before: Batch Size: 1024, column Num: 10, took 4 ms to write 10000 rows, flush stripe time: 0 ms, flush stripe count: 0 Batch Size: 1024, column Num: 100, took 43 ms to write 10000 rows, flush stripe time: 0 ms, flush stripe count: 0 Batch Size: 1024, column Num: 300, took 80 ms to write 10000 rows, flush stripe time: 0 ms, flush stripe count: 0 Batch Size: 1024, column Num: 500, took 111 ms to write 10000 rows, flush stripe time: 0 ms, flush stripe count: 0 Batch Size: 1024, column Num: 1000, took 234 ms to write 10000 rows, flush stripe time: 0 ms, flush stripe count: 0 Batch Size: 1024, column Num: 5000, took 1884 ms to write 10000 rows, flush stripe time: 632 ms, flush stripe count: 1 Batch Size: 1024, column Num: 10000, took 5132 ms to write 10000 rows, flush stripe time: 2391 ms, flush stripe count: 1 After: Batch Size: 1024, column Num: 10, took 3 ms to write 10000 rows, flush stripe time: 0 ms, flush stripe count: 0 Batch Size: 1024, column Num: 100, took 33 ms to write 10000 rows, flush stripe time: 0 ms, flush stripe count: 0 Batch Size: 1024, column Num: 300, took 71 ms to write 10000 rows, flush stripe time: 0 ms, flush stripe count: 0 Batch Size: 1024, column Num: 500, took 112 ms to write 10000 rows, flush stripe time: 0 ms, flush stripe count: 0 Batch Size: 1024, column Num: 1000, took 233 ms to write 10000 rows, flush stripe time: 0 ms, flush stripe count: 0 Batch Size: 1024, column Num: 5000, took 5984 ms to write 10000 rows, flush stripe time: 5100 ms, flush stripe count: 10 Batch Size: 1024, column Num: 10000, took 20473 ms to write 10000 rows, flush stripe time: 19308 ms, flush stripe count: 10 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@orc.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
