Rich-T-kid commented on PR #10044: URL: https://github.com/apache/arrow-rs/pull/10044#issuecomment-4635531181
I've been working on this all day and any further changes risk scope creep, so I'd like to split the IPC StreamWriter/FileWriter improvements and the arrow-flight work into separate PRs. Due to async polling it's hard to distinguish what copies happen at the tonic level versus in arrow-flight. I've been profiling this locally with an additional encode_to_send benchmark that measures the full path via do_put [will include in follow up PR] @alamb does that sound like a good idea? <img width="1263" height="262" alt="Image 6-5-26 at 4 58 PM" src="https://github.com/user-attachments/assets/3fb202c5-36e2-4941-8445-dd688342e583"; /> <img width="2444" height="474" alt="Image 6-5-26 at 4 57 PM (1)" src="https://github.com/user-attachments/assets/007221dd-a0cf-4741-92c0-9ca1ab55d75b"; /> <img width="2448" height="206" alt="Image 6-5-26 at 4 57 PM" src="https://github.com/user-attachments/assets/07550d6e-14d6-446b-af6d-93f9d7430989"; /> _One question_: does anyone have insight into why the benchmarks behave differently on the CI workers versus locally? I saw something similar in distributed DataFusion and it came down to thread count, but beyond thread count and CPU cache size I'm not sure what else could explain the difference at this scale. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
