HyukjinKwon commented on PR #59934: URL: https://github.com/apache/spark/pull/59934#issuecomment-6639682670
Thank you, @dongjoon-hyun. d40d64d addresses the eighth round, items 1 and 3 to 8; the replies are in the threads. Item 9 is fixed in #59931 (6ab9036) and merged here in 8a28f1d, and for item 2 I kept the current shape of `partition_id`, as you allowed. For item 5, I measured the callbacks on a native thread: going through the JVM adds 20 to 30 ns to a `get_next`, while Spark's own import of a batch costs 4 to 5 µs, and the first call on a thread attaches it in 40 to 90 µs; the numbers are in the thread and the guide. The native data source and columnar suites pass locally with a Java 25 test JVM. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
