HyukjinKwon commented on PR #59313: URL: https://github.com/apache/spark/pull/59313#issuecomment-6157138031
@LuciferYang Thanks for running the suites again. On memory, aa0f3ba4dd77 adds a Memory section to the docs page. It says that the memory of a library, the Arrow buffers that it exports and the batches that Spark writes are not accounted by Spark, and that this memory comes out of `spark.executor.memoryOverhead` (and `spark.driver.memoryOverhead` on the driver). It also says that the size of a read batch is up to the library. Thanks for the pointers to Gluten's `ManagedAllocationListener` and `ReservationListener`, which I'll look at for the design of memory accounting that @tdcmeehan asked for. @tokoko Thanks, that's an interesting direction for the C ABI of #59424. Version 1 needs a few things that ADBC does not have today: streaming reads with offsets, the protocol of distributed writes (task commits with messages, aborts, and epochs for streaming writes), and the pushdown semantics of Spark. With the ingest API that you are working on (apache/arrow-adbc#4317), building on ADBC with extensions where needed could be worth exploring for that interface, so that one driver serves several engines. @gengliangwang @gerashegalov Thanks, all addressed in aa0f3ba4dd77, with replies inline. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
