HyukjinKwon commented on PR #59934: URL: https://github.com/apache/spark/pull/59934#issuecomment-6646163586
Thank you, @dongjoon-hyun. 9711532 addresses the tenth round, together with #59931 (4a76b34), which d5b1e3d merges here; the replies are in the threads. `read` now returns exactly the columns of its schema to read, so both projections of Spark are gone, including the projection of the streams of engines. `ColumnarMicroBatchStream` plans each range once for every columnar stream reader, and the time zone member of the writers is removed. In this PR, a commit destroys the broadcasts of all the micro-batches up to it, also when the committed one kept its state inline. On a failed read, the reader closes the task after it released the stream, and the stream of an engine is released through Arrow again. The stream reader broadcasts with the `SparkContext` that it takes when it is created. I kept the conversion of the Arrow schema for each partition, which an earlier measurement put at about 5% of the export. The native and columnar suites pass locally with a Java 25 test JVM. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
