HyukjinKwon commented on PR #59934: URL: https://github.com/apache/spark/pull/59934#issuecomment-6643902775
Thank you, @dongjoon-hyun. 7b3fbb2 addresses the ninth round, and 466caed merges #59931 (e305457) for the two items of its code, the kept failure of `pruneColumns` and the inlined `ColumnarProjection`; the replies are in the threads. For a library that does not prune the columns, the batch scan of some of them now keeps the native path: the streams of engines project their schema and arrays in place to the read schema. A stream is released through FFM after a failure, and its task stays open if that fails, the broadcast of a micro-batch is destroyed once it is committed, the flags of the tasks are weakly keyed, the failure to create the callbacks names its root cause, and the counts of the contexts allow late closes. I kept the `Trigger.AvailableNow` end offset at the first `latestOffset`, which the Javadoc of `SupportsTriggerAvailableNow` recommends since SPARK-53560. The native and columnar suites pass locally with a Java 25 test JVM. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
