viirya commented on PR #59582: URL: https://github.com/apache/spark/pull/59582#issuecomment-6561237904
@dongjoon-hyun, about your [comment on the Arrow cache](https://github.com/apache/spark/pull/59581#discussion_r4318377875) on #59581: this PR takes the other option from #59551, closing a root only once none of its vectors is in use, rather than marking the cache vectors. `ArrowCachedBatchToColumnarBatchIterator` no longer closes the previous root in `next()`: each root is closed once all the columns of its batch are released through `closeIfFreeable`, and the roots still open are closed when the task completes. That keeps cache + UDF on the Arrow columnar path; the held roots stay in memory until their results come back, as for any input on that path. Marking the vectors with `ReusedArrowColumnVector` would be simpler, but would send cache + UDF to the row queue path. `SPARK-60325: Arrow cache: the pass-through columns match the UDF results` covers the case you described, over 200 batches of 1,000 rows. Would you prefer to keep this, or mark the vectors? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
