westonpace commented on issue #33640: URL: https://github.com/apache/arrow/issues/33640#issuecomment-1381330671
Turns out we hadn't added https://github.com/westonpace/arrow/commit/45791de8311b0c2e2525e72f4c4746cc3b4364e3 anyways. So I've combined both "AsofJoin backpressure" and "backpressure for large sources" into a single PR. As for the difference: The first PR (asofjoin backpressure) added backpressure to the asof join node. However, when run on large sources, it tended to crash. The main problem was the "can't shutdown plan from the process thread" problem, which has already been solved on master. So that is why it seemed that "backpressure for large sources" wasn't adding much. However, "backpressure for large sources" added unit tests to reproduce the issue, and these are valuable, so I've included them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
