westonpace opened a new pull request #10758: URL: https://github.com/apache/arrow/pull/10758
A couple of things were at play here. First, the parallel CSV reader added a cleanup step to the async vector generator which made it no longer sync-reentrant safe. This served as a useful canary to detect this issue. Second, there was a test in place to ensure that the merged generator did not pull from the individual subscriptions in a sync-reentrant manner. However, this test was not guarding the source generator. Finally, the bug itself was pretty straightforward. The source could be pulled whenever a subscription finished and so if two subscriptions finished at the same time it would cause a race condition (or if a subscription finished very quickly and the initial pull was still adding subscriptions). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
