westonpace opened a new pull request, #13036:
URL: https://github.com/apache/arrow/pull/13036
This builds on top of #13035 which is also important for avoiding
segmentation faults. On top of that there were a few more problems:
* The python was using `SourceNodeOptions::FromTable` which is a rather
dangerous method (mainly useful for unit testing) as it doesn't share ownership
of the input table (even worse, it takes a const ref). Python was not keeping
the table alive and it was maybe possible for the table to deleted out from
under the plan (I'm not entirely sure this was causing issues but it seemed
risky). I switched to TableSourceNode which shares ownership of the table (and
is a bit more efficient).
* Setting use_threads to False did nothing because `_perform_join` was not
passing the arg on to `execplan`.
* When fixing the above and running with `use_threads=False` it was
creating a single thread executor but the current best practice is to pass in
nullptr.
* Finally, the actual bug was my improper fix in #12894 . I had still left
a small window open for `End` to be called between `Submit` and `AddTask` which
would allow the task to be submitted but not participate in setting `finished`
on the node.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
