westonpace commented on pull request #10845:
URL: https://github.com/apache/arrow/pull/10845#issuecomment-901376192
@michalursa A couple of questions:
* For a "serial scan" can "serial" mean "thread pool of size 1"? Or does it
mean "use only the calling thread?" Or does it mean "only deliver one batch at
a time?" Also, in a serial scan is it ok that extra threads are used in the
I/O context for reading? I think it is easy to do a serial scan if serial
means "Only 1 CPU thread in the CPU thread pool" or "Do not call input received
until the previous call to input received has finished".
* It sounds like the root cause might be the unit test thread being reused.
I could see this happening with the existing code regardless of how we define
"serial scan" (and in fact, this would be necessary if we define a serial scan
as "use only the calling thread"). The "transfer" is probably not "taking"
because of future chaining issues. This is a very hard thing to explain.
However, an easy fix is to start the scan from a CPU thread pool thread. I can
try and look at this tomorrow. I don't think I can get to it today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
