ding-young commented on issue #16482:
URL: https://github.com/apache/datafusion/issues/16482#issuecomment-3004872124
Hi, I just came across this issue and wanted to leave a comment after
working around writing some test codes.
I wrote a test in my local branch where I tried to simulate the behavior of
`read_spill_as_stream` as closely as possible, while manually controlling the
producer and consumer stream behaviors.
- The producer stream is designed to always return Poll::Ready(Some(...))
immediately.
- The consumer stream loops tightly over the stream returned from
`spawn_buffered`, polling repeatedly as long as it returns Ready.
- The buffer capacity for `spawn_buffered` is intentionally set large to
avoid triggering a yield.
The goal is to approximate a scenario where the cooperative yielding
behavior must come from the stream wrapper (i.e., CooperativeStream) and not
from channel backpressure or natural I/O waits.
Could you confirm whether this matches the intention? Here's some code in my
[local
branch](https://github.com/apache/datafusion/compare/main...ding-young:datafusion:test-cancel-spill#diff-eba449dc026a6c39c12232a28b4a9b24380fc81ebbcb0fe279bd5b631c50badbR30)
I may not have fully understood it, so please feel free to correct me if I
got something wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
