Re: [I] Inability to cancel certain plans [datafusion]
pepijnve commented on issue #16193: URL: https://github.com/apache/datafusion/issues/16193#issuecomment-2915032807 I'll change the title to be less broad and focus on cancelling aggregations instead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] Inability to cancel certain plans [datafusion]
alamb commented on issue #16193: URL: https://github.com/apache/datafusion/issues/16193#issuecomment-2914274753 > [@alamb](https://github.com/alamb) how are you guys handling general housekeeping in the issue tracker? Should I close this issue since it's a specific example of [#15314](https://github.com/apache/datafusion/issues/15314)? That would be fine. Alternately, I think we can close this ticket once https://github.com/apache/datafusion/pull/16196 is merged too -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] Inability to cancel certain plans [datafusion]
pepijnve commented on issue #16193: URL: https://github.com/apache/datafusion/issues/16193#issuecomment-2913983580 @alamb how are you guys handling general housekeeping in the issue tracker? Should I close this issue since it's a specific example of #15314? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] Inability to cancel certain plans [datafusion]
pepijnve commented on issue #16193: URL: https://github.com/apache/datafusion/issues/16193#issuecomment-2913693236 Yes this is more or less the same issue. PR #14028 proposed adding a yield point at the leaf of the plan when moving from one file to the next. This PR adds yield points closer to the top of the plan tree just below the AggregateExec's stream by wrapping its input and then yields every 64 input batches. I was wondering if that should be row count or time interval based rather than batch count based. The comments on PR #14028 regarding Tokio's `yield_now` are interesting and relevant for PR #16196 as well. Seems like the code pattern should be ``` context::defer(cx.waker()); return Poll::Pending; ``` rather than ``` cx.waker().wake_by_ref(); return Poll::Pending; ``` I can run some tests to see what the actual behavior is in the ST and MT Tokio runtimes if that helps. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] Inability to cancel certain plans [datafusion]
alamb commented on issue #16193: URL: https://github.com/apache/datafusion/issues/16193#issuecomment-2913495619 I believe this sounds similar to from this PR from @jeffreyssmith2nd - https://github.com/apache/datafusion/issues/14036 @carols10cents has made a benchmark here: - https://github.com/apache/datafusion/pull/14818 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] Inability to cancel certain plans [datafusion]
zhuqi-lucas commented on issue #16193: URL: https://github.com/apache/datafusion/issues/16193#issuecomment-2912804691 Submitted the PR https://github.com/apache/datafusion/issues/16193 The PR is limited to solve aggregate with no group streaming, we can extend to more cases if it's not affecting performance? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [I] Inability to cancel certain plans [datafusion]
zhuqi-lucas commented on issue #16193: URL: https://github.com/apache/datafusion/issues/16193#issuecomment-2912756722 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
