comphead commented on PR #1211: URL: https://github.com/apache/datafusion-comet/pull/1211#issuecomment-2568592582
> > Hmm, so it is correctness issue too? > > The correctness tests in CI seem to be passing so far .. I wonder if the SMJ is producing lots of empty batches? That would explain the issue where the output row count and batch count are quite similar. I will try and confirm if this is happening. That is possible, when filtered SMJ introduced I feel this part is skipped in DF and batches gets emitted not for `batch_size` but less that that, that is good finding -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
