connec commented on issue #17169: URL: https://github.com/apache/datafusion/issues/17169#issuecomment-3191250106
`DISTINCT ON (...)` requires the query to be sorted on those fields. I guess that should allow the deduplication to happen in a streaming fashion on the sorted results? I.e. (very imprecise) calculate a key for each row as its visited. Remember just the latest key. Skip a row if the key matches the latest or update the latest key if not (plus merge deduplication if this is happening in multiple partitions). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
