Dandandan commented on issue #17169: URL: https://github.com/apache/datafusion/issues/17169#issuecomment-3191395841
> I.e. (very imprecise) calculate a key for each row as its visited. Remember just the latest key. Skip a row if the key matches the latest or update the latest key if not (plus merge deduplication if this is happening in multiple partitions). Yeah I think that could be how an efficient deduplicate operation could work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
