GitHub user daphnenhuch-at added a comment to the discussion: How does 'sort' interact with record batches?
More specifically, I have a number of files I'm merging into a single file. And I want this single file to store the fileRowNumbers. But if partitioning is enabled, then this seems impossible to end up with a file sorted by userPrimaryKey and fileRowNumber. Since I'm merging many files into one file, I don't need to keep them segregated by file but I do need the final result to be globally sorted across all the files by userPrimaryKey with fileRowNumber applied to that sorted result GitHub link: https://github.com/apache/datafusion/discussions/15711#discussioncomment-12982790 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
