shner-elmo opened a new issue, #35269: URL: https://github.com/apache/arrow/issues/35269
### Describe the usage question you have. Please include as many useful details as possible. Hello, I was wondering if there is a more eficcient way to partition a very large dataset (6B records) by a column that has many unique values. I have a column called ID that has 400k unique values (integer ranging from `0` to `9698193`) and I'm wondering if there is a way I can partition it with either the first N digits of a given ID, or create something like a B-tree where you group by all the records with the ID between X and Y in a directory. And in general is there a way to create custom partitioning? (like maybe by passing a function that takes as input the value and returns the value to use for partitioning, or subclassing a class) ### Component(s) Python -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
