shner-elmo opened a new issue, #35269:
URL: https://github.com/apache/arrow/issues/35269

   ### Describe the usage question you have. Please include as many useful 
details as  possible.
   
   
   Hello, I was wondering if there is a more eficcient way to partition a very 
large dataset (6B records) by a column that has many unique values.
   I have a column called ID that has 400k unique values (integer ranging from 
`0` to `9698193`) and I'm wondering if there is a way I can partition it with 
either the first N digits of a given ID, or create something like a B-tree 
where you group by all the records with the ID between X and Y in a directory.
   
   And in general is there a way to create custom partitioning? (like maybe by 
passing a function that takes as input the value and returns the value to use 
for partitioning, or subclassing a class)
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to