bvaradar commented on issue #2335:
URL: https://github.com/apache/hudi/issues/2335#issuecomment-745711266
@asharma4-lucid : The reason why any processing mechanism (hudi or others)
takes longer time is the second order effects that the partitioning style
causes. Having too many partitions implies the number of records per partition
(per file) is small. and the processing engine needs to create a ton of small
files (number of tasks increases).
There is no optimal number of partitions that could be suggested. It depends
on the workload. To copy from one table to another, you can use
spark.read.format("hudi")... and then write to another location in hudi format.
I have personally seen datasets with partitions in the range of 10K.
For deleting partitions, there is an open PR to support it efficiently
without individual records here : https://github.com/apache/hudi/pull/2254
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
