Hi all, First time posting here
I’m using MERGE INTO to upsert into a table with daily partitions. More recent days tend to have many more updates, which is causing skew in the write stage when write.distribution-mode=hash (the most recent day of data will get assigned to a single task, which takes much longer to finish than older days). I tried write.distribution-mode=range instead, but this only helps a little bit. I think this does a good job of splitting up the most recent days across multiple tasks, but probably clusters the very oldest/smallest days on a single task, which is slow due to opening and closing too many small files. I’m wondering if there’s a mode that works well for this use case that I may have missed, or if not, is there any appetite for supporting one? One idea is to add an option for a user-specified column in the clustering in SparkDistributionAndOrderingUtil. This would allow the caller to provide an additional column to split up large partitions while writing, without changing the table partitioning. Thanks in advance - Ed
