GitHub user leborchuk added a comment to the discussion: [Ideas] Adaptive join: correct a wrong Broadcast decision at runtime
Also good work is https://dl.acm.org/doi/pdf/10.1145/1376616.1376720 Yu Xu et. all, Teradata - Handling Data Skew in Parallel Joins in Shared-Nothing Systems. > One of the important challenges in parallel DBMSes is to effectively handle data skew in joins. Based on our observations of data warehouses in practice at various industries, we notice that skewed rows tend to be evenly partitioned on all parallel units. Motivated by the redistribution skew problem that arises naturally in business applications in the various industries, we propose an approach called PRPD (partial redistribution & partial duplication). To identify the skewed values, PRPD uses either collected or sampled statistics. Instead of redistributing these skewed rows as is done for the non-skewed rows, they are kept locally. The matching skewed value rows from the joined table are duplicated to each parallel unit to complete the join. The PRPD algorithm also handles the extreme cases where skewed rows are clustered in only one or a few parallel units by redistributing the skewed rows to all parallel units. Eliminating skewed processing eliminates system bottlenecks created by more conventional algorithms. Our experimental results have demonstrated the effectiveness of the PRPD algorithm in improving query execution time. We have also shown that the PRPD algorithm can be used in multiple joins. In addition, we are looking at handling skew dynamically to avoid reliance on collected or sampled statistics. I think it's worth studying it and trying to understand if it is applicable to Cloudberry or not. GitHub link: https://github.com/apache/cloudberry/discussions/1962#discussioncomment-18397378 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
