[GitHub] [spark] atronchi commented on pull request #18990: [SPARK-21782][Core] Repartition creates skews when numPartitions is a power of 2

[via GitHub]

atronchi commented on PR #18990:
URL: https://github.com/apache/spark/pull/18990#issuecomment-1467104804

   This issue appears to remain at large for the dataframe API which is used 
more broadly than RDD. What would it take to extend the fix to the dataframe 
API? 
   
   I verified this on Spark 3.2 using `df.repartition(1024)` on a dataframe 
with ~200k rows, which resulted in almost 30% EMPTY partitions, and the below 
shown skew of the remaining non-empty ones.
   <img width="419" alt="Screenshot 2023-03-13 at 10 09 24 AM" 
src="https://user-images.githubusercontent.com/4906224/224852367-a33564c8-21f2-4daa-8da2-cbca0389594a.png";>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org