Re: how to add a column for percent

2022-05-23 Thread Raghavendra Ganesh
withColumn takes a column as the second argument, not string. If you want formatting before show() you can use the round() function. -- Raghavendra On Mon, May 23, 2022 at 11:35 AM wilson wrote: > hello > > how to add a column for percent for the current row of counted data? > > scala> > df2.gr

Spark Push-Based Shuffle causing multiple stage failures

2022-05-23 Thread Han Altae-Tran
Hi, First of all, I am very thankful for all of the amazing work that goes into this project! It has opened up so many doors for me! I am a long time Spark user, and was very excited to start working with the push-based shuffle service for an academic paper we are working on, but I encountered som