Hi Srikanth, what is the spark version that you are using?
Can you tell us the data dictionary and the PK? Also if possible the data volumes that you are dealing with? Thanks and Regards, Gourav Sengupta On Thu, Dec 2, 2021 at 4:33 PM Shrikanth J R <shrika...@thinkdeeply.com> wrote: > Hi, > > I am facing an issue when exploding multiple array columns where few > columns have a huge element list. Repartition dataset before explode > doesn't work Getting memory and space issue though file size before explode > is ~16mb and Ram is 32gb with ~150gb. What is the efficient way of handling > > Regards, > Shrikanth R. >
