Hi,
I've a situation where the number of elements output by each partition from mapPartitions don't fit into the RAM even with the lowest number of rows in the partition (there is a hard lower limit on this value). What's the best way to address this problem? During the mapPartition phase, is there a way to convert intermediate results to a DF and save to a database? Rows saved to database don't need to be part of the output results from mapPartitions.
