Hi Matthias,
Thanks for the patch.
I have re-run the experiment and observed that there was indeed no more
memory pressure, but it still took ~90s for this simple script. I was
wondering what is the bottleneck for this case?
Total elapsed time: 94.800 sec.
Total compilation time: 1.826 sec.
Out of curiosity, I increased the driver memory to 10GB, and then all
operations were executed on CP. It took 37.166s but JVM GC took 30.534s. I
was wondering whether this is the expected behavior?
Total elapsed time: 38.093 sec.
Total compilation time: 0.926 sec.
Total execution time: 37.166
