Resource contention. Now all the CPU and I/O is competing and probably slows down
On Wed, Oct 26, 2022, 5:37 AM eab...@163.com <eab...@163.com> wrote: > Hi All, > > I have a CDH5.16.2 hadoop cluster with 1+3 nodes(64C/128G, 1NN/RM + > 3DN/NM), and yarn with 192C/240G. I used the following test scenario: > > 1.spark app resource with 2G driver memory/2C driver vcore/1 executor > nums/2G executor memory/2C executor vcore. > 2.one spark app will use 5G4C on yarn. > 3.first, I only run one spark app takes 40s. > 4.Then, I run 30 the same spark app at once, and each spark app takes 80s > on average. > > So, I want to know why the run time gap is so big, and how to optimize? > > Thanks > >
