The driver does not do processing (for the most part), but it does do scheduling and block management, so it can keep around a significant amount of metadata about all the stored RDD and broadcast blocks, as well as statistics about prior executions. Much of this is bounded in some way, though, so it is unusual for the driver to use too much memory. If you could get a heap dump (using jmap) of the driver when it's using a lot of memory, that would likely be very informative about what exactly is piling up.
On Mon, Apr 7, 2014 at 3:12 PM, Eduardo Costa Alfaia <e.costaalf...@unibs.it > wrote: > Hi Guys, > > I would like understanding why the Driver's RAM goes down, Does the > processing occur only in the workers? > Thanks > ##### Start Tests > computer1(Worker/Source Stream) > 23:57:18 up 12:03, 1 user, load average: 0.03, 0.31, 0.44 > total used free shared buffers cached > Mem: 3945 1084 2860 0 44 827 > -/+ buffers/cache: 212 3732 > Swap: 0 0 0 > computer8 (Driver/Master) > 23:57:18 up 11:53, 5 users, load average: 0.43, 1.19, 1.31 > total used free shared buffers cached > Mem: 5897 4430 1466 0 384 2662 > -/+ buffers/cache: 1382 4514 > Swap: 0 0 0 > computer10(Worker/Source Stream) > 23:57:18 up 12:02, 1 user, load average: 0.55, 1.34, 0.98 > total used free shared buffers cached > Mem: 5897 564 5332 0 18 358 > -/+ buffers/cache: 187 5709 > Swap: 0 0 0 > computer11(Worker/Source Stream) > 23:57:18 up 12:02, 1 user, load average: 0.07, 0.19, 0.29 > total used free shared buffers cached > Mem: 3945 603 3342 0 54 355 > -/+ buffers/cache: 193 3751 > Swap: 0 0 0 > > #### After 2 Minutes > > computer1 > 00:06:41 up 12:12, 1 user, load average: 3.11, 1.32, 0.73 > total used free shared buffers cached > Mem: 3945 2950 994 0 46 1095 > -/+ buffers/cache: 1808 2136 > Swap: 0 0 0 > computer8(Driver/Master) > 00:06:41 up 12:02, 5 users, load average: 1.16, 0.71, 0.96 > total used free shared buffers cached > Mem: 5897 5191 705 0 385 2792 > -/+ buffers/cache: 2014 3882 > Swap: 0 0 0 > computer10 > 00:06:41 up 12:11, 1 user, load average: 2.02, 1.07, 0.89 > total used free shared buffers cached > Mem: 5897 2567 3329 0 21 647 > -/+ buffers/cache: 1898 3998 > Swap: 0 0 0 > computer11 > 00:06:42 up 12:12, 1 user, load average: 3.96, 1.83, 0.88 > total used free shared buffers cached > Mem: 3945 3542 402 0 57 1099 > -/+ buffers/cache: 2385 1559 > Swap: 0 0 0 > > > Informativa sulla Privacy: http://www.unibs.it/node/8155
