Hi Alex, The memory is getting used at flume agent. Setting the variables MASTER_WATCHDOG & MASTERI_WATCHDOG at flume master will work for agent memory as well? Or do we have some other option as flume agent?
Regards, Deepak -----Original Message----- From: Alexander Alten-Lorenz [mailto:[email protected]] Sent: Thursday, January 17, 2013 2:18 AM To: [email protected] Subject: Re: Flume Service Takes Lot of Memory Hi, To add JVM options in flume we have to edit the watchdog line in bin/flume into: MASTER_WATCHDOG="java -XX:+HeapDumpOnOutOfMemoryError ${JOPTS} -Dpid=$$ -Dpidfile=$MASTERPID $WATCHDOG_CLASS" MASTERI_WATCHDOG="java -XX:+HeapDumpOnOutOfMemoryError ${JOPTS} -Dpid=$$ -Dpidfile=$MASTERPID $WATCHDOG_CLASS" and restart Flume master. flume-env.sh will not be used, because bin/flume honors only the variables FLUME_PID_DIR and FLUME_IDENT_STRING Ensure that you have provided the jvm with sufficient heap space. By default Flume starts the jvm with it's default heap allocation, which differs depending on the jvm version, the host type (os, 32/64 bit, etc...), total host memory available, as well as other issues. The environment variable UOPTS can be used to pass additional jvm parameters when running Flume. e.g. $ UOPTS="-Xms1g -Xmx2g" bin/flume node which starts a flume node with an initial heap of one gig and a max heap of two gig. See "java -h" or "java -X" for more details on available jvm options. cheers, Alex On Jan 17, 2013, at 8:10 AM, "Kumar, Deepak8" <[email protected]> wrote: > Hi Alex, > Thanks for assisting. I do have the following settings in flume-env.sh > file export UOPTS="-Xms256m -Xmx512m" > > But once I do top, it shows this much virtual memory > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 8571 root 21 0 1209m 424m 11m S 2.0 5.3 53:05.95 java > -Dflume.log.dir=/var/log/flume 4957 root > > Regards, > Deepak > > -----Original Message----- > From: Alexander Alten-Lorenz [mailto:[email protected]] > Sent: Thursday, January 17, 2013 1:58 AM > To: [email protected] > Subject: Re: Flume Service Takes Lot of Memory > > Hi Deepak, > > This is a normal behavior, you can tweak flume's (OG) settings by adding some > Xmx params: > > UOPTS="-Xms1g -Xmx2g" bin/flume node > > Of course you can also tweak the bin/flume script. > > - Alex > > On Jan 17, 2013, at 7:34 AM, "Kumar, Deepak8" <[email protected]> wrote: > >> Hi, >> Could you please guide? >> >> Regards, >> Deepak >> >> -----Original Message----- >> From: Kumar, Deepak8 [CCC-OT_IT NE] >> Sent: Wednesday, January 16, 2013 1:29 PM >> To: [email protected] >> Subject: Flume Service Takes Lot of Memory >> >> Hi, >> I am using flume OG 0.9, the flume agent seems to take almost 1Gig Virtual >> Memory. Is it uses this much normally or do we have some configuration >> settings? >> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >> 8571 root 21 0 1209m 424m 11m S 2.0 5.3 53:05.95 java >> -Dflume.log.dir=/var/log/flume 4957 root >> >> Regards, >> Deepak > > -- > Alexander Alten-Lorenz > http://mapredit.blogspot.com > German Hadoop LinkedIn Group: http://goo.gl/N8pCF > -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF
