Hari Hadoop config or in the flume config
> On Dec 10, 2013, at 11:30, Hari Shreedharan <[email protected]> wrote: > > The reason for this is the direct memory allocations by HDFS codecs. Reduce > your maxOpenFiles and idleTimeout to have the bucket writers garbage > collected regularly. > > > Thanks, > Hari > > >> On Tuesday, December 10, 2013 at 11:19 AM, Roshan Naik wrote: >> >> Flume version: 1.4 (compiled with hadoop 2) >> HDFS version: >> >> I have the following agent config: >> - 1 avro source, (threads = 24, deflate compression) >> - 1 file channel >> - 4 hdfs sinks (thread pool size 2, write to a new hdfs directory every 5 >> min, bzip2) >> >> Event size ~500bytes. >> Physical RAM : 64gb >> >> >> The java max heap size is capped at 8gb and the actual java heap consumption >> on the running instance is well below that (few hundred mb). >> However I am noticing in the 'top' output that the total virtual memory size >> and resident set size keep steadily increasing over time (well beyond 8gb). >> Once the total Resident set size of all the process comes close to about >> size of physical RAM (flume consuming 95% of it), the operating system nukes >> the flume process leaving no trace of this death in the flume logs. >> >> Here is the first sample top output. >> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >> >> 15625 root 20 0 30.1g 10g 53m S 2.0 16.0 40:45.93 java >> >> >> Here is one after a few hours >> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >> 15625 root 20 0 61.3g 36g 1072 S 12.0 57.4 126:10.98 java >> >> >> >> >> When i replace the HDFS sink with null sink, the problem goes away and the >> process remains very stable. So Fchannel does not seem to be the culprit. >> >> sample config is attached >> CONFIDENTIALITY NOTICE >> NOTICE: This message is intended for the use of the individual or entity to >> which it is addressed and may contain information that is confidential, >> privileged and exempt from disclosure under applicable law. If the reader of >> this message is not the intended recipient, you are hereby notified that any >> printing, copying, dissemination, distribution, disclosure or forwarding of >> this communication is strictly prohibited. If you have received this >> communication in error, please contact the sender immediately and delete it >> from your system. Thank You. >
