Sorry for the late reply.Thanks ATM, Colin and others. Indeed applying the
patch took care of most of the memory issues. Now we are tackling the sudden
increase in the number of active threads. We will post more concrete problem
definition shortly.
Regards,Mohammad
On Tuesday, September 22, 2015 9:29 AM, Colin P. McCabe
<cmcc...@apache.org> wrote:
Hi Mohammad,
Like ATM said, HDFS-8965 is an important fix in this area. We have
found that it prevents cases where INotify tries to read invalid
sequences of bytes (sometimes because the edit log was truncated or
corrupted; other times because it is in the middle of a write).
HDFS-8964 fixes the "attempt to read edit log entries that are still
being written." Both fixes have value when using INotify.
The other question is how many INotify clients you have
simultaneously? Right now, INotify works best with a small number of
clients, since it involves doing RPCs to the NameNode. There is some
work going on to scale out INotify to support many clients in
HDFS-8940.
best,
Colin
On Tue, Sep 15, 2015 at 11:52 AM, Mohammad Islam
<misla...@yahoo.com.invalid> wrote:
> Hi,We were using INotify feature in one of our internal service. Looks like
> it creates a lot of memory pressure on NN. Memory usage goes very high and
> remains the same causing expensive GC.
> Did anyone use this feature in any service? Is there any con to setup? We are
> using latest CDH.
> Regards,Mohammad
>
