[
https://issues.apache.org/jira/browse/HADOOP-3110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12582847#action_12582847
]
Pete Wyckoff commented on HADOOP-3110:
--------------------------------------
Not saying the NN is causing the JT lock up - just that it is doing the same
thing and can cause things to lock up.
This isn't really about optimizations - the main problem is that in general,
Hadoop does not treat synchronized methods as critical sections - which they
are. In theory, I would classify logging in a critical section as a bug. I
understand the desire to have log information about operations - most people do
this by logging after the critical section :) But, admittedly, that can cause
out of order problems but usually that's not required. Or people would use
something like an async appender or something like that.
-- pete
> NameNode does logging in critical sections just about everywhere
> ----------------------------------------------------------------
>
> Key: HADOOP-3110
> URL: https://issues.apache.org/jira/browse/HADOOP-3110
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.14.0, 0.14.1, 0.14.2, 0.14.3, 0.14.4, 0.15.0, 0.15.1,
> 0.15.2, 0.15.3, 0.16.0, 0.16.1
> Environment: All
> Reporter: Pete Wyckoff
>
> e.g., FSNameSystem.addStoredBlock (but almost every method has logging in its
> critical sections)
> This method is synchronized and it's spitting something out to Log.info every
> block stored. Normally not a big deal, but since this is in the name node and
> these are critical sections...
> We shouldn't even do any logging at all in critical sections, so even the
> info and warn are bad. But, in many places in the code, it would be hard to
> tease these out (although eventually they really should be), but the system
> could start using something like an AsyncAppender and see how it improves
> performance.
> Even though the log may have a buffer, the writing and doing the formatting
> and stuff cause a drag on performance with 100s/1000s of machines trying to
> talk to the name node.
> At a minimum, the most often triggered Log.info could be changed to
> Log.debug.
> for reference:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/AsyncAppender.html
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
