[
https://issues.apache.org/jira/browse/HDFS-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12868928#action_12868928
]
Konstantin Shvachko commented on HDFS-1112:
-------------------------------------------
A couple of somments.
# {{FSEditLog.isTimeToSync()}} should be {{shouldForceSync()}} with reversed
boolean return value, to make it consistent with the EditLogOutputStream method.
# Is it possible to reuse the flag {{isSyncRunning}} instead of new
{{isAutoSyncScheduled}}? The problem is that we {{waitForSyncToFinish()}} in
many operations, which makes sure that {{isSyncRunning == false}}. We should
probably do the same with {{isAutoSyncScheduled}} if there is no way to use one
flag for both conditions.
# This is the main concern. If {{logSync()}} fails, then
{{isAutoSyncScheduled}} will remain set. Nobody will be able to unset it, and
everybody will be waiting for the sync to finish when nobody is syncing.
# I see you had to add throwing IOException in a bunch of methods. The reason
is that {{logSync()}} throws IOEception, and therefore now the {{logEdit()}}
should do it too. I think we should fix {{logSync()}}, because it should not
throw IOException. The only reason {{logSync()}} throws is
{{setReadyToFlush()}}. But call to {{setReadyToFlush()}} should be treated the
same way as the call to {{flush()}}, that is it should catch the exception and
{{processIOError()}}. I think current treatment of {{setReadyToFlush()}} is
incorrect.
> Edit log buffer should not grow unboundedly
> -------------------------------------------
>
> Key: HDFS-1112
> URL: https://issues.apache.org/jira/browse/HDFS-1112
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 0.22.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.22.0
>
> Attachments: editLogBuf.patch, editLogBuf1.patch
>
>
> Currently HDFS does not impose an upper limit on the edit log buffer. In case
> there are a large number of open operations coming in with access time update
> on, since open does not call sync automatically, there is a possibility that
> the buffer grow to a large size, therefore causes memory leak and full GC in
> extreme cases as described in HDFS-1104.
> The edit log buffer should be automatically flushed when the buffer becomes
> full.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
