[
https://issues.apache.org/jira/browse/HADOOP-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
dhruba borthakur updated HADOOP-1942:
-------------------------------------
Attachment: transactionLogSync5.patch
Merged patch with latest trunk. I also removed resetting the ThreadLocal txid
in logSync(). In future, if we need to implement logSyncTillNow() then we can
implement that part of it. I also think that the time measurement to write the
transaction into memory (logEdit) might be helpful. Espeically, if in future we
decide to compare transaction log resident in ram vs. nvram. The two calls to
retrieve system time should not add much overhead.
About removing the "synchronized (editStream)": this one makes sense, but let
me ponder over it for a day.
> Increase the concurrency of transaction logging to edits log
> ------------------------------------------------------------
>
> Key: HADOOP-1942
> URL: https://issues.apache.org/jira/browse/HADOOP-1942
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Priority: Blocker
> Fix For: 0.15.0
>
> Attachments: transactionLogSync.patch, transactionLogSync2.patch,
> transactionLogSync3.patch, transactionLogSync4.patch,
> transactionLogSync5.patch
>
>
> For some typical workloads, the throughput of the namenode is bottlenecked by
> the rate of transactions that are being logged into tghe edits log. In the
> current code, a batching scheme implies that all transactions do not have to
> incur a sync of the edits log to disk. However, the existing batch-ing scheme
> can be improved.
> One option is to keep two buffers associated with edits file. Threads write
> to the primary buffer while holding the FSNamesystem lock. Then the thread
> release the FSNamesystem lock, acquires a new lock called the syncLock, swaps
> buffers, and flushes the old buffer to the persistent store. Since the
> buffers are swapped, new transactions continue to get logged into the new
> buffer. (Of course, the new transactions cannot complete before this new
> buffer is sync-ed).
> This approach does a better job of batching syncs to disk, thus improving
> performance.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
