[
https://issues.apache.org/jira/browse/HDFS-12978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16488293#comment-16488293
]
Konstantin Shvachko edited comment on HDFS-12978 at 5/24/18 1:48 AM:
---------------------------------------------------------------------
[~linyiqun] I don't think that {{Long.MAX_VALUE = 0x7fffffffffffffff}} is such
a long number. Even if it is, what's wrong with long numbers in configuration.
I am not a fan of special values like {{-1}}, why not {{-1177}}, as they don't
have meaning.
[~xkrogen] good point. I was targeting simplicity. There will be indeed
overhead re-initializing edits streams. My approach here is to get the feature
in, then measure the overhead and optimize as necessary.
Optimization doesn't look simple. When SBN is applying edits it holds like four
different locks, some of them redundantly. So some refactoring of this part
will be needed. But if the fast-pass works well we may not even need this.
I'll update the patch to take care of {{TestHdfsConfigFields}}, which hints to
adding the new config parameter into hdfs-default.xml.
I was thinking if we should keep it as undocumented parameter for now?
was (Author: shv):
[~linyiqun] I don't think that {{Long.MAX_VALUE = 0x7fffffffffffffff}} is such
a long number. Even if it is what's wrong with long numbers in configuration. I
not a fan of special values like {{-1}}, why not {{-1177}}, as they don't have
meaning.
[~xkrogen] good point. I was targeting simplicity. There will be indeed
overhead re-initializing edits streams. My approach here is to get the feature
in, then measure the overhead and optimize as necessary.
Optimization doesn't look simple. When SBN is applying edits it holds like four
different locks, some of them redundantly. So some refactoring of this part
will be needed. But if the fast-pass works well we may not even need this.
I'll update the patch to take care of {{TestHdfsConfigFields}}, which hints to
adding the new config parameter into hdfs-default.xml.
I was thinking if we should keep it as undocumented parameter for now?
> Fine-grained locking while consuming journal stream.
> ----------------------------------------------------
>
> Key: HDFS-12978
> URL: https://issues.apache.org/jira/browse/HDFS-12978
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Priority: Major
> Attachments: HDFS-12978.001.patch
>
>
> In current implementation SBN consumes the entire segment of transactions
> under a single namesystem lock, which does not allow reads over a long period
> of time until the segment is processed. We should break the lock into fine
> grained chunks. In extreme case each transaction should release the lock once
> it is applied.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
