[jira] [Commented] (HDFS-14370) Edit log tailing fast-path should allow for backoff
[ https://issues.apache.org/jira/browse/HDFS-14370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16906316#comment-16906316 ] Erik Krogen commented on HDFS-14370: Committed this to branch-2 now that HDFS-14204 has been committed. > Edit log tailing fast-path should allow for backoff > --- > > Key: HDFS-14370 > URL: https://issues.apache.org/jira/browse/HDFS-14370 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode, qjm >Affects Versions: 3.3.0 >Reporter: Erik Krogen >Assignee: Erik Krogen >Priority: Major > Fix For: 2.10.0, 3.0.4, 3.3.0, 3.2.1, 3.1.3 > > Attachments: HDFS-14370.000.patch, HDFS-14370.001.patch, > HDFS-14370.002.patch, HDFS-14370.003.patch, HDFS-14370.004.patch, > HDFS-14370.005.patch > > > As part of HDFS-13150, in-progress edit log tailing was changed to use an > RPC-based mechanism, thus allowing the edit log tailing frequency to be > turned way down, and allowing standby/observer NameNodes to be only a few > milliseconds stale as compared to the Active NameNode. > When there is a high volume of transactions on the system, each RPC fetches > transactions and takes some time to process them, self-rate-limiting how > frequently an RPC is submitted. In a lightly loaded cluster, however, most of > these RPCs return an empty set of transactions, consuming a high > (de)serialization overhead for very little benefit. This was reported by > [~jojochuang] in HDFS-14276 and I have also seen it on a test cluster where > the SbNN was submitting 8000 RPCs per second that returned empty. > I propose we add some sort of backoff to the tailing, so that if an empty > response is received, it will wait a longer period of time before submitting > a new RPC. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14370) Edit log tailing fast-path should allow for backoff
[ https://issues.apache.org/jira/browse/HDFS-14370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902228#comment-16902228 ] Erik Krogen commented on HDFS-14370: Thanks [~ayushtkn]. Addressed this last comment in v005 and went ahead to commit based on the three +1s in this thread. I committed to trunk and backported to the 3.x line. I think this needs to land in branch-2 as well once HDFS-14204 is completed, so I will leave this open for now. > Edit log tailing fast-path should allow for backoff > --- > > Key: HDFS-14370 > URL: https://issues.apache.org/jira/browse/HDFS-14370 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode, qjm >Affects Versions: 3.3.0 >Reporter: Erik Krogen >Assignee: Erik Krogen >Priority: Major > Attachments: HDFS-14370.000.patch, HDFS-14370.001.patch, > HDFS-14370.002.patch, HDFS-14370.003.patch, HDFS-14370.004.patch, > HDFS-14370.005.patch > > > As part of HDFS-13150, in-progress edit log tailing was changed to use an > RPC-based mechanism, thus allowing the edit log tailing frequency to be > turned way down, and allowing standby/observer NameNodes to be only a few > milliseconds stale as compared to the Active NameNode. > When there is a high volume of transactions on the system, each RPC fetches > transactions and takes some time to process them, self-rate-limiting how > frequently an RPC is submitted. In a lightly loaded cluster, however, most of > these RPCs return an empty set of transactions, consuming a high > (de)serialization overhead for very little benefit. This was reported by > [~jojochuang] in HDFS-14276 and I have also seen it on a test cluster where > the SbNN was submitting 8000 RPCs per second that returned empty. > I propose we add some sort of backoff to the tailing, so that if an empty > response is received, it will wait a longer period of time before submitting > a new RPC. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14370) Edit log tailing fast-path should allow for backoff
[ https://issues.apache.org/jira/browse/HDFS-14370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902219#comment-16902219 ] Hudson commented on HDFS-14370: --- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17058 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17058/]) HDFS-14370. Add exponential backoff to the edit log tailer to avoid (xkrogen: rev 827dbb11e24be294b40088a8aa46086ba8ca4ba8) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestEditLogTailer.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ObserverNameNode.md * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml > Edit log tailing fast-path should allow for backoff > --- > > Key: HDFS-14370 > URL: https://issues.apache.org/jira/browse/HDFS-14370 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode, qjm >Affects Versions: 3.3.0 >Reporter: Erik Krogen >Assignee: Erik Krogen >Priority: Major > Attachments: HDFS-14370.000.patch, HDFS-14370.001.patch, > HDFS-14370.002.patch, HDFS-14370.003.patch, HDFS-14370.004.patch, > HDFS-14370.005.patch > > > As part of HDFS-13150, in-progress edit log tailing was changed to use an > RPC-based mechanism, thus allowing the edit log tailing frequency to be > turned way down, and allowing standby/observer NameNodes to be only a few > milliseconds stale as compared to the Active NameNode. > When there is a high volume of transactions on the system, each RPC fetches > transactions and takes some time to process them, self-rate-limiting how > frequently an RPC is submitted. In a lightly loaded cluster, however, most of > these RPCs return an empty set of transactions, consuming a high > (de)serialization overhead for very little benefit. This was reported by > [~jojochuang] in HDFS-14276 and I have also seen it on a test cluster where > the SbNN was submitting 8000 RPCs per second that returned empty. > I propose we add some sort of backoff to the tailing, so that if an empty > response is received, it will wait a longer period of time before submitting > a new RPC. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14370) Edit log tailing fast-path should allow for backoff
[
https://issues.apache.org/jira/browse/HDFS-14370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902220#comment-16902220
]
Hadoop QA commented on HDFS-14370:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HDFS-14370 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-14370 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12976955/HDFS-14370.005.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/27441/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Edit log tailing fast-path should allow for backoff
> ---
>
> Key: HDFS-14370
> URL: https://issues.apache.org/jira/browse/HDFS-14370
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode, qjm
>Affects Versions: 3.3.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-14370.000.patch, HDFS-14370.001.patch,
> HDFS-14370.002.patch, HDFS-14370.003.patch, HDFS-14370.004.patch,
> HDFS-14370.005.patch
>
>
> As part of HDFS-13150, in-progress edit log tailing was changed to use an
> RPC-based mechanism, thus allowing the edit log tailing frequency to be
> turned way down, and allowing standby/observer NameNodes to be only a few
> milliseconds stale as compared to the Active NameNode.
> When there is a high volume of transactions on the system, each RPC fetches
> transactions and takes some time to process them, self-rate-limiting how
> frequently an RPC is submitted. In a lightly loaded cluster, however, most of
> these RPCs return an empty set of transactions, consuming a high
> (de)serialization overhead for very little benefit. This was reported by
> [~jojochuang] in HDFS-14276 and I have also seen it on a test cluster where
> the SbNN was submitting 8000 RPCs per second that returned empty.
> I propose we add some sort of backoff to the tailing, so that if an empty
> response is received, it will wait a longer period of time before submitting
> a new RPC.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14370) Edit log tailing fast-path should allow for backoff
[
https://issues.apache.org/jira/browse/HDFS-14370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901646#comment-16901646
]
Ayush Saxena commented on HDFS-14370:
-
Thanx [~xkrogen] for the patch.
{code:java}
+ maxSleepTimeMsTemp, DFSConfigKeys.DFS_HA_TAILEDITS_PERIOD_KEY);
+ maxSleepTimeMs = -1;
{code}
Here also we should set 0, to make it look consistent with other parts of code.
> Edit log tailing fast-path should allow for backoff
> ---
>
> Key: HDFS-14370
> URL: https://issues.apache.org/jira/browse/HDFS-14370
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode, qjm
>Affects Versions: 3.3.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-14370.000.patch, HDFS-14370.001.patch,
> HDFS-14370.002.patch, HDFS-14370.003.patch, HDFS-14370.004.patch
>
>
> As part of HDFS-13150, in-progress edit log tailing was changed to use an
> RPC-based mechanism, thus allowing the edit log tailing frequency to be
> turned way down, and allowing standby/observer NameNodes to be only a few
> milliseconds stale as compared to the Active NameNode.
> When there is a high volume of transactions on the system, each RPC fetches
> transactions and takes some time to process them, self-rate-limiting how
> frequently an RPC is submitted. In a lightly loaded cluster, however, most of
> these RPCs return an empty set of transactions, consuming a high
> (de)serialization overhead for very little benefit. This was reported by
> [~jojochuang] in HDFS-14276 and I have also seen it on a test cluster where
> the SbNN was submitting 8000 RPCs per second that returned empty.
> I propose we add some sort of backoff to the tailing, so that if an empty
> response is received, it will wait a longer period of time before submitting
> a new RPC.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14370) Edit log tailing fast-path should allow for backoff
[
https://issues.apache.org/jira/browse/HDFS-14370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901402#comment-16901402
]
Hadoop QA commented on HDFS-14370:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 39s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 37s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 478 unchanged - 0 fixed = 480 total (was 478) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 39s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 82m 53s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}137m 19s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestLargeBlockReport |
| | hadoop.hdfs.server.balancer.TestBalancer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14370 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12976833/HDFS-14370.004.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 59e7db8d7d1f 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / b8bf09b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/27418/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
[jira] [Commented] (HDFS-14370) Edit log tailing fast-path should allow for backoff
[ https://issues.apache.org/jira/browse/HDFS-14370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16901273#comment-16901273 ] Erik Krogen commented on HDFS-14370: Thanks for looking [~ayushtkn]. I've updated the documentation and changed the default to a 0 in v004. > Edit log tailing fast-path should allow for backoff > --- > > Key: HDFS-14370 > URL: https://issues.apache.org/jira/browse/HDFS-14370 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode, qjm >Affects Versions: 3.3.0 >Reporter: Erik Krogen >Assignee: Erik Krogen >Priority: Major > Attachments: HDFS-14370.000.patch, HDFS-14370.001.patch, > HDFS-14370.002.patch, HDFS-14370.003.patch, HDFS-14370.004.patch > > > As part of HDFS-13150, in-progress edit log tailing was changed to use an > RPC-based mechanism, thus allowing the edit log tailing frequency to be > turned way down, and allowing standby/observer NameNodes to be only a few > milliseconds stale as compared to the Active NameNode. > When there is a high volume of transactions on the system, each RPC fetches > transactions and takes some time to process them, self-rate-limiting how > frequently an RPC is submitted. In a lightly loaded cluster, however, most of > these RPCs return an empty set of transactions, consuming a high > (de)serialization overhead for very little benefit. This was reported by > [~jojochuang] in HDFS-14276 and I have also seen it on a test cluster where > the SbNN was submitting 8000 RPCs per second that returned empty. > I propose we add some sort of backoff to the tailing, so that if an empty > response is received, it will wait a longer period of time before submitting > a new RPC. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14370) Edit log tailing fast-path should allow for backoff
[
https://issues.apache.org/jira/browse/HDFS-14370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899288#comment-16899288
]
Ayush Saxena commented on HDFS-14370:
-
Thanx [~xkrogen] for the patch. Really sorry for the delay, Almost LGTM.
As per the code the back-off gets disabled with value 0 too.
{code:java}
+ * performed but no edits are available to be read. If this is less than or
+ * equal to 0, backoff is disabled.
{code}
Shouldn't we set the default as 0 itself rather than using negative value,
Makes more sense Mathematically, a Zero max compared to a negative max.
Secondly, we should mention this config in
{{ObserverNameNode.html#Configurations}}
> Edit log tailing fast-path should allow for backoff
> ---
>
> Key: HDFS-14370
> URL: https://issues.apache.org/jira/browse/HDFS-14370
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode, qjm
>Affects Versions: 3.3.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-14370.000.patch, HDFS-14370.001.patch,
> HDFS-14370.002.patch, HDFS-14370.003.patch
>
>
> As part of HDFS-13150, in-progress edit log tailing was changed to use an
> RPC-based mechanism, thus allowing the edit log tailing frequency to be
> turned way down, and allowing standby/observer NameNodes to be only a few
> milliseconds stale as compared to the Active NameNode.
> When there is a high volume of transactions on the system, each RPC fetches
> transactions and takes some time to process them, self-rate-limiting how
> frequently an RPC is submitted. In a lightly loaded cluster, however, most of
> these RPCs return an empty set of transactions, consuming a high
> (de)serialization overhead for very little benefit. This was reported by
> [~jojochuang] in HDFS-14276 and I have also seen it on a test cluster where
> the SbNN was submitting 8000 RPCs per second that returned empty.
> I propose we add some sort of backoff to the tailing, so that if an empty
> response is received, it will wait a longer period of time before submitting
> a new RPC.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14370) Edit log tailing fast-path should allow for backoff
[
https://issues.apache.org/jira/browse/HDFS-14370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899205#comment-16899205
]
Hadoop QA commented on HDFS-14370:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
2s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 17s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 52s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 477 unchanged - 0 fixed = 479 total (was 477) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 15s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}114m 32s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}177m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSClientSocketSize |
| | hadoop.hdfs.server.datanode.TestLargeBlockReport |
| | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier |
| | hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.7 Server=18.09.7 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | HDFS-14370 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12976560/HDFS-14370.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux dae25b73397e 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4
22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / b964b81 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Commented] (HDFS-14370) Edit log tailing fast-path should allow for backoff
[ https://issues.apache.org/jira/browse/HDFS-14370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899110#comment-16899110 ] Konstantin Shvachko commented on HDFS-14370: +1 on v3 patch. Looks good > Edit log tailing fast-path should allow for backoff > --- > > Key: HDFS-14370 > URL: https://issues.apache.org/jira/browse/HDFS-14370 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode, qjm >Affects Versions: 3.3.0 >Reporter: Erik Krogen >Assignee: Erik Krogen >Priority: Major > Attachments: HDFS-14370.000.patch, HDFS-14370.001.patch, > HDFS-14370.002.patch, HDFS-14370.003.patch > > > As part of HDFS-13150, in-progress edit log tailing was changed to use an > RPC-based mechanism, thus allowing the edit log tailing frequency to be > turned way down, and allowing standby/observer NameNodes to be only a few > milliseconds stale as compared to the Active NameNode. > When there is a high volume of transactions on the system, each RPC fetches > transactions and takes some time to process them, self-rate-limiting how > frequently an RPC is submitted. In a lightly loaded cluster, however, most of > these RPCs return an empty set of transactions, consuming a high > (de)serialization overhead for very little benefit. This was reported by > [~jojochuang] in HDFS-14276 and I have also seen it on a test cluster where > the SbNN was submitting 8000 RPCs per second that returned empty. > I propose we add some sort of backoff to the tailing, so that if an empty > response is received, it will wait a longer period of time before submitting > a new RPC. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14370) Edit log tailing fast-path should allow for backoff
[ https://issues.apache.org/jira/browse/HDFS-14370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899090#comment-16899090 ] Erik Krogen commented on HDFS-14370: Thanks [~vagarychen]! The description did mention that negative values disable the backoff behavior, but I made it more clear that this is the default. [~ayushtkn], I will commit this early next week unless I hear back from you, please let me know if you have any concerns with the v003 patch. > Edit log tailing fast-path should allow for backoff > --- > > Key: HDFS-14370 > URL: https://issues.apache.org/jira/browse/HDFS-14370 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode, qjm >Affects Versions: 3.3.0 >Reporter: Erik Krogen >Assignee: Erik Krogen >Priority: Major > Attachments: HDFS-14370.000.patch, HDFS-14370.001.patch, > HDFS-14370.002.patch, HDFS-14370.003.patch > > > As part of HDFS-13150, in-progress edit log tailing was changed to use an > RPC-based mechanism, thus allowing the edit log tailing frequency to be > turned way down, and allowing standby/observer NameNodes to be only a few > milliseconds stale as compared to the Active NameNode. > When there is a high volume of transactions on the system, each RPC fetches > transactions and takes some time to process them, self-rate-limiting how > frequently an RPC is submitted. In a lightly loaded cluster, however, most of > these RPCs return an empty set of transactions, consuming a high > (de)serialization overhead for very little benefit. This was reported by > [~jojochuang] in HDFS-14276 and I have also seen it on a test cluster where > the SbNN was submitting 8000 RPCs per second that returned empty. > I propose we add some sort of backoff to the tailing, so that if an empty > response is received, it will wait a longer period of time before submitting > a new RPC. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14370) Edit log tailing fast-path should allow for backoff
[ https://issues.apache.org/jira/browse/HDFS-14370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899066#comment-16899066 ] Chen Liang commented on HDFS-14370: --- Good discussion here! v002 patch LGTM, I ran the failed tests a couple times locally, TestLargeBlockReport failed without the patch as well, the other two patches both succeeded. Just one nit: maybe we should describe what does default value -1 mean for this key in hdfs-default description. With this addressed, +1. > Edit log tailing fast-path should allow for backoff > --- > > Key: HDFS-14370 > URL: https://issues.apache.org/jira/browse/HDFS-14370 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode, qjm >Affects Versions: 3.3.0 >Reporter: Erik Krogen >Assignee: Erik Krogen >Priority: Major > Attachments: HDFS-14370.000.patch, HDFS-14370.001.patch, > HDFS-14370.002.patch > > > As part of HDFS-13150, in-progress edit log tailing was changed to use an > RPC-based mechanism, thus allowing the edit log tailing frequency to be > turned way down, and allowing standby/observer NameNodes to be only a few > milliseconds stale as compared to the Active NameNode. > When there is a high volume of transactions on the system, each RPC fetches > transactions and takes some time to process them, self-rate-limiting how > frequently an RPC is submitted. In a lightly loaded cluster, however, most of > these RPCs return an empty set of transactions, consuming a high > (de)serialization overhead for very little benefit. This was reported by > [~jojochuang] in HDFS-14276 and I have also seen it on a test cluster where > the SbNN was submitting 8000 RPCs per second that returned empty. > I propose we add some sort of backoff to the tailing, so that if an empty > response is received, it will wait a longer period of time before submitting > a new RPC. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14370) Edit log tailing fast-path should allow for backoff
[
https://issues.apache.org/jira/browse/HDFS-14370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897582#comment-16897582
]
Hadoop QA commented on HDFS-14370:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
39s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 34s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 37s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 479 unchanged - 0 fixed = 481 total (was 479) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 7s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 47s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}131m 56s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestLargeBlockReport |
| | hadoop.hdfs.TestMaintenanceState |
| | hadoop.hdfs.TestMultipleNNPortQOP |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14370 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12976358/HDFS-14370.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux b9b82fb19716 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / b008072 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/27357/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
[jira] [Commented] (HDFS-14370) Edit log tailing fast-path should allow for backoff
[ https://issues.apache.org/jira/browse/HDFS-14370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897456#comment-16897456 ] Erik Krogen commented on HDFS-14370: Uploaded v002 patch to address the checkstyle issue. > Edit log tailing fast-path should allow for backoff > --- > > Key: HDFS-14370 > URL: https://issues.apache.org/jira/browse/HDFS-14370 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode, qjm >Affects Versions: 3.3.0 >Reporter: Erik Krogen >Assignee: Erik Krogen >Priority: Major > Attachments: HDFS-14370.000.patch, HDFS-14370.001.patch, > HDFS-14370.002.patch > > > As part of HDFS-13150, in-progress edit log tailing was changed to use an > RPC-based mechanism, thus allowing the edit log tailing frequency to be > turned way down, and allowing standby/observer NameNodes to be only a few > milliseconds stale as compared to the Active NameNode. > When there is a high volume of transactions on the system, each RPC fetches > transactions and takes some time to process them, self-rate-limiting how > frequently an RPC is submitted. In a lightly loaded cluster, however, most of > these RPCs return an empty set of transactions, consuming a high > (de)serialization overhead for very little benefit. This was reported by > [~jojochuang] in HDFS-14276 and I have also seen it on a test cluster where > the SbNN was submitting 8000 RPCs per second that returned empty. > I propose we add some sort of backoff to the tailing, so that if an empty > response is received, it will wait a longer period of time before submitting > a new RPC. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14370) Edit log tailing fast-path should allow for backoff
[
https://issues.apache.org/jira/browse/HDFS-14370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897455#comment-16897455
]
Hadoop QA commented on HDFS-14370:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 19s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 45s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 3 new + 478 unchanged - 0 fixed = 481 total (was 478) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 26s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 40s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}135m 31s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.server.datanode.TestLargeBlockReport |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14370 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12976351/HDFS-14370.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 41437e448151 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / a6f47b5 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/27356/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
[jira] [Commented] (HDFS-14370) Edit log tailing fast-path should allow for backoff
[
https://issues.apache.org/jira/browse/HDFS-14370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897319#comment-16897319
]
Erik Krogen commented on HDFS-14370:
I don't really like treating empty values specially, as it makes it impossible
to override back to the "empty" behavior when specifying options on the command
line (using {{-Dconfig.key=value}} flags). I've updated the patch to simply
consider any negative values (default of -1) as disabling backoff; I also made
it more clear in the code that backoff is explicitly disabled in this case.
v001 patch also includes a test case.
{quote}
For the 0 case may be we should consider, not allowing back-off, Doesn't as
such make sense, since fractions of no load and we are in back-off state, that
too the time to start with we have decided..but still can be discussed.
{quote}
I don't understand this part, can you explain further? Are you saying when the
config key is set to be 0?
> Edit log tailing fast-path should allow for backoff
> ---
>
> Key: HDFS-14370
> URL: https://issues.apache.org/jira/browse/HDFS-14370
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode, qjm
>Affects Versions: 3.3.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-14370.000.patch, HDFS-14370.001.patch
>
>
> As part of HDFS-13150, in-progress edit log tailing was changed to use an
> RPC-based mechanism, thus allowing the edit log tailing frequency to be
> turned way down, and allowing standby/observer NameNodes to be only a few
> milliseconds stale as compared to the Active NameNode.
> When there is a high volume of transactions on the system, each RPC fetches
> transactions and takes some time to process them, self-rate-limiting how
> frequently an RPC is submitted. In a lightly loaded cluster, however, most of
> these RPCs return an empty set of transactions, consuming a high
> (de)serialization overhead for very little benefit. This was reported by
> [~jojochuang] in HDFS-14276 and I have also seen it on a test cluster where
> the SbNN was submitting 8000 RPCs per second that returned empty.
> I propose we add some sort of backoff to the tailing, so that if an empty
> response is received, it will wait a longer period of time before submitting
> a new RPC.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14370) Edit log tailing fast-path should allow for backoff
[ https://issues.apache.org/jira/browse/HDFS-14370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896131#comment-16896131 ] Ayush Saxena commented on HDFS-14370: - On a quick thought, Can we keep the default empty, i.e don't have any default specified and in the execution check whether the value is specified or not, If the configuration isn't set, we can consider back-off to be turned off, and if a user explicitly specifies the value for the max back-off then only we enable it. For the 0 case may be we should consider, not allowing back-off, Doesn't as such make sense, since fractions of no load and we are in back-off state, that too the time to start with we have decided..but still can be discussed. > Edit log tailing fast-path should allow for backoff > --- > > Key: HDFS-14370 > URL: https://issues.apache.org/jira/browse/HDFS-14370 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode, qjm >Affects Versions: 3.3.0 >Reporter: Erik Krogen >Assignee: Erik Krogen >Priority: Major > Attachments: HDFS-14370.000.patch > > > As part of HDFS-13150, in-progress edit log tailing was changed to use an > RPC-based mechanism, thus allowing the edit log tailing frequency to be > turned way down, and allowing standby/observer NameNodes to be only a few > milliseconds stale as compared to the Active NameNode. > When there is a high volume of transactions on the system, each RPC fetches > transactions and takes some time to process them, self-rate-limiting how > frequently an RPC is submitted. In a lightly loaded cluster, however, most of > these RPCs return an empty set of transactions, consuming a high > (de)serialization overhead for very little benefit. This was reported by > [~jojochuang] in HDFS-14276 and I have also seen it on a test cluster where > the SbNN was submitting 8000 RPCs per second that returned empty. > I propose we add some sort of backoff to the tailing, so that if an empty > response is received, it will wait a longer period of time before submitting > a new RPC. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14370) Edit log tailing fast-path should allow for backoff
[ https://issues.apache.org/jira/browse/HDFS-14370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16895614#comment-16895614 ] Erik Krogen commented on HDFS-14370: Hey [~ayushtkn], thanks for taking a look. You raise great points. I am thinking we have two options: # Set the default value of the maximum time to be -1. If this value is encountered, disable the backoff by setting the maximum to be equal to the minimum time. I think the drawback here is that -1 could also be interpreted as "no maximum," so this behavior may be misleading to some users. # Change the backoff config to be a multiplier of the minimum sleep time. Set the default to be 1, effectively disabling backoff. This has the advantage of more consistent behavior, but determining a reasonable value may be a bit more difficult (more math involved). Let me know what you think. > Edit log tailing fast-path should allow for backoff > --- > > Key: HDFS-14370 > URL: https://issues.apache.org/jira/browse/HDFS-14370 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode, qjm >Affects Versions: 3.3.0 >Reporter: Erik Krogen >Assignee: Erik Krogen >Priority: Major > Attachments: HDFS-14370.000.patch > > > As part of HDFS-13150, in-progress edit log tailing was changed to use an > RPC-based mechanism, thus allowing the edit log tailing frequency to be > turned way down, and allowing standby/observer NameNodes to be only a few > milliseconds stale as compared to the Active NameNode. > When there is a high volume of transactions on the system, each RPC fetches > transactions and takes some time to process them, self-rate-limiting how > frequently an RPC is submitted. In a lightly loaded cluster, however, most of > these RPCs return an empty set of transactions, consuming a high > (de)serialization overhead for very little benefit. This was reported by > [~jojochuang] in HDFS-14276 and I have also seen it on a test cluster where > the SbNN was submitting 8000 RPCs per second that returned empty. > I propose we add some sort of backoff to the tailing, so that if an empty > response is received, it will wait a longer period of time before submitting > a new RPC. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14370) Edit log tailing fast-path should allow for backoff
[ https://issues.apache.org/jira/browse/HDFS-14370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16894392#comment-16894392 ] Ayush Saxena commented on HDFS-14370: - Thanx [~xkrogen] for the patch, Seems fair enough, A doubt, Is there any way to turn off this back-off mechanism? as if my requirement doesn't want me to have a backoff, usually we configure 0 interval for tailing edits for very loaded clusters, it may reach to a backoff stage may be easily, if for fractions the load is nil. And secondly if I don't configure the back-off time, The default it shall take and 1 Min, so if my in general time is 0, the next shall get triggered at 1 Min, I guess by default we shouldn't keep the back-off disabled and the value of max-backoff to be same as that of sleeptimeMs if the max-backoff time isn't specified. > Edit log tailing fast-path should allow for backoff > --- > > Key: HDFS-14370 > URL: https://issues.apache.org/jira/browse/HDFS-14370 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode, qjm >Affects Versions: 3.3.0 >Reporter: Erik Krogen >Assignee: Erik Krogen >Priority: Major > Attachments: HDFS-14370.000.patch > > > As part of HDFS-13150, in-progress edit log tailing was changed to use an > RPC-based mechanism, thus allowing the edit log tailing frequency to be > turned way down, and allowing standby/observer NameNodes to be only a few > milliseconds stale as compared to the Active NameNode. > When there is a high volume of transactions on the system, each RPC fetches > transactions and takes some time to process them, self-rate-limiting how > frequently an RPC is submitted. In a lightly loaded cluster, however, most of > these RPCs return an empty set of transactions, consuming a high > (de)serialization overhead for very little benefit. This was reported by > [~jojochuang] in HDFS-14276 and I have also seen it on a test cluster where > the SbNN was submitting 8000 RPCs per second that returned empty. > I propose we add some sort of backoff to the tailing, so that if an empty > response is received, it will wait a longer period of time before submitting > a new RPC. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14370) Edit log tailing fast-path should allow for backoff
[
https://issues.apache.org/jira/browse/HDFS-14370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16892396#comment-16892396
]
Hadoop QA commented on HDFS-14370:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
51s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 25s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 47s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 465 unchanged - 0 fixed = 466 total (was 465) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 51s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}107m 29s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}170m 4s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.tools.TestDFSZKFailoverController |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.0 Server=19.03.0 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | HDFS-14370 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12975722/HDFS-14370.000.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux fc259670fa69 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4
22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 62deab1 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Commented] (HDFS-14370) Edit log tailing fast-path should allow for backoff
[
https://issues.apache.org/jira/browse/HDFS-14370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16889633#comment-16889633
]
Ayush Saxena commented on HDFS-14370:
-
{quote} SbNN was submitting 8000 RPCs per second that returned empty.
{quote}
This is actually true, I too observed it while handling HDFS-14655.
Having a backoff in such scenario, could be of great help. We can for sure make
the parameters configurable, whether to back off or not. This may be helpful at
times of less load.
> Edit log tailing fast-path should allow for backoff
> ---
>
> Key: HDFS-14370
> URL: https://issues.apache.org/jira/browse/HDFS-14370
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode, qjm
>Affects Versions: 3.3.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
>
> As part of HDFS-13150, in-progress edit log tailing was changed to use an
> RPC-based mechanism, thus allowing the edit log tailing frequency to be
> turned way down, and allowing standby/observer NameNodes to be only a few
> milliseconds stale as compared to the Active NameNode.
> When there is a high volume of transactions on the system, each RPC fetches
> transactions and takes some time to process them, self-rate-limiting how
> frequently an RPC is submitted. In a lightly loaded cluster, however, most of
> these RPCs return an empty set of transactions, consuming a high
> (de)serialization overhead for very little benefit. This was reported by
> [~jojochuang] in HDFS-14276 and I have also seen it on a test cluster where
> the SbNN was submitting 8000 RPCs per second that returned empty.
> I propose we add some sort of backoff to the tailing, so that if an empty
> response is received, it will wait a longer period of time before submitting
> a new RPC.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14370) Edit log tailing fast-path should allow for backoff
[
https://issues.apache.org/jira/browse/HDFS-14370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807341#comment-16807341
]
star commented on HDFS-14370:
-
[~xkrogen]， agree. Server side handler as limited resource should not be
blocked.
{quote}This is similar to how the NN, when it wants a client to backoff due to
load, throws a backoff exception and expects the client to act accordingly.
{quote}
> Edit log tailing fast-path should allow for backoff
> ---
>
> Key: HDFS-14370
> URL: https://issues.apache.org/jira/browse/HDFS-14370
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode, qjm
>Affects Versions: 3.3.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
>
> As part of HDFS-13150, in-progress edit log tailing was changed to use an
> RPC-based mechanism, thus allowing the edit log tailing frequency to be
> turned way down, and allowing standby/observer NameNodes to be only a few
> milliseconds stale as compared to the Active NameNode.
> When there is a high volume of transactions on the system, each RPC fetches
> transactions and takes some time to process them, self-rate-limiting how
> frequently an RPC is submitted. In a lightly loaded cluster, however, most of
> these RPCs return an empty set of transactions, consuming a high
> (de)serialization overhead for very little benefit. This was reported by
> [~jojochuang] in HDFS-14276 and I have also seen it on a test cluster where
> the SbNN was submitting 8000 RPCs per second that returned empty.
> I propose we add some sort of backoff to the tailing, so that if an empty
> response is received, it will wait a longer period of time before submitting
> a new RPC.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14370) Edit log tailing fast-path should allow for backoff
[
https://issues.apache.org/jira/browse/HDFS-14370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806924#comment-16806924
]
Erik Krogen commented on HDFS-14370:
Copying [~starphin]'s comment from HDFS-14276:
{quote}
Maybe we can just sleep 10ms in rpc server side if there are no edit logs. As a
consequence that rpc thread will be blocked for 10ms.
{quote}
I think by "server side" you're referring to the JournalNode. I think I would
prefer to keep the sleep on the NN side, to avoid handler threads being tied up
on the JN. This is similar to how the NN, when it wants a client to backoff due
to load, throws a backoff exception and expects the client to act accordingly.
> Edit log tailing fast-path should allow for backoff
> ---
>
> Key: HDFS-14370
> URL: https://issues.apache.org/jira/browse/HDFS-14370
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode, qjm
>Affects Versions: 3.3.0
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
>
> As part of HDFS-13150, in-progress edit log tailing was changed to use an
> RPC-based mechanism, thus allowing the edit log tailing frequency to be
> turned way down, and allowing standby/observer NameNodes to be only a few
> milliseconds stale as compared to the Active NameNode.
> When there is a high volume of transactions on the system, each RPC fetches
> transactions and takes some time to process them, self-rate-limiting how
> frequently an RPC is submitted. In a lightly loaded cluster, however, most of
> these RPCs return an empty set of transactions, consuming a high
> (de)serialization overhead for very little benefit. This was reported by
> [~jojochuang] in HDFS-14276 and I have also seen it on a test cluster where
> the SbNN was submitting 8000 RPCs per second that returned empty.
> I propose we add some sort of backoff to the tailing, so that if an empty
> response is received, it will wait a longer period of time before submitting
> a new RPC.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14370) Edit log tailing fast-path should allow for backoff
[ https://issues.apache.org/jira/browse/HDFS-14370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16792096#comment-16792096 ] Erik Krogen commented on HDFS-14370: [~jojochuang] I would love your input on this > Edit log tailing fast-path should allow for backoff > --- > > Key: HDFS-14370 > URL: https://issues.apache.org/jira/browse/HDFS-14370 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode, qjm >Affects Versions: 3.3.0 >Reporter: Erik Krogen >Assignee: Erik Krogen >Priority: Major > > As part of HDFS-13150, in-progress edit log tailing was changed to use an > RPC-based mechanism, thus allowing the edit log tailing frequency to be > turned way down, and allowing standby/observer NameNodes to be only a few > milliseconds stale as compared to the Active NameNode. > When there is a high volume of transactions on the system, each RPC fetches > transactions and takes some time to process them, self-rate-limiting how > frequently an RPC is submitted. In a lightly loaded cluster, however, most of > these RPCs return an empty set of transactions, consuming a high > (de)serialization overhead for very little benefit. This was reported by > [~jojochuang] in HDFS-14276 and I have also reported it on a test cluster > where the SbNN was submitting 8000 RPCs per second that returned empty. > I propose we add some sort of backoff to the tailing, so that if an empty > response is received, it will wait a longer period of time before submitting > a new RPC. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
