[jira] [Commented] (HDFS-13331) Add lastSeenStateId to RpcRequestHeader.
[ https://issues.apache.org/jira/browse/HDFS-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16728476#comment-16728476 ] Hudson commented on HDFS-13331: --- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #15662 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15662/]) HDFS-13331. [SBN read] Add lastSeenStateId to RpcRequestHeader. (shv: rev 9bf0696c7348879678693176da0071628ca42566) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestStateAlignmentContext.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/AlignmentContext.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ProtoUtil.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/GlobalStateIdContext.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientGCIContext.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java * (edit) hadoop-common-project/hadoop-common/src/main/proto/RpcHeader.proto * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java > Add lastSeenStateId to RpcRequestHeader. > > > Key: HDFS-13331 > URL: https://issues.apache.org/jira/browse/HDFS-13331 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-12943 >Reporter: Plamen Jeliazkov >Assignee: Plamen Jeliazkov >Priority: Major > Fix For: HDFS-12943, 3.3.0 > > Attachments: HDFS-13331-HDFS-12943.002.patch, > HDFS-13331-HDFS-12943.003..patch, HDFS-13331-HDFS-12943.004.patch, > HDFS-13331-HDFS-12943.005.patch, HDFS-13331.trunk.001.patch, > HDFS_13331.trunk.000.patch > > > HDFS-12977 added a stateId into the RpcResponseHeader which is returned by > NameNode and stored by DFSClient. > This JIRA is to followup on that work and have the DFSClient send their > stored "lastSeenStateId" in the RpcRequestHeader so that ObserverNodes can > then compare with their own and act accordingly. > This JIRA work focuses on just the part of making DFSClient send their state > through RpcRequestHeader. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13331) Add lastSeenStateId to RpcRequestHeader.
[
https://issues.apache.org/jira/browse/HDFS-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426301#comment-16426301
]
Plamen Jeliazkov commented on HDFS-13331:
-
Thank you [~xkrogen]! And thanks [~shv] as well. I have made the follow-up JIRA
HDFS-13399 to address the static {{AlignmentContext}} as was discussed in this
JIRA.
> Add lastSeenStateId to RpcRequestHeader.
>
>
> Key: HDFS-13331
> URL: https://issues.apache.org/jira/browse/HDFS-13331
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-12943
>Reporter: Plamen Jeliazkov
>Assignee: Plamen Jeliazkov
>Priority: Major
> Fix For: HDFS-12943
>
> Attachments: HDFS-13331-HDFS-12943.002.patch,
> HDFS-13331-HDFS-12943.003..patch, HDFS-13331-HDFS-12943.004.patch,
> HDFS-13331-HDFS-12943.005.patch, HDFS-13331.trunk.001.patch,
> HDFS_13331.trunk.000.patch
>
>
> HDFS-12977 added a stateId into the RpcResponseHeader which is returned by
> NameNode and stored by DFSClient.
> This JIRA is to followup on that work and have the DFSClient send their
> stored "lastSeenStateId" in the RpcRequestHeader so that ObserverNodes can
> then compare with their own and act accordingly.
> This JIRA work focuses on just the part of making DFSClient send their state
> through RpcRequestHeader.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13331) Add lastSeenStateId to RpcRequestHeader.
[
https://issues.apache.org/jira/browse/HDFS-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426287#comment-16426287
]
Erik Krogen commented on HDFS-13331:
Hey Plamen, looks great. I just committed this to branch {{HDFS-12943}}. Thanks
for the work! Can you please make sure to file a follow-on JIRA for the static
{{AlignmentContext}} issues we discussed?
> Add lastSeenStateId to RpcRequestHeader.
>
>
> Key: HDFS-13331
> URL: https://issues.apache.org/jira/browse/HDFS-13331
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-12943
>Reporter: Plamen Jeliazkov
>Assignee: Plamen Jeliazkov
>Priority: Major
> Attachments: HDFS-13331-HDFS-12943.002.patch,
> HDFS-13331-HDFS-12943.003..patch, HDFS-13331-HDFS-12943.004.patch,
> HDFS-13331-HDFS-12943.005.patch, HDFS-13331.trunk.001.patch,
> HDFS_13331.trunk.000.patch
>
>
> HDFS-12977 added a stateId into the RpcResponseHeader which is returned by
> NameNode and stored by DFSClient.
> This JIRA is to followup on that work and have the DFSClient send their
> stored "lastSeenStateId" in the RpcRequestHeader so that ObserverNodes can
> then compare with their own and act accordingly.
> This JIRA work focuses on just the part of making DFSClient send their state
> through RpcRequestHeader.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13331) Add lastSeenStateId to RpcRequestHeader.
[
https://issues.apache.org/jira/browse/HDFS-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426128#comment-16426128
]
genericqa commented on HDFS-13331:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m
6s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-12943 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 6m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
7s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
18s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
10s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
52s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
22m 10s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
17s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
26s{color} | {color:green} HDFS-12943 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 12m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
13s{color} | {color:green} root: The patch generated 0 new + 363 unchanged - 1
fixed = 363 total (was 364) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 23s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m
8s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
16s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 84m 31s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
44s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}229m 34s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
| | hadoop.hdfs.server.namenode.ha.TestHAMetrics |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | HDFS-13331 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12917570/HDFS-13331-HDFS-12943.005.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient
[jira] [Commented] (HDFS-13331) Add lastSeenStateId to RpcRequestHeader.
[ https://issues.apache.org/jira/browse/HDFS-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425797#comment-16425797 ] Plamen Jeliazkov commented on HDFS-13331: - Oh, whoops. Good catch, [~xkrogen]. No that was not intentional. I've restored it and uploaded a new, v005 patch. > Add lastSeenStateId to RpcRequestHeader. > > > Key: HDFS-13331 > URL: https://issues.apache.org/jira/browse/HDFS-13331 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-12943 >Reporter: Plamen Jeliazkov >Assignee: Plamen Jeliazkov >Priority: Major > Attachments: HDFS-13331-HDFS-12943.002.patch, > HDFS-13331-HDFS-12943.003..patch, HDFS-13331-HDFS-12943.004.patch, > HDFS-13331-HDFS-12943.005.patch, HDFS-13331.trunk.001.patch, > HDFS_13331.trunk.000.patch > > > HDFS-12977 added a stateId into the RpcResponseHeader which is returned by > NameNode and stored by DFSClient. > This JIRA is to followup on that work and have the DFSClient send their > stored "lastSeenStateId" in the RpcRequestHeader so that ObserverNodes can > then compare with their own and act accordingly. > This JIRA work focuses on just the part of making DFSClient send their state > through RpcRequestHeader. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13331) Add lastSeenStateId to RpcRequestHeader.
[
https://issues.apache.org/jira/browse/HDFS-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424829#comment-16424829
]
Erik Krogen commented on HDFS-13331:
Cool, looks great to me, thanks [~zero45]! Last nit - I see in v004 that
{{getLastSeenStateId}} in {{ClientCGIContext}} is no longer public as it was in
v003, is that intentional?
> Add lastSeenStateId to RpcRequestHeader.
>
>
> Key: HDFS-13331
> URL: https://issues.apache.org/jira/browse/HDFS-13331
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-12943
>Reporter: Plamen Jeliazkov
>Assignee: Plamen Jeliazkov
>Priority: Major
> Attachments: HDFS-13331-HDFS-12943.002.patch,
> HDFS-13331-HDFS-12943.003..patch, HDFS-13331-HDFS-12943.004.patch,
> HDFS-13331.trunk.001.patch, HDFS_13331.trunk.000.patch
>
>
> HDFS-12977 added a stateId into the RpcResponseHeader which is returned by
> NameNode and stored by DFSClient.
> This JIRA is to followup on that work and have the DFSClient send their
> stored "lastSeenStateId" in the RpcRequestHeader so that ObserverNodes can
> then compare with their own and act accordingly.
> This JIRA work focuses on just the part of making DFSClient send their state
> through RpcRequestHeader.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13331) Add lastSeenStateId to RpcRequestHeader.
[
https://issues.apache.org/jira/browse/HDFS-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423219#comment-16423219
]
genericqa commented on HDFS-13331:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-12943 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 5m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
34s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
9s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
16s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
6s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 20s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
9s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
26s{color} | {color:green} HDFS-12943 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 12m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
17s{color} | {color:green} root: The patch generated 0 new + 363 unchanged - 1
fixed = 363 total (was 364) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 23s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
3s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
34s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 95m 48s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}212m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy
|
| | hadoop.hdfs.server.namenode.TestReconstructStripedBlocks |
| | hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | HDFS-13331 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12917232/HDFS-13331-HDFS-12943.004.patch
|
| Optional Tests | asflicense compile
[jira] [Commented] (HDFS-13331) Add lastSeenStateId to RpcRequestHeader.
[
https://issues.apache.org/jira/browse/HDFS-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16422986#comment-16422986
]
Plamen Jeliazkov commented on HDFS-13331:
-
On second thought I decided to implement (1) anyway and make use of
{{LogCapturer}}. My justification is that we can address any concerns in the
follow-up work anyway; and it's possible we keep the log statement but maybe
just change its structure once further server implementations are done.
I have attached a new patch addressing your points.
> Add lastSeenStateId to RpcRequestHeader.
>
>
> Key: HDFS-13331
> URL: https://issues.apache.org/jira/browse/HDFS-13331
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-12943
>Reporter: Plamen Jeliazkov
>Assignee: Plamen Jeliazkov
>Priority: Major
> Attachments: HDFS-13331-HDFS-12943.002.patch,
> HDFS-13331-HDFS-12943.003..patch, HDFS-13331-HDFS-12943.004.patch,
> HDFS-13331.trunk.001.patch, HDFS_13331.trunk.000.patch
>
>
> HDFS-12977 added a stateId into the RpcResponseHeader which is returned by
> NameNode and stored by DFSClient.
> This JIRA is to followup on that work and have the DFSClient send their
> stored "lastSeenStateId" in the RpcRequestHeader so that ObserverNodes can
> then compare with their own and act accordingly.
> This JIRA work focuses on just the part of making DFSClient send their state
> through RpcRequestHeader.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13331) Add lastSeenStateId to RpcRequestHeader.
[
https://issues.apache.org/jira/browse/HDFS-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16422883#comment-16422883
]
Plamen Jeliazkov commented on HDFS-13331:
-
Thanks [~xkrogen] -- I debated whether to do the static field work now or in
follow-up for a while and I think in follow-up will help make my patch more
focused. I think I am able to do it by passing it through {{Call}} object
though.
As for your points, I agree with them except that for (1) I do not intend the
log statement to last long and therefore would opt not to rely on a log
capture. Log capture tests are a little hacky as well. If you would really like
me to remove the {{@Ignored}} however I can just remove it for now and update
the test later.
> Add lastSeenStateId to RpcRequestHeader.
>
>
> Key: HDFS-13331
> URL: https://issues.apache.org/jira/browse/HDFS-13331
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-12943
>Reporter: Plamen Jeliazkov
>Assignee: Plamen Jeliazkov
>Priority: Major
> Attachments: HDFS-13331-HDFS-12943.002.patch,
> HDFS-13331-HDFS-12943.003..patch, HDFS-13331.trunk.001.patch,
> HDFS_13331.trunk.000.patch
>
>
> HDFS-12977 added a stateId into the RpcResponseHeader which is returned by
> NameNode and stored by DFSClient.
> This JIRA is to followup on that work and have the DFSClient send their
> stored "lastSeenStateId" in the RpcRequestHeader so that ObserverNodes can
> then compare with their own and act accordingly.
> This JIRA work focuses on just the part of making DFSClient send their state
> through RpcRequestHeader.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13331) Add lastSeenStateId to RpcRequestHeader.
[
https://issues.apache.org/jira/browse/HDFS-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16421037#comment-16421037
]
Erik Krogen commented on HDFS-13331:
That seems reasonable to me, [~zero45] I will leave it up to you if you want to
address here or in a follow-up. My last comments on the patch:
* For {{testClientSendsGreaterState()}}, instead of {{@Ignored}}, can we use a
log capture to verify that the correct statement is logged? This still verifies
the correct code path was taken.
* style nit in {{Server}}:
{code}
if(alignmentContext != null) {
alignmentContext.receiveRequestState(header);
}
{code}
There should be a space between if and the parenthesis?
* re: the above code, I don't think {{checkRpcHeaders()}} is the right place
for it. The method name implies it is only doing a check, not any processing.
It seems that {{processRpcRequest()}} is a better place, since this is
processing the request and taking action on it.
> Add lastSeenStateId to RpcRequestHeader.
>
>
> Key: HDFS-13331
> URL: https://issues.apache.org/jira/browse/HDFS-13331
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-12943
>Reporter: Plamen Jeliazkov
>Assignee: Plamen Jeliazkov
>Priority: Major
> Attachments: HDFS-13331-HDFS-12943.002.patch,
> HDFS-13331-HDFS-12943.003..patch, HDFS-13331.trunk.001.patch,
> HDFS_13331.trunk.000.patch
>
>
> HDFS-12977 added a stateId into the RpcResponseHeader which is returned by
> NameNode and stored by DFSClient.
> This JIRA is to followup on that work and have the DFSClient send their
> stored "lastSeenStateId" in the RpcRequestHeader so that ObserverNodes can
> then compare with their own and act accordingly.
> This JIRA work focuses on just the part of making DFSClient send their state
> through RpcRequestHeader.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13331) Add lastSeenStateId to RpcRequestHeader.
[ https://issues.apache.org/jira/browse/HDFS-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420967#comment-16420967 ] Konstantin Shvachko commented on HDFS-13331: Would it make sense to commit this and solve the context issue in the next jira? My motivation is to make room for parallel development: with all fields available after this patch we will be able to work on client ans server parts independently. Since this is a dev branch, we don't need to make everything ideal in the first pass. > Add lastSeenStateId to RpcRequestHeader. > > > Key: HDFS-13331 > URL: https://issues.apache.org/jira/browse/HDFS-13331 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-12943 >Reporter: Plamen Jeliazkov >Assignee: Plamen Jeliazkov >Priority: Major > Attachments: HDFS-13331-HDFS-12943.002.patch, > HDFS-13331-HDFS-12943.003..patch, HDFS-13331.trunk.001.patch, > HDFS_13331.trunk.000.patch > > > HDFS-12977 added a stateId into the RpcResponseHeader which is returned by > NameNode and stored by DFSClient. > This JIRA is to followup on that work and have the DFSClient send their > stored "lastSeenStateId" in the RpcRequestHeader so that ObserverNodes can > then compare with their own and act accordingly. > This JIRA work focuses on just the part of making DFSClient send their state > through RpcRequestHeader. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13331) Add lastSeenStateId to RpcRequestHeader.
[
https://issues.apache.org/jira/browse/HDFS-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420913#comment-16420913
]
Plamen Jeliazkov commented on HDFS-13331:
-
I agree we should try to leverage {{Connection}} objects and that ThreadLocal
would be hacky. I've taken the {{Connection}} approach but I am currently
hitting an issue in that DFSClients re-use {{Connection}}s, but with multiple
DFSClients having their own {{AlignmentContext}} then only the first
instantiated {{Connection}} will have its {{AlignmentContext}} updated.
I am thinking now that it needs to live in the {{Call}} object instead... I
will continue experimenting however and report next week.
Also, no need to apologize. We are on the same page / team here. :) In fact its
probably more fitting to address this as part of this JIRA then the last anyway
as now its easier to focus on the client side logic.
> Add lastSeenStateId to RpcRequestHeader.
>
>
> Key: HDFS-13331
> URL: https://issues.apache.org/jira/browse/HDFS-13331
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-12943
>Reporter: Plamen Jeliazkov
>Assignee: Plamen Jeliazkov
>Priority: Major
> Attachments: HDFS-13331-HDFS-12943.002.patch,
> HDFS-13331-HDFS-12943.003..patch, HDFS-13331.trunk.001.patch,
> HDFS_13331.trunk.000.patch
>
>
> HDFS-12977 added a stateId into the RpcResponseHeader which is returned by
> NameNode and stored by DFSClient.
> This JIRA is to followup on that work and have the DFSClient send their
> stored "lastSeenStateId" in the RpcRequestHeader so that ObserverNodes can
> then compare with their own and act accordingly.
> This JIRA work focuses on just the part of making DFSClient send their state
> through RpcRequestHeader.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13331) Add lastSeenStateId to RpcRequestHeader.
[
https://issues.apache.org/jira/browse/HDFS-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419948#comment-16419948
]
Erik Krogen commented on HDFS-13331:
One way to achieve it would be to make {{alignmentContext}} a ThreadLocal in
{{Client}}. This matches well with how other state is maintained within
{{Client}}. Then, before calling into the {{ClientProtocol}}, {{DFSClient}} can
set the proper {{alignmentContext}}. It can (optionally?) reset it after the
call completes. We could facilitate this by bundling it into a {{Closable}} so
that it could be done in a try-with-resource statement like:
{code:java}
try (Client.setAlignmentContext(alignmentContext)) {
namenode.doWhateverCall(...)
}
{code}
This requires the least changes outside of DFSClient, but feels kind of hacky.
Given the {{Client}} already maintains a map of {{ConnectionId}} ->
{{Connection}} mapping, it seems we could leverage this to achieve a cleaner
solution. It seems that really the place this reference should be stored is
within {{Connection}}, or maybe {{ConnectionId}} but that seems a little less
clean. The {{Connection}}s are created via {{Client#getConnection}}, which is
called by {{Client#call}} -> {{ProtobufRpcEngine#invoke}}, so we could have the
{{DFSClient}} pass in its {{alignmentContext}} when it creates its proxy. Any
thoughts on this approach? Sorry for not chiming in on the last JIRA, btw.
> Add lastSeenStateId to RpcRequestHeader.
>
>
> Key: HDFS-13331
> URL: https://issues.apache.org/jira/browse/HDFS-13331
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-12943
>Reporter: Plamen Jeliazkov
>Assignee: Plamen Jeliazkov
>Priority: Major
> Attachments: HDFS-13331-HDFS-12943.002.patch,
> HDFS-13331-HDFS-12943.003..patch, HDFS-13331.trunk.001.patch,
> HDFS_13331.trunk.000.patch
>
>
> HDFS-12977 added a stateId into the RpcResponseHeader which is returned by
> NameNode and stored by DFSClient.
> This JIRA is to followup on that work and have the DFSClient send their
> stored "lastSeenStateId" in the RpcRequestHeader so that ObserverNodes can
> then compare with their own and act accordingly.
> This JIRA work focuses on just the part of making DFSClient send their state
> through RpcRequestHeader.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13331) Add lastSeenStateId to RpcRequestHeader.
[ https://issues.apache.org/jira/browse/HDFS-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419788#comment-16419788 ] Erik Krogen commented on HDFS-13331: I see, sorry I skimmed the comments on the last patch but didn't look closely enough to notice your concerns. I'll take a look as well and see if any idea strikes me.. > Add lastSeenStateId to RpcRequestHeader. > > > Key: HDFS-13331 > URL: https://issues.apache.org/jira/browse/HDFS-13331 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-12943 >Reporter: Plamen Jeliazkov >Assignee: Plamen Jeliazkov >Priority: Major > Attachments: HDFS-13331-HDFS-12943.002.patch, > HDFS-13331-HDFS-12943.003..patch, HDFS-13331.trunk.001.patch, > HDFS_13331.trunk.000.patch > > > HDFS-12977 added a stateId into the RpcResponseHeader which is returned by > NameNode and stored by DFSClient. > This JIRA is to followup on that work and have the DFSClient send their > stored "lastSeenStateId" in the RpcRequestHeader so that ObserverNodes can > then compare with their own and act accordingly. > This JIRA work focuses on just the part of making DFSClient send their state > through RpcRequestHeader. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13331) Add lastSeenStateId to RpcRequestHeader.
[ https://issues.apache.org/jira/browse/HDFS-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419704#comment-16419704 ] Plamen Jeliazkov commented on HDFS-13331: - Yes you are correct. I brought up that concern in the last patch as well stating that I was not happy with the use of a static method in Client as a way to handle alignmentContext, but it seems that point was glossed over. I mentioned this: https://issues.apache.org/jira/browse/HDFS-12977?focusedCommentId=16372125=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16372125 And here: https://issues.apache.org/jira/browse/HDFS-12977?focusedCommentId=16386960=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16386960 My issue was that I had difficulty finding a way to pass in alignmentContext into Client constructor. I am happy to try again though. > Add lastSeenStateId to RpcRequestHeader. > > > Key: HDFS-13331 > URL: https://issues.apache.org/jira/browse/HDFS-13331 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-12943 >Reporter: Plamen Jeliazkov >Assignee: Plamen Jeliazkov >Priority: Major > Attachments: HDFS-13331-HDFS-12943.002.patch, > HDFS-13331-HDFS-12943.003..patch, HDFS-13331.trunk.001.patch, > HDFS_13331.trunk.000.patch > > > HDFS-12977 added a stateId into the RpcResponseHeader which is returned by > NameNode and stored by DFSClient. > This JIRA is to followup on that work and have the DFSClient send their > stored "lastSeenStateId" in the RpcRequestHeader so that ObserverNodes can > then compare with their own and act accordingly. > This JIRA work focuses on just the part of making DFSClient send their state > through RpcRequestHeader. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13331) Add lastSeenStateId to RpcRequestHeader.
[
https://issues.apache.org/jira/browse/HDFS-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419655#comment-16419655
]
Erik Krogen commented on HDFS-13331:
Hey [~zero45], just looked through the most recent patch. I have a question
about using a static {{alignmentContext}} in {{Client}}. We can potentially
have two {{DFSClient}} objects in the same JVM which are communicating with
completely difference namespaces, each of which should have their own
{{alignmentContext}}. The current implementation would not support this,
AFAICT. Am I correct? I don't think that is acceptable IIUC.
> Add lastSeenStateId to RpcRequestHeader.
>
>
> Key: HDFS-13331
> URL: https://issues.apache.org/jira/browse/HDFS-13331
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-12943
>Reporter: Plamen Jeliazkov
>Assignee: Plamen Jeliazkov
>Priority: Major
> Attachments: HDFS-13331-HDFS-12943.002.patch,
> HDFS-13331-HDFS-12943.003..patch, HDFS-13331.trunk.001.patch,
> HDFS_13331.trunk.000.patch
>
>
> HDFS-12977 added a stateId into the RpcResponseHeader which is returned by
> NameNode and stored by DFSClient.
> This JIRA is to followup on that work and have the DFSClient send their
> stored "lastSeenStateId" in the RpcRequestHeader so that ObserverNodes can
> then compare with their own and act accordingly.
> This JIRA work focuses on just the part of making DFSClient send their state
> through RpcRequestHeader.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13331) Add lastSeenStateId to RpcRequestHeader.
[ https://issues.apache.org/jira/browse/HDFS-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16419521#comment-16419521 ] Plamen Jeliazkov commented on HDFS-13331: - The failed unit tests appear to be unrelated to the patch. I verified locally that they all pass. I saw some of these fail during the Jenkins run of the previous HDFS-12977 patch as well. > Add lastSeenStateId to RpcRequestHeader. > > > Key: HDFS-13331 > URL: https://issues.apache.org/jira/browse/HDFS-13331 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-12943 >Reporter: Plamen Jeliazkov >Assignee: Plamen Jeliazkov >Priority: Major > Attachments: HDFS-13331-HDFS-12943.002.patch, > HDFS-13331-HDFS-12943.003..patch, HDFS-13331.trunk.001.patch, > HDFS_13331.trunk.000.patch > > > HDFS-12977 added a stateId into the RpcResponseHeader which is returned by > NameNode and stored by DFSClient. > This JIRA is to followup on that work and have the DFSClient send their > stored "lastSeenStateId" in the RpcRequestHeader so that ObserverNodes can > then compare with their own and act accordingly. > This JIRA work focuses on just the part of making DFSClient send their state > through RpcRequestHeader. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13331) Add lastSeenStateId to RpcRequestHeader.
[
https://issues.apache.org/jira/browse/HDFS-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416248#comment-16416248
]
genericqa commented on HDFS-13331:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m
41s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-12943 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 5m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
16s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
46s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
11s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
29s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 9s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
53s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
26s{color} | {color:green} HDFS-12943 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 15m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
12s{color} | {color:green} root: The patch generated 0 new + 363 unchanged - 1
fixed = 363 total (was 364) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 2s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
36s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
42s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}111m 24s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}254m 27s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce
[jira] [Commented] (HDFS-13331) Add lastSeenStateId to RpcRequestHeader.
[ https://issues.apache.org/jira/browse/HDFS-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16415903#comment-16415903 ] Plamen Jeliazkov commented on HDFS-13331: - Attaching new patch to address whitespace and checkstyle. The checkstyle failure was on ClientGCIContext.lastSeenStateId being package private so I made it private and added a public method "getLastSeenStateId()" which is used in the unit tests. > Add lastSeenStateId to RpcRequestHeader. > > > Key: HDFS-13331 > URL: https://issues.apache.org/jira/browse/HDFS-13331 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-12943 >Reporter: Plamen Jeliazkov >Assignee: Plamen Jeliazkov >Priority: Major > Attachments: HDFS-13331-HDFS-12943.002.patch, > HDFS-13331-HDFS-12943.003..patch, HDFS-13331.trunk.001.patch, > HDFS_13331.trunk.000.patch > > > HDFS-12977 added a stateId into the RpcResponseHeader which is returned by > NameNode and stored by DFSClient. > This JIRA is to followup on that work and have the DFSClient send their > stored "lastSeenStateId" in the RpcRequestHeader so that ObserverNodes can > then compare with their own and act accordingly. > This JIRA work focuses on just the part of making DFSClient send their state > through RpcRequestHeader. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13331) Add lastSeenStateId to RpcRequestHeader.
[
https://issues.apache.org/jira/browse/HDFS-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16414833#comment-16414833
]
genericqa commented on HDFS-13331:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
47s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-12943 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 5m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
51s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
11s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
45s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
38s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 39s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
41s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
37s{color} | {color:green} HDFS-12943 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
38s{color} | {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
45s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 12m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
2s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 27s{color} | {color:orange} root: The patch generated 1 new + 363 unchanged
- 1 fixed = 364 total (was 364) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
57s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 29s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} hadoop-hdfs-project_hadoop-hdfs-client generated 0
new + 0 unchanged - 2 fixed = 0 total (was 2) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} hadoop-hdfs-project_hadoop-hdfs generated 0 new + 1
unchanged - 2 fixed = 1 total (was 3) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
6s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
36s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 57s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch
[jira] [Commented] (HDFS-13331) Add lastSeenStateId to RpcRequestHeader.
[ https://issues.apache.org/jira/browse/HDFS-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16414722#comment-16414722 ] Plamen Jeliazkov commented on HDFS-13331: - Hi [~xkrogen]. Yes, absolutely. I'm attaching a renamed patch now and will start Jenkins. Apologies as I did not know what the standard was before. > Add lastSeenStateId to RpcRequestHeader. > > > Key: HDFS-13331 > URL: https://issues.apache.org/jira/browse/HDFS-13331 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Plamen Jeliazkov >Assignee: Plamen Jeliazkov >Priority: Major > Attachments: HDFS-13331-HDFS-12943.002.patch, > HDFS-13331.trunk.001.patch, HDFS_13331.trunk.000.patch > > > HDFS-12977 added a stateId into the RpcResponseHeader which is returned by > NameNode and stored by DFSClient. > This JIRA is to followup on that work and have the DFSClient send their > stored "lastSeenStateId" in the RpcRequestHeader so that ObserverNodes can > then compare with their own and act accordingly. > This JIRA work focuses on just the part of making DFSClient send their state > through RpcRequestHeader. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13331) Add lastSeenStateId to RpcRequestHeader.
[ https://issues.apache.org/jira/browse/HDFS-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16414704#comment-16414704 ] Erik Krogen commented on HDFS-13331: Hey [~zero45], can you upload the patch with the correct naming to apply to HDFS-12943 branch so we can get a Jenkins run? > Add lastSeenStateId to RpcRequestHeader. > > > Key: HDFS-13331 > URL: https://issues.apache.org/jira/browse/HDFS-13331 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Plamen Jeliazkov >Assignee: Plamen Jeliazkov >Priority: Major > Attachments: HDFS-13331.trunk.001.patch, HDFS_13331.trunk.000.patch > > > HDFS-12977 added a stateId into the RpcResponseHeader which is returned by > NameNode and stored by DFSClient. > This JIRA is to followup on that work and have the DFSClient send their > stored "lastSeenStateId" in the RpcRequestHeader so that ObserverNodes can > then compare with their own and act accordingly. > This JIRA work focuses on just the part of making DFSClient send their state > through RpcRequestHeader. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13331) Add lastSeenStateId to RpcRequestHeader.
[ https://issues.apache.org/jira/browse/HDFS-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16414695#comment-16414695 ] Konstantin Shvachko commented on HDFS-13331: Liked the patch +1. > Add lastSeenStateId to RpcRequestHeader. > > > Key: HDFS-13331 > URL: https://issues.apache.org/jira/browse/HDFS-13331 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Plamen Jeliazkov >Assignee: Plamen Jeliazkov >Priority: Major > Attachments: HDFS-13331.trunk.001.patch, HDFS_13331.trunk.000.patch > > > HDFS-12977 added a stateId into the RpcResponseHeader which is returned by > NameNode and stored by DFSClient. > This JIRA is to followup on that work and have the DFSClient send their > stored "lastSeenStateId" in the RpcRequestHeader so that ObserverNodes can > then compare with their own and act accordingly. > This JIRA work focuses on just the part of making DFSClient send their state > through RpcRequestHeader. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13331) Add lastSeenStateId to RpcRequestHeader.
[ https://issues.apache.org/jira/browse/HDFS-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16414405#comment-16414405 ] Plamen Jeliazkov commented on HDFS-13331: - Thanks Konstantin. I found I could do the following: 1) Remove getLastSeenStateId() and just make lastSeenStateId package-private. Then I could just make 1 method "getAlignmentContext" in DFSClient and that would satisfy everything I need. Now I can get lastSeenStateId from the ClientGCIContext in tests and expose the AlignmentContext to those using DFSClient directly. 2) I did add the basic server-side logic with simple checking. I had to add another new interface method "receiveRequestState". I use the FSNamesystem LOG to log a warning. I also added a unit test to demonstrate this but marked it as ignored until we implement the full logic. You can run the test yourself to see the warning message however. I also simplified the test "testClientSendsState" -- previously it had some logic that tested to ensure spied AlignmentContext had the same lastSeenStateId as the original object. Obviously we don't really need to test Mockito so I removed that portion. I also added a segment that checks that the RpcRequestHeaders increase in stateId and that the first and last RpcRequestHeaders are not equal. This should be enough to show that state must have "increased". > Add lastSeenStateId to RpcRequestHeader. > > > Key: HDFS-13331 > URL: https://issues.apache.org/jira/browse/HDFS-13331 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Plamen Jeliazkov >Assignee: Plamen Jeliazkov >Priority: Major > Attachments: HDFS-13331.trunk.001.patch, HDFS_13331.trunk.000.patch > > > HDFS-12977 added a stateId into the RpcResponseHeader which is returned by > NameNode and stored by DFSClient. > This JIRA is to followup on that work and have the DFSClient send their > stored "lastSeenStateId" in the RpcRequestHeader so that ObserverNodes can > then compare with their own and act accordingly. > This JIRA work focuses on just the part of making DFSClient send their state > through RpcRequestHeader. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13331) Add lastSeenStateId to RpcRequestHeader.
[
https://issues.apache.org/jira/browse/HDFS-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412318#comment-16412318
]
Konstantin Shvachko commented on HDFS-13331:
Thanks, Plamen. The changes looks good.
# I would make {{getLastSeenStateId()}} public in GCI contexts. I mean who are
we hiding it from. Then you don't need @VisibleForTesting I guess.
# You should also add some server-side logic in the call receiving the context.
I think we can just compare it the received {{stateId}} with the {{stateId}} of
the node and log a warning if the received Id is higher. It does nothing for
now, but sets the tone for next steps.
> Add lastSeenStateId to RpcRequestHeader.
>
>
> Key: HDFS-13331
> URL: https://issues.apache.org/jira/browse/HDFS-13331
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Plamen Jeliazkov
>Assignee: Plamen Jeliazkov
>Priority: Major
> Attachments: HDFS_13331.trunk.000.patch
>
>
> HDFS-12977 added a stateId into the RpcResponseHeader which is returned by
> NameNode and stored by DFSClient.
> This JIRA is to followup on that work and have the DFSClient send their
> stored "lastSeenStateId" in the RpcRequestHeader so that ObserverNodes can
> then compare with their own and act accordingly.
> This JIRA work focuses on just the part of making DFSClient send their state
> through RpcRequestHeader.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13331) Add lastSeenStateId to RpcRequestHeader.
[ https://issues.apache.org/jira/browse/HDFS-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411849#comment-16411849 ] Plamen Jeliazkov commented on HDFS-13331: - Attaching initial patch (based on top of HDFS-12943 branch). Couple major changes to observe from HDFS-12977: 1) I removed the long field lastSeenStateId from DFSClient and replaced it with a field for the AlignmentContext. This fixed the last checkstyle and simplified from having to store the lastSeenStateId twice (in ClientGCIContext and DFSClient). 2) ClientGCIContext no longer needs dependency on DFSClient as a result. No constructor necessary. 3) I added a new method in AlignmentContext that takes RpcRequestHeaderProto.Builder and updates the stateId proto field. Does nothing on server side. 4) Visible for testing methods added: Client.getAlignmentContext in order to facilitate a new test with Mockito spying. ClientGCIContext.getLastSeenStateId and DFSClient.getLastSeenSateId in facilitate (1) and keep tests simple. > Add lastSeenStateId to RpcRequestHeader. > > > Key: HDFS-13331 > URL: https://issues.apache.org/jira/browse/HDFS-13331 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Plamen Jeliazkov >Assignee: Plamen Jeliazkov >Priority: Major > Attachments: HDFS_13331.trunk.000.patch > > > HDFS-12977 added a stateId into the RpcResponseHeader which is returned by > NameNode and stored by DFSClient. > This JIRA is to followup on that work and have the DFSClient send their > stored "lastSeenStateId" in the RpcRequestHeader so that ObserverNodes can > then compare with their own and act accordingly. > This JIRA work focuses on just the part of making DFSClient send their state > through RpcRequestHeader. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
