[jira] [Commented] (HDFS-15251) Add new zookeeper event type case after zk updated to 3.5.x
[
https://issues.apache.org/jira/browse/HDFS-15251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071912#comment-17071912
]
Jianfei Jiang commented on HDFS-15251:
--
I found the new zk state when we update zk dependency of our own HBASE to 3.5,
some commands throw or log the unknown state. meanwhile the dependency of HBASE
in community has not updated yet. So I check the hdfs code and find this issue.
I am not sure the the closed state should act like disconnect or expired. So I
add two patches and i am puzzled. when zkfc will close the connection actively.
If only the moment of graceful shutdown(i will build 3.3.0-snapshot version to
make sure if it actually do close at the shutdown) close
the connection, it may be unnecessary to rejoin a election as its target is
just to shutdown.
> Add new zookeeper event type case after zk updated to 3.5.x
> ---
>
> Key: HDFS-15251
> URL: https://issues.apache.org/jira/browse/HDFS-15251
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.3.0
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
>Priority: Major
> Attachments: HDFS-15251.001.patch, HDFS-15251.002.patch
>
>
> In zookeeper 3.5.x, KeeperState add a new one named Closed, so should add
> Close case to the swich as it is not an unexpected Zookeeper watch event
> state.
> {code:java}
> /** @deprecated */
> @Deprecated
> Unknown(-1),
> Disconnected(0),
> /** @deprecated */
> @Deprecated
> NoSyncConnected(1),
> SyncConnected(3),
> AuthFailed(4),
> ConnectedReadOnly(5),
> SaslAuthenticated(6),
> Expired(-112),
> Closed(7);{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15251) Add new zookeeper event type case after zk updated to 3.5.x
[
https://issues.apache.org/jira/browse/HDFS-15251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071778#comment-17071778
]
Mate Szalay-Beko commented on HDFS-15251:
-
[~jiangjianfei], [~weichiu]
I guess this logic is needed for triggering a new election to decide who should
be the active / stand-by name node. I am not very familiar with the HDFS code,
so can not review that part, but I can give you some background info about the
CLOSED state.
It was introduced by [ZOOKEEPER-2368
|https://issues.apache.org/jira/browse/ZOOKEEPER-2368]. The ZooKeeper watcher
gets notified when the connection was broken by the ZooKeeper server and the
connection state is DISCONNECTED in this case. The new behaviour in 3.5.5+ is
that a new watcher event gets triggered even if the ZooKeeper client was the
one closing the connection in which case the connection state will be CLOSED.
So (as far as I can tell) it is never possible to get two watcher event when
the connection is closing. There will be only a single event and the state
should be either DISCONNECTED or CLOSED. Depending on who initiated the closing
of the connection. This makes the proposed patch logical. Handling this watcher
event definitely makes sense (at least to log it).
On the other hand I am not sure what is the expected behaviour in HDFS failover
controller when HDFS is closing the ZooKeeper connection. When do we call
ZooKeeper.close() on the connection in the HDFS code? I guess HDFS might do
this during some graceful shutdown in the failover controller process. Are we
sure we want to go to neutral mode and rejoin to election during shutdown? I
really don't know the background, so I let you to decide.
> Add new zookeeper event type case after zk updated to 3.5.x
> ---
>
> Key: HDFS-15251
> URL: https://issues.apache.org/jira/browse/HDFS-15251
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.3.0
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
>Priority: Major
> Attachments: HDFS-15251.001.patch, HDFS-15251.002.patch
>
>
> In zookeeper 3.5.x, KeeperState add a new one named Closed, so should add
> Close case to the swich as it is not an unexpected Zookeeper watch event
> state.
> {code:java}
> /** @deprecated */
> @Deprecated
> Unknown(-1),
> Disconnected(0),
> /** @deprecated */
> @Deprecated
> NoSyncConnected(1),
> SyncConnected(3),
> AuthFailed(4),
> ConnectedReadOnly(5),
> SaslAuthenticated(6),
> Expired(-112),
> Closed(7);{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15251) Add new zookeeper event type case after zk updated to 3.5.x
[
https://issues.apache.org/jira/browse/HDFS-15251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071680#comment-17071680
]
Hadoop QA commented on HDFS-15251:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
1s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 30m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 25m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
23m 8s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 20m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 12s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
56s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}136m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.8 Server=19.03.8 Image:yetus/hadoop:4454c6d14b7 |
| JIRA Issue | HDFS-15251 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12998311/HDFS-15251.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 98700e324774 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 80b877a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/29064/testReport/ |
| Max. process+thread count | 1486 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/29064/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add new zookeeper event type case after zk updated to 3.5.x
> ---
>
> Key: HDFS-15251
>
[jira] [Commented] (HDFS-15251) Add new zookeeper event type case after zk updated to 3.5.x
[
https://issues.apache.org/jira/browse/HDFS-15251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071486#comment-17071486
]
Jianfei Jiang commented on HDFS-15251:
--
oops, it may be a issue with prefix of hadoop but not hdfs.
> Add new zookeeper event type case after zk updated to 3.5.x
> ---
>
> Key: HDFS-15251
> URL: https://issues.apache.org/jira/browse/HDFS-15251
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.3.0
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
>Priority: Major
> Attachments: HDFS-15251.001.patch
>
>
> In zookeeper 3.5.x, KeeperState add a new one named Closed, so should add
> Close case to the swich as it is not an unexpected Zookeeper watch event
> state.
> {code:java}
> /** @deprecated */
> @Deprecated
> Unknown(-1),
> Disconnected(0),
> /** @deprecated */
> @Deprecated
> NoSyncConnected(1),
> SyncConnected(3),
> AuthFailed(4),
> ConnectedReadOnly(5),
> SaslAuthenticated(6),
> Expired(-112),
> Closed(7);{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15251) Add new zookeeper event type case after zk updated to 3.5.x
[
https://issues.apache.org/jira/browse/HDFS-15251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071458#comment-17071458
]
Hadoop QA commented on HDFS-15251:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 29m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 14s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 17s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
4s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}113m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.8 Server=19.03.8 Image:yetus/hadoop:4454c6d14b7 |
| JIRA Issue | HDFS-15251 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12998283/HDFS-15251.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux f6932bc9e976 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 80b877a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/29062/testReport/ |
| Max. process+thread count | 3161 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/29062/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add new zookeeper event type case after zk updated to 3.5.x
> ---
>
> Key: HDFS-15251
>
[jira] [Commented] (HDFS-15251) Add new zookeeper event type case after zk updated to 3.5.x
[
https://issues.apache.org/jira/browse/HDFS-15251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071422#comment-17071422
]
Jianfei Jiang commented on HDFS-15251:
--
Thank you [~weichiu], yes it only affect unreleased 3.3.0, change the version
is absolutely correct.
> Add new zookeeper event type case after zk updated to 3.5.x
> ---
>
> Key: HDFS-15251
> URL: https://issues.apache.org/jira/browse/HDFS-15251
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.3.0
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
>Priority: Major
> Attachments: HDFS-15251.001.patch
>
>
> In zookeeper 3.5.x, KeeperState add a new one named Closed, so should add
> Close case to the swich as it is not an unexpected Zookeeper watch event
> state.
> {code:java}
> /** @deprecated */
> @Deprecated
> Unknown(-1),
> Disconnected(0),
> /** @deprecated */
> @Deprecated
> NoSyncConnected(1),
> SyncConnected(3),
> AuthFailed(4),
> ConnectedReadOnly(5),
> SaslAuthenticated(6),
> Expired(-112),
> Closed(7);{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15251) Add new zookeeper event type case after zk updated to 3.5.x
[
https://issues.apache.org/jira/browse/HDFS-15251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071420#comment-17071420
]
Wei-Chiu Chuang commented on HDFS-15251:
Updated Affected Version to 3.3.0 since zk dependency was only updated in 3.3.0
(via HADOOP-16579). Let me know if I didn't understand it correctly.
> Add new zookeeper event type case after zk updated to 3.5.x
> ---
>
> Key: HDFS-15251
> URL: https://issues.apache.org/jira/browse/HDFS-15251
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.3.0
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
>Priority: Major
> Attachments: HDFS-15251.001.patch
>
>
> In zookeeper 3.5.x, KeeperState add a new one named Closed, so should add
> Close case to the swich as it is not an unexpected Zookeeper watch event
> state.
> {code:java}
> /** @deprecated */
> @Deprecated
> Unknown(-1),
> Disconnected(0),
> /** @deprecated */
> @Deprecated
> NoSyncConnected(1),
> SyncConnected(3),
> AuthFailed(4),
> ConnectedReadOnly(5),
> SaslAuthenticated(6),
> Expired(-112),
> Closed(7);{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15251) Add new zookeeper event type case after zk updated to 3.5.x
[
https://issues.apache.org/jira/browse/HDFS-15251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071419#comment-17071419
]
Wei-Chiu Chuang commented on HDFS-15251:
[~nkalmar] [~symat] fyi can you review? thanks
> Add new zookeeper event type case after zk updated to 3.5.x
> ---
>
> Key: HDFS-15251
> URL: https://issues.apache.org/jira/browse/HDFS-15251
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.2.1
>Reporter: Jianfei Jiang
>Assignee: Jianfei Jiang
>Priority: Major
> Attachments: HDFS-15251.001.patch
>
>
> In zookeeper 3.5.x, KeeperState add a new one named Closed, so should add
> Close case to the swich as it is not an unexpected Zookeeper watch event
> state.
> {code:java}
> /** @deprecated */
> @Deprecated
> Unknown(-1),
> Disconnected(0),
> /** @deprecated */
> @Deprecated
> NoSyncConnected(1),
> SyncConnected(3),
> AuthFailed(4),
> ConnectedReadOnly(5),
> SaslAuthenticated(6),
> Expired(-112),
> Closed(7);{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
