[
https://issues.apache.org/jira/browse/HDFS-11308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
tangshangwen updated HDFS-11308:
--------------------------------
Description:
In our Cluster, I found some abnormal in ZKFC log
{noformat}
[2017-01-10T01:42:37.168+08:00] [INFO]
hadoop.ha.SshFenceByTcpPort.doFence(SshFenceByTcpPort.java 147) [Health Monitor
for NameNode at xxx-xxx-172xxx.hadoop.xxx.com/xxx.xxx.172.xxx:8021-EventThread]
: Indeterminate response from trying to kill service. Verifying whether it is
running using nc...
[2017-01-10T01:42:37.234+08:00] [WARN]
hadoop.ha.SshFenceByTcpPort.pump(StreamPumper.java 88) [nc -z
xxx-xxx-172xx.hadoop.xx.com 8021 via ssh: StreamPumper for STDERR] : nc -z
xxx-xxx-172xx.hadoop.xxx.com 8021 via ssh: nc: invalid option -- 'z'
[2017-01-10T01:42:37.235+08:00] [WARN]
hadoop.ha.SshFenceByTcpPort.pump(StreamPumper.java 88) [nc -z
xxx-xxx-172xx.hadoop.xxx.com 8021 via ssh: StreamPumper for STDERR] : nc -z
xxx-xxx-17224.hadoop.xxx.com 8021 via ssh: Ncat: Try `--help' or man(1) ncat
for more information, usage options and help. QUITTING.
{noformat}
When I perform nc an exception occurs, the return value is 2, and cannot
confirm sshfence success，this may lead to some problems
{code:title=SshFenceByTcpPort.java|borderStyle=solid}
rc = execCommand(session, "nc -z " + serviceAddr.getHostName() +
" " + serviceAddr.getPort());
if (rc == 0) {
// the service is still listening - we are unable to fence
LOG.warn("Unable to fence - it is running but we cannot kill it");
return false;
} else {
LOG.info("Verified that the service is down.");
return true;
}
{code}
was:
In our Cluster, I found some abnormal in ZKFC log
{noformat}
[2017-01-10T01:42:37.168+08:00] [INFO]
hadoop.ha.SshFenceByTcpPort.doFence(SshFenceByTcpPort.java 147) [Health Monitor
for NameNode at xxx-xxx-172xxx.hadoop.xxx.com/xxx.xxx.172.xxx:8021-EventThread]
: Indeterminate response from trying to kill service. Verifying whether it is
running using nc...
[2017-01-10T01:42:37.234+08:00] [WARN]
hadoop.ha.SshFenceByTcpPort.pump(StreamPumper.java 88) [nc -z
BJYF-Druid-17224.hadoop.jd.local 8021 via ssh: StreamPumper for STDERR] : nc -z
xxx-xxx-172xx.hadoop.xxx.com 8021 via ssh: nc: invalid option -- 'z'
[2017-01-10T01:42:37.235+08:00] [WARN]
hadoop.ha.SshFenceByTcpPort.pump(StreamPumper.java 88) [nc -z
xxx-xxx-172xx.hadoop.xxx.com 8021 via ssh: StreamPumper for STDERR] : nc -z
BJYF-Druid-17224.hadoop.jd.local 8021 via ssh: Ncat: Try `--help' or man(1)
ncat for more information, usage options and help. QUITTING.
{noformat}
When I perform nc an exception occurs, the return value is 2, and cannot
confirm sshfence success，this may lead to some problems
{code:title=SshFenceByTcpPort.java|borderStyle=solid}
rc = execCommand(session, "nc -z " + serviceAddr.getHostName() +
" " + serviceAddr.getPort());
if (rc == 0) {
// the service is still listening - we are unable to fence
LOG.warn("Unable to fence - it is running but we cannot kill it");
return false;
} else {
LOG.info("Verified that the service is down.");
return true;
}
{code}
> NameNode doFence state judgment problem
> ---------------------------------------
>
> Key: HDFS-11308
> URL: https://issues.apache.org/jira/browse/HDFS-11308
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: auto-failover
> Affects Versions: 2.7.1
> Environment: CentOS Linux release 7.1.1503 (Core)
> Reporter: tangshangwen
>
> In our Cluster, I found some abnormal in ZKFC log
> {noformat}
> [2017-01-10T01:42:37.168+08:00] [INFO]
> hadoop.ha.SshFenceByTcpPort.doFence(SshFenceByTcpPort.java 147) [Health
> Monitor for NameNode at
> xxx-xxx-172xxx.hadoop.xxx.com/xxx.xxx.172.xxx:8021-EventThread] :
> Indeterminate response from trying to kill service. Verifying whether it is
> running using nc...
> [2017-01-10T01:42:37.234+08:00] [WARN]
> hadoop.ha.SshFenceByTcpPort.pump(StreamPumper.java 88) [nc -z
> xxx-xxx-172xx.hadoop.xx.com 8021 via ssh: StreamPumper for STDERR] : nc -z
> xxx-xxx-172xx.hadoop.xxx.com 8021 via ssh: nc: invalid option -- 'z'
> [2017-01-10T01:42:37.235+08:00] [WARN]
> hadoop.ha.SshFenceByTcpPort.pump(StreamPumper.java 88) [nc -z
> xxx-xxx-172xx.hadoop.xxx.com 8021 via ssh: StreamPumper for STDERR] : nc -z
> xxx-xxx-17224.hadoop.xxx.com 8021 via ssh: Ncat: Try `--help' or man(1) ncat
> for more information, usage options and help. QUITTING.
> {noformat}
> When I perform nc an exception occurs, the return value is 2, and cannot
> confirm sshfence success，this may lead to some problems
> {code:title=SshFenceByTcpPort.java|borderStyle=solid}
> rc = execCommand(session, "nc -z " + serviceAddr.getHostName() +
> " " + serviceAddr.getPort());
> if (rc == 0) {
> // the service is still listening - we are unable to fence
> LOG.warn("Unable to fence - it is running but we cannot kill it");
> return false;
> } else {
> LOG.info("Verified that the service is down.");
> return true;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
