[jira] [Commented] (HBASE-15856) Cached Connection instances can wind up with addresses never resolved
[ https://issues.apache.org/jira/browse/HBASE-15856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15307695#comment-15307695 ] Yu Li commented on HBASE-15856: --- This problem is the same with HBASE-13960, let me mark 13960 duplicated of this one. Thanks for fix and get it in [~ghelmling]! > Cached Connection instances can wind up with addresses never resolved > - > > Key: HBASE-15856 > URL: https://issues.apache.org/jira/browse/HBASE-15856 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.2, 0.98.20, 1.1.6 > > Attachments: HBASE-15856.001.patch, HBASE-15856.002.patch, > HBASE-15856.003.patch, HBASE-15856.addendum.patch > > > During periods where DNS is not working properly, we can wind up caching > connections to master or regionservers where the initial hostname resolution > and the resolution is never re-attempted. This means that clients will > forever get UnknownHostException for any calls. > When constructing a BlockingRpcChannelImplementation, we instantiate the > InetSocketAddress to use for the connection. This instance is then used in > the rpc client connection, where we check isUnresolved() and throw an > UnknownHostException if that returns true. However, at this point the rpc > channel is already cached in the HConnectionImplementation map of stubs. So > at this point it will never be resolved. > Setting the config for hbase.resolve.hostnames.on.failure masks this issue, > since the stub key used is modified to contain the address. However, even in > that case, if DNS fails, an rpc channel instance with unresolved ISA will > still be cached in the stubs under the hostname only key. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15856) Cached Connection instances can wind up with addresses never resolved
[ https://issues.apache.org/jira/browse/HBASE-15856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295152#comment-15295152 ] Hudson commented on HBASE-15856: FAILURE: Integrated in HBase-0.98-matrix #347 (See [https://builds.apache.org/job/HBase-0.98-matrix/347/]) HBASE-15856 Don't cache unresolved addresses for connections (garyh: rev 5d2cd28c606b928ec4725e7c3fe194b8d1b9b82b) * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestConnectionImplementation.java * hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RpcClient.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientTimeouts.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java > Cached Connection instances can wind up with addresses never resolved > - > > Key: HBASE-15856 > URL: https://issues.apache.org/jira/browse/HBASE-15856 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.2, 0.98.20, 1.1.6 > > Attachments: HBASE-15856.001.patch, HBASE-15856.002.patch, > HBASE-15856.003.patch, HBASE-15856.addendum.patch > > > During periods where DNS is not working properly, we can wind up caching > connections to master or regionservers where the initial hostname resolution > and the resolution is never re-attempted. This means that clients will > forever get UnknownHostException for any calls. > When constructing a BlockingRpcChannelImplementation, we instantiate the > InetSocketAddress to use for the connection. This instance is then used in > the rpc client connection, where we check isUnresolved() and throw an > UnknownHostException if that returns true. However, at this point the rpc > channel is already cached in the HConnectionImplementation map of stubs. So > at this point it will never be resolved. > Setting the config for hbase.resolve.hostnames.on.failure masks this issue, > since the stub key used is modified to contain the address. However, even in > that case, if DNS fails, an rpc channel instance with unresolved ISA will > still be cached in the stubs under the hostname only key. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15856) Cached Connection instances can wind up with addresses never resolved
[ https://issues.apache.org/jira/browse/HBASE-15856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295145#comment-15295145 ] Hudson commented on HBASE-15856: FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1219 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1219/]) HBASE-15856 Don't cache unresolved addresses for connections (garyh: rev 5d2cd28c606b928ec4725e7c3fe194b8d1b9b82b) * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientTimeouts.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java * hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RpcClient.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestConnectionImplementation.java > Cached Connection instances can wind up with addresses never resolved > - > > Key: HBASE-15856 > URL: https://issues.apache.org/jira/browse/HBASE-15856 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.2, 0.98.20, 1.1.6 > > Attachments: HBASE-15856.001.patch, HBASE-15856.002.patch, > HBASE-15856.003.patch, HBASE-15856.addendum.patch > > > During periods where DNS is not working properly, we can wind up caching > connections to master or regionservers where the initial hostname resolution > and the resolution is never re-attempted. This means that clients will > forever get UnknownHostException for any calls. > When constructing a BlockingRpcChannelImplementation, we instantiate the > InetSocketAddress to use for the connection. This instance is then used in > the rpc client connection, where we check isUnresolved() and throw an > UnknownHostException if that returns true. However, at this point the rpc > channel is already cached in the HConnectionImplementation map of stubs. So > at this point it will never be resolved. > Setting the config for hbase.resolve.hostnames.on.failure masks this issue, > since the stub key used is modified to contain the address. However, even in > that case, if DNS fails, an rpc channel instance with unresolved ISA will > still be cached in the stubs under the hostname only key. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15856) Cached Connection instances can wind up with addresses never resolved
[ https://issues.apache.org/jira/browse/HBASE-15856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294839#comment-15294839 ] Hudson commented on HBASE-15856: SUCCESS: Integrated in HBase-1.1-JDK7 #1719 (See [https://builds.apache.org/job/HBase-1.1-JDK7/1719/]) HBASE-15856 Addendum Fix UnknownHostException import in MetaTableLocator (garyh: rev 9379345fa1807f9f5b665f72638c1d75d0b2d081) * hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/MetaTableLocator.java > Cached Connection instances can wind up with addresses never resolved > - > > Key: HBASE-15856 > URL: https://issues.apache.org/jira/browse/HBASE-15856 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.2, 0.98.20, 1.1.6 > > Attachments: HBASE-15856.001.patch, HBASE-15856.002.patch, > HBASE-15856.003.patch, HBASE-15856.addendum.patch > > > During periods where DNS is not working properly, we can wind up caching > connections to master or regionservers where the initial hostname resolution > and the resolution is never re-attempted. This means that clients will > forever get UnknownHostException for any calls. > When constructing a BlockingRpcChannelImplementation, we instantiate the > InetSocketAddress to use for the connection. This instance is then used in > the rpc client connection, where we check isUnresolved() and throw an > UnknownHostException if that returns true. However, at this point the rpc > channel is already cached in the HConnectionImplementation map of stubs. So > at this point it will never be resolved. > Setting the config for hbase.resolve.hostnames.on.failure masks this issue, > since the stub key used is modified to contain the address. However, even in > that case, if DNS fails, an rpc channel instance with unresolved ISA will > still be cached in the stubs under the hostname only key. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15856) Cached Connection instances can wind up with addresses never resolved
[ https://issues.apache.org/jira/browse/HBASE-15856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294836#comment-15294836 ] Hudson commented on HBASE-15856: FAILURE: Integrated in HBase-Trunk_matrix #938 (See [https://builds.apache.org/job/HBase-Trunk_matrix/938/]) HBASE-15856 Addendum Fix UnknownHostException import in MetaTableLocator (garyh: rev b1bf622e69b885219bfbf9ec59ee7beab1d3e08b) * hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/MetaTableLocator.java > Cached Connection instances can wind up with addresses never resolved > - > > Key: HBASE-15856 > URL: https://issues.apache.org/jira/browse/HBASE-15856 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.2, 0.98.20, 1.1.6 > > Attachments: HBASE-15856.001.patch, HBASE-15856.002.patch, > HBASE-15856.003.patch, HBASE-15856.addendum.patch > > > During periods where DNS is not working properly, we can wind up caching > connections to master or regionservers where the initial hostname resolution > and the resolution is never re-attempted. This means that clients will > forever get UnknownHostException for any calls. > When constructing a BlockingRpcChannelImplementation, we instantiate the > InetSocketAddress to use for the connection. This instance is then used in > the rpc client connection, where we check isUnresolved() and throw an > UnknownHostException if that returns true. However, at this point the rpc > channel is already cached in the HConnectionImplementation map of stubs. So > at this point it will never be resolved. > Setting the config for hbase.resolve.hostnames.on.failure masks this issue, > since the stub key used is modified to contain the address. However, even in > that case, if DNS fails, an rpc channel instance with unresolved ISA will > still be cached in the stubs under the hostname only key. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15856) Cached Connection instances can wind up with addresses never resolved
[ https://issues.apache.org/jira/browse/HBASE-15856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294834#comment-15294834 ] Hudson commented on HBASE-15856: SUCCESS: Integrated in HBase-1.1-JDK8 #1805 (See [https://builds.apache.org/job/HBase-1.1-JDK8/1805/]) HBASE-15856 Addendum Fix UnknownHostException import in MetaTableLocator (garyh: rev 9379345fa1807f9f5b665f72638c1d75d0b2d081) * hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/MetaTableLocator.java > Cached Connection instances can wind up with addresses never resolved > - > > Key: HBASE-15856 > URL: https://issues.apache.org/jira/browse/HBASE-15856 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.2, 0.98.20, 1.1.6 > > Attachments: HBASE-15856.001.patch, HBASE-15856.002.patch, > HBASE-15856.003.patch, HBASE-15856.addendum.patch > > > During periods where DNS is not working properly, we can wind up caching > connections to master or regionservers where the initial hostname resolution > and the resolution is never re-attempted. This means that clients will > forever get UnknownHostException for any calls. > When constructing a BlockingRpcChannelImplementation, we instantiate the > InetSocketAddress to use for the connection. This instance is then used in > the rpc client connection, where we check isUnresolved() and throw an > UnknownHostException if that returns true. However, at this point the rpc > channel is already cached in the HConnectionImplementation map of stubs. So > at this point it will never be resolved. > Setting the config for hbase.resolve.hostnames.on.failure masks this issue, > since the stub key used is modified to contain the address. However, even in > that case, if DNS fails, an rpc channel instance with unresolved ISA will > still be cached in the stubs under the hostname only key. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15856) Cached Connection instances can wind up with addresses never resolved
[ https://issues.apache.org/jira/browse/HBASE-15856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294794#comment-15294794 ] Hudson commented on HBASE-15856: SUCCESS: Integrated in HBase-1.3-IT #672 (See [https://builds.apache.org/job/HBase-1.3-IT/672/]) HBASE-15856 Addendum Fix UnknownHostException import in MetaTableLocator (garyh: rev df7e5b4f99d6ad18a5bd39c71c290c46106c5026) * hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/MetaTableLocator.java > Cached Connection instances can wind up with addresses never resolved > - > > Key: HBASE-15856 > URL: https://issues.apache.org/jira/browse/HBASE-15856 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.2, 0.98.20, 1.1.6 > > Attachments: HBASE-15856.001.patch, HBASE-15856.002.patch, > HBASE-15856.003.patch, HBASE-15856.addendum.patch > > > During periods where DNS is not working properly, we can wind up caching > connections to master or regionservers where the initial hostname resolution > and the resolution is never re-attempted. This means that clients will > forever get UnknownHostException for any calls. > When constructing a BlockingRpcChannelImplementation, we instantiate the > InetSocketAddress to use for the connection. This instance is then used in > the rpc client connection, where we check isUnresolved() and throw an > UnknownHostException if that returns true. However, at this point the rpc > channel is already cached in the HConnectionImplementation map of stubs. So > at this point it will never be resolved. > Setting the config for hbase.resolve.hostnames.on.failure masks this issue, > since the stub key used is modified to contain the address. However, even in > that case, if DNS fails, an rpc channel instance with unresolved ISA will > still be cached in the stubs under the hostname only key. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15856) Cached Connection instances can wind up with addresses never resolved
[ https://issues.apache.org/jira/browse/HBASE-15856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294780#comment-15294780 ] Hudson commented on HBASE-15856: SUCCESS: Integrated in HBase-1.2 #631 (See [https://builds.apache.org/job/HBase-1.2/631/]) HBASE-15856 Addendum Fix UnknownHostException import in MetaTableLocator (garyh: rev 83db381fb8a3f88e06349bf7afd43e062ada6239) * hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/MetaTableLocator.java > Cached Connection instances can wind up with addresses never resolved > - > > Key: HBASE-15856 > URL: https://issues.apache.org/jira/browse/HBASE-15856 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.2, 0.98.20, 1.1.6 > > Attachments: HBASE-15856.001.patch, HBASE-15856.002.patch, > HBASE-15856.003.patch, HBASE-15856.addendum.patch > > > During periods where DNS is not working properly, we can wind up caching > connections to master or regionservers where the initial hostname resolution > and the resolution is never re-attempted. This means that clients will > forever get UnknownHostException for any calls. > When constructing a BlockingRpcChannelImplementation, we instantiate the > InetSocketAddress to use for the connection. This instance is then used in > the rpc client connection, where we check isUnresolved() and throw an > UnknownHostException if that returns true. However, at this point the rpc > channel is already cached in the HConnectionImplementation map of stubs. So > at this point it will never be resolved. > Setting the config for hbase.resolve.hostnames.on.failure masks this issue, > since the stub key used is modified to contain the address. However, even in > that case, if DNS fails, an rpc channel instance with unresolved ISA will > still be cached in the stubs under the hostname only key. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15856) Cached Connection instances can wind up with addresses never resolved
[ https://issues.apache.org/jira/browse/HBASE-15856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294769#comment-15294769 ] Hudson commented on HBASE-15856: FAILURE: Integrated in HBase-1.3 #709 (See [https://builds.apache.org/job/HBase-1.3/709/]) HBASE-15856 Addendum Fix UnknownHostException import in MetaTableLocator (garyh: rev df7e5b4f99d6ad18a5bd39c71c290c46106c5026) * hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/MetaTableLocator.java > Cached Connection instances can wind up with addresses never resolved > - > > Key: HBASE-15856 > URL: https://issues.apache.org/jira/browse/HBASE-15856 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.2, 0.98.20, 1.1.6 > > Attachments: HBASE-15856.001.patch, HBASE-15856.002.patch, > HBASE-15856.003.patch, HBASE-15856.addendum.patch > > > During periods where DNS is not working properly, we can wind up caching > connections to master or regionservers where the initial hostname resolution > and the resolution is never re-attempted. This means that clients will > forever get UnknownHostException for any calls. > When constructing a BlockingRpcChannelImplementation, we instantiate the > InetSocketAddress to use for the connection. This instance is then used in > the rpc client connection, where we check isUnresolved() and throw an > UnknownHostException if that returns true. However, at this point the rpc > channel is already cached in the HConnectionImplementation map of stubs. So > at this point it will never be resolved. > Setting the config for hbase.resolve.hostnames.on.failure masks this issue, > since the stub key used is modified to contain the address. However, even in > that case, if DNS fails, an rpc channel instance with unresolved ISA will > still be cached in the stubs under the hostname only key. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15856) Cached Connection instances can wind up with addresses never resolved
[ https://issues.apache.org/jira/browse/HBASE-15856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294746#comment-15294746 ] Hudson commented on HBASE-15856: SUCCESS: Integrated in HBase-1.4 #168 (See [https://builds.apache.org/job/HBase-1.4/168/]) HBASE-15856 Addendum Fix UnknownHostException import in MetaTableLocator (garyh: rev aacc816ac76edd9dd6d1f069d828729d7ee1fc31) * hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/MetaTableLocator.java > Cached Connection instances can wind up with addresses never resolved > - > > Key: HBASE-15856 > URL: https://issues.apache.org/jira/browse/HBASE-15856 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.2, 0.98.20, 1.1.6 > > Attachments: HBASE-15856.001.patch, HBASE-15856.002.patch, > HBASE-15856.003.patch, HBASE-15856.addendum.patch > > > During periods where DNS is not working properly, we can wind up caching > connections to master or regionservers where the initial hostname resolution > and the resolution is never re-attempted. This means that clients will > forever get UnknownHostException for any calls. > When constructing a BlockingRpcChannelImplementation, we instantiate the > InetSocketAddress to use for the connection. This instance is then used in > the rpc client connection, where we check isUnresolved() and throw an > UnknownHostException if that returns true. However, at this point the rpc > channel is already cached in the HConnectionImplementation map of stubs. So > at this point it will never be resolved. > Setting the config for hbase.resolve.hostnames.on.failure masks this issue, > since the stub key used is modified to contain the address. However, even in > that case, if DNS fails, an rpc channel instance with unresolved ISA will > still be cached in the stubs under the hostname only key. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15856) Cached Connection instances can wind up with addresses never resolved
[ https://issues.apache.org/jira/browse/HBASE-15856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294679#comment-15294679 ] Hudson commented on HBASE-15856: SUCCESS: Integrated in HBase-1.2-IT #513 (See [https://builds.apache.org/job/HBase-1.2-IT/513/]) HBASE-15856 Addendum Fix UnknownHostException import in MetaTableLocator (garyh: rev 83db381fb8a3f88e06349bf7afd43e062ada6239) * hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/MetaTableLocator.java > Cached Connection instances can wind up with addresses never resolved > - > > Key: HBASE-15856 > URL: https://issues.apache.org/jira/browse/HBASE-15856 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.2, 0.98.20, 1.1.6 > > Attachments: HBASE-15856.001.patch, HBASE-15856.002.patch, > HBASE-15856.003.patch > > > During periods where DNS is not working properly, we can wind up caching > connections to master or regionservers where the initial hostname resolution > and the resolution is never re-attempted. This means that clients will > forever get UnknownHostException for any calls. > When constructing a BlockingRpcChannelImplementation, we instantiate the > InetSocketAddress to use for the connection. This instance is then used in > the rpc client connection, where we check isUnresolved() and throw an > UnknownHostException if that returns true. However, at this point the rpc > channel is already cached in the HConnectionImplementation map of stubs. So > at this point it will never be resolved. > Setting the config for hbase.resolve.hostnames.on.failure masks this issue, > since the stub key used is modified to contain the address. However, even in > that case, if DNS fails, an rpc channel instance with unresolved ISA will > still be cached in the stubs under the hostname only key. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15856) Cached Connection instances can wind up with addresses never resolved
[ https://issues.apache.org/jira/browse/HBASE-15856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294566#comment-15294566 ] Gary Helmling commented on HBASE-15856: --- Looks like this may have broken TestMasterFailover in branch-1 and older. I'm taking a look at it. If I can't come up with a solution tonight, I'll revert from those branches. > Cached Connection instances can wind up with addresses never resolved > - > > Key: HBASE-15856 > URL: https://issues.apache.org/jira/browse/HBASE-15856 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.2, 0.98.20, 1.1.6 > > Attachments: HBASE-15856.001.patch, HBASE-15856.002.patch, > HBASE-15856.003.patch > > > During periods where DNS is not working properly, we can wind up caching > connections to master or regionservers where the initial hostname resolution > and the resolution is never re-attempted. This means that clients will > forever get UnknownHostException for any calls. > When constructing a BlockingRpcChannelImplementation, we instantiate the > InetSocketAddress to use for the connection. This instance is then used in > the rpc client connection, where we check isUnresolved() and throw an > UnknownHostException if that returns true. However, at this point the rpc > channel is already cached in the HConnectionImplementation map of stubs. So > at this point it will never be resolved. > Setting the config for hbase.resolve.hostnames.on.failure masks this issue, > since the stub key used is modified to contain the address. However, even in > that case, if DNS fails, an rpc channel instance with unresolved ISA will > still be cached in the stubs under the hostname only key. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15856) Cached Connection instances can wind up with addresses never resolved
[ https://issues.apache.org/jira/browse/HBASE-15856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294554#comment-15294554 ] Hudson commented on HBASE-15856: FAILURE: Integrated in HBase-1.1-JDK7 #1718 (See [https://builds.apache.org/job/HBase-1.1-JDK7/1718/]) HBASE-15856 Don't cache unresolved addresses for connections (garyh: rev 976f3bc29df33a152bb23f1507f89d8e2c84d61c) * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientTimeouts.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestConnectionImplementation.java * hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AbstractRpcClient.java > Cached Connection instances can wind up with addresses never resolved > - > > Key: HBASE-15856 > URL: https://issues.apache.org/jira/browse/HBASE-15856 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.2, 0.98.20, 1.1.6 > > Attachments: HBASE-15856.001.patch, HBASE-15856.002.patch, > HBASE-15856.003.patch > > > During periods where DNS is not working properly, we can wind up caching > connections to master or regionservers where the initial hostname resolution > and the resolution is never re-attempted. This means that clients will > forever get UnknownHostException for any calls. > When constructing a BlockingRpcChannelImplementation, we instantiate the > InetSocketAddress to use for the connection. This instance is then used in > the rpc client connection, where we check isUnresolved() and throw an > UnknownHostException if that returns true. However, at this point the rpc > channel is already cached in the HConnectionImplementation map of stubs. So > at this point it will never be resolved. > Setting the config for hbase.resolve.hostnames.on.failure masks this issue, > since the stub key used is modified to contain the address. However, even in > that case, if DNS fails, an rpc channel instance with unresolved ISA will > still be cached in the stubs under the hostname only key. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15856) Cached Connection instances can wind up with addresses never resolved
[ https://issues.apache.org/jira/browse/HBASE-15856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294549#comment-15294549 ] Hudson commented on HBASE-15856: FAILURE: Integrated in HBase-1.1-JDK8 #1804 (See [https://builds.apache.org/job/HBase-1.1-JDK8/1804/]) HBASE-15856 Don't cache unresolved addresses for connections (garyh: rev 976f3bc29df33a152bb23f1507f89d8e2c84d61c) * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestConnectionImplementation.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java * hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AbstractRpcClient.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientTimeouts.java > Cached Connection instances can wind up with addresses never resolved > - > > Key: HBASE-15856 > URL: https://issues.apache.org/jira/browse/HBASE-15856 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.2, 0.98.20, 1.1.6 > > Attachments: HBASE-15856.001.patch, HBASE-15856.002.patch, > HBASE-15856.003.patch > > > During periods where DNS is not working properly, we can wind up caching > connections to master or regionservers where the initial hostname resolution > and the resolution is never re-attempted. This means that clients will > forever get UnknownHostException for any calls. > When constructing a BlockingRpcChannelImplementation, we instantiate the > InetSocketAddress to use for the connection. This instance is then used in > the rpc client connection, where we check isUnresolved() and throw an > UnknownHostException if that returns true. However, at this point the rpc > channel is already cached in the HConnectionImplementation map of stubs. So > at this point it will never be resolved. > Setting the config for hbase.resolve.hostnames.on.failure masks this issue, > since the stub key used is modified to contain the address. However, even in > that case, if DNS fails, an rpc channel instance with unresolved ISA will > still be cached in the stubs under the hostname only key. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15856) Cached Connection instances can wind up with addresses never resolved
[ https://issues.apache.org/jira/browse/HBASE-15856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294347#comment-15294347 ] Hudson commented on HBASE-15856: FAILURE: Integrated in HBase-1.3 #707 (See [https://builds.apache.org/job/HBase-1.3/707/]) HBASE-15856 Don't cache unresolved addresses for connections (garyh: rev efdc0e10096413460fff9c23929da0f980ddf941) * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientTimeouts.java * hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AbstractRpcClient.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestConnectionImplementation.java > Cached Connection instances can wind up with addresses never resolved > - > > Key: HBASE-15856 > URL: https://issues.apache.org/jira/browse/HBASE-15856 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.2, 0.98.20, 1.1.6 > > Attachments: HBASE-15856.001.patch, HBASE-15856.002.patch, > HBASE-15856.003.patch > > > During periods where DNS is not working properly, we can wind up caching > connections to master or regionservers where the initial hostname resolution > and the resolution is never re-attempted. This means that clients will > forever get UnknownHostException for any calls. > When constructing a BlockingRpcChannelImplementation, we instantiate the > InetSocketAddress to use for the connection. This instance is then used in > the rpc client connection, where we check isUnresolved() and throw an > UnknownHostException if that returns true. However, at this point the rpc > channel is already cached in the HConnectionImplementation map of stubs. So > at this point it will never be resolved. > Setting the config for hbase.resolve.hostnames.on.failure masks this issue, > since the stub key used is modified to contain the address. However, even in > that case, if DNS fails, an rpc channel instance with unresolved ISA will > still be cached in the stubs under the hostname only key. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15856) Cached Connection instances can wind up with addresses never resolved
[ https://issues.apache.org/jira/browse/HBASE-15856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294328#comment-15294328 ] Hudson commented on HBASE-15856: FAILURE: Integrated in HBase-1.4 #166 (See [https://builds.apache.org/job/HBase-1.4/166/]) HBASE-15856 Don't cache unresolved addresses for connections (garyh: rev df40739ae7217568b5dfeb9eb8fe922f6242042a) * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientTimeouts.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestConnectionImplementation.java * hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AbstractRpcClient.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java > Cached Connection instances can wind up with addresses never resolved > - > > Key: HBASE-15856 > URL: https://issues.apache.org/jira/browse/HBASE-15856 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.2, 0.98.20, 1.1.6 > > Attachments: HBASE-15856.001.patch, HBASE-15856.002.patch, > HBASE-15856.003.patch > > > During periods where DNS is not working properly, we can wind up caching > connections to master or regionservers where the initial hostname resolution > and the resolution is never re-attempted. This means that clients will > forever get UnknownHostException for any calls. > When constructing a BlockingRpcChannelImplementation, we instantiate the > InetSocketAddress to use for the connection. This instance is then used in > the rpc client connection, where we check isUnresolved() and throw an > UnknownHostException if that returns true. However, at this point the rpc > channel is already cached in the HConnectionImplementation map of stubs. So > at this point it will never be resolved. > Setting the config for hbase.resolve.hostnames.on.failure masks this issue, > since the stub key used is modified to contain the address. However, even in > that case, if DNS fails, an rpc channel instance with unresolved ISA will > still be cached in the stubs under the hostname only key. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15856) Cached Connection instances can wind up with addresses never resolved
[ https://issues.apache.org/jira/browse/HBASE-15856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294330#comment-15294330 ] Hudson commented on HBASE-15856: FAILURE: Integrated in HBase-1.2 #629 (See [https://builds.apache.org/job/HBase-1.2/629/]) HBASE-15856 Don't cache unresolved addresses for connections (garyh: rev 88c415c2f9af360fe67bf0805bcee06a0f5b8ab6) * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestConnectionImplementation.java * hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AbstractRpcClient.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientTimeouts.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java > Cached Connection instances can wind up with addresses never resolved > - > > Key: HBASE-15856 > URL: https://issues.apache.org/jira/browse/HBASE-15856 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.2, 0.98.20, 1.1.6 > > Attachments: HBASE-15856.001.patch, HBASE-15856.002.patch, > HBASE-15856.003.patch > > > During periods where DNS is not working properly, we can wind up caching > connections to master or regionservers where the initial hostname resolution > and the resolution is never re-attempted. This means that clients will > forever get UnknownHostException for any calls. > When constructing a BlockingRpcChannelImplementation, we instantiate the > InetSocketAddress to use for the connection. This instance is then used in > the rpc client connection, where we check isUnresolved() and throw an > UnknownHostException if that returns true. However, at this point the rpc > channel is already cached in the HConnectionImplementation map of stubs. So > at this point it will never be resolved. > Setting the config for hbase.resolve.hostnames.on.failure masks this issue, > since the stub key used is modified to contain the address. However, even in > that case, if DNS fails, an rpc channel instance with unresolved ISA will > still be cached in the stubs under the hostname only key. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15856) Cached Connection instances can wind up with addresses never resolved
[ https://issues.apache.org/jira/browse/HBASE-15856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294064#comment-15294064 ] Hudson commented on HBASE-15856: SUCCESS: Integrated in HBase-1.2-IT #511 (See [https://builds.apache.org/job/HBase-1.2-IT/511/]) HBASE-15856 Don't cache unresolved addresses for connections (garyh: rev 88c415c2f9af360fe67bf0805bcee06a0f5b8ab6) * hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AbstractRpcClient.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestConnectionImplementation.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientTimeouts.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java > Cached Connection instances can wind up with addresses never resolved > - > > Key: HBASE-15856 > URL: https://issues.apache.org/jira/browse/HBASE-15856 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.2, 0.98.20, 1.1.6 > > Attachments: HBASE-15856.001.patch, HBASE-15856.002.patch, > HBASE-15856.003.patch > > > During periods where DNS is not working properly, we can wind up caching > connections to master or regionservers where the initial hostname resolution > and the resolution is never re-attempted. This means that clients will > forever get UnknownHostException for any calls. > When constructing a BlockingRpcChannelImplementation, we instantiate the > InetSocketAddress to use for the connection. This instance is then used in > the rpc client connection, where we check isUnresolved() and throw an > UnknownHostException if that returns true. However, at this point the rpc > channel is already cached in the HConnectionImplementation map of stubs. So > at this point it will never be resolved. > Setting the config for hbase.resolve.hostnames.on.failure masks this issue, > since the stub key used is modified to contain the address. However, even in > that case, if DNS fails, an rpc channel instance with unresolved ISA will > still be cached in the stubs under the hostname only key. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15856) Cached Connection instances can wind up with addresses never resolved
[ https://issues.apache.org/jira/browse/HBASE-15856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294025#comment-15294025 ] Hudson commented on HBASE-15856: FAILURE: Integrated in HBase-Trunk_matrix #936 (See [https://builds.apache.org/job/HBase-Trunk_matrix/936/]) HBASE-15856 Don't cache unresolved addresses for connections (garyh: rev 4793988aef849625c565e1cc410af39040881f6d) * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientTimeouts.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestConnectionImplementation.java * hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AbstractRpcClient.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java > Cached Connection instances can wind up with addresses never resolved > - > > Key: HBASE-15856 > URL: https://issues.apache.org/jira/browse/HBASE-15856 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.2 > > Attachments: HBASE-15856.001.patch, HBASE-15856.002.patch, > HBASE-15856.003.patch > > > During periods where DNS is not working properly, we can wind up caching > connections to master or regionservers where the initial hostname resolution > and the resolution is never re-attempted. This means that clients will > forever get UnknownHostException for any calls. > When constructing a BlockingRpcChannelImplementation, we instantiate the > InetSocketAddress to use for the connection. This instance is then used in > the rpc client connection, where we check isUnresolved() and throw an > UnknownHostException if that returns true. However, at this point the rpc > channel is already cached in the HConnectionImplementation map of stubs. So > at this point it will never be resolved. > Setting the config for hbase.resolve.hostnames.on.failure masks this issue, > since the stub key used is modified to contain the address. However, even in > that case, if DNS fails, an rpc channel instance with unresolved ISA will > still be cached in the stubs under the hostname only key. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15856) Cached Connection instances can wind up with addresses never resolved
[ https://issues.apache.org/jira/browse/HBASE-15856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15293976#comment-15293976 ] Hudson commented on HBASE-15856: SUCCESS: Integrated in HBase-1.3-IT #670 (See [https://builds.apache.org/job/HBase-1.3-IT/670/]) HBASE-15856 Don't cache unresolved addresses for connections (garyh: rev efdc0e10096413460fff9c23929da0f980ddf941) * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestConnectionImplementation.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientTimeouts.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java * hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AbstractRpcClient.java > Cached Connection instances can wind up with addresses never resolved > - > > Key: HBASE-15856 > URL: https://issues.apache.org/jira/browse/HBASE-15856 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.2 > > Attachments: HBASE-15856.001.patch, HBASE-15856.002.patch, > HBASE-15856.003.patch > > > During periods where DNS is not working properly, we can wind up caching > connections to master or regionservers where the initial hostname resolution > and the resolution is never re-attempted. This means that clients will > forever get UnknownHostException for any calls. > When constructing a BlockingRpcChannelImplementation, we instantiate the > InetSocketAddress to use for the connection. This instance is then used in > the rpc client connection, where we check isUnresolved() and throw an > UnknownHostException if that returns true. However, at this point the rpc > channel is already cached in the HConnectionImplementation map of stubs. So > at this point it will never be resolved. > Setting the config for hbase.resolve.hostnames.on.failure masks this issue, > since the stub key used is modified to contain the address. However, even in > that case, if DNS fails, an rpc channel instance with unresolved ISA will > still be cached in the stubs under the hostname only key. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15856) Cached Connection instances can wind up with addresses never resolved
[ https://issues.apache.org/jira/browse/HBASE-15856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15293175#comment-15293175 ] stack commented on HBASE-15856: --- +1 Excellent. > Cached Connection instances can wind up with addresses never resolved > - > > Key: HBASE-15856 > URL: https://issues.apache.org/jira/browse/HBASE-15856 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.2 > > Attachments: HBASE-15856.001.patch, HBASE-15856.002.patch, > HBASE-15856.003.patch > > > During periods where DNS is not working properly, we can wind up caching > connections to master or regionservers where the initial hostname resolution > and the resolution is never re-attempted. This means that clients will > forever get UnknownHostException for any calls. > When constructing a BlockingRpcChannelImplementation, we instantiate the > InetSocketAddress to use for the connection. This instance is then used in > the rpc client connection, where we check isUnresolved() and throw an > UnknownHostException if that returns true. However, at this point the rpc > channel is already cached in the HConnectionImplementation map of stubs. So > at this point it will never be resolved. > Setting the config for hbase.resolve.hostnames.on.failure masks this issue, > since the stub key used is modified to contain the address. However, even in > that case, if DNS fails, an rpc channel instance with unresolved ISA will > still be cached in the stubs under the hostname only key. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15856) Cached Connection instances can wind up with addresses never resolved
[
https://issues.apache.org/jira/browse/HBASE-15856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292770#comment-15292770
]
Hadoop QA commented on HBASE-15856:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
1s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 23s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
56s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 24s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 8m
47s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1
2.5.2 2.6.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 55s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 103m 24s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 135m 34s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12805124/HBASE-15856.003.patch
|
| JIRA Issue | HBASE-15856 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-15856) Cached Connection instances can wind up with addresses never resolved
[
https://issues.apache.org/jira/browse/HBASE-15856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292481#comment-15292481
]
Gary Helmling commented on HBASE-15856:
---
In {{testFailover()}} the master is failing to start with:
{noformat}
2016-05-19 17:48:53,221 FATAL [localhost:37057.activeMasterManager]
master.HMaster$1(1769): Failed to become active master
java.net.UnknownHostException: 0.example.org
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.(AbstractRpcClient.java:340)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.createBlockingRpcChannel(AbstractRpcClient.java:271)
at
org.apache.hadoop.hbase.client.ConnectionImplementation.getAdmin(ConnectionImplementation.java:1265)
at
org.apache.hadoop.hbase.client.ConnectionUtils$1.getAdmin(ConnectionUtils.java:135)
at
org.apache.hadoop.hbase.client.ConnectionImplementation.getAdmin(ConnectionImplementation.java:1246)
at
org.apache.hadoop.hbase.zookeeper.MetaTableLocator.getCachedConnection(MetaTableLocator.java:387)
at
org.apache.hadoop.hbase.zookeeper.MetaTableLocator.getMetaServerConnection(MetaTableLocator.java:368)
at
org.apache.hadoop.hbase.zookeeper.MetaTableLocator.verifyMetaRegionLocation(MetaTableLocator.java:283)
at org.apache.hadoop.hbase.master.HMaster.assignMeta(HMaster.java:921)
at
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:759)
at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:194)
at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1765)
at java.lang.Thread.run(Thread.java:745)
{noformat}
So it seems like the {{MetaTableLocator.verifyMetaRegionLocation()}} call was
not really effective for UnknownHostException prior to this patch, since UHE
wouldn't actually be triggered.
> Cached Connection instances can wind up with addresses never resolved
> -
>
> Key: HBASE-15856
> URL: https://issues.apache.org/jira/browse/HBASE-15856
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Gary Helmling
>Assignee: Gary Helmling
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.2
>
> Attachments: HBASE-15856.001.patch, HBASE-15856.002.patch
>
>
> During periods where DNS is not working properly, we can wind up caching
> connections to master or regionservers where the initial hostname resolution
> and the resolution is never re-attempted. This means that clients will
> forever get UnknownHostException for any calls.
> When constructing a BlockingRpcChannelImplementation, we instantiate the
> InetSocketAddress to use for the connection. This instance is then used in
> the rpc client connection, where we check isUnresolved() and throw an
> UnknownHostException if that returns true. However, at this point the rpc
> channel is already cached in the HConnectionImplementation map of stubs. So
> at this point it will never be resolved.
> Setting the config for hbase.resolve.hostnames.on.failure masks this issue,
> since the stub key used is modified to contain the address. However, even in
> that case, if DNS fails, an rpc channel instance with unresolved ISA will
> still be cached in the stubs under the hostname only key.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15856) Cached Connection instances can wind up with addresses never resolved
[
https://issues.apache.org/jira/browse/HBASE-15856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292452#comment-15292452
]
Gary Helmling commented on HBASE-15856:
---
The failure in {{TestMasterNoCluster}} is actually caused by this patch. In
{{testFailover()}} it's using fake, unresolvable, hostnames, which this change
is probably breaking. Looks like that will need to be rewritten.
> Cached Connection instances can wind up with addresses never resolved
> -
>
> Key: HBASE-15856
> URL: https://issues.apache.org/jira/browse/HBASE-15856
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Gary Helmling
>Assignee: Gary Helmling
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.2
>
> Attachments: HBASE-15856.001.patch, HBASE-15856.002.patch
>
>
> During periods where DNS is not working properly, we can wind up caching
> connections to master or regionservers where the initial hostname resolution
> and the resolution is never re-attempted. This means that clients will
> forever get UnknownHostException for any calls.
> When constructing a BlockingRpcChannelImplementation, we instantiate the
> InetSocketAddress to use for the connection. This instance is then used in
> the rpc client connection, where we check isUnresolved() and throw an
> UnknownHostException if that returns true. However, at this point the rpc
> channel is already cached in the HConnectionImplementation map of stubs. So
> at this point it will never be resolved.
> Setting the config for hbase.resolve.hostnames.on.failure masks this issue,
> since the stub key used is modified to contain the address. However, even in
> that case, if DNS fails, an rpc channel instance with unresolved ISA will
> still be cached in the stubs under the hostname only key.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15856) Cached Connection instances can wind up with addresses never resolved
[
https://issues.apache.org/jira/browse/HBASE-15856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292413#comment-15292413
]
Hadoop QA commented on HBASE-15856:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 58s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 8m
32s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1
2.5.2 2.6.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 56s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 114m 4s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 145m 29s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.TestMasterNoCluster |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12805053/HBASE-15856.002.patch
|
| JIRA Issue | HBASE-15856 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64
[jira] [Commented] (HBASE-15856) Cached Connection instances can wind up with addresses never resolved
[
https://issues.apache.org/jira/browse/HBASE-15856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292123#comment-15292123
]
Hadoop QA commented on HBASE-15856:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
5s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 8m
50s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1
2.5.2 2.6.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 56s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 4s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 48m 37s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.TestCheckTestClasses |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12805035/HBASE-15856.001.patch
|
| JIRA Issue | HBASE-15856 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64
[jira] [Commented] (HBASE-15856) Cached Connection instances can wind up with addresses never resolved
[
https://issues.apache.org/jira/browse/HBASE-15856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15292056#comment-15292056
]
Ted Yu commented on HBASE-15856:
lgtm
{code}
+ */
+public class TestConnectionImplementation {
{code}
Please add test category.
> Cached Connection instances can wind up with addresses never resolved
> -
>
> Key: HBASE-15856
> URL: https://issues.apache.org/jira/browse/HBASE-15856
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Gary Helmling
>Assignee: Gary Helmling
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.2
>
> Attachments: HBASE-15856.001.patch
>
>
> During periods where DNS is not working properly, we can wind up caching
> connections to master or regionservers where the initial hostname resolution
> and the resolution is never re-attempted. This means that clients will
> forever get UnknownHostException for any calls.
> When constructing a BlockingRpcChannelImplementation, we instantiate the
> InetSocketAddress to use for the connection. This instance is then used in
> the rpc client connection, where we check isUnresolved() and throw an
> UnknownHostException if that returns true. However, at this point the rpc
> channel is already cached in the HConnectionImplementation map of stubs. So
> at this point it will never be resolved.
> Setting the config for hbase.resolve.hostnames.on.failure masks this issue,
> since the stub key used is modified to contain the address. However, even in
> that case, if DNS fails, an rpc channel instance with unresolved ISA will
> still be cached in the stubs under the hostname only key.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
