[jira] [Updated] (HBASE-15957) RpcClientImpl.close never ends in some circumstances
[
https://issues.apache.org/jira/browse/HBASE-15957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-15957:
---
Fix Version/s: (was: 1.4.0)
> RpcClientImpl.close never ends in some circumstances
>
>
> Key: HBASE-15957
> URL: https://issues.apache.org/jira/browse/HBASE-15957
> Project: HBase
> Issue Type: Bug
> Components: Client, rpc
>Affects Versions: 1.1.2
>Reporter: Sergey Soldatov
>Assignee: Sergey Soldatov
> Fix For: 2.0.0, 1.3.0, 1.2.2, 1.1.6
>
> Attachments: HBASE-15957-2.patch, HBASE-15957.patch
>
>
> This bug is related to HBASE-14241 and HBASE-13851.
> Fix for HBASE-13851 introduced the check for non alive connections and if
> connection is not alive, it close it:
> {noformat}
> if (!conn.isAlive()) {
> if (connsToClose == null) {
> connsToClose = new HashSet();
> }
> connsToClose.add(conn);
> }
>
> if (connsToClose != null) {
> for (Connection conn : connsToClose) {
> if (conn.markClosed(new InterruptedIOException("RpcClient is
> closing"))) {
> conn.close();
> }
> }
> }
> {noformat}
> That worked fine until fix for HBASE-14241 introduced handling for interrupt
> in writer thread:
> {noformat}
> try {
> cts = callsToWrite.take();
> } catch (InterruptedException e) {
> markClosed(new InterruptedIOException());
> }
> {noformat}
> So, if writer thread is running, but connection thread is not started yet,
> interrupt will cause calling of markClosed which will set
> shouldCloseConnection flag for the parent connection. And the next time
> during the handling of non alive connections markClosed will return false and
> close will not be called. As the result connection will not be removed from
> the connections pool and RpcClientImpl.close never finish.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-15957) RpcClientImpl.close never ends in some circumstances
[
https://issues.apache.org/jira/browse/HBASE-15957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-15957:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 1.1.6
1.2.2
1.4.0
1.3.0
2.0.0
Status: Resolved (was: Patch Available)
I've committed this to 1.1+. Thanks Sergey.
> RpcClientImpl.close never ends in some circumstances
>
>
> Key: HBASE-15957
> URL: https://issues.apache.org/jira/browse/HBASE-15957
> Project: HBase
> Issue Type: Bug
> Components: Client, rpc
>Affects Versions: 1.1.2
>Reporter: Sergey Soldatov
>Assignee: Sergey Soldatov
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.2, 1.1.6
>
> Attachments: HBASE-15957-2.patch, HBASE-15957.patch
>
>
> This bug is related to HBASE-14241 and HBASE-13851.
> Fix for HBASE-13851 introduced the check for non alive connections and if
> connection is not alive, it close it:
> {noformat}
> if (!conn.isAlive()) {
> if (connsToClose == null) {
> connsToClose = new HashSet();
> }
> connsToClose.add(conn);
> }
>
> if (connsToClose != null) {
> for (Connection conn : connsToClose) {
> if (conn.markClosed(new InterruptedIOException("RpcClient is
> closing"))) {
> conn.close();
> }
> }
> }
> {noformat}
> That worked fine until fix for HBASE-14241 introduced handling for interrupt
> in writer thread:
> {noformat}
> try {
> cts = callsToWrite.take();
> } catch (InterruptedException e) {
> markClosed(new InterruptedIOException());
> }
> {noformat}
> So, if writer thread is running, but connection thread is not started yet,
> interrupt will cause calling of markClosed which will set
> shouldCloseConnection flag for the parent connection. And the next time
> during the handling of non alive connections markClosed will return false and
> close will not be called. As the result connection will not be removed from
> the connections pool and RpcClientImpl.close never finish.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15957) RpcClientImpl.close never ends in some circumstances
[
https://issues.apache.org/jira/browse/HBASE-15957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Soldatov updated HBASE-15957:
Attachment: HBASE-15957-2.patch
Added test case. It opens rpcClient connections, tries to send request and
close rpcClient immediately. Since we're trying to catch a race condition, I
repeat it 1000 times. usually it takes less than 100 attempts to reproduce it,
but sometimes it may be even 300-400.
[~enis] as for the check, as I already mentioned offline, it's the only place
where we check the result of markClosed. And definitely in this case it's safe
to call close.
> RpcClientImpl.close never ends in some circumstances
>
>
> Key: HBASE-15957
> URL: https://issues.apache.org/jira/browse/HBASE-15957
> Project: HBase
> Issue Type: Bug
> Components: Client, rpc
>Affects Versions: 1.1.2
>Reporter: Sergey Soldatov
>Assignee: Sergey Soldatov
> Attachments: HBASE-15957-2.patch, HBASE-15957.patch
>
>
> This bug is related to HBASE-14241 and HBASE-13851.
> Fix for HBASE-13851 introduced the check for non alive connections and if
> connection is not alive, it close it:
> {noformat}
> if (!conn.isAlive()) {
> if (connsToClose == null) {
> connsToClose = new HashSet();
> }
> connsToClose.add(conn);
> }
>
> if (connsToClose != null) {
> for (Connection conn : connsToClose) {
> if (conn.markClosed(new InterruptedIOException("RpcClient is
> closing"))) {
> conn.close();
> }
> }
> }
> {noformat}
> That worked fine until fix for HBASE-14241 introduced handling for interrupt
> in writer thread:
> {noformat}
> try {
> cts = callsToWrite.take();
> } catch (InterruptedException e) {
> markClosed(new InterruptedIOException());
> }
> {noformat}
> So, if writer thread is running, but connection thread is not started yet,
> interrupt will cause calling of markClosed which will set
> shouldCloseConnection flag for the parent connection. And the next time
> during the handling of non alive connections markClosed will return false and
> close will not be called. As the result connection will not be removed from
> the connections pool and RpcClientImpl.close never finish.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15957) RpcClientImpl.close never ends in some circumstances
[
https://issues.apache.org/jira/browse/HBASE-15957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Soldatov updated HBASE-15957:
Status: Patch Available (was: Open)
> RpcClientImpl.close never ends in some circumstances
>
>
> Key: HBASE-15957
> URL: https://issues.apache.org/jira/browse/HBASE-15957
> Project: HBase
> Issue Type: Bug
> Components: Client, rpc
>Affects Versions: 1.1.2
>Reporter: Sergey Soldatov
>Assignee: Sergey Soldatov
> Attachments: HBASE-15957.patch
>
>
> This bug is related to HBASE-14241 and HBASE-13851.
> Fix for HBASE-13851 introduced the check for non alive connections and if
> connection is not alive, it close it:
> {noformat}
> if (!conn.isAlive()) {
> if (connsToClose == null) {
> connsToClose = new HashSet();
> }
> connsToClose.add(conn);
> }
>
> if (connsToClose != null) {
> for (Connection conn : connsToClose) {
> if (conn.markClosed(new InterruptedIOException("RpcClient is
> closing"))) {
> conn.close();
> }
> }
> }
> {noformat}
> That worked fine until fix for HBASE-14241 introduced handling for interrupt
> in writer thread:
> {noformat}
> try {
> cts = callsToWrite.take();
> } catch (InterruptedException e) {
> markClosed(new InterruptedIOException());
> }
> {noformat}
> So, if writer thread is running, but connection thread is not started yet,
> interrupt will cause calling of markClosed which will set
> shouldCloseConnection flag for the parent connection. And the next time
> during the handling of non alive connections markClosed will return false and
> close will not be called. As the result connection will not be removed from
> the connections pool and RpcClientImpl.close never finish.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15957) RpcClientImpl.close never ends in some circumstances
[
https://issues.apache.org/jira/browse/HBASE-15957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Soldatov updated HBASE-15957:
Attachment: HBASE-15957.patch
The fix is pretty simple. Since the connection is not alive that mean it's not
started or already ended (and all resources were released) we can call
{{close}} without looking on the state shouldCloseConnection.
[~enis], [~te...@apache.org] did I miss something?
> RpcClientImpl.close never ends in some circumstances
>
>
> Key: HBASE-15957
> URL: https://issues.apache.org/jira/browse/HBASE-15957
> Project: HBase
> Issue Type: Bug
> Components: Client, rpc
>Affects Versions: 1.1.2
>Reporter: Sergey Soldatov
>Assignee: Sergey Soldatov
> Attachments: HBASE-15957.patch
>
>
> This bug is related to HBASE-14241 and HBASE-13851.
> Fix for HBASE-13851 introduced the check for non alive connections and if
> connection is not alive, it close it:
> {noformat}
> if (!conn.isAlive()) {
> if (connsToClose == null) {
> connsToClose = new HashSet();
> }
> connsToClose.add(conn);
> }
>
> if (connsToClose != null) {
> for (Connection conn : connsToClose) {
> if (conn.markClosed(new InterruptedIOException("RpcClient is
> closing"))) {
> conn.close();
> }
> }
> }
> {noformat}
> That worked fine until fix for HBASE-14241 introduced handling for interrupt
> in writer thread:
> {noformat}
> try {
> cts = callsToWrite.take();
> } catch (InterruptedException e) {
> markClosed(new InterruptedIOException());
> }
> {noformat}
> So, if writer thread is running, but connection thread is not started yet,
> interrupt will cause calling of markClosed which will set
> shouldCloseConnection flag for the parent connection. And the next time
> during the handling of non alive connections markClosed will return false and
> close will not be called. As the result connection will not be removed from
> the connections pool and RpcClientImpl.close never finish.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
