[jira] [Commented] (HBASE-16752) Upgrading from 1.2 to 1.3 can lead to replication failures due to difference in RPC size limit

2016-10-20 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-16752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593677#comment-15593677
 ] 

Hudson commented on HBASE-16752:


SUCCESS: Integrated in Jenkins build HBase-1.3-JDK7 #48 (See 
[https://builds.apache.org/job/HBase-1.3-JDK7/48/])
HBASE-16752 addendum. Do not retry large request for client versions (garyh: 
rev 3efa3b911294fe0a42440e2aef3865898d91acc0)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/RequestTooBigException.java


> Upgrading from 1.2 to 1.3 can lead to replication failures due to difference 
> in RPC size limit
> --
>
> Key: HBASE-16752
> URL: https://issues.apache.org/jira/browse/HBASE-16752
> Project: HBase
>  Issue Type: Bug
>  Components: Replication, rpc
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-16752.V1.patch, HBASE-16752.V2.patch, 
> HBASE-16752.addendum.patch
>
>
> In HBase 1.2, we don't limit size of a single RPC but in 1.3 we limit it by 
> default to 256 MB.  This means that during upgrade scenarios (or when source 
> is 1.2 peer is already on 1.3), it's possible to encounter a situation where 
> we try to send an rpc with size greater than 256 MB because we never unroll a 
> WALEdit while sending replication traffic.
> RpcServer throws the underlying exception locally, but closes the connection 
> with returning the underlying error to the client, and client only sees a 
> "Broken pipe" error.
> I am not sure what is the proper fix here (or if one is needed) to make sure 
> this does not happen, but we should return the underlying exception to the 
> RpcClient, because without it, it can be difficult to diagnose the problem, 
> especially for someone new to HBase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-16752) Upgrading from 1.2 to 1.3 can lead to replication failures due to difference in RPC size limit

2016-10-20 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-16752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593396#comment-15593396
 ] 

Hudson commented on HBASE-16752:


FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1823 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/1823/])
HBASE-16752 addendum. Do not retry large request for client versions (garyh: 
rev f1691e3d0d0b650d569540f4711f1f0ba2b5701e)
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/RequestTooBigException.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java


> Upgrading from 1.2 to 1.3 can lead to replication failures due to difference 
> in RPC size limit
> --
>
> Key: HBASE-16752
> URL: https://issues.apache.org/jira/browse/HBASE-16752
> Project: HBase
>  Issue Type: Bug
>  Components: Replication, rpc
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-16752.V1.patch, HBASE-16752.V2.patch, 
> HBASE-16752.addendum.patch
>
>
> In HBase 1.2, we don't limit size of a single RPC but in 1.3 we limit it by 
> default to 256 MB.  This means that during upgrade scenarios (or when source 
> is 1.2 peer is already on 1.3), it's possible to encounter a situation where 
> we try to send an rpc with size greater than 256 MB because we never unroll a 
> WALEdit while sending replication traffic.
> RpcServer throws the underlying exception locally, but closes the connection 
> with returning the underlying error to the client, and client only sees a 
> "Broken pipe" error.
> I am not sure what is the proper fix here (or if one is needed) to make sure 
> this does not happen, but we should return the underlying exception to the 
> RpcClient, because without it, it can be difficult to diagnose the problem, 
> especially for someone new to HBase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-16752) Upgrading from 1.2 to 1.3 can lead to replication failures due to difference in RPC size limit

2016-10-20 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-16752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593210#comment-15593210
 ] 

Hudson commented on HBASE-16752:


FAILURE: Integrated in Jenkins build HBase-1.4 #483 (See 
[https://builds.apache.org/job/HBase-1.4/483/])
HBASE-16752 addendum. Do not retry large request for client versions (garyh: 
rev 0117ed9d78820abecf066d742782de1dd49f309e)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/RequestTooBigException.java


> Upgrading from 1.2 to 1.3 can lead to replication failures due to difference 
> in RPC size limit
> --
>
> Key: HBASE-16752
> URL: https://issues.apache.org/jira/browse/HBASE-16752
> Project: HBase
>  Issue Type: Bug
>  Components: Replication, rpc
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-16752.V1.patch, HBASE-16752.V2.patch, 
> HBASE-16752.addendum.patch
>
>
> In HBase 1.2, we don't limit size of a single RPC but in 1.3 we limit it by 
> default to 256 MB.  This means that during upgrade scenarios (or when source 
> is 1.2 peer is already on 1.3), it's possible to encounter a situation where 
> we try to send an rpc with size greater than 256 MB because we never unroll a 
> WALEdit while sending replication traffic.
> RpcServer throws the underlying exception locally, but closes the connection 
> with returning the underlying error to the client, and client only sees a 
> "Broken pipe" error.
> I am not sure what is the proper fix here (or if one is needed) to make sure 
> this does not happen, but we should return the underlying exception to the 
> RpcClient, because without it, it can be difficult to diagnose the problem, 
> especially for someone new to HBase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-16752) Upgrading from 1.2 to 1.3 can lead to replication failures due to difference in RPC size limit

2016-10-20 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-16752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593168#comment-15593168
 ] 

Hudson commented on HBASE-16752:


SUCCESS: Integrated in Jenkins build HBase-1.3-JDK8 #54 (See 
[https://builds.apache.org/job/HBase-1.3-JDK8/54/])
HBASE-16752 addendum. Do not retry large request for client versions (garyh: 
rev 3efa3b911294fe0a42440e2aef3865898d91acc0)
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/RequestTooBigException.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java


> Upgrading from 1.2 to 1.3 can lead to replication failures due to difference 
> in RPC size limit
> --
>
> Key: HBASE-16752
> URL: https://issues.apache.org/jira/browse/HBASE-16752
> Project: HBase
>  Issue Type: Bug
>  Components: Replication, rpc
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-16752.V1.patch, HBASE-16752.V2.patch, 
> HBASE-16752.addendum.patch
>
>
> In HBase 1.2, we don't limit size of a single RPC but in 1.3 we limit it by 
> default to 256 MB.  This means that during upgrade scenarios (or when source 
> is 1.2 peer is already on 1.3), it's possible to encounter a situation where 
> we try to send an rpc with size greater than 256 MB because we never unroll a 
> WALEdit while sending replication traffic.
> RpcServer throws the underlying exception locally, but closes the connection 
> with returning the underlying error to the client, and client only sees a 
> "Broken pipe" error.
> I am not sure what is the proper fix here (or if one is needed) to make sure 
> this does not happen, but we should return the underlying exception to the 
> RpcClient, because without it, it can be difficult to diagnose the problem, 
> especially for someone new to HBase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-16752) Upgrading from 1.2 to 1.3 can lead to replication failures due to difference in RPC size limit

2016-10-20 Thread Gary Helmling (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-16752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592477#comment-15592477
 ] 

Gary Helmling commented on HBASE-16752:
---

Committed the addendum to branch-1.3+.

> Upgrading from 1.2 to 1.3 can lead to replication failures due to difference 
> in RPC size limit
> --
>
> Key: HBASE-16752
> URL: https://issues.apache.org/jira/browse/HBASE-16752
> Project: HBase
>  Issue Type: Bug
>  Components: Replication, rpc
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-16752.V1.patch, HBASE-16752.V2.patch, 
> HBASE-16752.addendum.patch
>
>
> In HBase 1.2, we don't limit size of a single RPC but in 1.3 we limit it by 
> default to 256 MB.  This means that during upgrade scenarios (or when source 
> is 1.2 peer is already on 1.3), it's possible to encounter a situation where 
> we try to send an rpc with size greater than 256 MB because we never unroll a 
> WALEdit while sending replication traffic.
> RpcServer throws the underlying exception locally, but closes the connection 
> with returning the underlying error to the client, and client only sees a 
> "Broken pipe" error.
> I am not sure what is the proper fix here (or if one is needed) to make sure 
> this does not happen, but we should return the underlying exception to the 
> RpcClient, because without it, it can be difficult to diagnose the problem, 
> especially for someone new to HBase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-16752) Upgrading from 1.2 to 1.3 can lead to replication failures due to difference in RPC size limit

2016-10-20 Thread Mikhail Antonov (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-16752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592443#comment-15592443
 ] 

Mikhail Antonov commented on HBASE-16752:
-

+1

> Upgrading from 1.2 to 1.3 can lead to replication failures due to difference 
> in RPC size limit
> --
>
> Key: HBASE-16752
> URL: https://issues.apache.org/jira/browse/HBASE-16752
> Project: HBase
>  Issue Type: Bug
>  Components: Replication, rpc
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-16752.V1.patch, HBASE-16752.V2.patch, 
> HBASE-16752.addendum.patch
>
>
> In HBase 1.2, we don't limit size of a single RPC but in 1.3 we limit it by 
> default to 256 MB.  This means that during upgrade scenarios (or when source 
> is 1.2 peer is already on 1.3), it's possible to encounter a situation where 
> we try to send an rpc with size greater than 256 MB because we never unroll a 
> WALEdit while sending replication traffic.
> RpcServer throws the underlying exception locally, but closes the connection 
> with returning the underlying error to the client, and client only sees a 
> "Broken pipe" error.
> I am not sure what is the proper fix here (or if one is needed) to make sure 
> this does not happen, but we should return the underlying exception to the 
> RpcClient, because without it, it can be difficult to diagnose the problem, 
> especially for someone new to HBase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-16752) Upgrading from 1.2 to 1.3 can lead to replication failures due to difference in RPC size limit

2016-10-20 Thread Gary Helmling (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-16752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15592402#comment-15592402
 ] 

Gary Helmling commented on HBASE-16752:
---

+1 on the addendum.  Should avoid the retrying we discussed on 1.2 clients.

> Upgrading from 1.2 to 1.3 can lead to replication failures due to difference 
> in RPC size limit
> --
>
> Key: HBASE-16752
> URL: https://issues.apache.org/jira/browse/HBASE-16752
> Project: HBase
>  Issue Type: Bug
>  Components: Replication, rpc
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-16752.V1.patch, HBASE-16752.V2.patch, 
> HBASE-16752.addendum.patch
>
>
> In HBase 1.2, we don't limit size of a single RPC but in 1.3 we limit it by 
> default to 256 MB.  This means that during upgrade scenarios (or when source 
> is 1.2 peer is already on 1.3), it's possible to encounter a situation where 
> we try to send an rpc with size greater than 256 MB because we never unroll a 
> WALEdit while sending replication traffic.
> RpcServer throws the underlying exception locally, but closes the connection 
> with returning the underlying error to the client, and client only sees a 
> "Broken pipe" error.
> I am not sure what is the proper fix here (or if one is needed) to make sure 
> this does not happen, but we should return the underlying exception to the 
> RpcClient, because without it, it can be difficult to diagnose the problem, 
> especially for someone new to HBase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-16752) Upgrading from 1.2 to 1.3 can lead to replication failures due to difference in RPC size limit

2016-10-20 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-16752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15591056#comment-15591056
 ] 

Hudson commented on HBASE-16752:


SUCCESS: Integrated in Jenkins build HBase-1.3-JDK8 #53 (See 
[https://builds.apache.org/job/HBase-1.3-JDK8/53/])
HBASE-16752 Return error back to rpc client on exceeding rpc request (garyh: 
rev 29a60d0eeb1ac0856d727754b8fa52a9d137bc9d)
* (add) 
hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/RequestTooBigException.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/ClientExceptionsUtil.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/AbstractTestIPC.java


> Upgrading from 1.2 to 1.3 can lead to replication failures due to difference 
> in RPC size limit
> --
>
> Key: HBASE-16752
> URL: https://issues.apache.org/jira/browse/HBASE-16752
> Project: HBase
>  Issue Type: Bug
>  Components: Replication, rpc
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-16752.V1.patch, HBASE-16752.V2.patch, 
> HBASE-16752.addendum.patch
>
>
> In HBase 1.2, we don't limit size of a single RPC but in 1.3 we limit it by 
> default to 256 MB.  This means that during upgrade scenarios (or when source 
> is 1.2 peer is already on 1.3), it's possible to encounter a situation where 
> we try to send an rpc with size greater than 256 MB because we never unroll a 
> WALEdit while sending replication traffic.
> RpcServer throws the underlying exception locally, but closes the connection 
> with returning the underlying error to the client, and client only sees a 
> "Broken pipe" error.
> I am not sure what is the proper fix here (or if one is needed) to make sure 
> this does not happen, but we should return the underlying exception to the 
> RpcClient, because without it, it can be difficult to diagnose the problem, 
> especially for someone new to HBase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-16752) Upgrading from 1.2 to 1.3 can lead to replication failures due to difference in RPC size limit

2016-10-19 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-16752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590644#comment-15590644
 ] 

Hudson commented on HBASE-16752:


FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1818 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/1818/])
HBASE-16752 Return error back to rpc client on exceeding rpc request (garyh: 
rev ee1123b0696e290fe406e3b7de30052f6dd06ba7)
* (add) 
hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/RequestTooBigException.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/AbstractTestIPC.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/ClientExceptionsUtil.java


> Upgrading from 1.2 to 1.3 can lead to replication failures due to difference 
> in RPC size limit
> --
>
> Key: HBASE-16752
> URL: https://issues.apache.org/jira/browse/HBASE-16752
> Project: HBase
>  Issue Type: Bug
>  Components: Replication, rpc
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-16752.V1.patch, HBASE-16752.V2.patch, 
> HBASE-16752.addendum.patch
>
>
> In HBase 1.2, we don't limit size of a single RPC but in 1.3 we limit it by 
> default to 256 MB.  This means that during upgrade scenarios (or when source 
> is 1.2 peer is already on 1.3), it's possible to encounter a situation where 
> we try to send an rpc with size greater than 256 MB because we never unroll a 
> WALEdit while sending replication traffic.
> RpcServer throws the underlying exception locally, but closes the connection 
> with returning the underlying error to the client, and client only sees a 
> "Broken pipe" error.
> I am not sure what is the proper fix here (or if one is needed) to make sure 
> this does not happen, but we should return the underlying exception to the 
> RpcClient, because without it, it can be difficult to diagnose the problem, 
> especially for someone new to HBase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-16752) Upgrading from 1.2 to 1.3 can lead to replication failures due to difference in RPC size limit

2016-10-19 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-16752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15590572#comment-15590572
 ] 

Hudson commented on HBASE-16752:


FAILURE: Integrated in Jenkins build HBase-1.4 #479 (See 
[https://builds.apache.org/job/HBase-1.4/479/])
HBASE-16752 Return error back to rpc client on exceeding rpc request (garyh: 
rev 33e89fa9cfd4036fe972ff892df957603ae25bbf)
* (add) 
hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/RequestTooBigException.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/ClientExceptionsUtil.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/AbstractTestIPC.java


> Upgrading from 1.2 to 1.3 can lead to replication failures due to difference 
> in RPC size limit
> --
>
> Key: HBASE-16752
> URL: https://issues.apache.org/jira/browse/HBASE-16752
> Project: HBase
>  Issue Type: Bug
>  Components: Replication, rpc
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-16752.V1.patch, HBASE-16752.V2.patch, 
> HBASE-16752.addendum.patch
>
>
> In HBase 1.2, we don't limit size of a single RPC but in 1.3 we limit it by 
> default to 256 MB.  This means that during upgrade scenarios (or when source 
> is 1.2 peer is already on 1.3), it's possible to encounter a situation where 
> we try to send an rpc with size greater than 256 MB because we never unroll a 
> WALEdit while sending replication traffic.
> RpcServer throws the underlying exception locally, but closes the connection 
> with returning the underlying error to the client, and client only sees a 
> "Broken pipe" error.
> I am not sure what is the proper fix here (or if one is needed) to make sure 
> this does not happen, but we should return the underlying exception to the 
> RpcClient, because without it, it can be difficult to diagnose the problem, 
> especially for someone new to HBase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-16752) Upgrading from 1.2 to 1.3 can lead to replication failures due to difference in RPC size limit

2016-10-14 Thread Gary Helmling (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-16752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15575676#comment-15575676
 ] 

Gary Helmling commented on HBASE-16752:
---

+1 on the v2 patch.  I'll commit later this morning unless there are any 
objections.

> Upgrading from 1.2 to 1.3 can lead to replication failures due to difference 
> in RPC size limit
> --
>
> Key: HBASE-16752
> URL: https://issues.apache.org/jira/browse/HBASE-16752
> Project: HBase
>  Issue Type: Bug
>  Components: Replication, rpc
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
> Attachments: HBASE-16752.V1.patch, HBASE-16752.V2.patch
>
>
> In HBase 1.2, we don't limit size of a single RPC but in 1.3 we limit it by 
> default to 256 MB.  This means that during upgrade scenarios (or when source 
> is 1.2 peer is already on 1.3), it's possible to encounter a situation where 
> we try to send an rpc with size greater than 256 MB because we never unroll a 
> WALEdit while sending replication traffic.
> RpcServer throws the underlying exception locally, but closes the connection 
> with returning the underlying error to the client, and client only sees a 
> "Broken pipe" error.
> I am not sure what is the proper fix here (or if one is needed) to make sure 
> this does not happen, but we should return the underlying exception to the 
> RpcClient, because without it, it can be difficult to diagnose the problem, 
> especially for someone new to HBase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-16752) Upgrading from 1.2 to 1.3 can lead to replication failures due to difference in RPC size limit

2016-10-12 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-16752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570828#comment-15570828
 ] 

Hadoop QA commented on HBASE-16752:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
6s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
28s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
3s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
26m 43s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 53s 
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 152m 35s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
24s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 196m 35s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestHRegion |
| Timed out junit tests | org.apache.hadoop.hbase.client.TestReplicasClient |
|   | org.apache.hadoop.hbase.client.TestMetaWithReplicas |
|   | org.apache.hadoop.hbase.client.TestEnableTable |
|   | org.apache.hadoop.hbase.client.TestMobRestoreSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12833011/HBASE-16752.V2.patch |
| JIRA Issue | HBASE-16752 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 2a64ca92b9dd 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 92ef234 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit | 

[jira] [Commented] (HBASE-16752) Upgrading from 1.2 to 1.3 can lead to replication failures due to difference in RPC size limit

2016-10-12 Thread Ashu Pachauri (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-16752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570394#comment-15570394
 ] 

Ashu Pachauri commented on HBASE-16752:
---

[~ghelmling] Yeah you raise a valid point, we need to address the remaining 
part of the request on the channel in some way. Reading the garbage off the 
network just to discard the data sound like overkill. Even now we close the 
connection; with this change we'll have a better feedback to the client even if 
we do close the connection. So, I'll just modify the code to close the 
connection.

> Upgrading from 1.2 to 1.3 can lead to replication failures due to difference 
> in RPC size limit
> --
>
> Key: HBASE-16752
> URL: https://issues.apache.org/jira/browse/HBASE-16752
> Project: HBase
>  Issue Type: Bug
>  Components: Replication, rpc
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
> Attachments: HBASE-16752.V1.patch
>
>
> In HBase 1.2, we don't limit size of a single RPC but in 1.3 we limit it by 
> default to 256 MB.  This means that during upgrade scenarios (or when source 
> is 1.2 peer is already on 1.3), it's possible to encounter a situation where 
> we try to send an rpc with size greater than 256 MB because we never unroll a 
> WALEdit while sending replication traffic.
> RpcServer throws the underlying exception locally, but closes the connection 
> with returning the underlying error to the client, and client only sees a 
> "Broken pipe" error.
> I am not sure what is the proper fix here (or if one is needed) to make sure 
> this does not happen, but we should return the underlying exception to the 
> RpcClient, because without it, it can be difficult to diagnose the problem, 
> especially for someone new to HBase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-16752) Upgrading from 1.2 to 1.3 can lead to replication failures due to difference in RPC size limit

2016-10-12 Thread Gary Helmling (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-16752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570353#comment-15570353
 ] 

Gary Helmling commented on HBASE-16752:
---

Please add javadoc for RequestTooBigException.

It occurs to me that we also need to do something with the remaining request 
data (the body) that is sitting in the socket channel.  Either we'll need to 
read it off into something like a garbage buffer and throw it away, or else 
maybe we should actually close the connection.  If we do close the connection, 
then does this change still buy us something?

> Upgrading from 1.2 to 1.3 can lead to replication failures due to difference 
> in RPC size limit
> --
>
> Key: HBASE-16752
> URL: https://issues.apache.org/jira/browse/HBASE-16752
> Project: HBase
>  Issue Type: Bug
>  Components: Replication, rpc
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
> Attachments: HBASE-16752.V1.patch
>
>
> In HBase 1.2, we don't limit size of a single RPC but in 1.3 we limit it by 
> default to 256 MB.  This means that during upgrade scenarios (or when source 
> is 1.2 peer is already on 1.3), it's possible to encounter a situation where 
> we try to send an rpc with size greater than 256 MB because we never unroll a 
> WALEdit while sending replication traffic.
> RpcServer throws the underlying exception locally, but closes the connection 
> with returning the underlying error to the client, and client only sees a 
> "Broken pipe" error.
> I am not sure what is the proper fix here (or if one is needed) to make sure 
> this does not happen, but we should return the underlying exception to the 
> RpcClient, because without it, it can be difficult to diagnose the problem, 
> especially for someone new to HBase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-16752) Upgrading from 1.2 to 1.3 can lead to replication failures due to difference in RPC size limit

2016-10-12 Thread Ashu Pachauri (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-16752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15570299#comment-15570299
 ] 

Ashu Pachauri commented on HBASE-16752:
---

Ran the timed out test locally and also had a look at the test code too; they 
seem unrelated to the patch.

> Upgrading from 1.2 to 1.3 can lead to replication failures due to difference 
> in RPC size limit
> --
>
> Key: HBASE-16752
> URL: https://issues.apache.org/jira/browse/HBASE-16752
> Project: HBase
>  Issue Type: Bug
>  Components: Replication, rpc
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
> Attachments: HBASE-16752.V1.patch
>
>
> In HBase 1.2, we don't limit size of a single RPC but in 1.3 we limit it by 
> default to 256 MB.  This means that during upgrade scenarios (or when source 
> is 1.2 peer is already on 1.3), it's possible to encounter a situation where 
> we try to send an rpc with size greater than 256 MB because we never unroll a 
> WALEdit while sending replication traffic.
> RpcServer throws the underlying exception locally, but closes the connection 
> with returning the underlying error to the client, and client only sees a 
> "Broken pipe" error.
> I am not sure what is the proper fix here (or if one is needed) to make sure 
> this does not happen, but we should return the underlying exception to the 
> RpcClient, because without it, it can be difficult to diagnose the problem, 
> especially for someone new to HBase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-16752) Upgrading from 1.2 to 1.3 can lead to replication failures due to difference in RPC size limit

2016-10-12 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-16752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15568491#comment-15568491
 ] 

Hadoop QA commented on HBASE-16752:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
30s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
31s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
8s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 58s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
30m 44s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 3s 
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 95m 43s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
23s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 145m 22s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | 
org.apache.hadoop.hbase.master.procedure.TestDisableTableProcedure |
|   | org.apache.hadoop.hbase.master.procedure.TestEnableTableProcedure |
|   | org.apache.hadoop.hbase.master.procedure.TestCloneSnapshotProcedure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12832852/HBASE-16752.V1.patch |
| JIRA Issue | HBASE-16752 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 269c1715c4b4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
 |
| git revision | master / b76b751 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/3958/artifact/patchprocess/patch-unit-hbase-server.txt
 |
| unit test logs |  

[jira] [Commented] (HBASE-16752) Upgrading from 1.2 to 1.3 can lead to replication failures due to difference in RPC size limit

2016-10-12 Thread Ted Yu (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-16752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15568387#comment-15568387
 ] 

Ted Yu commented on HBASE-16752:


lgtm

> Upgrading from 1.2 to 1.3 can lead to replication failures due to difference 
> in RPC size limit
> --
>
> Key: HBASE-16752
> URL: https://issues.apache.org/jira/browse/HBASE-16752
> Project: HBase
>  Issue Type: Bug
>  Components: Replication, rpc
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
> Attachments: HBASE-16752.V1.patch
>
>
> In HBase 1.2, we don't limit size of a single RPC but in 1.3 we limit it by 
> default to 256 MB.  This means that during upgrade scenarios (or when source 
> is 1.2 peer is already on 1.3), it's possible to encounter a situation where 
> we try to send an rpc with size greater than 256 MB because we never unroll a 
> WALEdit while sending replication traffic.
> RpcServer throws the underlying exception locally, but closes the connection 
> with returning the underlying error to the client, and client only sees a 
> "Broken pipe" error.
> I am not sure what is the proper fix here (or if one is needed) to make sure 
> this does not happen, but we should return the underlying exception to the 
> RpcClient, because without it, it can be difficult to diagnose the problem, 
> especially for someone new to HBase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-16752) Upgrading from 1.2 to 1.3 can lead to replication failures due to difference in RPC size limit

2016-10-12 Thread Ashu Pachauri (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-16752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15568079#comment-15568079
 ] 

Ashu Pachauri commented on HBASE-16752:
---

[~anoop.hbase] The way it's implemented right now is that there is no feedback 
to the client (there is no RequestTooBigException), and the connection is 
simply dropped. This has two side effects:
1. Client only sees connection drops without any reason, which may be hard to 
debug for people not very familiar with HBase codebase. Even if I do try to 
return a RequestTooBigException (a new exception), client simply discards this 
because server sends an incorrect call ID that it's not expecting (Server has 
an incorrect call ID because it does not want to read the whole request  as 
it's too large).
2. Client will retry the same rpc again and again and keep failing (until 
retries are exhausted or forever in case of replication).

The implication on replication is that if the destination peer is upgraded to 
1.3 (where servers enforce this limit), replication can fail because source can 
take large RPCs while peer cannot. A temporary fix here is that the HBase admin 
override this rpc size limit on the peer. We could also change the default on 
HBase 1.3 (currently 256 MB per call, 1 GB total call queue size ) to match max 
call queue size on HBase 1.2 (1 GB), but then it defeats the purpose of this 
config.

That said, I do not plan to fix the replication problem, just to give better 
feedback to the client so that this can be easily diagnosed, temporary fix can 
be applied and clients can be modified to respect the rpc size limit.


> Upgrading from 1.2 to 1.3 can lead to replication failures due to difference 
> in RPC size limit
> --
>
> Key: HBASE-16752
> URL: https://issues.apache.org/jira/browse/HBASE-16752
> Project: HBase
>  Issue Type: Bug
>  Components: Replication, rpc
>Affects Versions: 1.3.0
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
>
> In HBase 1.2, we don't limit size of a single RPC but in 1.3 we limit it by 
> default to 256 MB.  This means that during upgrade scenarios (or when source 
> is 1.2 peer is already on 1.3), it's possible to encounter a situation where 
> we try to send an rpc with size greater than 256 MB because we never unroll a 
> WALEdit while sending replication traffic.
> RpcServer throws the underlying exception locally, but closes the connection 
> with returning the underlying error to the client, and client only sees a 
> "Broken pipe" error.
> I am not sure what is the proper fix here (or if one is needed) to make sure 
> this does not happen, but we should return the underlying exception to the 
> RpcClient, because without it, it can be difficult to diagnose the problem, 
> especially for someone new to HBase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-16752) Upgrading from 1.2 to 1.3 can lead to replication failures due to difference in RPC size limit

2016-10-11 Thread Anoop Sam John (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-16752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15567588#comment-15567588
 ] 

Anoop Sam John commented on HBASE-16752:


Right now what client is doing when it is seeing RequestTooBigException ? I 
dont mean in case of replication. (In normal user facing client).  Trying to 
get what the callId is used for.  What is async RPC client?

So the idea on the replication part is that unless the src is also upgraded, 
the replication will fail (?) Do u plan to have some special mechanism at src 
end to temp halt it?

> Upgrading from 1.2 to 1.3 can lead to replication failures due to difference 
> in RPC size limit
> --
>
> Key: HBASE-16752
> URL: https://issues.apache.org/jira/browse/HBASE-16752
> Project: HBase
>  Issue Type: Bug
>  Components: Replication, rpc
>Affects Versions: 1.3.0
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
>
> In HBase 1.2, we don't limit size of a single RPC but in 1.3 we limit it by 
> default to 256 MB.  This means that during upgrade scenarios (or when source 
> is 1.2 peer is already on 1.3), it's possible to encounter a situation where 
> we try to send an rpc with size greater than 256 MB because we never unroll a 
> WALEdit while sending replication traffic.
> RpcServer throws the underlying exception locally, but closes the connection 
> with returning the underlying error to the client, and client only sees a 
> "Broken pipe" error.
> I am not sure what is the proper fix here (or if one is needed) to make sure 
> this does not happen, but we should return the underlying exception to the 
> RpcClient, because without it, it can be difficult to diagnose the problem, 
> especially for someone new to HBase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-16752) Upgrading from 1.2 to 1.3 can lead to replication failures due to difference in RPC size limit

2016-10-10 Thread Ashu Pachauri (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-16752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563323#comment-15563323
 ] 

Ashu Pachauri commented on HBASE-16752:
---

The current plan I have is to return a RequestTooBigException to the client. 
The behavior of the server currently when it encounters a giant request is to 
locally log the message and drop the connection after reading the data length, 
without having to read the entire rpc off the wire.
I plan to maintain the current behavior. However, to return the response I need 
to read the request off the wire to get the RPC Call ID. I can think of two 
ways to tackle this:
1. Read only first few bytes of the rpc to try to decode the request header 
from them. This makes assumptions on request header format and size, and will 
need to be kept in sync with the changes to the request format.
2. Instead of returning response with correct Call ID, return with a special 
call ID (negative) for responses that are to be following by dropping the 
connection and modify the rpc connection implementation to handle there 
responses in a special way. This breaks the fundamental client server model, 
because server is sending special instructions to the client which the client 
did not ask for.
Personally, I like the second approach because it just sets a special contract 
and does not require any funky stuff to decode the header, but I am open to 
suggestions.

> Upgrading from 1.2 to 1.3 can lead to replication failures due to difference 
> in RPC size limit
> --
>
> Key: HBASE-16752
> URL: https://issues.apache.org/jira/browse/HBASE-16752
> Project: HBase
>  Issue Type: Bug
>  Components: Replication, rpc
>Affects Versions: 1.3.0
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
>
> In HBase 1.2, we don't limit size of a single RPC but in 1.3 we limit it by 
> default to 256 MB.  This means that during upgrade scenarios (or when source 
> is 1.2 peer is already on 1.3), it's possible to encounter a situation where 
> we try to send an rpc with size greater than 256 MB because we never unroll a 
> WALEdit while sending replication traffic.
> RpcServer throws the underlying exception locally, but closes the connection 
> with returning the underlying error to the client, and client only sees a 
> "Broken pipe" error.
> I am not sure what is the proper fix here (or if one is needed) to make sure 
> this does not happen, but we should return the underlying exception to the 
> RpcClient, because without it, it can be difficult to diagnose the problem, 
> especially for someone new to HBase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-16752) Upgrading from 1.2 to 1.3 can lead to replication failures due to difference in RPC size limit

2016-10-05 Thread Ashu Pachauri (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-16752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15547888#comment-15547888
 ] 

Ashu Pachauri commented on HBASE-16752:
---

Picking it up

> Upgrading from 1.2 to 1.3 can lead to replication failures due to difference 
> in RPC size limit
> --
>
> Key: HBASE-16752
> URL: https://issues.apache.org/jira/browse/HBASE-16752
> Project: HBase
>  Issue Type: Bug
>  Components: Replication, rpc
>Affects Versions: 1.3.0
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
>
> In HBase 1.2, we don't limit size of a single RPC but in 1.3 we limit it by 
> default to 256 MB.  This means that during upgrade scenarios (or when source 
> is 1.2 peer is already on 1.3), it's possible to encounter a situation where 
> we try to send an rpc with size greater than 256 MB because we never unroll a 
> WALEdit while sending replication traffic.
> RpcServer throws the underlying exception locally, but closes the connection 
> with returning the underlying error to the client, and client only sees a 
> "Broken pipe" error.
> I am not sure what is the proper fix here (or if one is needed) to make sure 
> this does not happen, but we should return the underlying exception to the 
> RpcClient, because without it, it can be difficult to diagnose the problem, 
> especially for someone new to HBase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HBASE-16752) Upgrading from 1.2 to 1.3 can lead to replication failures due to difference in RPC size limit

2016-10-04 Thread Mikhail Antonov (JIRA)

[ 
https://issues.apache.org/jira/browse/HBASE-16752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15546684#comment-15546684
 ] 

Mikhail Antonov commented on HBASE-16752:
-

I would say it should be fine to return meaningful debuggable exception to the 
client and just clearly state in the release notes that in the unlikely 
scenario that you encounter this problem the remediation would be to 
temporarily bump this limit to something bigger.

> Upgrading from 1.2 to 1.3 can lead to replication failures due to difference 
> in RPC size limit
> --
>
> Key: HBASE-16752
> URL: https://issues.apache.org/jira/browse/HBASE-16752
> Project: HBase
>  Issue Type: Bug
>  Components: Replication, rpc
>Affects Versions: 1.3.0
>Reporter: Ashu Pachauri
>
> In HBase 1.2, we don't limit size of a single RPC but in 1.3 we limit it by 
> default to 256 MB.  This means that during upgrade scenarios (or when source 
> is 1.2 peer is already on 1.3), it's possible to encounter a situation where 
> we try to send an rpc with size greater than 256 MB because we never unroll a 
> WALEdit while sending replication traffic.
> RpcServer throws the underlying exception locally, but closes the connection 
> with returning the underlying error to the client, and client only sees a 
> "Broken pipe" error.
> I am not sure what is the proper fix here (or if one is needed) to make sure 
> this does not happen, but we should return the underlying exception to the 
> RpcClient, because without it, it can be difficult to diagnose the problem, 
> especially for someone new to HBase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)