[jira] [Commented] (HBASE-16388) Prevent client threads being blocked by only one slow region server
[ https://issues.apache.org/jira/browse/HBASE-16388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416563#comment-15416563 ] stack commented on HBASE-16388: --- Thank you [~yangzhe1991] Start simple I'd say. Will be hard enough to test. Would be sweet if we could avoid another layer of accounting in the client. There is a bunch already going on in AsyncProcess by region and by server. Can this be leveraged? I like the way you characterize how queue-per-client is at other end of the spectrum from what this patch is trying to do. Looking forward to it Phil Yang. > Prevent client threads being blocked by only one slow region server > --- > > Key: HBASE-16388 > URL: https://issues.apache.org/jira/browse/HBASE-16388 > Project: HBase > Issue Type: New Feature >Reporter: Phil Yang >Assignee: Phil Yang > > It is a general use case for HBase's users that they have several > threads/handlers in their service, and each handler has its own Table/HTable > instance. Generally users think each handler is independent and won't > interact each other. > However, in an extreme case, if a region server is very slow, every requests > to this RS will timeout, handlers of users' service may be occupied by the > long-waiting requests even requests belong to other RS will also be timeout. > For example: > If we have 100 handlers in a client service(timeout is 1000ms) and HBase has > 10 region servers whose average response time is 50ms. If no region server is > slow, we can handle 2000 requests per second. > Now this service's QPS is 1000. If there is one region server very slow and > all requests to it will be timeout. Users hope that only 10% requests failed, > and 90% requests' response time is still 50ms, because only 10% requests are > located to the slow RS. However, each second we have 100 long-waiting > requests which exactly occupies all 100 handles. So all handlers is blocked, > the availability of this service is almost zero. > To prevent this case, we can limit the max concurrent requests to one RS in > process-level. Requests exceeding the limit will throws > ServerBusyException(extends DoNotRetryIOE) immediately to users. In the above > case, if we set this limit to 20, only 20 handlers will be occupied and other > 80 handlers can still handle requests to other RS. The availability of this > service is 90% as expected. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16267) Remove commons-httpclient dependency from hbase-rest module
[
https://issues.apache.org/jira/browse/HBASE-16267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416560#comment-15416560
]
Hadoop QA commented on HBASE-16267:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 2s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s {color}
| {color:red} HBASE-16267 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12823172/16267.v15.txt |
| JIRA Issue | HBASE-16267 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3061/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Remove commons-httpclient dependency from hbase-rest module
> ---
>
> Key: HBASE-16267
> URL: https://issues.apache.org/jira/browse/HBASE-16267
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 16267.v10.txt, 16267.v11.txt, 16267.v12.txt,
> 16267.v13.txt, 16267.v14.txt, 16267.v15.txt, 16267.v15.txt, 16267.v2.txt,
> 16267.v4.txt, 16267.v6.txt, 16267.v8.txt, 16267.v9.txt
>
>
> hbase-rest module still has imports from org.apache.commons.httpclient .
> There is more work to be done after HBASE-15767 was integrated.
> In master branch, there seems to be transitive dependency which allows the
> code to compile:
> {code}
> [INFO] +- org.apache.hadoop:hadoop-common:jar:2.7.1:compile
> [INFO] | +- org.apache.hadoop:hadoop-annotations:jar:2.7.1:compile
> [INFO] | +- commons-cli:commons-cli:jar:1.2:compile
> [INFO] | +- org.apache.commons:commons-math3:jar:3.1.1:compile
> [INFO] | +- xmlenc:xmlenc:jar:0.52:compile
> [INFO] | +- commons-httpclient:commons-httpclient:jar:3.1:compile
> {code}
> HADOOP-12767
> to move the uses of httpclient HADOOP-10105
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-5262 :
> http/conn/ssl/SSLConnectionSocketFactory.java in Apache HttpComponents
> HttpClient before 4.3.6 ignores the http.socket.timeout configuration setting
> during an SSL handshake, which allows remote attackers to cause a denial of
> service (HTTPS call hang) via unspecified vectors.
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-6153
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-5783
> Apache Commons HttpClient 3.x, as used in Amazon Flexible Payments
> Service (FPS) merchant Java SDK and other products, does not verify that the
> server hostname matches a domain name in the subject's Common Name (CN) or
> subjectAltName field of the X.509 certificate, which allows man-in-the-middle
> attackers to spoof SSL servers via an arbitrary valid certificate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16267) Remove commons-httpclient dependency from hbase-rest module
[
https://issues.apache.org/jira/browse/HBASE-16267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-16267:
--
Attachment: 16267.v15.txt
Retry to see if tests are flakey and how long build takes.
> Remove commons-httpclient dependency from hbase-rest module
> ---
>
> Key: HBASE-16267
> URL: https://issues.apache.org/jira/browse/HBASE-16267
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 16267.v10.txt, 16267.v11.txt, 16267.v12.txt,
> 16267.v13.txt, 16267.v14.txt, 16267.v15.txt, 16267.v15.txt, 16267.v2.txt,
> 16267.v4.txt, 16267.v6.txt, 16267.v8.txt, 16267.v9.txt
>
>
> hbase-rest module still has imports from org.apache.commons.httpclient .
> There is more work to be done after HBASE-15767 was integrated.
> In master branch, there seems to be transitive dependency which allows the
> code to compile:
> {code}
> [INFO] +- org.apache.hadoop:hadoop-common:jar:2.7.1:compile
> [INFO] | +- org.apache.hadoop:hadoop-annotations:jar:2.7.1:compile
> [INFO] | +- commons-cli:commons-cli:jar:1.2:compile
> [INFO] | +- org.apache.commons:commons-math3:jar:3.1.1:compile
> [INFO] | +- xmlenc:xmlenc:jar:0.52:compile
> [INFO] | +- commons-httpclient:commons-httpclient:jar:3.1:compile
> {code}
> HADOOP-12767
> to move the uses of httpclient HADOOP-10105
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-5262 :
> http/conn/ssl/SSLConnectionSocketFactory.java in Apache HttpComponents
> HttpClient before 4.3.6 ignores the http.socket.timeout configuration setting
> during an SSL handshake, which allows remote attackers to cause a denial of
> service (HTTPS call hang) via unspecified vectors.
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-6153
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-5783
> Apache Commons HttpClient 3.x, as used in Amazon Flexible Payments
> Service (FPS) merchant Java SDK and other products, does not verify that the
> server hostname matches a domain name in the subject's Common Name (CN) or
> subjectAltName field of the X.509 certificate, which allows man-in-the-middle
> attackers to spoof SSL servers via an arbitrary valid certificate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16267) Remove commons-httpclient dependency from hbase-rest module
[
https://issues.apache.org/jira/browse/HBASE-16267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416557#comment-15416557
]
stack commented on HBASE-16267:
---
I took a look. Build times seem erratic:
https://builds.apache.org/view/H-L/view/HBase/job/PreCommit-HBASE-Build/buildTimeTrend
There was a 6 hour build in the mix. I am wary after HBASE-16308 went in. Will
keep an eye on it.
> Remove commons-httpclient dependency from hbase-rest module
> ---
>
> Key: HBASE-16267
> URL: https://issues.apache.org/jira/browse/HBASE-16267
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 16267.v10.txt, 16267.v11.txt, 16267.v12.txt,
> 16267.v13.txt, 16267.v14.txt, 16267.v15.txt, 16267.v2.txt, 16267.v4.txt,
> 16267.v6.txt, 16267.v8.txt, 16267.v9.txt
>
>
> hbase-rest module still has imports from org.apache.commons.httpclient .
> There is more work to be done after HBASE-15767 was integrated.
> In master branch, there seems to be transitive dependency which allows the
> code to compile:
> {code}
> [INFO] +- org.apache.hadoop:hadoop-common:jar:2.7.1:compile
> [INFO] | +- org.apache.hadoop:hadoop-annotations:jar:2.7.1:compile
> [INFO] | +- commons-cli:commons-cli:jar:1.2:compile
> [INFO] | +- org.apache.commons:commons-math3:jar:3.1.1:compile
> [INFO] | +- xmlenc:xmlenc:jar:0.52:compile
> [INFO] | +- commons-httpclient:commons-httpclient:jar:3.1:compile
> {code}
> HADOOP-12767
> to move the uses of httpclient HADOOP-10105
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-5262 :
> http/conn/ssl/SSLConnectionSocketFactory.java in Apache HttpComponents
> HttpClient before 4.3.6 ignores the http.socket.timeout configuration setting
> during an SSL handshake, which allows remote attackers to cause a denial of
> service (HTTPS call hang) via unspecified vectors.
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-6153
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-5783
> Apache Commons HttpClient 3.x, as used in Amazon Flexible Payments
> Service (FPS) merchant Java SDK and other products, does not verify that the
> server hostname matches a domain name in the subject's Common Name (CN) or
> subjectAltName field of the X.509 certificate, which allows man-in-the-middle
> attackers to spoof SSL servers via an arbitrary valid certificate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14921) Memory optimizations
[ https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416542#comment-15416542 ] ramkrishna.s.vasudevan commented on HBASE-14921: [~anastas], [~eshcar] Any updates/feedback here? > Memory optimizations > > > Key: HBASE-14921 > URL: https://issues.apache.org/jira/browse/HBASE-14921 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Eshcar Hillel >Assignee: Anastasia Braginsky > Attachments: CellBlocksSegmentInMemStore.pdf, > CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, > HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, > HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, > HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, > IntroductiontoNewFlatandCompactMemStore.pdf, MemstoreItrCountissue.patch, > NewCompactingMemStoreFlow.pptx > > > Memory optimizations including compressed format representation and offheap > allocations -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16394) What cause "Compaction is trying to add a bad range",and Should stop the regionserver?
[ https://issues.apache.org/jira/browse/HBASE-16394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416541#comment-15416541 ] qgxiaozhan commented on HBASE-16394: Thanks ! I saw that before , but could'n find any useful information . Anyway , i will try to found out the solution. > What cause "Compaction is trying to add a bad range",and Should stop the > regionserver? > -- > > Key: HBASE-16394 > URL: https://issues.apache.org/jira/browse/HBASE-16394 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 1.1.2 > Environment: hadoop-2.6.1 hbase-1.1.2 >Reporter: qgxiaozhan > > My cluster dead one regionserver because of "Compaction is trying to add a > bad range" > Here the log: > [2016-08-09T18:30:19.094+08:00] [INFO] regionserver.ReplicationSource : Log > hdfs://athene/hbase/oldWALs/MJQ-HBASE-ATHENE-11139%2C16020%2C1470729882622.default.1470736608897 > was moved to hdfs://athene/hbase/oldWA > Ls/MJQ-HBASE-ATHENE%2C16020%2C1470729882622.default.1470736608897 > [2016-08-09T18:30:30.225+08:00] [INFO] regionserver.MemStoreFlusher : Waited > 90070ms on a compaction to clean up 'TOO MANY STORE FILES'; waited long > enough... proceeding with flush of > tjs4:popt_info,160608008474430,147073716071 > 1.7900baab5204e4f36fa49379c30cd584. > [2016-08-09T18:30:30.226+08:00] [INFO] regionserver.HRegion : Started > memstore flush for > tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584., > current region memstore size 769.41 MB, and 1/1 column fam ilies' > memstores are being flushed. > [2016-08-09T18:30:30.549+08:00] [INFO] regionserver.StripeStoreFileManager : > 3 conflicting files (likely created by a flush) of size 156153021 are moved > to L0 due to concurrent stripe change > [2016-08-09T18:30:31.199+08:00] [INFO] regionserver.HStore : Completed > compaction of 203 file(s) in c of > tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584. > into 20347d203d09442cac30c42b424adda6(size= 3.0 G), > ded362eab9cf4a819675cd35992d4974(size=3.0 G), > 281b1039ed2643679e5b0a3820f5059d(size=2.4 G), total size for store is 8.6 G. > This selection was in queue for 0sec, and took 10mins, 16sec to execute. > [2016-08-09T18:30:31.200+08:00] [INFO] regionserver.CompactSplitThread : > Completed compaction: Request = > regionName=tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584., > storeName=c, fileCount=203, fil eSize=7.2 G, priority=-3, > time=5388162916126535; duration=10mins, 16sec > [2016-08-09T18:30:31.201+08:00] [INFO] regionserver.HRegion : Starting > compaction on ci in region > ad_union:union_click,3487f383ad484bcbb5cef727b69cec2a,1466484980245.c5772fc60c54f64cc977ba9cc01d74ad. > [2016-08-09T18:30:31.201+08:00] [INFO] regionserver.HStore : Starting > compaction of 14 file(s) in ci of > ad_union:union_click,3487f383ad484bcbb5cef727b69cec2a,1466484980245.c5772fc60c54f64cc977ba9cc01d74ad. > into tmpdir=hdfs://at > hene/hbase/data/ad_union/union_click/c5772fc60c54f64cc977ba9cc01d74ad/.tmp, > totalSize=75.0 M > [2016-08-09T18:30:31.206+08:00] [INFO] hfile.CacheConfig : > blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@52659482, > cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, > cacheBloomsOnWrite=fal se, cacheEvictOnClose=false, > cacheDataCompressed=false, prefetchOnOpen=false > [2016-08-09T18:30:32.893+08:00] [INFO] regionserver.ReplicationSource : Log > hdfs://athene/hbase/oldWALs/MJQ-HBASE-ATHENE-11139l%2C16020%2C1470729882622.default.1470736612825 > was moved to hdfs://athene/hbase/oldWA > Ls/MJQ-HBASE-ATHENE-11139.%2C16020%2C1470729882622.default.1470736612825 > [2016-08-09T18:30:34.373+08:00] [INFO] regionserver.HStore : Added > hdfs://athene/hbase/data/tjs4/popt_info/7900baab5204e4f36fa49379c30cd584/c/775e8956cd2a48aaae70b9eded4457e9, > entries=4336457, sequenceid=582528, filesize=48.7 M > [2016-08-09T18:30:34.373+08:00] [FATAL] regionserver.HRegionServer : ABORTING > region server MJQ-HBASE-ATHENE-11139.,16020,1470729882622: Replay of WAL > required. Forcing server shutdown > org.apache.hadoop.hbase.DroppedSnapshotException: region: > tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584. > at > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2354) > at > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2057) > at > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2019) > at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1911) > at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1837) > at >
[jira] [Commented] (HBASE-16394) What cause "Compaction is trying to add a bad range",and Should stop the regionserver?
[ https://issues.apache.org/jira/browse/HBASE-16394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416537#comment-15416537 ] Dima Spivak commented on HBASE-16394: - :) Don't worry, I took care of that already. Along with sending an email to [user@|mailto:u...@hbase.apache.org], it might be worth looking through the Apache mail archives to see if anyone has had advice about this in the past. A quick Google search [led me to this thread|http://mail-archives.apache.org/mod_mbox/hbase-dev/201604.mbox/%3C2E4914CB1DF61A4382C3DCB0A7516A4F592830AE@mbx05.360buyAD.local%3E], though it doesn't look like there was much of a resolution there. > What cause "Compaction is trying to add a bad range",and Should stop the > regionserver? > -- > > Key: HBASE-16394 > URL: https://issues.apache.org/jira/browse/HBASE-16394 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 1.1.2 > Environment: hadoop-2.6.1 hbase-1.1.2 >Reporter: qgxiaozhan > > My cluster dead one regionserver because of "Compaction is trying to add a > bad range" > Here the log: > [2016-08-09T18:30:19.094+08:00] [INFO] regionserver.ReplicationSource : Log > hdfs://athene/hbase/oldWALs/MJQ-HBASE-ATHENE-11139%2C16020%2C1470729882622.default.1470736608897 > was moved to hdfs://athene/hbase/oldWA > Ls/MJQ-HBASE-ATHENE%2C16020%2C1470729882622.default.1470736608897 > [2016-08-09T18:30:30.225+08:00] [INFO] regionserver.MemStoreFlusher : Waited > 90070ms on a compaction to clean up 'TOO MANY STORE FILES'; waited long > enough... proceeding with flush of > tjs4:popt_info,160608008474430,147073716071 > 1.7900baab5204e4f36fa49379c30cd584. > [2016-08-09T18:30:30.226+08:00] [INFO] regionserver.HRegion : Started > memstore flush for > tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584., > current region memstore size 769.41 MB, and 1/1 column fam ilies' > memstores are being flushed. > [2016-08-09T18:30:30.549+08:00] [INFO] regionserver.StripeStoreFileManager : > 3 conflicting files (likely created by a flush) of size 156153021 are moved > to L0 due to concurrent stripe change > [2016-08-09T18:30:31.199+08:00] [INFO] regionserver.HStore : Completed > compaction of 203 file(s) in c of > tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584. > into 20347d203d09442cac30c42b424adda6(size= 3.0 G), > ded362eab9cf4a819675cd35992d4974(size=3.0 G), > 281b1039ed2643679e5b0a3820f5059d(size=2.4 G), total size for store is 8.6 G. > This selection was in queue for 0sec, and took 10mins, 16sec to execute. > [2016-08-09T18:30:31.200+08:00] [INFO] regionserver.CompactSplitThread : > Completed compaction: Request = > regionName=tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584., > storeName=c, fileCount=203, fil eSize=7.2 G, priority=-3, > time=5388162916126535; duration=10mins, 16sec > [2016-08-09T18:30:31.201+08:00] [INFO] regionserver.HRegion : Starting > compaction on ci in region > ad_union:union_click,3487f383ad484bcbb5cef727b69cec2a,1466484980245.c5772fc60c54f64cc977ba9cc01d74ad. > [2016-08-09T18:30:31.201+08:00] [INFO] regionserver.HStore : Starting > compaction of 14 file(s) in ci of > ad_union:union_click,3487f383ad484bcbb5cef727b69cec2a,1466484980245.c5772fc60c54f64cc977ba9cc01d74ad. > into tmpdir=hdfs://at > hene/hbase/data/ad_union/union_click/c5772fc60c54f64cc977ba9cc01d74ad/.tmp, > totalSize=75.0 M > [2016-08-09T18:30:31.206+08:00] [INFO] hfile.CacheConfig : > blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@52659482, > cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, > cacheBloomsOnWrite=fal se, cacheEvictOnClose=false, > cacheDataCompressed=false, prefetchOnOpen=false > [2016-08-09T18:30:32.893+08:00] [INFO] regionserver.ReplicationSource : Log > hdfs://athene/hbase/oldWALs/MJQ-HBASE-ATHENE-11139l%2C16020%2C1470729882622.default.1470736612825 > was moved to hdfs://athene/hbase/oldWA > Ls/MJQ-HBASE-ATHENE-11139.%2C16020%2C1470729882622.default.1470736612825 > [2016-08-09T18:30:34.373+08:00] [INFO] regionserver.HStore : Added > hdfs://athene/hbase/data/tjs4/popt_info/7900baab5204e4f36fa49379c30cd584/c/775e8956cd2a48aaae70b9eded4457e9, > entries=4336457, sequenceid=582528, filesize=48.7 M > [2016-08-09T18:30:34.373+08:00] [FATAL] regionserver.HRegionServer : ABORTING > region server MJQ-HBASE-ATHENE-11139.,16020,1470729882622: Replay of WAL > required. Forcing server shutdown > org.apache.hadoop.hbase.DroppedSnapshotException: region: > tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584. > at > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2354) > at
[jira] [Commented] (HBASE-16394) What cause "Compaction is trying to add a bad range",and Should stop the regionserver?
[ https://issues.apache.org/jira/browse/HBASE-16394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416533#comment-15416533 ] qgxiaozhan commented on HBASE-16394: I am sorry,should i close it ? > What cause "Compaction is trying to add a bad range",and Should stop the > regionserver? > -- > > Key: HBASE-16394 > URL: https://issues.apache.org/jira/browse/HBASE-16394 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 1.1.2 > Environment: hadoop-2.6.1 hbase-1.1.2 >Reporter: qgxiaozhan > > My cluster dead one regionserver because of "Compaction is trying to add a > bad range" > Here the log: > [2016-08-09T18:30:19.094+08:00] [INFO] regionserver.ReplicationSource : Log > hdfs://athene/hbase/oldWALs/MJQ-HBASE-ATHENE-11139%2C16020%2C1470729882622.default.1470736608897 > was moved to hdfs://athene/hbase/oldWA > Ls/MJQ-HBASE-ATHENE%2C16020%2C1470729882622.default.1470736608897 > [2016-08-09T18:30:30.225+08:00] [INFO] regionserver.MemStoreFlusher : Waited > 90070ms on a compaction to clean up 'TOO MANY STORE FILES'; waited long > enough... proceeding with flush of > tjs4:popt_info,160608008474430,147073716071 > 1.7900baab5204e4f36fa49379c30cd584. > [2016-08-09T18:30:30.226+08:00] [INFO] regionserver.HRegion : Started > memstore flush for > tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584., > current region memstore size 769.41 MB, and 1/1 column fam ilies' > memstores are being flushed. > [2016-08-09T18:30:30.549+08:00] [INFO] regionserver.StripeStoreFileManager : > 3 conflicting files (likely created by a flush) of size 156153021 are moved > to L0 due to concurrent stripe change > [2016-08-09T18:30:31.199+08:00] [INFO] regionserver.HStore : Completed > compaction of 203 file(s) in c of > tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584. > into 20347d203d09442cac30c42b424adda6(size= 3.0 G), > ded362eab9cf4a819675cd35992d4974(size=3.0 G), > 281b1039ed2643679e5b0a3820f5059d(size=2.4 G), total size for store is 8.6 G. > This selection was in queue for 0sec, and took 10mins, 16sec to execute. > [2016-08-09T18:30:31.200+08:00] [INFO] regionserver.CompactSplitThread : > Completed compaction: Request = > regionName=tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584., > storeName=c, fileCount=203, fil eSize=7.2 G, priority=-3, > time=5388162916126535; duration=10mins, 16sec > [2016-08-09T18:30:31.201+08:00] [INFO] regionserver.HRegion : Starting > compaction on ci in region > ad_union:union_click,3487f383ad484bcbb5cef727b69cec2a,1466484980245.c5772fc60c54f64cc977ba9cc01d74ad. > [2016-08-09T18:30:31.201+08:00] [INFO] regionserver.HStore : Starting > compaction of 14 file(s) in ci of > ad_union:union_click,3487f383ad484bcbb5cef727b69cec2a,1466484980245.c5772fc60c54f64cc977ba9cc01d74ad. > into tmpdir=hdfs://at > hene/hbase/data/ad_union/union_click/c5772fc60c54f64cc977ba9cc01d74ad/.tmp, > totalSize=75.0 M > [2016-08-09T18:30:31.206+08:00] [INFO] hfile.CacheConfig : > blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@52659482, > cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, > cacheBloomsOnWrite=fal se, cacheEvictOnClose=false, > cacheDataCompressed=false, prefetchOnOpen=false > [2016-08-09T18:30:32.893+08:00] [INFO] regionserver.ReplicationSource : Log > hdfs://athene/hbase/oldWALs/MJQ-HBASE-ATHENE-11139l%2C16020%2C1470729882622.default.1470736612825 > was moved to hdfs://athene/hbase/oldWA > Ls/MJQ-HBASE-ATHENE-11139.%2C16020%2C1470729882622.default.1470736612825 > [2016-08-09T18:30:34.373+08:00] [INFO] regionserver.HStore : Added > hdfs://athene/hbase/data/tjs4/popt_info/7900baab5204e4f36fa49379c30cd584/c/775e8956cd2a48aaae70b9eded4457e9, > entries=4336457, sequenceid=582528, filesize=48.7 M > [2016-08-09T18:30:34.373+08:00] [FATAL] regionserver.HRegionServer : ABORTING > region server MJQ-HBASE-ATHENE-11139.,16020,1470729882622: Replay of WAL > required. Forcing server shutdown > org.apache.hadoop.hbase.DroppedSnapshotException: region: > tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584. > at > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2354) > at > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2057) > at > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2019) > at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1911) > at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1837) > at > org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:510) > at >
[jira] [Commented] (HBASE-16363) Correct javadoc for qualifier length and value length in Cell interface
[
https://issues.apache.org/jira/browse/HBASE-16363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416529#comment-15416529
]
Hudson commented on HBASE-16363:
FAILURE: Integrated in HBase-1.3 #816 (See
[https://builds.apache.org/job/HBase-1.3/816/])
HBASE-16363 Correct javadoc for qualifier length and value (Xiang Li) (stack:
rev 9aed64173b4f360e3029a0936ae986cd5c916062)
* hbase-common/src/main/java/org/apache/hadoop/hbase/Cell.java
> Correct javadoc for qualifier length and value length in Cell interface
> ---
>
> Key: HBASE-16363
> URL: https://issues.apache.org/jira/browse/HBASE-16363
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.2.3
>
> Attachments: HBASE-16363.master.000.patch
>
>
> In Cell interface
> (hbase-common/src/main/java/org/apache/hadoop/hbase/Cell.java)
> (1) We have the following comment for "qualifier"
> {code}
> /**
> * Contiguous raw bytes that may start at any index in the containing array.
> Max length is
> * Short.MAX_VALUE which is 32,767 bytes.
> * @return The array containing the qualifier bytes.
> */
> {code}
> But getQualifierLength() returns int
> {code}
> int getQualifierLength();
> {code}
> (2) We have the following comment for "value"
> {code}
> /**
>* Contiguous raw bytes that may start at any index in the containing
> array. Max length is
>* Integer.MAX_VALUE which is 2,147,483,648 bytes.
>* @return The array containing the value bytes.
>*/
> {code}
> Integer.MAX_VALUE is not 2,147,483,648 (2^31), but 2,147,483,647 (2^31-1)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16368) test*WhenRegionMove in TestPartialResultsFromClientSide is flaky
[
https://issues.apache.org/jira/browse/HBASE-16368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416530#comment-15416530
]
Hudson commented on HBASE-16368:
FAILURE: Integrated in HBase-1.3 #816 (See
[https://builds.apache.org/job/HBase-1.3/816/])
HBASE-16368 test*WhenRegionMove in TestPartialResultsFromClientSide is
(zhangduo: rev 8f5b70ac3235a463708245ac83baa23ddc5b5867)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/TestPartialResultsFromClientSide.java
> test*WhenRegionMove in TestPartialResultsFromClientSide is flaky
>
>
> Key: HBASE-16368
> URL: https://issues.apache.org/jira/browse/HBASE-16368
> Project: HBase
> Issue Type: Bug
> Components: Scanners
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.5, 1.2.2
>Reporter: Guanghao Zhang
>Assignee: Phil Yang
> Labels: trivial
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.6, 1.2.3
>
> Attachments: HBASE-16368-v1.patch, HBASE-16368-v1.patch,
> HBASE-16368-v1.patch
>
>
> This test fail when Hadoop QA run preCommit:
> https://builds.apache.org/job/PreCommit-HBASE-Build/2971/testReport/org.apache.hadoop.hbase/TestPartialResultsFromClientSide/testReversedCompleteResultWhenRegionMove/.
> And I found it is in Flaky Tests Dashboard:
> http://hbase.x10host.com/flaky-tests/. I run it in my local machine and it
> may fail, too.
> Test results show that the region location didn't update when scanner
> callable get a NotServingRegionException or RegionMovedException.
> {code}
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=36, exceptions:
> Sat Aug 06 05:55:52 UTC 2016, null, java.net.SocketTimeoutException:
> callTimeout=2000, callDuration=2157:
> org.apache.hadoop.hbase.NotServingRegionException:
> testReversedCompleteResultWhenRegionMove,,1470462949504.5069bd63bf6eda5108acec4fcc087b0e.
> is closing
> at
> org.apache.hadoop.hbase.regionserver.HRegion.startRegionOperation(HRegion.java:8233)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2634)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2629)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2623)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2490)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34950)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2264)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:118)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:189)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:169)
> row '' on table 'testReversedCompleteResultWhenRegionMove' at
> region=testReversedCompleteResultWhenRegionMove,,1470462949504.5069bd63bf6eda5108acec4fcc087b0e.,
> hostname=asf907.gq1.ygridcore.net,38914,1470462943053, seqNum=2
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:281)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:213)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:61)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:212)
> at
> org.apache.hadoop.hbase.client.ReversedClientScanner.nextScanner(ReversedClientScanner.java:118)
> at
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:166)
> at
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:161)
> at
> org.apache.hadoop.hbase.client.ReversedClientScanner.(ReversedClientScanner.java:56)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:785)
> at
> org.apache.hadoop.hbase.TestPartialResultsFromClientSide.testReversedCompleteResultWhenRegionMove(TestPartialResultsFromClientSide.java:986)
> {code}
> {code}
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=36, exceptions:
> Sat Aug 06 16:27:22 CST 2016, null, java.net.SocketTimeoutException:
> callTimeout=2000, callDuration=3035: Region moved to: hostname=localhost
> port=58351 startCode=1470472007714. As of locationSeqNum=6. row 'testRow0' on
> table 'testPartialResultWhenRegionMove' at
> region=testPartialResultWhenRegionMove,,1470472035048.977faf05c1d6d9990b5559b17aa18913.,
> hostname=localhost,40425,1470472007646, seqNum=2
> at
>
[jira] [Resolved] (HBASE-16394) What cause "Compaction is trying to add a bad range",and Should stop the regionserver?
[ https://issues.apache.org/jira/browse/HBASE-16394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dima Spivak resolved HBASE-16394. - Resolution: Invalid Hey [~qgxiaozhan], HBase only uses JIRA to track development work. You'll have much more luck getting help with the issue you're seeing if you email [u...@hbase.apache.org|mailto:u...@hbase.apache.org] with details of the problem you're having. A link to a Pastebin/Gist with log entries from just before the issue happens would also be really useful. Cheers! > What cause "Compaction is trying to add a bad range",and Should stop the > regionserver? > -- > > Key: HBASE-16394 > URL: https://issues.apache.org/jira/browse/HBASE-16394 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 1.1.2 > Environment: hadoop-2.6.1 hbase-1.1.2 >Reporter: qgxiaozhan > > My cluster dead one regionserver because of "Compaction is trying to add a > bad range" > Here the log: > [2016-08-09T18:30:19.094+08:00] [INFO] regionserver.ReplicationSource : Log > hdfs://athene/hbase/oldWALs/MJQ-HBASE-ATHENE-11139%2C16020%2C1470729882622.default.1470736608897 > was moved to hdfs://athene/hbase/oldWA > Ls/MJQ-HBASE-ATHENE%2C16020%2C1470729882622.default.1470736608897 > [2016-08-09T18:30:30.225+08:00] [INFO] regionserver.MemStoreFlusher : Waited > 90070ms on a compaction to clean up 'TOO MANY STORE FILES'; waited long > enough... proceeding with flush of > tjs4:popt_info,160608008474430,147073716071 > 1.7900baab5204e4f36fa49379c30cd584. > [2016-08-09T18:30:30.226+08:00] [INFO] regionserver.HRegion : Started > memstore flush for > tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584., > current region memstore size 769.41 MB, and 1/1 column fam ilies' > memstores are being flushed. > [2016-08-09T18:30:30.549+08:00] [INFO] regionserver.StripeStoreFileManager : > 3 conflicting files (likely created by a flush) of size 156153021 are moved > to L0 due to concurrent stripe change > [2016-08-09T18:30:31.199+08:00] [INFO] regionserver.HStore : Completed > compaction of 203 file(s) in c of > tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584. > into 20347d203d09442cac30c42b424adda6(size= 3.0 G), > ded362eab9cf4a819675cd35992d4974(size=3.0 G), > 281b1039ed2643679e5b0a3820f5059d(size=2.4 G), total size for store is 8.6 G. > This selection was in queue for 0sec, and took 10mins, 16sec to execute. > [2016-08-09T18:30:31.200+08:00] [INFO] regionserver.CompactSplitThread : > Completed compaction: Request = > regionName=tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584., > storeName=c, fileCount=203, fil eSize=7.2 G, priority=-3, > time=5388162916126535; duration=10mins, 16sec > [2016-08-09T18:30:31.201+08:00] [INFO] regionserver.HRegion : Starting > compaction on ci in region > ad_union:union_click,3487f383ad484bcbb5cef727b69cec2a,1466484980245.c5772fc60c54f64cc977ba9cc01d74ad. > [2016-08-09T18:30:31.201+08:00] [INFO] regionserver.HStore : Starting > compaction of 14 file(s) in ci of > ad_union:union_click,3487f383ad484bcbb5cef727b69cec2a,1466484980245.c5772fc60c54f64cc977ba9cc01d74ad. > into tmpdir=hdfs://at > hene/hbase/data/ad_union/union_click/c5772fc60c54f64cc977ba9cc01d74ad/.tmp, > totalSize=75.0 M > [2016-08-09T18:30:31.206+08:00] [INFO] hfile.CacheConfig : > blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@52659482, > cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, > cacheBloomsOnWrite=fal se, cacheEvictOnClose=false, > cacheDataCompressed=false, prefetchOnOpen=false > [2016-08-09T18:30:32.893+08:00] [INFO] regionserver.ReplicationSource : Log > hdfs://athene/hbase/oldWALs/MJQ-HBASE-ATHENE-11139l%2C16020%2C1470729882622.default.1470736612825 > was moved to hdfs://athene/hbase/oldWA > Ls/MJQ-HBASE-ATHENE-11139.%2C16020%2C1470729882622.default.1470736612825 > [2016-08-09T18:30:34.373+08:00] [INFO] regionserver.HStore : Added > hdfs://athene/hbase/data/tjs4/popt_info/7900baab5204e4f36fa49379c30cd584/c/775e8956cd2a48aaae70b9eded4457e9, > entries=4336457, sequenceid=582528, filesize=48.7 M > [2016-08-09T18:30:34.373+08:00] [FATAL] regionserver.HRegionServer : ABORTING > region server MJQ-HBASE-ATHENE-11139.,16020,1470729882622: Replay of WAL > required. Forcing server shutdown > org.apache.hadoop.hbase.DroppedSnapshotException: region: > tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584. > at > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2354) > at > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2057) > at >
[jira] [Commented] (HBASE-16267) Remove commons-httpclient dependency from hbase-rest module
[
https://issues.apache.org/jira/browse/HBASE-16267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416524#comment-15416524
]
Ted Yu commented on HBASE-16267:
Timed out tests were not related to patch.
I ran a few (TestMasterFailoverWithProcedures) locally with patch v15 and they
passed.
> Remove commons-httpclient dependency from hbase-rest module
> ---
>
> Key: HBASE-16267
> URL: https://issues.apache.org/jira/browse/HBASE-16267
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 16267.v10.txt, 16267.v11.txt, 16267.v12.txt,
> 16267.v13.txt, 16267.v14.txt, 16267.v15.txt, 16267.v2.txt, 16267.v4.txt,
> 16267.v6.txt, 16267.v8.txt, 16267.v9.txt
>
>
> hbase-rest module still has imports from org.apache.commons.httpclient .
> There is more work to be done after HBASE-15767 was integrated.
> In master branch, there seems to be transitive dependency which allows the
> code to compile:
> {code}
> [INFO] +- org.apache.hadoop:hadoop-common:jar:2.7.1:compile
> [INFO] | +- org.apache.hadoop:hadoop-annotations:jar:2.7.1:compile
> [INFO] | +- commons-cli:commons-cli:jar:1.2:compile
> [INFO] | +- org.apache.commons:commons-math3:jar:3.1.1:compile
> [INFO] | +- xmlenc:xmlenc:jar:0.52:compile
> [INFO] | +- commons-httpclient:commons-httpclient:jar:3.1:compile
> {code}
> HADOOP-12767
> to move the uses of httpclient HADOOP-10105
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-5262 :
> http/conn/ssl/SSLConnectionSocketFactory.java in Apache HttpComponents
> HttpClient before 4.3.6 ignores the http.socket.timeout configuration setting
> during an SSL handshake, which allows remote attackers to cause a denial of
> service (HTTPS call hang) via unspecified vectors.
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-6153
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-5783
> Apache Commons HttpClient 3.x, as used in Amazon Flexible Payments
> Service (FPS) merchant Java SDK and other products, does not verify that the
> server hostname matches a domain name in the subject's Common Name (CN) or
> subjectAltName field of the X.509 certificate, which allows man-in-the-middle
> attackers to spoof SSL servers via an arbitrary valid certificate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16394) What cause "Compaction is trying to add a bad range",and Should stop the regionserver?
[ https://issues.apache.org/jira/browse/HBASE-16394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qgxiaozhan updated HBASE-16394: --- Description: My cluster dead one regionserver because of "Compaction is trying to add a bad range" Here the log: [2016-08-09T18:30:19.094+08:00] [INFO] regionserver.ReplicationSource : Log hdfs://athene/hbase/oldWALs/MJQ-HBASE-ATHENE-11139%2C16020%2C1470729882622.default.1470736608897 was moved to hdfs://athene/hbase/oldWA Ls/MJQ-HBASE-ATHENE%2C16020%2C1470729882622.default.1470736608897 [2016-08-09T18:30:30.225+08:00] [INFO] regionserver.MemStoreFlusher : Waited 90070ms on a compaction to clean up 'TOO MANY STORE FILES'; waited long enough... proceeding with flush of tjs4:popt_info,160608008474430,147073716071 1.7900baab5204e4f36fa49379c30cd584. [2016-08-09T18:30:30.226+08:00] [INFO] regionserver.HRegion : Started memstore flush for tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584., current region memstore size 769.41 MB, and 1/1 column fam ilies' memstores are being flushed. [2016-08-09T18:30:30.549+08:00] [INFO] regionserver.StripeStoreFileManager : 3 conflicting files (likely created by a flush) of size 156153021 are moved to L0 due to concurrent stripe change [2016-08-09T18:30:31.199+08:00] [INFO] regionserver.HStore : Completed compaction of 203 file(s) in c of tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584. into 20347d203d09442cac30c42b424adda6(size= 3.0 G), ded362eab9cf4a819675cd35992d4974(size=3.0 G), 281b1039ed2643679e5b0a3820f5059d(size=2.4 G), total size for store is 8.6 G. This selection was in queue for 0sec, and took 10mins, 16sec to execute. [2016-08-09T18:30:31.200+08:00] [INFO] regionserver.CompactSplitThread : Completed compaction: Request = regionName=tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584., storeName=c, fileCount=203, fil eSize=7.2 G, priority=-3, time=5388162916126535; duration=10mins, 16sec [2016-08-09T18:30:31.201+08:00] [INFO] regionserver.HRegion : Starting compaction on ci in region ad_union:union_click,3487f383ad484bcbb5cef727b69cec2a,1466484980245.c5772fc60c54f64cc977ba9cc01d74ad. [2016-08-09T18:30:31.201+08:00] [INFO] regionserver.HStore : Starting compaction of 14 file(s) in ci of ad_union:union_click,3487f383ad484bcbb5cef727b69cec2a,1466484980245.c5772fc60c54f64cc977ba9cc01d74ad. into tmpdir=hdfs://at hene/hbase/data/ad_union/union_click/c5772fc60c54f64cc977ba9cc01d74ad/.tmp, totalSize=75.0 M [2016-08-09T18:30:31.206+08:00] [INFO] hfile.CacheConfig : blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@52659482, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=fal se, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false [2016-08-09T18:30:32.893+08:00] [INFO] regionserver.ReplicationSource : Log hdfs://athene/hbase/oldWALs/MJQ-HBASE-ATHENE-11139l%2C16020%2C1470729882622.default.1470736612825 was moved to hdfs://athene/hbase/oldWA Ls/MJQ-HBASE-ATHENE-11139.%2C16020%2C1470729882622.default.1470736612825 [2016-08-09T18:30:34.373+08:00] [INFO] regionserver.HStore : Added hdfs://athene/hbase/data/tjs4/popt_info/7900baab5204e4f36fa49379c30cd584/c/775e8956cd2a48aaae70b9eded4457e9, entries=4336457, sequenceid=582528, filesize=48.7 M [2016-08-09T18:30:34.373+08:00] [FATAL] regionserver.HRegionServer : ABORTING region server MJQ-HBASE-ATHENE-11139.,16020,1470729882622: Replay of WAL required. Forcing server shutdown org.apache.hadoop.hbase.DroppedSnapshotException: region: tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584. at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2057) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2019) at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1911) at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1837) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:510) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:471) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$800(MemStoreFlusher.java:75) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:259) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Compaction is trying to add a bad range. at org.apache.hadoop.hbase.regionserver.StripeStoreFileManager$CompactionOrFlushMergeCopy.processNewCandidateStripes(StripeStoreFileManager.java:837) at org.apache.hadoop.hbase.regionserver.StripeStoreFileManager$CompactionOrFlushMergeCopy.mergeResults(StripeStoreFileManager.java:672) at
[jira] [Updated] (HBASE-16394) What cause "Compaction is trying to add a bad range",and Should stop the regionserver?
[ https://issues.apache.org/jira/browse/HBASE-16394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qgxiaozhan updated HBASE-16394: --- Description: My cluster dead one regionserver because of "Compaction is trying to add a bad range" Here the log: ``` [2016-08-09T18:30:19.094+08:00] [INFO] regionserver.ReplicationSource : Log hdfs://athene/hbase/oldWALs/MJQ-HBASE-ATHENE-11139%2C16020%2C1470729882622.default.1470736608897 was moved to hdfs://athene/hbase/oldWA Ls/MJQ-HBASE-ATHENE%2C16020%2C1470729882622.default.1470736608897 [2016-08-09T18:30:30.225+08:00] [INFO] regionserver.MemStoreFlusher : Waited 90070ms on a compaction to clean up 'TOO MANY STORE FILES'; waited long enough... proceeding with flush of tjs4:popt_info,160608008474430,147073716071 1.7900baab5204e4f36fa49379c30cd584. [2016-08-09T18:30:30.226+08:00] [INFO] regionserver.HRegion : Started memstore flush for tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584., current region memstore size 769.41 MB, and 1/1 column fam ilies' memstores are being flushed. [2016-08-09T18:30:30.549+08:00] [INFO] regionserver.StripeStoreFileManager : 3 conflicting files (likely created by a flush) of size 156153021 are moved to L0 due to concurrent stripe change [2016-08-09T18:30:31.199+08:00] [INFO] regionserver.HStore : Completed compaction of 203 file(s) in c of tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584. into 20347d203d09442cac30c42b424adda6(size= 3.0 G), ded362eab9cf4a819675cd35992d4974(size=3.0 G), 281b1039ed2643679e5b0a3820f5059d(size=2.4 G), total size for store is 8.6 G. This selection was in queue for 0sec, and took 10mins, 16sec to execute. [2016-08-09T18:30:31.200+08:00] [INFO] regionserver.CompactSplitThread : Completed compaction: Request = regionName=tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584., storeName=c, fileCount=203, fil eSize=7.2 G, priority=-3, time=5388162916126535; duration=10mins, 16sec [2016-08-09T18:30:31.201+08:00] [INFO] regionserver.HRegion : Starting compaction on ci in region ad_union:union_click,3487f383ad484bcbb5cef727b69cec2a,1466484980245.c5772fc60c54f64cc977ba9cc01d74ad. [2016-08-09T18:30:31.201+08:00] [INFO] regionserver.HStore : Starting compaction of 14 file(s) in ci of ad_union:union_click,3487f383ad484bcbb5cef727b69cec2a,1466484980245.c5772fc60c54f64cc977ba9cc01d74ad. into tmpdir=hdfs://at hene/hbase/data/ad_union/union_click/c5772fc60c54f64cc977ba9cc01d74ad/.tmp, totalSize=75.0 M [2016-08-09T18:30:31.206+08:00] [INFO] hfile.CacheConfig : blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@52659482, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=fal se, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false [2016-08-09T18:30:32.893+08:00] [INFO] regionserver.ReplicationSource : Log hdfs://athene/hbase/oldWALs/MJQ-HBASE-ATHENE-11139l%2C16020%2C1470729882622.default.1470736612825 was moved to hdfs://athene/hbase/oldWA Ls/MJQ-HBASE-ATHENE-11139.%2C16020%2C1470729882622.default.1470736612825 [2016-08-09T18:30:34.373+08:00] [INFO] regionserver.HStore : Added hdfs://athene/hbase/data/tjs4/popt_info/7900baab5204e4f36fa49379c30cd584/c/775e8956cd2a48aaae70b9eded4457e9, entries=4336457, sequenceid=582528, filesize=48.7 M [2016-08-09T18:30:34.373+08:00] [FATAL] regionserver.HRegionServer : ABORTING region server MJQ-HBASE-ATHENE-11139.,16020,1470729882622: Replay of WAL required. Forcing server shutdown org.apache.hadoop.hbase.DroppedSnapshotException: region: tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584. at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2057) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2019) at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1911) at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1837) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:510) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:471) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$800(MemStoreFlusher.java:75) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:259) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Compaction is trying to add a bad range. at org.apache.hadoop.hbase.regionserver.StripeStoreFileManager$CompactionOrFlushMergeCopy.processNewCandidateStripes(StripeStoreFileManager.java:837) at org.apache.hadoop.hbase.regionserver.StripeStoreFileManager$CompactionOrFlushMergeCopy.mergeResults(StripeStoreFileManager.java:672) at
[jira] [Updated] (HBASE-16394) What cause "Compaction is trying to add a bad range",and Should stop the regionserver?
[ https://issues.apache.org/jira/browse/HBASE-16394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qgxiaozhan updated HBASE-16394: --- Description: My cluster dead one regionserver because of "Compaction is trying to add a bad range" Here the log: [2016-08-09T18:30:19.094+08:00] [INFO] regionserver.ReplicationSource : Log hdfs://athene/hbase/oldWALs/MJQ-HBASE-ATHENE-11139%2C16020%2C1470729882622.default.1470736608897 was moved to hdfs://athene/hbase/oldWA Ls/MJQ-HBASE-ATHENE%2C16020%2C1470729882622.default.1470736608897 [2016-08-09T18:30:30.225+08:00] [INFO] regionserver.MemStoreFlusher : Waited 90070ms on a compaction to clean up 'TOO MANY STORE FILES'; waited long enough... proceeding with flush of tjs4:popt_info,160608008474430,147073716071 1.7900baab5204e4f36fa49379c30cd584. [2016-08-09T18:30:30.226+08:00] [INFO] regionserver.HRegion : Started memstore flush for tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584., current region memstore size 769.41 MB, and 1/1 column fam ilies' memstores are being flushed. [2016-08-09T18:30:30.549+08:00] [INFO] regionserver.StripeStoreFileManager : 3 conflicting files (likely created by a flush) of size 156153021 are moved to L0 due to concurrent stripe change [2016-08-09T18:30:31.199+08:00] [INFO] regionserver.HStore : Completed compaction of 203 file(s) in c of tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584. into 20347d203d09442cac30c42b424adda6(size= 3.0 G), ded362eab9cf4a819675cd35992d4974(size=3.0 G), 281b1039ed2643679e5b0a3820f5059d(size=2.4 G), total size for store is 8.6 G. This selection was in queue for 0sec, and took 10mins, 16sec to execute. [2016-08-09T18:30:31.200+08:00] [INFO] regionserver.CompactSplitThread : Completed compaction: Request = regionName=tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584., storeName=c, fileCount=203, fil eSize=7.2 G, priority=-3, time=5388162916126535; duration=10mins, 16sec [2016-08-09T18:30:31.201+08:00] [INFO] regionserver.HRegion : Starting compaction on ci in region ad_union:union_click,3487f383ad484bcbb5cef727b69cec2a,1466484980245.c5772fc60c54f64cc977ba9cc01d74ad. [2016-08-09T18:30:31.201+08:00] [INFO] regionserver.HStore : Starting compaction of 14 file(s) in ci of ad_union:union_click,3487f383ad484bcbb5cef727b69cec2a,1466484980245.c5772fc60c54f64cc977ba9cc01d74ad. into tmpdir=hdfs://at hene/hbase/data/ad_union/union_click/c5772fc60c54f64cc977ba9cc01d74ad/.tmp, totalSize=75.0 M [2016-08-09T18:30:31.206+08:00] [INFO] hfile.CacheConfig : blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@52659482, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=fal se, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false [2016-08-09T18:30:32.893+08:00] [INFO] regionserver.ReplicationSource : Log hdfs://athene/hbase/oldWALs/MJQ-HBASE-ATHENE-11139l%2C16020%2C1470729882622.default.1470736612825 was moved to hdfs://athene/hbase/oldWA Ls/MJQ-HBASE-ATHENE-11139.%2C16020%2C1470729882622.default.1470736612825 [2016-08-09T18:30:34.373+08:00] [INFO] regionserver.HStore : Added hdfs://athene/hbase/data/tjs4/popt_info/7900baab5204e4f36fa49379c30cd584/c/775e8956cd2a48aaae70b9eded4457e9, entries=4336457, sequenceid=582528, filesize=48.7 M [2016-08-09T18:30:34.373+08:00] [FATAL] regionserver.HRegionServer : ABORTING region server MJQ-HBASE-ATHENE-11139.,16020,1470729882622: Replay of WAL required. Forcing server shutdown org.apache.hadoop.hbase.DroppedSnapshotException: region: tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584. at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2057) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2019) at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1911) at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1837) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:510) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:471) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$800(MemStoreFlusher.java:75) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:259) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Compaction is trying to add a bad range. at org.apache.hadoop.hbase.regionserver.StripeStoreFileManager$CompactionOrFlushMergeCopy.processNewCandidateStripes(StripeStoreFileManager.java:837) at org.apache.hadoop.hbase.regionserver.StripeStoreFileManager$CompactionOrFlushMergeCopy.mergeResults(StripeStoreFileManager.java:672) at
[jira] [Updated] (HBASE-16394) What cause "Compaction is trying to add a bad range",and Should stop the regionserver?
[ https://issues.apache.org/jira/browse/HBASE-16394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qgxiaozhan updated HBASE-16394: --- Summary: What cause "Compaction is trying to add a bad range",and Should stop the regionserver? (was: Compaction is trying to add a bad range) > What cause "Compaction is trying to add a bad range",and Should stop the > regionserver? > -- > > Key: HBASE-16394 > URL: https://issues.apache.org/jira/browse/HBASE-16394 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 1.1.2 > Environment: hadoop-2.6.1 hbase-1.1.2 >Reporter: qgxiaozhan > > My cluster dead because of "Compaction is trying to add a bad range" > Here the log: > [2016-08-09T18:30:19.094+08:00] [INFO] regionserver.ReplicationSource : Log > hdfs://athene/hbase/oldWALs/MJQ-HBASE-ATHENE-11139%2C16020%2C1470729882622.default.1470736608897 > was moved to hdfs://athene/hbase/oldWA > Ls/MJQ-HBASE-ATHENE%2C16020%2C1470729882622.default.1470736608897 > [2016-08-09T18:30:30.225+08:00] [INFO] regionserver.MemStoreFlusher : Waited > 90070ms on a compaction to clean up 'TOO MANY STORE FILES'; waited long > enough... proceeding with flush of > tjs4:popt_info,160608008474430,147073716071 > 1.7900baab5204e4f36fa49379c30cd584. > [2016-08-09T18:30:30.226+08:00] [INFO] regionserver.HRegion : Started > memstore flush for > tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584., > current region memstore size 769.41 MB, and 1/1 column fam ilies' > memstores are being flushed. > [2016-08-09T18:30:30.549+08:00] [INFO] regionserver.StripeStoreFileManager : > 3 conflicting files (likely created by a flush) of size 156153021 are moved > to L0 due to concurrent stripe change > [2016-08-09T18:30:31.199+08:00] [INFO] regionserver.HStore : Completed > compaction of 203 file(s) in c of > tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584. > into 20347d203d09442cac30c42b424adda6(size= 3.0 G), > ded362eab9cf4a819675cd35992d4974(size=3.0 G), > 281b1039ed2643679e5b0a3820f5059d(size=2.4 G), total size for store is 8.6 G. > This selection was in queue for 0sec, and took 10mins, 16sec to execute. > [2016-08-09T18:30:31.200+08:00] [INFO] regionserver.CompactSplitThread : > Completed compaction: Request = > regionName=tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584., > storeName=c, fileCount=203, fil eSize=7.2 G, priority=-3, > time=5388162916126535; duration=10mins, 16sec > [2016-08-09T18:30:31.201+08:00] [INFO] regionserver.HRegion : Starting > compaction on ci in region > ad_union:union_click,3487f383ad484bcbb5cef727b69cec2a,1466484980245.c5772fc60c54f64cc977ba9cc01d74ad. > [2016-08-09T18:30:31.201+08:00] [INFO] regionserver.HStore : Starting > compaction of 14 file(s) in ci of > ad_union:union_click,3487f383ad484bcbb5cef727b69cec2a,1466484980245.c5772fc60c54f64cc977ba9cc01d74ad. > into tmpdir=hdfs://at > hene/hbase/data/ad_union/union_click/c5772fc60c54f64cc977ba9cc01d74ad/.tmp, > totalSize=75.0 M > [2016-08-09T18:30:31.206+08:00] [INFO] hfile.CacheConfig : > blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@52659482, > cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, > cacheBloomsOnWrite=fal se, cacheEvictOnClose=false, > cacheDataCompressed=false, prefetchOnOpen=false > [2016-08-09T18:30:32.893+08:00] [INFO] regionserver.ReplicationSource : Log > hdfs://athene/hbase/oldWALs/MJQ-HBASE-ATHENE-11139l%2C16020%2C1470729882622.default.1470736612825 > was moved to hdfs://athene/hbase/oldWA > Ls/MJQ-HBASE-ATHENE-11139.%2C16020%2C1470729882622.default.1470736612825 > [2016-08-09T18:30:34.373+08:00] [INFO] regionserver.HStore : Added > hdfs://athene/hbase/data/tjs4/popt_info/7900baab5204e4f36fa49379c30cd584/c/775e8956cd2a48aaae70b9eded4457e9, > entries=4336457, sequenceid=582528, filesize=48.7 M > [2016-08-09T18:30:34.373+08:00] [FATAL] regionserver.HRegionServer : ABORTING > region server MJQ-HBASE-ATHENE-11139.,16020,1470729882622: Replay of WAL > required. Forcing server shutdown > org.apache.hadoop.hbase.DroppedSnapshotException: region: > tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584. > at > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2354) > at > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2057) > at > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2019) > at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1911) > at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1837) > at >
[jira] [Commented] (HBASE-16393) Improve computeHDFSBlocksDistribution
[
https://issues.apache.org/jira/browse/HBASE-16393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416520#comment-15416520
]
Hadoop QA commented on HBASE-16393:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 23s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} master passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} master passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 0s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 113m 51s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 162m 41s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures |
| Timed out junit tests | org.apache.hadoop.hbase.wal.TestWALSplitCompressed |
| | org.apache.hadoop.hbase.io.encoding.TestChangingEncoding |
| | org.apache.hadoop.hbase.zookeeper.lock.TestZKInterProcessReadWriteLock |
| | org.apache.hadoop.hbase.TestIOFencing |
| | org.apache.hadoop.hbase.wal.TestWALSplit |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.0 Server=1.12.0 Image:yetus/hbase:date2016-08-11 |
| JIRA Patch URL |
[jira] [Created] (HBASE-16394) Compaction is trying to add a bad range
qgxiaozhan created HBASE-16394: -- Summary: Compaction is trying to add a bad range Key: HBASE-16394 URL: https://issues.apache.org/jira/browse/HBASE-16394 Project: HBase Issue Type: Bug Components: Compaction Affects Versions: 1.1.2 Environment: hadoop-2.6.1 hbase-1.1.2 Reporter: qgxiaozhan My cluster dead because of "Compaction is trying to add a bad range" Here the log: [2016-08-09T18:30:19.094+08:00] [INFO] regionserver.ReplicationSource : Log hdfs://athene/hbase/oldWALs/MJQ-HBASE-ATHENE-11139%2C16020%2C1470729882622.default.1470736608897 was moved to hdfs://athene/hbase/oldWA Ls/MJQ-HBASE-ATHENE%2C16020%2C1470729882622.default.1470736608897 [2016-08-09T18:30:30.225+08:00] [INFO] regionserver.MemStoreFlusher : Waited 90070ms on a compaction to clean up 'TOO MANY STORE FILES'; waited long enough... proceeding with flush of tjs4:popt_info,160608008474430,147073716071 1.7900baab5204e4f36fa49379c30cd584. [2016-08-09T18:30:30.226+08:00] [INFO] regionserver.HRegion : Started memstore flush for tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584., current region memstore size 769.41 MB, and 1/1 column fam ilies' memstores are being flushed. [2016-08-09T18:30:30.549+08:00] [INFO] regionserver.StripeStoreFileManager : 3 conflicting files (likely created by a flush) of size 156153021 are moved to L0 due to concurrent stripe change [2016-08-09T18:30:31.199+08:00] [INFO] regionserver.HStore : Completed compaction of 203 file(s) in c of tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584. into 20347d203d09442cac30c42b424adda6(size= 3.0 G), ded362eab9cf4a819675cd35992d4974(size=3.0 G), 281b1039ed2643679e5b0a3820f5059d(size=2.4 G), total size for store is 8.6 G. This selection was in queue for 0sec, and took 10mins, 16sec to execute. [2016-08-09T18:30:31.200+08:00] [INFO] regionserver.CompactSplitThread : Completed compaction: Request = regionName=tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584., storeName=c, fileCount=203, fil eSize=7.2 G, priority=-3, time=5388162916126535; duration=10mins, 16sec [2016-08-09T18:30:31.201+08:00] [INFO] regionserver.HRegion : Starting compaction on ci in region ad_union:union_click,3487f383ad484bcbb5cef727b69cec2a,1466484980245.c5772fc60c54f64cc977ba9cc01d74ad. [2016-08-09T18:30:31.201+08:00] [INFO] regionserver.HStore : Starting compaction of 14 file(s) in ci of ad_union:union_click,3487f383ad484bcbb5cef727b69cec2a,1466484980245.c5772fc60c54f64cc977ba9cc01d74ad. into tmpdir=hdfs://at hene/hbase/data/ad_union/union_click/c5772fc60c54f64cc977ba9cc01d74ad/.tmp, totalSize=75.0 M [2016-08-09T18:30:31.206+08:00] [INFO] hfile.CacheConfig : blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@52659482, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=fal se, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false [2016-08-09T18:30:32.893+08:00] [INFO] regionserver.ReplicationSource : Log hdfs://athene/hbase/oldWALs/MJQ-HBASE-ATHENE-11139l%2C16020%2C1470729882622.default.1470736612825 was moved to hdfs://athene/hbase/oldWA Ls/MJQ-HBASE-ATHENE-11139.%2C16020%2C1470729882622.default.1470736612825 [2016-08-09T18:30:34.373+08:00] [INFO] regionserver.HStore : Added hdfs://athene/hbase/data/tjs4/popt_info/7900baab5204e4f36fa49379c30cd584/c/775e8956cd2a48aaae70b9eded4457e9, entries=4336457, sequenceid=582528, filesize=48.7 M [2016-08-09T18:30:34.373+08:00] [FATAL] regionserver.HRegionServer : ABORTING region server MJQ-HBASE-ATHENE-11139.,16020,1470729882622: Replay of WAL required. Forcing server shutdown org.apache.hadoop.hbase.DroppedSnapshotException: region: tjs4:popt_info,160608008474430,1470737160711.7900baab5204e4f36fa49379c30cd584. at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2354) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2057) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2019) at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1911) at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1837) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:510) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:471) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$800(MemStoreFlusher.java:75) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:259) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Compaction is trying to add a bad range. at
[jira] [Commented] (HBASE-16267) Remove commons-httpclient dependency from hbase-rest module
[
https://issues.apache.org/jira/browse/HBASE-16267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416513#comment-15416513
]
Hadoop QA commented on HBASE-16267:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 21m 2s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 3s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 3m 46s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 22s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 45s
{color} | {color:green} master passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
24s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 36s
{color} | {color:red} hbase-rest in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 41s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 25s
{color} | {color:green} master passed with JDK v1.7.0_101 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 18s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 39s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
32m 40s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:red}-1{color} | {color:red} hbaseprotoc {color} | {color:red} 0m 14s
{color} | {color:red} root in the patch failed. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
21s {color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s {color} | {color:green} hbase-rest generated 0 new + 0 unchanged - 1 fixed
= 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 40s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} |
[jira] [Commented] (HBASE-16363) Correct javadoc for qualifier length and value length in Cell interface
[
https://issues.apache.org/jira/browse/HBASE-16363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416510#comment-15416510
]
Hudson commented on HBASE-16363:
FAILURE: Integrated in HBase-1.4 #350 (See
[https://builds.apache.org/job/HBase-1.4/350/])
HBASE-16363 Correct javadoc for qualifier length and value (Xiang Li) (stack:
rev cf9878f77e5176296c107e3850f8a2c01e48c812)
* hbase-common/src/main/java/org/apache/hadoop/hbase/Cell.java
> Correct javadoc for qualifier length and value length in Cell interface
> ---
>
> Key: HBASE-16363
> URL: https://issues.apache.org/jira/browse/HBASE-16363
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.2.3
>
> Attachments: HBASE-16363.master.000.patch
>
>
> In Cell interface
> (hbase-common/src/main/java/org/apache/hadoop/hbase/Cell.java)
> (1) We have the following comment for "qualifier"
> {code}
> /**
> * Contiguous raw bytes that may start at any index in the containing array.
> Max length is
> * Short.MAX_VALUE which is 32,767 bytes.
> * @return The array containing the qualifier bytes.
> */
> {code}
> But getQualifierLength() returns int
> {code}
> int getQualifierLength();
> {code}
> (2) We have the following comment for "value"
> {code}
> /**
>* Contiguous raw bytes that may start at any index in the containing
> array. Max length is
>* Integer.MAX_VALUE which is 2,147,483,648 bytes.
>* @return The array containing the value bytes.
>*/
> {code}
> Integer.MAX_VALUE is not 2,147,483,648 (2^31), but 2,147,483,647 (2^31-1)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16363) Correct javadoc for qualifier length and value length in Cell interface
[
https://issues.apache.org/jira/browse/HBASE-16363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416496#comment-15416496
]
Hudson commented on HBASE-16363:
FAILURE: Integrated in HBase-1.2 #700 (See
[https://builds.apache.org/job/HBase-1.2/700/])
HBASE-16363 Correct javadoc for qualifier length and value (Xiang Li) (stack:
rev 470667420caff382963cc7510b2f71d9b679a116)
* hbase-common/src/main/java/org/apache/hadoop/hbase/Cell.java
> Correct javadoc for qualifier length and value length in Cell interface
> ---
>
> Key: HBASE-16363
> URL: https://issues.apache.org/jira/browse/HBASE-16363
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.2.3
>
> Attachments: HBASE-16363.master.000.patch
>
>
> In Cell interface
> (hbase-common/src/main/java/org/apache/hadoop/hbase/Cell.java)
> (1) We have the following comment for "qualifier"
> {code}
> /**
> * Contiguous raw bytes that may start at any index in the containing array.
> Max length is
> * Short.MAX_VALUE which is 32,767 bytes.
> * @return The array containing the qualifier bytes.
> */
> {code}
> But getQualifierLength() returns int
> {code}
> int getQualifierLength();
> {code}
> (2) We have the following comment for "value"
> {code}
> /**
>* Contiguous raw bytes that may start at any index in the containing
> array. Max length is
>* Integer.MAX_VALUE which is 2,147,483,648 bytes.
>* @return The array containing the value bytes.
>*/
> {code}
> Integer.MAX_VALUE is not 2,147,483,648 (2^31), but 2,147,483,647 (2^31-1)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16363) Correct javadoc for qualifier length and value length in Cell interface
[
https://issues.apache.org/jira/browse/HBASE-16363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416485#comment-15416485
]
Hudson commented on HBASE-16363:
FAILURE: Integrated in HBase-1.2-IT #578 (See
[https://builds.apache.org/job/HBase-1.2-IT/578/])
HBASE-16363 Correct javadoc for qualifier length and value (Xiang Li) (stack:
rev 470667420caff382963cc7510b2f71d9b679a116)
* hbase-common/src/main/java/org/apache/hadoop/hbase/Cell.java
> Correct javadoc for qualifier length and value length in Cell interface
> ---
>
> Key: HBASE-16363
> URL: https://issues.apache.org/jira/browse/HBASE-16363
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.2.3
>
> Attachments: HBASE-16363.master.000.patch
>
>
> In Cell interface
> (hbase-common/src/main/java/org/apache/hadoop/hbase/Cell.java)
> (1) We have the following comment for "qualifier"
> {code}
> /**
> * Contiguous raw bytes that may start at any index in the containing array.
> Max length is
> * Short.MAX_VALUE which is 32,767 bytes.
> * @return The array containing the qualifier bytes.
> */
> {code}
> But getQualifierLength() returns int
> {code}
> int getQualifierLength();
> {code}
> (2) We have the following comment for "value"
> {code}
> /**
>* Contiguous raw bytes that may start at any index in the containing
> array. Max length is
>* Integer.MAX_VALUE which is 2,147,483,648 bytes.
>* @return The array containing the value bytes.
>*/
> {code}
> Integer.MAX_VALUE is not 2,147,483,648 (2^31), but 2,147,483,647 (2^31-1)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16267) Remove commons-httpclient dependency from hbase-rest module
[
https://issues.apache.org/jira/browse/HBASE-16267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416460#comment-15416460
]
Ted Yu commented on HBASE-16267:
I wonder what took root test(s) over two hours (4 hours since tests started - 2
hours in hbase-server).
>From https://builds.apache.org/job/PreCommit-HBASE-Build/3057/console :
mvn -Dmaven.repo.local=/home/jenkins/yetus-m2/hbase-master-patch-0
-DHBasePatchProcess -PrunAllTests clean test -fae >
/testptch/patchprocess/patch-unit-root.txt 2>&1
> Remove commons-httpclient dependency from hbase-rest module
> ---
>
> Key: HBASE-16267
> URL: https://issues.apache.org/jira/browse/HBASE-16267
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 16267.v10.txt, 16267.v11.txt, 16267.v12.txt,
> 16267.v13.txt, 16267.v14.txt, 16267.v15.txt, 16267.v2.txt, 16267.v4.txt,
> 16267.v6.txt, 16267.v8.txt, 16267.v9.txt
>
>
> hbase-rest module still has imports from org.apache.commons.httpclient .
> There is more work to be done after HBASE-15767 was integrated.
> In master branch, there seems to be transitive dependency which allows the
> code to compile:
> {code}
> [INFO] +- org.apache.hadoop:hadoop-common:jar:2.7.1:compile
> [INFO] | +- org.apache.hadoop:hadoop-annotations:jar:2.7.1:compile
> [INFO] | +- commons-cli:commons-cli:jar:1.2:compile
> [INFO] | +- org.apache.commons:commons-math3:jar:3.1.1:compile
> [INFO] | +- xmlenc:xmlenc:jar:0.52:compile
> [INFO] | +- commons-httpclient:commons-httpclient:jar:3.1:compile
> {code}
> HADOOP-12767
> to move the uses of httpclient HADOOP-10105
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-5262 :
> http/conn/ssl/SSLConnectionSocketFactory.java in Apache HttpComponents
> HttpClient before 4.3.6 ignores the http.socket.timeout configuration setting
> during an SSL handshake, which allows remote attackers to cause a denial of
> service (HTTPS call hang) via unspecified vectors.
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-6153
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-5783
> Apache Commons HttpClient 3.x, as used in Amazon Flexible Payments
> Service (FPS) merchant Java SDK and other products, does not verify that the
> server hostname matches a domain name in the subject's Common Name (CN) or
> subjectAltName field of the X.509 certificate, which allows man-in-the-middle
> attackers to spoof SSL servers via an arbitrary valid certificate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16388) Prevent client threads being blocked by only one slow region server
[
https://issues.apache.org/jira/browse/HBASE-16388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416456#comment-15416456
]
Phil Yang commented on HBASE-16388:
---
[~stack] Thanks for your reply.
{quote}
Where will you throttle the requests in the process?
{quote}
In a lower level, tend to AbstractRpcClient. Use a static guava cache(it helps
us cleaning up removed RS) to save AtomicLongs for each server address. One
question is whether we need differentiate RPC requests to master and to RS,
whether we need differentiate requests to meta table and user table. If not,
the logic will be simple. And I think the concurrent requests to master or meta
table is not very high, so a suitable threshold may not fail requests to master
server or meta table, right?
{quote}
do a queue per client on the regionserver
{quote}
Sounds reasonable, it can prevent server being "DDOS" by a client. And in this
issue we can prevent a client being blocked by a slow server. They are two
approaches that both can increase the availability in extreme case.
> Prevent client threads being blocked by only one slow region server
> ---
>
> Key: HBASE-16388
> URL: https://issues.apache.org/jira/browse/HBASE-16388
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
>
> It is a general use case for HBase's users that they have several
> threads/handlers in their service, and each handler has its own Table/HTable
> instance. Generally users think each handler is independent and won't
> interact each other.
> However, in an extreme case, if a region server is very slow, every requests
> to this RS will timeout, handlers of users' service may be occupied by the
> long-waiting requests even requests belong to other RS will also be timeout.
> For example:
> If we have 100 handlers in a client service(timeout is 1000ms) and HBase has
> 10 region servers whose average response time is 50ms. If no region server is
> slow, we can handle 2000 requests per second.
> Now this service's QPS is 1000. If there is one region server very slow and
> all requests to it will be timeout. Users hope that only 10% requests failed,
> and 90% requests' response time is still 50ms, because only 10% requests are
> located to the slow RS. However, each second we have 100 long-waiting
> requests which exactly occupies all 100 handles. So all handlers is blocked,
> the availability of this service is almost zero.
> To prevent this case, we can limit the max concurrent requests to one RS in
> process-level. Requests exceeding the limit will throws
> ServerBusyException(extends DoNotRetryIOE) immediately to users. In the above
> case, if we set this limit to 20, only 20 handlers will be occupied and other
> 80 handlers can still handle requests to other RS. The availability of this
> service is 90% as expected.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16389) Thread leak in CoprocessorHost#getTable(TableName) API
[
https://issues.apache.org/jira/browse/HBASE-16389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16389:
---
Resolution: Fixed
Assignee: Ankit Singhal
Hadoop Flags: Reviewed
Fix Version/s: 2.0.0
Status: Resolved (was: Patch Available)
Didn't find this method in branch-1.
Thanks for the patch, Ankit.
Thanks for the review, Stack.
> Thread leak in CoprocessorHost#getTable(TableName) API
> --
>
> Key: HBASE-16389
> URL: https://issues.apache.org/jira/browse/HBASE-16389
> Project: HBase
> Issue Type: Bug
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
> Fix For: 2.0.0
>
> Attachments: HBASE-16389.patch
>
>
> There is a thread leak in below method, we are creating a default pool while
> calling
> {code}
> @Override
> public HTableInterface getTable(TableName tableName) throws IOException {
> return this.getTable(tableName,
> HTable.getDefaultExecutor(getConfiguration()));
> }
> {code}
> which will never be shutdown because in HTable, we are setting
> this.cleanupPoolOnClose to false
> {code}
> @InterfaceAudience.Private
> public HTable(TableName tableName, final ClusterConnection connection,
> final TableConfiguration tableConfig,
> final RpcRetryingCallerFactory rpcCallerFactory,
> final RpcControllerFactory rpcControllerFactory,
> final ExecutorService pool) throws IOException {
> if (connection == null || connection.isClosed()) {
> throw new IllegalArgumentException("Connection is null or closed.");
> }
> this.tableName = tableName;
> this.cleanupConnectionOnClose = false;
> this.connection = connection;
> this.configuration = connection.getConfiguration();
> this.tableConfiguration = tableConfig;
> this.pool = pool;
> if (pool == null) {
> this.pool = getDefaultExecutor(this.configuration);
> this.cleanupPoolOnClose = true;
> } else {
> this.cleanupPoolOnClose = false;
> }
> {code}
> resulted in pool to stay forever ,which eventually can lead other processes
> or the same process to starve for a threads.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16363) Correct javadoc for qualifier length and value length in Cell interface
[
https://issues.apache.org/jira/browse/HBASE-16363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416438#comment-15416438
]
Hudson commented on HBASE-16363:
FAILURE: Integrated in HBase-1.3-IT #790 (See
[https://builds.apache.org/job/HBase-1.3-IT/790/])
HBASE-16363 Correct javadoc for qualifier length and value (Xiang Li) (stack:
rev 9aed64173b4f360e3029a0936ae986cd5c916062)
* hbase-common/src/main/java/org/apache/hadoop/hbase/Cell.java
> Correct javadoc for qualifier length and value length in Cell interface
> ---
>
> Key: HBASE-16363
> URL: https://issues.apache.org/jira/browse/HBASE-16363
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.2.3
>
> Attachments: HBASE-16363.master.000.patch
>
>
> In Cell interface
> (hbase-common/src/main/java/org/apache/hadoop/hbase/Cell.java)
> (1) We have the following comment for "qualifier"
> {code}
> /**
> * Contiguous raw bytes that may start at any index in the containing array.
> Max length is
> * Short.MAX_VALUE which is 32,767 bytes.
> * @return The array containing the qualifier bytes.
> */
> {code}
> But getQualifierLength() returns int
> {code}
> int getQualifierLength();
> {code}
> (2) We have the following comment for "value"
> {code}
> /**
>* Contiguous raw bytes that may start at any index in the containing
> array. Max length is
>* Integer.MAX_VALUE which is 2,147,483,648 bytes.
>* @return The array containing the value bytes.
>*/
> {code}
> Integer.MAX_VALUE is not 2,147,483,648 (2^31), but 2,147,483,647 (2^31-1)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16385) Have hbase-rest pull hbase.rest.port from Constants.java
[
https://issues.apache.org/jira/browse/HBASE-16385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416440#comment-15416440
]
Hudson commented on HBASE-16385:
FAILURE: Integrated in HBase-1.3-IT #790 (See
[https://builds.apache.org/job/HBase-1.3-IT/790/])
HBASE-16385 Have hbase-rest pull hbase.rest.port from Constants.java (Yi
(stack: rev 2831850de713d3e044f582394a2683a1b5da9a5f)
* hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/RESTServer.java
> Have hbase-rest pull hbase.rest.port from Constants.java
>
>
> Key: HBASE-16385
> URL: https://issues.apache.org/jira/browse/HBASE-16385
> Project: HBase
> Issue Type: Improvement
> Components: REST
>Reporter: Dima Spivak
>Assignee: Yi Liang
>Priority: Minor
> Labels: beginner
> Fix For: 2.0.0, 1.3.0, 1.2.3
>
> Attachments: HBASE-16385-V1.patch
>
>
> [A while back|https://issues.apache.org/jira/browse/HBASE-3272], we
> [hardcoded|https://github.com/apache/hbase/blob/master/hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/RESTServer.java#L266]
> the {{hbase.rest.port}} instead of getting it from
> [Constants.java|https://github.com/apache/hbase/blob/master/hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/Constants.java#L36].
> We should fix that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14345) Consolidate printUsage in IntegrationTestLoadAndVerify
[
https://issues.apache.org/jira/browse/HBASE-14345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416442#comment-15416442
]
Hudson commented on HBASE-14345:
FAILURE: Integrated in HBase-1.3-IT #790 (See
[https://builds.apache.org/job/HBase-1.3-IT/790/])
HBASE-14345 Consolidate printUsage in IntegrationTestLoadAndVerify (Reid (appy:
rev 2759b4a4a36b62f0d5595785ab5e91bd7a863a83)
*
hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestLoadAndVerify.java
*
hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestWithCellVisibilityLoadAndVerify.java
> Consolidate printUsage in IntegrationTestLoadAndVerify
> --
>
> Key: HBASE-14345
> URL: https://issues.apache.org/jira/browse/HBASE-14345
> Project: HBase
> Issue Type: Improvement
> Components: integration tests
>Reporter: Nick Dimiduk
>Assignee: Reid Chan
>Priority: Trivial
> Labels: beginner
> Fix For: 2.0.0, 1.3.0, 1.0.4, 1.4.0, 1.1.6, 1.2.3
>
> Attachments: HBASE-14345.002.patch, HBASE-14345.003.patch,
> HBASE-14345.004.patch, HBASE-14345.master.001.patch,
> HBASE-14345.master.002.patch, HBASE-14345.patch, itlav-2016-07-07.png,
> itlv.png
>
>
> Investigating the use of {{itlav}} is a little screwy. Subclasses are not
> overriding the {{printUsage()}} methods correctly, so you have to pass
> {{--help}} to get some info and no arguments to get the rest.
> {noformat}
> [hbase@ndimiduk-112rc2-7 ~]$ hbase
> org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify --help
> usage: bin/hbase org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify
>
> Options:
> -h,--help Show usage
> -m,--monkey Which chaos monkey to run
> -monkeyProps The properties file for specifying chaos monkey
> properties.
> -ncc,--noClusterCleanUp Don't clean up the cluster at the end
> [hbase@ndimiduk-112rc2-7 ~]$ hbase
> org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify
> IntegrationTestLoadAndVerify [-Doptions]
> Loads a table with row dependencies and verifies the dependency chains
> Options
> -Dloadmapper.table=Table to write/verify (default autogen)
> -Dloadmapper.backrefs=Number of backreferences per row (default
> 50)
> -Dloadmapper.num_to_write=Number of rows per mapper (default 100,000
> per mapper)
> -Dloadmapper.deleteAfter= Delete after a successful verify (default
> true)
> -Dloadmapper.numPresplits=Number of presplit regions to start with
> (default 40)
> -Dloadmapper.map.tasks= Number of map tasks for load (default 200)
> -Dverify.reduce.tasks=Number of reduce tasks for verify (default
> 35)
> -Dverify.scannercaching= Number hbase scanner caching rows to read
> (default 50)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16368) test*WhenRegionMove in TestPartialResultsFromClientSide is flaky
[
https://issues.apache.org/jira/browse/HBASE-16368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416441#comment-15416441
]
Hudson commented on HBASE-16368:
FAILURE: Integrated in HBase-1.3-IT #790 (See
[https://builds.apache.org/job/HBase-1.3-IT/790/])
HBASE-16368 test*WhenRegionMove in TestPartialResultsFromClientSide is
(zhangduo: rev 8f5b70ac3235a463708245ac83baa23ddc5b5867)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/TestPartialResultsFromClientSide.java
> test*WhenRegionMove in TestPartialResultsFromClientSide is flaky
>
>
> Key: HBASE-16368
> URL: https://issues.apache.org/jira/browse/HBASE-16368
> Project: HBase
> Issue Type: Bug
> Components: Scanners
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.5, 1.2.2
>Reporter: Guanghao Zhang
>Assignee: Phil Yang
> Labels: trivial
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.6, 1.2.3
>
> Attachments: HBASE-16368-v1.patch, HBASE-16368-v1.patch,
> HBASE-16368-v1.patch
>
>
> This test fail when Hadoop QA run preCommit:
> https://builds.apache.org/job/PreCommit-HBASE-Build/2971/testReport/org.apache.hadoop.hbase/TestPartialResultsFromClientSide/testReversedCompleteResultWhenRegionMove/.
> And I found it is in Flaky Tests Dashboard:
> http://hbase.x10host.com/flaky-tests/. I run it in my local machine and it
> may fail, too.
> Test results show that the region location didn't update when scanner
> callable get a NotServingRegionException or RegionMovedException.
> {code}
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=36, exceptions:
> Sat Aug 06 05:55:52 UTC 2016, null, java.net.SocketTimeoutException:
> callTimeout=2000, callDuration=2157:
> org.apache.hadoop.hbase.NotServingRegionException:
> testReversedCompleteResultWhenRegionMove,,1470462949504.5069bd63bf6eda5108acec4fcc087b0e.
> is closing
> at
> org.apache.hadoop.hbase.regionserver.HRegion.startRegionOperation(HRegion.java:8233)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2634)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2629)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2623)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2490)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34950)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2264)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:118)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:189)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:169)
> row '' on table 'testReversedCompleteResultWhenRegionMove' at
> region=testReversedCompleteResultWhenRegionMove,,1470462949504.5069bd63bf6eda5108acec4fcc087b0e.,
> hostname=asf907.gq1.ygridcore.net,38914,1470462943053, seqNum=2
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:281)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:213)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:61)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:212)
> at
> org.apache.hadoop.hbase.client.ReversedClientScanner.nextScanner(ReversedClientScanner.java:118)
> at
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:166)
> at
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:161)
> at
> org.apache.hadoop.hbase.client.ReversedClientScanner.(ReversedClientScanner.java:56)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:785)
> at
> org.apache.hadoop.hbase.TestPartialResultsFromClientSide.testReversedCompleteResultWhenRegionMove(TestPartialResultsFromClientSide.java:986)
> {code}
> {code}
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=36, exceptions:
> Sat Aug 06 16:27:22 CST 2016, null, java.net.SocketTimeoutException:
> callTimeout=2000, callDuration=3035: Region moved to: hostname=localhost
> port=58351 startCode=1470472007714. As of locationSeqNum=6. row 'testRow0' on
> table 'testPartialResultWhenRegionMove' at
> region=testPartialResultWhenRegionMove,,1470472035048.977faf05c1d6d9990b5559b17aa18913.,
> hostname=localhost,40425,1470472007646, seqNum=2
> at
>
[jira] [Commented] (HBASE-16285) Drop RPC requests if it must be considered as timeout at client
[ https://issues.apache.org/jira/browse/HBASE-16285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416439#comment-15416439 ] Hudson commented on HBASE-16285: FAILURE: Integrated in HBase-1.3-IT #790 (See [https://builds.apache.org/job/HBase-1.3-IT/790/]) HBASE-16285 Drop RPC requests if it must be considered as timeout at (zhangduo: rev 974dab0f03c5e3db3234a7425c532157b660b10e) * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServerInterface.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/CallRunner.java > Drop RPC requests if it must be considered as timeout at client > --- > > Key: HBASE-16285 > URL: https://issues.apache.org/jira/browse/HBASE-16285 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0, 1.3.0, 1.4.0 >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-16285-branch-1-v1.patch, > HBASE-16285-branch-1-v2.patch, HBASE-16285-branch-1-v3.patch, > HBASE-16285-branch-1-v4.patch, HBASE-16285-branch-1-v5.patch, > HBASE-16285-branch-1-v5.patch, HBASE-16285-branch-1-v6.patch, > HBASE-16285-v1.patch, HBASE-16285-v2.patch, HBASE-16285-v3.patch, > HBASE-16285-v4.patch, HBASE-16285-v5.patch, HBASE-16285-v6.patch, > HBASE-16285-v7.patch, HBASE-16285-v7.patch, HBASE-16285-v8.patch, > HBASE-16285-v9.patch > > > After HBASE-15593, we have a timeout param in header of RPC requests. We can > use it in more scenes. > A straightforward scene is to drop requests if it has waited so long in RPC > queue and has been dropped by client. Even if we handle this request and send > the response back, it will not be used any more. And client may have sent a > retry. In an extreme case, if the server is slow, all requests may be timeout > or queue-full-exception because we should handle previous requests which have > been dropped by client and many resources at server are wasted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16379) [replication] Minor improvement to replication/copy_tables_desc.rb
[ https://issues.apache.org/jira/browse/HBASE-16379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416443#comment-15416443 ] Hudson commented on HBASE-16379: FAILURE: Integrated in HBase-1.3-IT #790 (See [https://builds.apache.org/job/HBase-1.3-IT/790/]) HBASE-16379 [replication] Minor improvement to (esteban: rev 66c23e37003a97778840f12cb25f65f51ce677fe) * bin/replication/copy_tables_desc.rb > [replication] Minor improvement to replication/copy_tables_desc.rb > -- > > Key: HBASE-16379 > URL: https://issues.apache.org/jira/browse/HBASE-16379 > Project: HBase > Issue Type: Improvement > Components: Replication, shell >Affects Versions: 1.3.0, 1.2.2 >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez >Priority: Trivial > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3, 1.1.7 > > Attachments: > 0001-HBASE-16379-replication-Minor-improvement-to-replica.patch, > 0001-HBASE-16379-replication-Minor-improvement-to-replica.patch, > 0001-HBASE-16379-replication-Minor-improvement-to-replica.patch, > HBASE-16379.patch > > > copy_tables_desc.rb is helpful for quickly setting up a table remotely based > on an existing schema. However it does copy by default all tables. Now you > can pass a list of tables as an optional third argument and it will also > display what table descriptors where copied. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16389) Thread leak in CoprocessorHost#getTable(TableName) API
[
https://issues.apache.org/jira/browse/HBASE-16389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16389:
---
Summary: Thread leak in CoprocessorHost#getTable(TableName) API (was:
Thread leak in CoprocessorHost#getTable(TableName tableName) API)
> Thread leak in CoprocessorHost#getTable(TableName) API
> --
>
> Key: HBASE-16389
> URL: https://issues.apache.org/jira/browse/HBASE-16389
> Project: HBase
> Issue Type: Bug
>Reporter: Ankit Singhal
> Attachments: HBASE-16389.patch
>
>
> There is a thread leak in below method, we are creating a default pool while
> calling
> {code}
> @Override
> public HTableInterface getTable(TableName tableName) throws IOException {
> return this.getTable(tableName,
> HTable.getDefaultExecutor(getConfiguration()));
> }
> {code}
> which will never be shutdown because in HTable, we are setting
> this.cleanupPoolOnClose to false
> {code}
> @InterfaceAudience.Private
> public HTable(TableName tableName, final ClusterConnection connection,
> final TableConfiguration tableConfig,
> final RpcRetryingCallerFactory rpcCallerFactory,
> final RpcControllerFactory rpcControllerFactory,
> final ExecutorService pool) throws IOException {
> if (connection == null || connection.isClosed()) {
> throw new IllegalArgumentException("Connection is null or closed.");
> }
> this.tableName = tableName;
> this.cleanupConnectionOnClose = false;
> this.connection = connection;
> this.configuration = connection.getConfiguration();
> this.tableConfiguration = tableConfig;
> this.pool = pool;
> if (pool == null) {
> this.pool = getDefaultExecutor(this.configuration);
> this.cleanupPoolOnClose = true;
> } else {
> this.cleanupPoolOnClose = false;
> }
> {code}
> resulted in pool to stay forever ,which eventually can lead other processes
> or the same process to starve for a threads.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16389) Thread leak in CoprocessorHost#getTable(TableName tableName) API
[
https://issues.apache.org/jira/browse/HBASE-16389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416430#comment-15416430
]
Hadoop QA commented on HBASE-16389:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} master passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 49s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 101m 22s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 143m 47s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-08-11 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12823068/HBASE-16389.patch |
| JIRA Issue | HBASE-16389 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 1c68460a6634 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-16388) Prevent client threads being blocked by only one slow region server
[ https://issues.apache.org/jira/browse/HBASE-16388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416419#comment-15416419 ] binlijin commented on HBASE-16388: -- May be we can using NettyRpcServer with it more easy by just adding a throttle Handler? Use netty we can control write to client and read from client. http://normanmaurer.me/presentations/2014-facebook-eng-netty/slides.html#9.0 http://normanmaurer.me/presentations/2014-facebook-eng-netty/slides.html#33.0 > Prevent client threads being blocked by only one slow region server > --- > > Key: HBASE-16388 > URL: https://issues.apache.org/jira/browse/HBASE-16388 > Project: HBase > Issue Type: New Feature >Reporter: Phil Yang >Assignee: Phil Yang > > It is a general use case for HBase's users that they have several > threads/handlers in their service, and each handler has its own Table/HTable > instance. Generally users think each handler is independent and won't > interact each other. > However, in an extreme case, if a region server is very slow, every requests > to this RS will timeout, handlers of users' service may be occupied by the > long-waiting requests even requests belong to other RS will also be timeout. > For example: > If we have 100 handlers in a client service(timeout is 1000ms) and HBase has > 10 region servers whose average response time is 50ms. If no region server is > slow, we can handle 2000 requests per second. > Now this service's QPS is 1000. If there is one region server very slow and > all requests to it will be timeout. Users hope that only 10% requests failed, > and 90% requests' response time is still 50ms, because only 10% requests are > located to the slow RS. However, each second we have 100 long-waiting > requests which exactly occupies all 100 handles. So all handlers is blocked, > the availability of this service is almost zero. > To prevent this case, we can limit the max concurrent requests to one RS in > process-level. Requests exceeding the limit will throws > ServerBusyException(extends DoNotRetryIOE) immediately to users. In the above > case, if we set this limit to 20, only 20 handlers will be occupied and other > 80 handlers can still handle requests to other RS. The availability of this > service is 90% as expected. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16363) Correct javadoc for qualifier length and value length in Cell interface
[
https://issues.apache.org/jira/browse/HBASE-16363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416391#comment-15416391
]
Hudson commented on HBASE-16363:
FAILURE: Integrated in HBase-Trunk_matrix #1393 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1393/])
HBASE-16363 Correct javadoc for qualifier length and value (Xiang Li) (stack:
rev 7e3251051313d6ef8fd173c7b530c1822667410e)
* hbase-common/src/main/java/org/apache/hadoop/hbase/Cell.java
> Correct javadoc for qualifier length and value length in Cell interface
> ---
>
> Key: HBASE-16363
> URL: https://issues.apache.org/jira/browse/HBASE-16363
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.2.3
>
> Attachments: HBASE-16363.master.000.patch
>
>
> In Cell interface
> (hbase-common/src/main/java/org/apache/hadoop/hbase/Cell.java)
> (1) We have the following comment for "qualifier"
> {code}
> /**
> * Contiguous raw bytes that may start at any index in the containing array.
> Max length is
> * Short.MAX_VALUE which is 32,767 bytes.
> * @return The array containing the qualifier bytes.
> */
> {code}
> But getQualifierLength() returns int
> {code}
> int getQualifierLength();
> {code}
> (2) We have the following comment for "value"
> {code}
> /**
>* Contiguous raw bytes that may start at any index in the containing
> array. Max length is
>* Integer.MAX_VALUE which is 2,147,483,648 bytes.
>* @return The array containing the value bytes.
>*/
> {code}
> Integer.MAX_VALUE is not 2,147,483,648 (2^31), but 2,147,483,647 (2^31-1)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16385) Have hbase-rest pull hbase.rest.port from Constants.java
[
https://issues.apache.org/jira/browse/HBASE-16385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416392#comment-15416392
]
Hudson commented on HBASE-16385:
FAILURE: Integrated in HBase-Trunk_matrix #1393 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1393/])
HBASE-16385 Have hbase-rest pull hbase.rest.port from Constants.java (Yi
(stack: rev 88956676d7e7164f3a5b0dfbb31ad67bd3d5ed87)
* hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/RESTServer.java
> Have hbase-rest pull hbase.rest.port from Constants.java
>
>
> Key: HBASE-16385
> URL: https://issues.apache.org/jira/browse/HBASE-16385
> Project: HBase
> Issue Type: Improvement
> Components: REST
>Reporter: Dima Spivak
>Assignee: Yi Liang
>Priority: Minor
> Labels: beginner
> Fix For: 2.0.0, 1.3.0, 1.2.3
>
> Attachments: HBASE-16385-V1.patch
>
>
> [A while back|https://issues.apache.org/jira/browse/HBASE-3272], we
> [hardcoded|https://github.com/apache/hbase/blob/master/hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/RESTServer.java#L266]
> the {{hbase.rest.port}} instead of getting it from
> [Constants.java|https://github.com/apache/hbase/blob/master/hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/Constants.java#L36].
> We should fix that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16393) Improve computeHDFSBlocksDistribution
[ https://issues.apache.org/jira/browse/HBASE-16393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16393: --- Status: Patch Available (was: Open) > Improve computeHDFSBlocksDistribution > - > > Key: HBASE-16393 > URL: https://issues.apache.org/jira/browse/HBASE-16393 > Project: HBase > Issue Type: Improvement >Reporter: binlijin > Attachments: HBASE-16393.patch > > > With our cluster is big, i can see the balancer is slow from time to time. > And the balancer will be called on master startup, so we can see the startup > is slow also. > The first thing i think whether if we can parallel compute different region's > HDFSBlocksDistribution. > The second i think we can improve compute single region's > HDFSBlocksDistribution. > When to compute a storefile's HDFSBlocksDistribution first we call > FileSystem#getFileStatus(path) and then > FileSystem#getFileBlockLocations(status, start, length), so two namenode rpc > call for every storefile. Instead we can use FileSystem#listLocatedStatus to > get a LocatedFileStatus for the information we need, so reduce the namenode > rpc call to one. This can speed the computeHDFSBlocksDistribution, but also > send out less rpc call to namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16393) Improve computeHDFSBlocksDistribution
[ https://issues.apache.org/jira/browse/HBASE-16393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416327#comment-15416327 ] Guanghao Zhang commented on HBASE-16393: +1 on this idea. We found this in our production cluster, too. The balancer is too slow when there are a lot of regions. And some default balancer configs is too small for big cluster. Maybe we can make the default config value related to regions number. > Improve computeHDFSBlocksDistribution > - > > Key: HBASE-16393 > URL: https://issues.apache.org/jira/browse/HBASE-16393 > Project: HBase > Issue Type: Improvement >Reporter: binlijin > Attachments: HBASE-16393.patch > > > With our cluster is big, i can see the balancer is slow from time to time. > And the balancer will be called on master startup, so we can see the startup > is slow also. > The first thing i think whether if we can parallel compute different region's > HDFSBlocksDistribution. > The second i think we can improve compute single region's > HDFSBlocksDistribution. > When to compute a storefile's HDFSBlocksDistribution first we call > FileSystem#getFileStatus(path) and then > FileSystem#getFileBlockLocations(status, start, length), so two namenode rpc > call for every storefile. Instead we can use FileSystem#listLocatedStatus to > get a LocatedFileStatus for the information we need, so reduce the namenode > rpc call to one. This can speed the computeHDFSBlocksDistribution, but also > send out less rpc call to namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14345) Consolidate printUsage in IntegrationTestLoadAndVerify
[
https://issues.apache.org/jira/browse/HBASE-14345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416316#comment-15416316
]
Hudson commented on HBASE-14345:
SUCCESS: Integrated in HBase-1.1-JDK7 #1763 (See
[https://builds.apache.org/job/HBase-1.1-JDK7/1763/])
HBASE-14345 Consolidate printUsage in IntegrationTestLoadAndVerify (Reid (appy:
rev e5697bdb1bc63e832db57542420181348605b475)
*
hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestLoadAndVerify.java
*
hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestWithCellVisibilityLoadAndVerify.java
> Consolidate printUsage in IntegrationTestLoadAndVerify
> --
>
> Key: HBASE-14345
> URL: https://issues.apache.org/jira/browse/HBASE-14345
> Project: HBase
> Issue Type: Improvement
> Components: integration tests
>Reporter: Nick Dimiduk
>Assignee: Reid Chan
>Priority: Trivial
> Labels: beginner
> Fix For: 2.0.0, 1.3.0, 1.0.4, 1.4.0, 1.1.6, 1.2.3
>
> Attachments: HBASE-14345.002.patch, HBASE-14345.003.patch,
> HBASE-14345.004.patch, HBASE-14345.master.001.patch,
> HBASE-14345.master.002.patch, HBASE-14345.patch, itlav-2016-07-07.png,
> itlv.png
>
>
> Investigating the use of {{itlav}} is a little screwy. Subclasses are not
> overriding the {{printUsage()}} methods correctly, so you have to pass
> {{--help}} to get some info and no arguments to get the rest.
> {noformat}
> [hbase@ndimiduk-112rc2-7 ~]$ hbase
> org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify --help
> usage: bin/hbase org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify
>
> Options:
> -h,--help Show usage
> -m,--monkey Which chaos monkey to run
> -monkeyProps The properties file for specifying chaos monkey
> properties.
> -ncc,--noClusterCleanUp Don't clean up the cluster at the end
> [hbase@ndimiduk-112rc2-7 ~]$ hbase
> org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify
> IntegrationTestLoadAndVerify [-Doptions]
> Loads a table with row dependencies and verifies the dependency chains
> Options
> -Dloadmapper.table=Table to write/verify (default autogen)
> -Dloadmapper.backrefs=Number of backreferences per row (default
> 50)
> -Dloadmapper.num_to_write=Number of rows per mapper (default 100,000
> per mapper)
> -Dloadmapper.deleteAfter= Delete after a successful verify (default
> true)
> -Dloadmapper.numPresplits=Number of presplit regions to start with
> (default 40)
> -Dloadmapper.map.tasks= Number of map tasks for load (default 200)
> -Dverify.reduce.tasks=Number of reduce tasks for verify (default
> 35)
> -Dverify.scannercaching= Number hbase scanner caching rows to read
> (default 50)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16379) [replication] Minor improvement to replication/copy_tables_desc.rb
[ https://issues.apache.org/jira/browse/HBASE-16379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416317#comment-15416317 ] Hudson commented on HBASE-16379: SUCCESS: Integrated in HBase-1.1-JDK7 #1763 (See [https://builds.apache.org/job/HBase-1.1-JDK7/1763/]) HBASE-16379 [replication] Minor improvement to (esteban: rev 3259f03a592b42abe05ae1b89540b15ee4245cbe) * bin/replication/copy_tables_desc.rb > [replication] Minor improvement to replication/copy_tables_desc.rb > -- > > Key: HBASE-16379 > URL: https://issues.apache.org/jira/browse/HBASE-16379 > Project: HBase > Issue Type: Improvement > Components: Replication, shell >Affects Versions: 1.3.0, 1.2.2 >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez >Priority: Trivial > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3, 1.1.7 > > Attachments: > 0001-HBASE-16379-replication-Minor-improvement-to-replica.patch, > 0001-HBASE-16379-replication-Minor-improvement-to-replica.patch, > 0001-HBASE-16379-replication-Minor-improvement-to-replica.patch, > HBASE-16379.patch > > > copy_tables_desc.rb is helpful for quickly setting up a table remotely based > on an existing schema. However it does copy by default all tables. Now you > can pass a list of tables as an optional third argument and it will also > display what table descriptors where copied. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16393) Improve computeHDFSBlocksDistribution
[ https://issues.apache.org/jira/browse/HBASE-16393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416311#comment-15416311 ] binlijin commented on HBASE-16393: -- I do not get the numbers yet, will do it. > Improve computeHDFSBlocksDistribution > - > > Key: HBASE-16393 > URL: https://issues.apache.org/jira/browse/HBASE-16393 > Project: HBase > Issue Type: Improvement >Reporter: binlijin > Attachments: HBASE-16393.patch > > > With our cluster is big, i can see the balancer is slow from time to time. > And the balancer will be called on master startup, so we can see the startup > is slow also. > The first thing i think whether if we can parallel compute different region's > HDFSBlocksDistribution. > The second i think we can improve compute single region's > HDFSBlocksDistribution. > When to compute a storefile's HDFSBlocksDistribution first we call > FileSystem#getFileStatus(path) and then > FileSystem#getFileBlockLocations(status, start, length), so two namenode rpc > call for every storefile. Instead we can use FileSystem#listLocatedStatus to > get a LocatedFileStatus for the information we need, so reduce the namenode > rpc call to one. This can speed the computeHDFSBlocksDistribution, but also > send out less rpc call to namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16393) Improve computeHDFSBlocksDistribution
[ https://issues.apache.org/jira/browse/HBASE-16393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416302#comment-15416302 ] binlijin edited comment on HBASE-16393 at 8/11/16 1:00 AM: --- I know there is other place can improve by the same way, first improve StoreFileInfo#computeHDFSBlocksDistribution. I like to improve other places by subtask. was (Author: aoxiang): I know there is other place can improve by the same way, first improve StoreFileInfo#computeHDFSBlocksDistribution. > Improve computeHDFSBlocksDistribution > - > > Key: HBASE-16393 > URL: https://issues.apache.org/jira/browse/HBASE-16393 > Project: HBase > Issue Type: Improvement >Reporter: binlijin > Attachments: HBASE-16393.patch > > > With our cluster is big, i can see the balancer is slow from time to time. > And the balancer will be called on master startup, so we can see the startup > is slow also. > The first thing i think whether if we can parallel compute different region's > HDFSBlocksDistribution. > The second i think we can improve compute single region's > HDFSBlocksDistribution. > When to compute a storefile's HDFSBlocksDistribution first we call > FileSystem#getFileStatus(path) and then > FileSystem#getFileBlockLocations(status, start, length), so two namenode rpc > call for every storefile. Instead we can use FileSystem#listLocatedStatus to > get a LocatedFileStatus for the information we need, so reduce the namenode > rpc call to one. This can speed the computeHDFSBlocksDistribution, but also > send out less rpc call to namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16393) Improve computeHDFSBlocksDistribution
[ https://issues.apache.org/jira/browse/HBASE-16393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416309#comment-15416309 ] Ted Yu commented on HBASE-16393: Do you have performance improvement comparison with vs. without the patch ? Thanks > Improve computeHDFSBlocksDistribution > - > > Key: HBASE-16393 > URL: https://issues.apache.org/jira/browse/HBASE-16393 > Project: HBase > Issue Type: Improvement >Reporter: binlijin > Attachments: HBASE-16393.patch > > > With our cluster is big, i can see the balancer is slow from time to time. > And the balancer will be called on master startup, so we can see the startup > is slow also. > The first thing i think whether if we can parallel compute different region's > HDFSBlocksDistribution. > The second i think we can improve compute single region's > HDFSBlocksDistribution. > When to compute a storefile's HDFSBlocksDistribution first we call > FileSystem#getFileStatus(path) and then > FileSystem#getFileBlockLocations(status, start, length), so two namenode rpc > call for every storefile. Instead we can use FileSystem#listLocatedStatus to > get a LocatedFileStatus for the information we need, so reduce the namenode > rpc call to one. This can speed the computeHDFSBlocksDistribution, but also > send out less rpc call to namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16363) Correct javadoc for qualifier length and value length in Cell interface
[
https://issues.apache.org/jira/browse/HBASE-16363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-16363:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 1.2.3
1.3.0
2.0.0
Status: Resolved (was: Patch Available)
Nice. Thanks [~Octivian]
> Correct javadoc for qualifier length and value length in Cell interface
> ---
>
> Key: HBASE-16363
> URL: https://issues.apache.org/jira/browse/HBASE-16363
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Reporter: Xiang Li
>Assignee: Xiang Li
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.2.3
>
> Attachments: HBASE-16363.master.000.patch
>
>
> In Cell interface
> (hbase-common/src/main/java/org/apache/hadoop/hbase/Cell.java)
> (1) We have the following comment for "qualifier"
> {code}
> /**
> * Contiguous raw bytes that may start at any index in the containing array.
> Max length is
> * Short.MAX_VALUE which is 32,767 bytes.
> * @return The array containing the qualifier bytes.
> */
> {code}
> But getQualifierLength() returns int
> {code}
> int getQualifierLength();
> {code}
> (2) We have the following comment for "value"
> {code}
> /**
>* Contiguous raw bytes that may start at any index in the containing
> array. Max length is
>* Integer.MAX_VALUE which is 2,147,483,648 bytes.
>* @return The array containing the value bytes.
>*/
> {code}
> Integer.MAX_VALUE is not 2,147,483,648 (2^31), but 2,147,483,647 (2^31-1)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16254) hbase:backup table should have proper description in system table UI
[ https://issues.apache.org/jira/browse/HBASE-16254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16254: --- Labels: backup ui (was: backup) > hbase:backup table should have proper description in system table UI > > > Key: HBASE-16254 > URL: https://issues.apache.org/jira/browse/HBASE-16254 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Priority: Minor > Labels: backup, ui > > In MasterStatusTmpl.jamon, there is handling for system tables except for > hbase:backup. > We should add description for hbase:backup table as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16393) Improve computeHDFSBlocksDistribution
[ https://issues.apache.org/jira/browse/HBASE-16393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] binlijin updated HBASE-16393: - Description: With our cluster is big, i can see the balancer is slow from time to time. And the balancer will be called on master startup, so we can see the startup is slow also. The first thing i think whether if we can parallel compute different region's HDFSBlocksDistribution. The second i think we can improve compute single region's HDFSBlocksDistribution. When to compute a storefile's HDFSBlocksDistribution first we call FileSystem#getFileStatus(path) and then FileSystem#getFileBlockLocations(status, start, length), so two namenode rpc call for every storefile. Instead we can use FileSystem#listLocatedStatus to get a LocatedFileStatus for the information we need, so reduce the namenode rpc call to one. This can speed the computeHDFSBlocksDistribution, but also send out less rpc call to namenode. was: With our cluster is big, i can see the balancer is slow from time to time. And the balancer will be called on master startup, so we can see the startup is slow also. The first thing i think whether if we can parallel compute different region's HDFSBlocksDistribution. The second i think we can improve compute single region's HDFSBlocksDistribution. When to compute a storefile's HDFSBlocksDistribution first we call FileSystem#getFileStatus(path) and then FileSystem#getFileBlockLocations(status, start, length), so two namenode rpc call for every storefile. Instead we can use FileSystem#listLocatedStatus to get a LocatedFileStatus for the information we need, so reduce the namenode rpc call to one. > Improve computeHDFSBlocksDistribution > - > > Key: HBASE-16393 > URL: https://issues.apache.org/jira/browse/HBASE-16393 > Project: HBase > Issue Type: Improvement >Reporter: binlijin > Attachments: HBASE-16393.patch > > > With our cluster is big, i can see the balancer is slow from time to time. > And the balancer will be called on master startup, so we can see the startup > is slow also. > The first thing i think whether if we can parallel compute different region's > HDFSBlocksDistribution. > The second i think we can improve compute single region's > HDFSBlocksDistribution. > When to compute a storefile's HDFSBlocksDistribution first we call > FileSystem#getFileStatus(path) and then > FileSystem#getFileBlockLocations(status, start, length), so two namenode rpc > call for every storefile. Instead we can use FileSystem#listLocatedStatus to > get a LocatedFileStatus for the information we need, so reduce the namenode > rpc call to one. This can speed the computeHDFSBlocksDistribution, but also > send out less rpc call to namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16347) Unevaluated expressions in book
[
https://issues.apache.org/jira/browse/HBASE-16347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416303#comment-15416303
]
Nick Dimiduk commented on HBASE-16347:
--
Looks like the change has been pushed. Thank you mysterious site-pushing people
:)
> Unevaluated expressions in book
> ---
>
> Key: HBASE-16347
> URL: https://issues.apache.org/jira/browse/HBASE-16347
> Project: HBase
> Issue Type: Bug
> Components: documentation, site
>Reporter: Nick Dimiduk
>Assignee: Dima Spivak
> Fix For: 2.0.0
>
> Attachments: HBASE-16347.patch, after-patch.png
>
>
> Have a look at the [quickstart
> guide|http://hbase.apache.org/book.html#quickstart], step two
> {noformat}
> $ tar xzvf hbase--bin.tar.gz
> $ cd hbase-/
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16393) Improve computeHDFSBlocksDistribution
[ https://issues.apache.org/jira/browse/HBASE-16393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416302#comment-15416302 ] binlijin commented on HBASE-16393: -- I know there is other place can improve by the same way, first improve StoreFileInfo#computeHDFSBlocksDistribution. > Improve computeHDFSBlocksDistribution > - > > Key: HBASE-16393 > URL: https://issues.apache.org/jira/browse/HBASE-16393 > Project: HBase > Issue Type: Improvement >Reporter: binlijin > Attachments: HBASE-16393.patch > > > With our cluster is big, i can see the balancer is slow from time to time. > And the balancer will be called on master startup, so we can see the startup > is slow also. > The first thing i think whether if we can parallel compute different region's > HDFSBlocksDistribution. > The second i think we can improve compute single region's > HDFSBlocksDistribution. > When to compute a storefile's HDFSBlocksDistribution first we call > FileSystem#getFileStatus(path) and then > FileSystem#getFileBlockLocations(status, start, length), so two namenode rpc > call for every storefile. Instead we can use FileSystem#listLocatedStatus to > get a LocatedFileStatus for the information we need, so reduce the namenode > rpc call to one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16393) Improve computeHDFSBlocksDistribution
[ https://issues.apache.org/jira/browse/HBASE-16393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] binlijin updated HBASE-16393: - Attachment: HBASE-16393.patch > Improve computeHDFSBlocksDistribution > - > > Key: HBASE-16393 > URL: https://issues.apache.org/jira/browse/HBASE-16393 > Project: HBase > Issue Type: Improvement >Reporter: binlijin > Attachments: HBASE-16393.patch > > > With our cluster is big, i can see the balancer is slow from time to time. > And the balancer will be called on master startup, so we can see the startup > is slow also. > The first thing i think whether if we can parallel compute different region's > HDFSBlocksDistribution. > The second i think we can improve compute single region's > HDFSBlocksDistribution. > When to compute a storefile's HDFSBlocksDistribution first we call > FileSystem#getFileStatus(path) and then > FileSystem#getFileBlockLocations(status, start, length), so two namenode rpc > call for every storefile. Instead we can use FileSystem#listLocatedStatus to > get a LocatedFileStatus for the information we need, so reduce the namenode > rpc call to one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16393) Improve computeHDFSBlocksDistribution
[
https://issues.apache.org/jira/browse/HBASE-16393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416295#comment-15416295
]
binlijin commented on HBASE-16393:
--
Master balancer's jstack
{code}
hbase(main):002:0> balancer
ERROR: Call id=3, waitTime=180001, operationTimeout=18 expired.
{code}
{code}
"B.defaultRpcServer.handler=31,queue=5,port=60100" daemon prio=10
tid=0x7f3e2aec1800 nid=0x369b2 in Object.wait() [0x7f3e1affd000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at org.apache.hadoop.ipc.Client.call(Client.java:1484)
- locked <0x000603eb5738> (a org.apache.hadoop.ipc.Client$Call)
at org.apache.hadoop.ipc.Client.call(Client.java:1429)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy15.getBlockLocations(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:254)
at sun.reflect.GeneratedMethodAccessor75.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy16.getBlockLocations(Unknown Source)
at sun.reflect.GeneratedMethodAccessor75.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:330)
at com.sun.proxy.$Proxy17.getBlockLocations(Unknown Source)
at sun.reflect.GeneratedMethodAccessor75.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:330)
at com.sun.proxy.$Proxy17.getBlockLocations(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1205)
at
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1195)
at
org.apache.hadoop.hdfs.DFSClient.getBlockLocations(DFSClient.java:1245)
at
org.apache.hadoop.hdfs.DistributedFileSystem$1.doCall(DistributedFileSystem.java:220)
at
org.apache.hadoop.hdfs.DistributedFileSystem$1.doCall(DistributedFileSystem.java:216)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileBlockLocations(DistributedFileSystem.java:216)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileBlockLocations(DistributedFileSystem.java:208)
at
org.apache.hadoop.hbase.util.FSUtils.computeHDFSBlocksDistribution(FSUtils.java:1042)
at
org.apache.hadoop.hbase.regionserver.StoreFileInfo.computeHDFSBlocksDistributionInternal(StoreFileInfo.java:294)
at
org.apache.hadoop.hbase.regionserver.StoreFileInfo.computeHDFSBlocksDistribution(StoreFileInfo.java:284)
at
org.apache.hadoop.hbase.regionserver.HRegion.computeHDFSBlocksDistribution(HRegion.java:1083)
at
org.apache.hadoop.hbase.regionserver.HRegion.computeHDFSBlocksDistribution(HRegion.java:1058)
at
org.apache.hadoop.hbase.master.balancer.RegionLocationFinder.internalGetTopBlockLocation(RegionLocationFinder.java:127)
at
org.apache.hadoop.hbase.master.balancer.RegionLocationFinder$1.load(RegionLocationFinder.java:65)
at
org.apache.hadoop.hbase.master.balancer.RegionLocationFinder$1.load(RegionLocationFinder.java:61)
at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3584)
at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2372)
at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2335)
- locked <0x000603eabc40> (a
com.google.common.cache.LocalCache$StrongAccessEntry)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2250)
at com.google.common.cache.LocalCache.get(LocalCache.java:3985)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3989)
at
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4873)
at
org.apache.hadoop.hbase.master.balancer.RegionLocationFinder.getTopBlockLocations(RegionLocationFinder.java:105)
at
[jira] [Comment Edited] (HBASE-16393) Improve computeHDFSBlocksDistribution
[
https://issues.apache.org/jira/browse/HBASE-16393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416293#comment-15416293
]
binlijin edited comment on HBASE-16393 at 8/11/16 12:45 AM:
Master start up's jstack
{code}
"hostmaster:60100.activeMasterManager" daemon prio=10 tid=0x7f3df900a000
nid=0x36a51 in Object.wait() [0x7f3df86fc000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at org.apache.hadoop.ipc.Client.call(Client.java:1484)
- locked <0x0005fdaea6c8> (a org.apache.hadoop.ipc.Client$Call)
at org.apache.hadoop.ipc.Client.call(Client.java:1429)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy16.getFileInfo(Unknown Source)
at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:330)
at com.sun.proxy.$Proxy17.getFileInfo(Unknown Source)
at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:330)
at com.sun.proxy.$Proxy17.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1974)
at
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1128)
at
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
at
org.apache.hadoop.hbase.regionserver.StoreFileInfo.getReferencedFileStatus(StoreFileInfo.java:337)
at
org.apache.hadoop.hbase.regionserver.StoreFileInfo.computeHDFSBlocksDistributionInternal(StoreFileInfo.java:290)
at
org.apache.hadoop.hbase.regionserver.StoreFileInfo.computeHDFSBlocksDistribution(StoreFileInfo.java:284)
at
org.apache.hadoop.hbase.regionserver.HRegion.computeHDFSBlocksDistribution(HRegion.java:1083)
at
org.apache.hadoop.hbase.regionserver.HRegion.computeHDFSBlocksDistribution(HRegion.java:1058)
at
org.apache.hadoop.hbase.master.balancer.RegionLocationFinder.internalGetTopBlockLocation(RegionLocationFinder.java:127)
at
org.apache.hadoop.hbase.master.balancer.RegionLocationFinder$1.load(RegionLocationFinder.java:65)
at
org.apache.hadoop.hbase.master.balancer.RegionLocationFinder$1.load(RegionLocationFinder.java:61)
at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3584)
at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2372)
at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2335)
- locked <0x0005fda4e288> (a
com.google.common.cache.LocalCache$StrongAccessEntry)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2250)
at com.google.common.cache.LocalCache.get(LocalCache.java:3985)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3989)
at
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4873)
at
org.apache.hadoop.hbase.master.balancer.RegionLocationFinder.getTopBlockLocations(RegionLocationFinder.java:105)
at
org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.registerRegion(BaseLoadBalancer.java:433)
at
org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.(BaseLoadBalancer.java:281)
at
org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer.createCluster(BaseLoadBalancer.java:1098)
at
[jira] [Commented] (HBASE-16393) Improve computeHDFSBlocksDistribution
[
https://issues.apache.org/jira/browse/HBASE-16393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416293#comment-15416293
]
binlijin commented on HBASE-16393:
--
{code}
"hostmaster:60100.activeMasterManager" daemon prio=10 tid=0x7f3df900a000
nid=0x36a51 in Object.wait() [0x7f3df86fc000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at org.apache.hadoop.ipc.Client.call(Client.java:1484)
- locked <0x0005fdaea6c8> (a org.apache.hadoop.ipc.Client$Call)
at org.apache.hadoop.ipc.Client.call(Client.java:1429)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy15.getFileInfo(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy16.getFileInfo(Unknown Source)
at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:330)
at com.sun.proxy.$Proxy17.getFileInfo(Unknown Source)
at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:330)
at com.sun.proxy.$Proxy17.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1974)
at
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1128)
at
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
at
org.apache.hadoop.hbase.regionserver.StoreFileInfo.getReferencedFileStatus(StoreFileInfo.java:337)
at
org.apache.hadoop.hbase.regionserver.StoreFileInfo.computeHDFSBlocksDistributionInternal(StoreFileInfo.java:290)
at
org.apache.hadoop.hbase.regionserver.StoreFileInfo.computeHDFSBlocksDistribution(StoreFileInfo.java:284)
at
org.apache.hadoop.hbase.regionserver.HRegion.computeHDFSBlocksDistribution(HRegion.java:1083)
at
org.apache.hadoop.hbase.regionserver.HRegion.computeHDFSBlocksDistribution(HRegion.java:1058)
at
org.apache.hadoop.hbase.master.balancer.RegionLocationFinder.internalGetTopBlockLocation(RegionLocationFinder.java:127)
at
org.apache.hadoop.hbase.master.balancer.RegionLocationFinder$1.load(RegionLocationFinder.java:65)
at
org.apache.hadoop.hbase.master.balancer.RegionLocationFinder$1.load(RegionLocationFinder.java:61)
at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3584)
at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2372)
at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2335)
- locked <0x0005fda4e288> (a
com.google.common.cache.LocalCache$StrongAccessEntry)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2250)
at com.google.common.cache.LocalCache.get(LocalCache.java:3985)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3989)
at
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4873)
at
org.apache.hadoop.hbase.master.balancer.RegionLocationFinder.getTopBlockLocations(RegionLocationFinder.java:105)
at
org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.registerRegion(BaseLoadBalancer.java:433)
at
org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer$Cluster.(BaseLoadBalancer.java:281)
at
org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer.createCluster(BaseLoadBalancer.java:1098)
at
org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer.retainAssignment(BaseLoadBalancer.java:1235)
at
[jira] [Created] (HBASE-16393) Improve computeHDFSBlocksDistribution
binlijin created HBASE-16393: Summary: Improve computeHDFSBlocksDistribution Key: HBASE-16393 URL: https://issues.apache.org/jira/browse/HBASE-16393 Project: HBase Issue Type: Improvement Reporter: binlijin With our cluster is big, i can see the balancer is slow from time to time. And the balancer will be called on master startup, so we can see the startup is slow also. The first thing i think whether if we can parallel compute different region's HDFSBlocksDistribution. The second i think we can improve compute single region's HDFSBlocksDistribution. When to compute a storefile's HDFSBlocksDistribution first we call FileSystem#getFileStatus(path) and then FileSystem#getFileBlockLocations(status, start, length), so two namenode rpc call for every storefile. Instead we can use FileSystem#listLocatedStatus to get a LocatedFileStatus for the information we need, so reduce the namenode rpc call to one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16388) Prevent client threads being blocked by only one slow region server
[ https://issues.apache.org/jira/browse/HBASE-16388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416281#comment-15416281 ] stack commented on HBASE-16388: --- Tell us more [~yangzhe1991] Where will you throttle the requests in the process? Hopefully it is not more on AsyncProcess (smile). But sounds great. Related, an interesting suggestion I heard last week was to do a queue per client on the regionserver. This would help when a 'bad client' tends to hog or swap the RegionServer... When the 'bad client' fills its queue, we can reject further requests. Meantime, other clients can make progress. > Prevent client threads being blocked by only one slow region server > --- > > Key: HBASE-16388 > URL: https://issues.apache.org/jira/browse/HBASE-16388 > Project: HBase > Issue Type: New Feature >Reporter: Phil Yang >Assignee: Phil Yang > > It is a general use case for HBase's users that they have several > threads/handlers in their service, and each handler has its own Table/HTable > instance. Generally users think each handler is independent and won't > interact each other. > However, in an extreme case, if a region server is very slow, every requests > to this RS will timeout, handlers of users' service may be occupied by the > long-waiting requests even requests belong to other RS will also be timeout. > For example: > If we have 100 handlers in a client service(timeout is 1000ms) and HBase has > 10 region servers whose average response time is 50ms. If no region server is > slow, we can handle 2000 requests per second. > Now this service's QPS is 1000. If there is one region server very slow and > all requests to it will be timeout. Users hope that only 10% requests failed, > and 90% requests' response time is still 50ms, because only 10% requests are > located to the slow RS. However, each second we have 100 long-waiting > requests which exactly occupies all 100 handles. So all handlers is blocked, > the availability of this service is almost zero. > To prevent this case, we can limit the max concurrent requests to one RS in > process-level. Requests exceeding the limit will throws > ServerBusyException(extends DoNotRetryIOE) immediately to users. In the above > case, if we set this limit to 20, only 20 handlers will be occupied and other > 80 handlers can still handle requests to other RS. The availability of this > service is 90% as expected. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16384) Update report-flakies.py script to allow specifying a list of build ids to be excluded
[
https://issues.apache.org/jira/browse/HBASE-16384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416279#comment-15416279
]
Hadoop QA commented on HBASE-16384:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} pylint {color} | {color:red} 0m 6s {color}
| {color:red} The patch generated 9 new + 23 unchanged - 11 fixed = 32 total
(was 34) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 1s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:red}-1{color} | {color:red} hbaseprotoc {color} | {color:red} 0m 10s
{color} | {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 54s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-08-11 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12823147/HBASE-16384.master.003.patch
|
| JIRA Issue | HBASE-16384 |
| Optional Tests | asflicense pylint |
| uname | Linux 041cc4a28bc7 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 8895667 |
| pylint | v1.6.4 |
| pylint |
https://builds.apache.org/job/PreCommit-HBASE-Build/3058/artifact/patchprocess/diff-patch-pylint.txt
|
| hbaseprotoc |
https://builds.apache.org/job/PreCommit-HBASE-Build/3058/artifact/patchprocess/patch-hbaseprotoc-root.txt
|
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3058/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Update report-flakies.py script to allow specifying a list of build ids to be
> excluded
> --
>
> Key: HBASE-16384
> URL: https://issues.apache.org/jira/browse/HBASE-16384
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-16384.master.001.patch,
> HBASE-16384.master.002.patch, HBASE-16384.master.003.patch
>
>
> Sometimes, builds fail mysteriously and leave lots of tests hanging. This
> makes [flaky
> list|https://builds.apache.org/job/HBase-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html]
> go crazy.
> This patch adds that feature to specify build ids to exclude in
> report-flakies.py.
> If we find that a build screwed up, we can exclude it using "exclude=" option
> in --urls param and rerun the job to fix the flaky list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16385) Have hbase-rest pull hbase.rest.port from Constants.java
[
https://issues.apache.org/jira/browse/HBASE-16385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416275#comment-15416275
]
Hudson commented on HBASE-16385:
FAILURE: Integrated in HBase-1.3 #815 (See
[https://builds.apache.org/job/HBase-1.3/815/])
HBASE-16385 Have hbase-rest pull hbase.rest.port from Constants.java (Yi
(stack: rev 2831850de713d3e044f582394a2683a1b5da9a5f)
* hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/RESTServer.java
> Have hbase-rest pull hbase.rest.port from Constants.java
>
>
> Key: HBASE-16385
> URL: https://issues.apache.org/jira/browse/HBASE-16385
> Project: HBase
> Issue Type: Improvement
> Components: REST
>Reporter: Dima Spivak
>Assignee: Yi Liang
>Priority: Minor
> Labels: beginner
> Fix For: 2.0.0, 1.3.0, 1.2.3
>
> Attachments: HBASE-16385-V1.patch
>
>
> [A while back|https://issues.apache.org/jira/browse/HBASE-3272], we
> [hardcoded|https://github.com/apache/hbase/blob/master/hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/RESTServer.java#L266]
> the {{hbase.rest.port}} instead of getting it from
> [Constants.java|https://github.com/apache/hbase/blob/master/hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/Constants.java#L36].
> We should fix that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16389) Thread leak in CoprocessorHost#getTable(TableName tableName) API
[
https://issues.apache.org/jira/browse/HBASE-16389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416271#comment-15416271
]
stack commented on HBASE-16389:
---
+1 on patch.
On our facilitating being able to create a Table instance in CP context, I have
my qualms (but not your issue Ankit).
Thanks for the fix.
> Thread leak in CoprocessorHost#getTable(TableName tableName) API
>
>
> Key: HBASE-16389
> URL: https://issues.apache.org/jira/browse/HBASE-16389
> Project: HBase
> Issue Type: Bug
>Reporter: Ankit Singhal
> Attachments: HBASE-16389.patch
>
>
> There is a thread leak in below method, we are creating a default pool while
> calling
> {code}
> @Override
> public HTableInterface getTable(TableName tableName) throws IOException {
> return this.getTable(tableName,
> HTable.getDefaultExecutor(getConfiguration()));
> }
> {code}
> which will never be shutdown because in HTable, we are setting
> this.cleanupPoolOnClose to false
> {code}
> @InterfaceAudience.Private
> public HTable(TableName tableName, final ClusterConnection connection,
> final TableConfiguration tableConfig,
> final RpcRetryingCallerFactory rpcCallerFactory,
> final RpcControllerFactory rpcControllerFactory,
> final ExecutorService pool) throws IOException {
> if (connection == null || connection.isClosed()) {
> throw new IllegalArgumentException("Connection is null or closed.");
> }
> this.tableName = tableName;
> this.cleanupConnectionOnClose = false;
> this.connection = connection;
> this.configuration = connection.getConfiguration();
> this.tableConfiguration = tableConfig;
> this.pool = pool;
> if (pool == null) {
> this.pool = getDefaultExecutor(this.configuration);
> this.cleanupPoolOnClose = true;
> } else {
> this.cleanupPoolOnClose = false;
> }
> {code}
> resulted in pool to stay forever ,which eventually can lead other processes
> or the same process to starve for a threads.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16389) Thread leak in CoprocessorHost#getTable(TableName tableName) API
[
https://issues.apache.org/jira/browse/HBASE-16389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-16389:
--
Status: Patch Available (was: Open)
> Thread leak in CoprocessorHost#getTable(TableName tableName) API
>
>
> Key: HBASE-16389
> URL: https://issues.apache.org/jira/browse/HBASE-16389
> Project: HBase
> Issue Type: Bug
>Reporter: Ankit Singhal
> Attachments: HBASE-16389.patch
>
>
> There is a thread leak in below method, we are creating a default pool while
> calling
> {code}
> @Override
> public HTableInterface getTable(TableName tableName) throws IOException {
> return this.getTable(tableName,
> HTable.getDefaultExecutor(getConfiguration()));
> }
> {code}
> which will never be shutdown because in HTable, we are setting
> this.cleanupPoolOnClose to false
> {code}
> @InterfaceAudience.Private
> public HTable(TableName tableName, final ClusterConnection connection,
> final TableConfiguration tableConfig,
> final RpcRetryingCallerFactory rpcCallerFactory,
> final RpcControllerFactory rpcControllerFactory,
> final ExecutorService pool) throws IOException {
> if (connection == null || connection.isClosed()) {
> throw new IllegalArgumentException("Connection is null or closed.");
> }
> this.tableName = tableName;
> this.cleanupConnectionOnClose = false;
> this.connection = connection;
> this.configuration = connection.getConfiguration();
> this.tableConfiguration = tableConfig;
> this.pool = pool;
> if (pool == null) {
> this.pool = getDefaultExecutor(this.configuration);
> this.cleanupPoolOnClose = true;
> } else {
> this.cleanupPoolOnClose = false;
> }
> {code}
> resulted in pool to stay forever ,which eventually can lead other processes
> or the same process to starve for a threads.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16285) Drop RPC requests if it must be considered as timeout at client
[ https://issues.apache.org/jira/browse/HBASE-16285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416263#comment-15416263 ] stack commented on HBASE-16285: --- Belated +1. Patch is excellent. > Drop RPC requests if it must be considered as timeout at client > --- > > Key: HBASE-16285 > URL: https://issues.apache.org/jira/browse/HBASE-16285 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0, 1.3.0, 1.4.0 >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-16285-branch-1-v1.patch, > HBASE-16285-branch-1-v2.patch, HBASE-16285-branch-1-v3.patch, > HBASE-16285-branch-1-v4.patch, HBASE-16285-branch-1-v5.patch, > HBASE-16285-branch-1-v5.patch, HBASE-16285-branch-1-v6.patch, > HBASE-16285-v1.patch, HBASE-16285-v2.patch, HBASE-16285-v3.patch, > HBASE-16285-v4.patch, HBASE-16285-v5.patch, HBASE-16285-v6.patch, > HBASE-16285-v7.patch, HBASE-16285-v7.patch, HBASE-16285-v8.patch, > HBASE-16285-v9.patch > > > After HBASE-15593, we have a timeout param in header of RPC requests. We can > use it in more scenes. > A straightforward scene is to drop requests if it has waited so long in RPC > queue and has been dropped by client. Even if we handle this request and send > the response back, it will not be used any more. And client may have sent a > retry. In an extreme case, if the server is slow, all requests may be timeout > or queue-full-exception because we should handle previous requests which have > been dropped by client and many resources at server are wasted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16213) A new HFileBlock structure for fast random get
[ https://issues.apache.org/jira/browse/HBASE-16213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416261#comment-15416261 ] binlijin commented on HBASE-16213: -- Yes sir, you can try alter table DATA_BLOCK_ENCODING from 'NONE' to ‘ROW_INDEX_V1' to test it. > A new HFileBlock structure for fast random get > -- > > Key: HBASE-16213 > URL: https://issues.apache.org/jira/browse/HBASE-16213 > Project: HBase > Issue Type: New Feature > Components: Performance >Reporter: binlijin >Assignee: binlijin > Attachments: HBASE-16213-master_v1.patch, HBASE-16213.patch, > HBASE-16213_branch1_v3.patch, HBASE-16213_v2.patch, hfile-cpu.png, > hfile_block_performance.pptx, hfile_block_performance2.pptx, > new-hfile-block.xlsx > > > HFileBlock store cells sequential, current when to get a row from the block, > it scan from the first cell until the row's cell. > The new structure store every row's start offset with data, so it can find > the exact row with binarySearch. > I use EncodedSeekPerformanceTest test the performance. > First use ycsb write 100w data, every row have only one qualifier, and > valueLength=16B/64/256B/1k. > Then use EncodedSeekPerformanceTest to test random read 1w or 100w row, and > also record HFileBlock's dataSize/dataWithMetaSize in the encoding. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16368) test*WhenRegionMove in TestPartialResultsFromClientSide is flaky
[
https://issues.apache.org/jira/browse/HBASE-16368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416258#comment-15416258
]
stack commented on HBASE-16368:
---
+1
> test*WhenRegionMove in TestPartialResultsFromClientSide is flaky
>
>
> Key: HBASE-16368
> URL: https://issues.apache.org/jira/browse/HBASE-16368
> Project: HBase
> Issue Type: Bug
> Components: Scanners
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.5, 1.2.2
>Reporter: Guanghao Zhang
>Assignee: Phil Yang
> Labels: trivial
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.6, 1.2.3
>
> Attachments: HBASE-16368-v1.patch, HBASE-16368-v1.patch,
> HBASE-16368-v1.patch
>
>
> This test fail when Hadoop QA run preCommit:
> https://builds.apache.org/job/PreCommit-HBASE-Build/2971/testReport/org.apache.hadoop.hbase/TestPartialResultsFromClientSide/testReversedCompleteResultWhenRegionMove/.
> And I found it is in Flaky Tests Dashboard:
> http://hbase.x10host.com/flaky-tests/. I run it in my local machine and it
> may fail, too.
> Test results show that the region location didn't update when scanner
> callable get a NotServingRegionException or RegionMovedException.
> {code}
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=36, exceptions:
> Sat Aug 06 05:55:52 UTC 2016, null, java.net.SocketTimeoutException:
> callTimeout=2000, callDuration=2157:
> org.apache.hadoop.hbase.NotServingRegionException:
> testReversedCompleteResultWhenRegionMove,,1470462949504.5069bd63bf6eda5108acec4fcc087b0e.
> is closing
> at
> org.apache.hadoop.hbase.regionserver.HRegion.startRegionOperation(HRegion.java:8233)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2634)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2629)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2623)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2490)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34950)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2264)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:118)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:189)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:169)
> row '' on table 'testReversedCompleteResultWhenRegionMove' at
> region=testReversedCompleteResultWhenRegionMove,,1470462949504.5069bd63bf6eda5108acec4fcc087b0e.,
> hostname=asf907.gq1.ygridcore.net,38914,1470462943053, seqNum=2
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:281)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:213)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:61)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:212)
> at
> org.apache.hadoop.hbase.client.ReversedClientScanner.nextScanner(ReversedClientScanner.java:118)
> at
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:166)
> at
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:161)
> at
> org.apache.hadoop.hbase.client.ReversedClientScanner.(ReversedClientScanner.java:56)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:785)
> at
> org.apache.hadoop.hbase.TestPartialResultsFromClientSide.testReversedCompleteResultWhenRegionMove(TestPartialResultsFromClientSide.java:986)
> {code}
> {code}
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=36, exceptions:
> Sat Aug 06 16:27:22 CST 2016, null, java.net.SocketTimeoutException:
> callTimeout=2000, callDuration=3035: Region moved to: hostname=localhost
> port=58351 startCode=1470472007714. As of locationSeqNum=6. row 'testRow0' on
> table 'testPartialResultWhenRegionMove' at
> region=testPartialResultWhenRegionMove,,1470472035048.977faf05c1d6d9990b5559b17aa18913.,
> hostname=localhost,40425,1470472007646, seqNum=2
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:281)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:213)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:61)
> at
>
[jira] [Commented] (HBASE-16384) Update report-flakies.py script to allow specifying a list of build ids to be excluded
[ https://issues.apache.org/jira/browse/HBASE-16384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416254#comment-15416254 ] Appy commented on HBASE-16384: -- :) Updated the patch as [~dimaspivak] suggested. > Update report-flakies.py script to allow specifying a list of build ids to be > excluded > -- > > Key: HBASE-16384 > URL: https://issues.apache.org/jira/browse/HBASE-16384 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: HBASE-16384.master.001.patch, > HBASE-16384.master.002.patch, HBASE-16384.master.003.patch > > > Sometimes, builds fail mysteriously and leave lots of tests hanging. This > makes [flaky > list|https://builds.apache.org/job/HBase-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html] > go crazy. > This patch adds that feature to specify build ids to exclude in > report-flakies.py. > If we find that a build screwed up, we can exclude it using "exclude=" option > in --urls param and rerun the job to fix the flaky list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16379) [replication] Minor improvement to replication/copy_tables_desc.rb
[ https://issues.apache.org/jira/browse/HBASE-16379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416250#comment-15416250 ] Hudson commented on HBASE-16379: FAILURE: Integrated in HBase-1.4 #349 (See [https://builds.apache.org/job/HBase-1.4/349/]) HBASE-16379 [replication] Minor improvement to (esteban: rev 429f398c0769bfc76eddefcaa0230f6c3de40933) * bin/replication/copy_tables_desc.rb > [replication] Minor improvement to replication/copy_tables_desc.rb > -- > > Key: HBASE-16379 > URL: https://issues.apache.org/jira/browse/HBASE-16379 > Project: HBase > Issue Type: Improvement > Components: Replication, shell >Affects Versions: 1.3.0, 1.2.2 >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez >Priority: Trivial > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3, 1.1.7 > > Attachments: > 0001-HBASE-16379-replication-Minor-improvement-to-replica.patch, > 0001-HBASE-16379-replication-Minor-improvement-to-replica.patch, > 0001-HBASE-16379-replication-Minor-improvement-to-replica.patch, > HBASE-16379.patch > > > copy_tables_desc.rb is helpful for quickly setting up a table remotely based > on an existing schema. However it does copy by default all tables. Now you > can pass a list of tables as an optional third argument and it will also > display what table descriptors where copied. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16385) Have hbase-rest pull hbase.rest.port from Constants.java
[
https://issues.apache.org/jira/browse/HBASE-16385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416248#comment-15416248
]
Hudson commented on HBASE-16385:
FAILURE: Integrated in HBase-1.4 #349 (See
[https://builds.apache.org/job/HBase-1.4/349/])
HBASE-16385 Have hbase-rest pull hbase.rest.port from Constants.java (Yi
(stack: rev 2933795822d28a6da0889e90b0bed1f9868e667f)
* hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/RESTServer.java
> Have hbase-rest pull hbase.rest.port from Constants.java
>
>
> Key: HBASE-16385
> URL: https://issues.apache.org/jira/browse/HBASE-16385
> Project: HBase
> Issue Type: Improvement
> Components: REST
>Reporter: Dima Spivak
>Assignee: Yi Liang
>Priority: Minor
> Labels: beginner
> Fix For: 2.0.0, 1.3.0, 1.2.3
>
> Attachments: HBASE-16385-V1.patch
>
>
> [A while back|https://issues.apache.org/jira/browse/HBASE-3272], we
> [hardcoded|https://github.com/apache/hbase/blob/master/hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/RESTServer.java#L266]
> the {{hbase.rest.port}} instead of getting it from
> [Constants.java|https://github.com/apache/hbase/blob/master/hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/Constants.java#L36].
> We should fix that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16384) Update report-flakies.py script to allow specifying a list of build ids to be excluded
[ https://issues.apache.org/jira/browse/HBASE-16384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16384: - Attachment: HBASE-16384.master.003.patch > Update report-flakies.py script to allow specifying a list of build ids to be > excluded > -- > > Key: HBASE-16384 > URL: https://issues.apache.org/jira/browse/HBASE-16384 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: HBASE-16384.master.001.patch, > HBASE-16384.master.002.patch, HBASE-16384.master.003.patch > > > Sometimes, builds fail mysteriously and leave lots of tests hanging. This > makes [flaky > list|https://builds.apache.org/job/HBase-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html] > go crazy. > This patch adds that feature to specify build ids to exclude in > report-flakies.py. > If we find that a build screwed up, we can exclude it using "exclude=" option > in --urls param and rerun the job to fix the flaky list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14345) Consolidate printUsage in IntegrationTestLoadAndVerify
[
https://issues.apache.org/jira/browse/HBASE-14345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416249#comment-15416249
]
Hudson commented on HBASE-14345:
FAILURE: Integrated in HBase-1.4 #349 (See
[https://builds.apache.org/job/HBase-1.4/349/])
HBASE-14345 Consolidate printUsage in IntegrationTestLoadAndVerify (Reid (appy:
rev 89e3373096b3adfc6357b0aba42cbf3b06532485)
*
hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestLoadAndVerify.java
*
hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestWithCellVisibilityLoadAndVerify.java
> Consolidate printUsage in IntegrationTestLoadAndVerify
> --
>
> Key: HBASE-14345
> URL: https://issues.apache.org/jira/browse/HBASE-14345
> Project: HBase
> Issue Type: Improvement
> Components: integration tests
>Reporter: Nick Dimiduk
>Assignee: Reid Chan
>Priority: Trivial
> Labels: beginner
> Fix For: 2.0.0, 1.3.0, 1.0.4, 1.4.0, 1.1.6, 1.2.3
>
> Attachments: HBASE-14345.002.patch, HBASE-14345.003.patch,
> HBASE-14345.004.patch, HBASE-14345.master.001.patch,
> HBASE-14345.master.002.patch, HBASE-14345.patch, itlav-2016-07-07.png,
> itlv.png
>
>
> Investigating the use of {{itlav}} is a little screwy. Subclasses are not
> overriding the {{printUsage()}} methods correctly, so you have to pass
> {{--help}} to get some info and no arguments to get the rest.
> {noformat}
> [hbase@ndimiduk-112rc2-7 ~]$ hbase
> org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify --help
> usage: bin/hbase org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify
>
> Options:
> -h,--help Show usage
> -m,--monkey Which chaos monkey to run
> -monkeyProps The properties file for specifying chaos monkey
> properties.
> -ncc,--noClusterCleanUp Don't clean up the cluster at the end
> [hbase@ndimiduk-112rc2-7 ~]$ hbase
> org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify
> IntegrationTestLoadAndVerify [-Doptions]
> Loads a table with row dependencies and verifies the dependency chains
> Options
> -Dloadmapper.table=Table to write/verify (default autogen)
> -Dloadmapper.backrefs=Number of backreferences per row (default
> 50)
> -Dloadmapper.num_to_write=Number of rows per mapper (default 100,000
> per mapper)
> -Dloadmapper.deleteAfter= Delete after a successful verify (default
> true)
> -Dloadmapper.numPresplits=Number of presplit regions to start with
> (default 40)
> -Dloadmapper.map.tasks= Number of map tasks for load (default 200)
> -Dverify.reduce.tasks=Number of reduce tasks for verify (default
> 35)
> -Dverify.scannercaching= Number hbase scanner caching rows to read
> (default 50)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16213) A new HFileBlock structure for fast random get
[ https://issues.apache.org/jira/browse/HBASE-16213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416245#comment-15416245 ] binlijin commented on HBASE-16213: -- Yes. > A new HFileBlock structure for fast random get > -- > > Key: HBASE-16213 > URL: https://issues.apache.org/jira/browse/HBASE-16213 > Project: HBase > Issue Type: New Feature > Components: Performance >Reporter: binlijin >Assignee: binlijin > Attachments: HBASE-16213-master_v1.patch, HBASE-16213.patch, > HBASE-16213_branch1_v3.patch, HBASE-16213_v2.patch, hfile-cpu.png, > hfile_block_performance.pptx, hfile_block_performance2.pptx, > new-hfile-block.xlsx > > > HFileBlock store cells sequential, current when to get a row from the block, > it scan from the first cell until the row's cell. > The new structure store every row's start offset with data, so it can find > the exact row with binarySearch. > I use EncodedSeekPerformanceTest test the performance. > First use ycsb write 100w data, every row have only one qualifier, and > valueLength=16B/64/256B/1k. > Then use EncodedSeekPerformanceTest to test random read 1w or 100w row, and > also record HFileBlock's dataSize/dataWithMetaSize in the encoding. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16385) Have hbase-rest pull hbase.rest.port from Constants.java
[
https://issues.apache.org/jira/browse/HBASE-16385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416246#comment-15416246
]
Hudson commented on HBASE-16385:
FAILURE: Integrated in HBase-1.2 #699 (See
[https://builds.apache.org/job/HBase-1.2/699/])
HBASE-16385 Have hbase-rest pull hbase.rest.port from Constants.java (Yi
(stack: rev 3de063cf03b6475f02f938fd751e42799df297f4)
* hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/RESTServer.java
> Have hbase-rest pull hbase.rest.port from Constants.java
>
>
> Key: HBASE-16385
> URL: https://issues.apache.org/jira/browse/HBASE-16385
> Project: HBase
> Issue Type: Improvement
> Components: REST
>Reporter: Dima Spivak
>Assignee: Yi Liang
>Priority: Minor
> Labels: beginner
> Fix For: 2.0.0, 1.3.0, 1.2.3
>
> Attachments: HBASE-16385-V1.patch
>
>
> [A while back|https://issues.apache.org/jira/browse/HBASE-3272], we
> [hardcoded|https://github.com/apache/hbase/blob/master/hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/RESTServer.java#L266]
> the {{hbase.rest.port}} instead of getting it from
> [Constants.java|https://github.com/apache/hbase/blob/master/hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/Constants.java#L36].
> We should fix that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16213) A new HFileBlock structure for fast random get
[ https://issues.apache.org/jira/browse/HBASE-16213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416244#comment-15416244 ] stack commented on HBASE-16213: --- This makes sense to me. The latency improvements are probably so small as to go unnoticed (relative) but yeah, less savings in a bunch of cpu... I can measure too > A new HFileBlock structure for fast random get > -- > > Key: HBASE-16213 > URL: https://issues.apache.org/jira/browse/HBASE-16213 > Project: HBase > Issue Type: New Feature > Components: Performance >Reporter: binlijin >Assignee: binlijin > Attachments: HBASE-16213-master_v1.patch, HBASE-16213.patch, > HBASE-16213_branch1_v3.patch, HBASE-16213_v2.patch, hfile-cpu.png, > hfile_block_performance.pptx, hfile_block_performance2.pptx, > new-hfile-block.xlsx > > > HFileBlock store cells sequential, current when to get a row from the block, > it scan from the first cell until the row's cell. > The new structure store every row's start offset with data, so it can find > the exact row with binarySearch. > I use EncodedSeekPerformanceTest test the performance. > First use ycsb write 100w data, every row have only one qualifier, and > valueLength=16B/64/256B/1k. > Then use EncodedSeekPerformanceTest to test random read 1w or 100w row, and > also record HFileBlock's dataSize/dataWithMetaSize in the encoding. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16385) Have hbase-rest pull hbase.rest.port from Constants.java
[
https://issues.apache.org/jira/browse/HBASE-16385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416238#comment-15416238
]
Hudson commented on HBASE-16385:
FAILURE: Integrated in HBase-1.2-IT #577 (See
[https://builds.apache.org/job/HBase-1.2-IT/577/])
HBASE-16385 Have hbase-rest pull hbase.rest.port from Constants.java (stack:
rev 3de063cf03b6475f02f938fd751e42799df297f4)
* hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/RESTServer.java
> Have hbase-rest pull hbase.rest.port from Constants.java
>
>
> Key: HBASE-16385
> URL: https://issues.apache.org/jira/browse/HBASE-16385
> Project: HBase
> Issue Type: Improvement
> Components: REST
>Reporter: Dima Spivak
>Assignee: Yi Liang
>Priority: Minor
> Labels: beginner
> Fix For: 2.0.0, 1.3.0, 1.2.3
>
> Attachments: HBASE-16385-V1.patch
>
>
> [A while back|https://issues.apache.org/jira/browse/HBASE-3272], we
> [hardcoded|https://github.com/apache/hbase/blob/master/hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/RESTServer.java#L266]
> the {{hbase.rest.port}} instead of getting it from
> [Constants.java|https://github.com/apache/hbase/blob/master/hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/Constants.java#L36].
> We should fix that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16213) A new HFileBlock structure for fast random get
[ https://issues.apache.org/jira/browse/HBASE-16213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416234#comment-15416234 ] binlijin commented on HBASE-16213: -- I will test it when the machine is ready. The last time when i get the hfile-cpu.png, i do not see the throughput improved, but cut down much CPU usage. > A new HFileBlock structure for fast random get > -- > > Key: HBASE-16213 > URL: https://issues.apache.org/jira/browse/HBASE-16213 > Project: HBase > Issue Type: New Feature > Components: Performance >Reporter: binlijin >Assignee: binlijin > Attachments: HBASE-16213-master_v1.patch, HBASE-16213.patch, > HBASE-16213_branch1_v3.patch, HBASE-16213_v2.patch, hfile-cpu.png, > hfile_block_performance.pptx, hfile_block_performance2.pptx, > new-hfile-block.xlsx > > > HFileBlock store cells sequential, current when to get a row from the block, > it scan from the first cell until the row's cell. > The new structure store every row's start offset with data, so it can find > the exact row with binarySearch. > I use EncodedSeekPerformanceTest test the performance. > First use ycsb write 100w data, every row have only one qualifier, and > valueLength=16B/64/256B/1k. > Then use EncodedSeekPerformanceTest to test random read 1w or 100w row, and > also record HFileBlock's dataSize/dataWithMetaSize in the encoding. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16213) A new HFileBlock structure for fast random get
[ https://issues.apache.org/jira/browse/HBASE-16213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416230#comment-15416230 ] binlijin commented on HBASE-16213: -- Yes sir, we can turn this feature on always when this is stable. Yes，the data is all cached. > A new HFileBlock structure for fast random get > -- > > Key: HBASE-16213 > URL: https://issues.apache.org/jira/browse/HBASE-16213 > Project: HBase > Issue Type: New Feature > Components: Performance >Reporter: binlijin >Assignee: binlijin > Attachments: HBASE-16213-master_v1.patch, HBASE-16213.patch, > HBASE-16213_branch1_v3.patch, HBASE-16213_v2.patch, hfile-cpu.png, > hfile_block_performance.pptx, hfile_block_performance2.pptx, > new-hfile-block.xlsx > > > HFileBlock store cells sequential, current when to get a row from the block, > it scan from the first cell until the row's cell. > The new structure store every row's start offset with data, so it can find > the exact row with binarySearch. > I use EncodedSeekPerformanceTest test the performance. > First use ycsb write 100w data, every row have only one qualifier, and > valueLength=16B/64/256B/1k. > Then use EncodedSeekPerformanceTest to test random read 1w or 100w row, and > also record HFileBlock's dataSize/dataWithMetaSize in the encoding. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14345) Consolidate printUsage in IntegrationTestLoadAndVerify
[
https://issues.apache.org/jira/browse/HBASE-14345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416215#comment-15416215
]
Hudson commented on HBASE-14345:
SUCCESS: Integrated in HBase-1.1-JDK8 #1849 (See
[https://builds.apache.org/job/HBase-1.1-JDK8/1849/])
HBASE-14345 Consolidate printUsage in IntegrationTestLoadAndVerify (Reid (appy:
rev e5697bdb1bc63e832db57542420181348605b475)
*
hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestLoadAndVerify.java
*
hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestWithCellVisibilityLoadAndVerify.java
> Consolidate printUsage in IntegrationTestLoadAndVerify
> --
>
> Key: HBASE-14345
> URL: https://issues.apache.org/jira/browse/HBASE-14345
> Project: HBase
> Issue Type: Improvement
> Components: integration tests
>Reporter: Nick Dimiduk
>Assignee: Reid Chan
>Priority: Trivial
> Labels: beginner
> Fix For: 2.0.0, 1.3.0, 1.0.4, 1.4.0, 1.1.6, 1.2.3
>
> Attachments: HBASE-14345.002.patch, HBASE-14345.003.patch,
> HBASE-14345.004.patch, HBASE-14345.master.001.patch,
> HBASE-14345.master.002.patch, HBASE-14345.patch, itlav-2016-07-07.png,
> itlv.png
>
>
> Investigating the use of {{itlav}} is a little screwy. Subclasses are not
> overriding the {{printUsage()}} methods correctly, so you have to pass
> {{--help}} to get some info and no arguments to get the rest.
> {noformat}
> [hbase@ndimiduk-112rc2-7 ~]$ hbase
> org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify --help
> usage: bin/hbase org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify
>
> Options:
> -h,--help Show usage
> -m,--monkey Which chaos monkey to run
> -monkeyProps The properties file for specifying chaos monkey
> properties.
> -ncc,--noClusterCleanUp Don't clean up the cluster at the end
> [hbase@ndimiduk-112rc2-7 ~]$ hbase
> org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify
> IntegrationTestLoadAndVerify [-Doptions]
> Loads a table with row dependencies and verifies the dependency chains
> Options
> -Dloadmapper.table=Table to write/verify (default autogen)
> -Dloadmapper.backrefs=Number of backreferences per row (default
> 50)
> -Dloadmapper.num_to_write=Number of rows per mapper (default 100,000
> per mapper)
> -Dloadmapper.deleteAfter= Delete after a successful verify (default
> true)
> -Dloadmapper.numPresplits=Number of presplit regions to start with
> (default 40)
> -Dloadmapper.map.tasks= Number of map tasks for load (default 200)
> -Dverify.reduce.tasks=Number of reduce tasks for verify (default
> 35)
> -Dverify.scannercaching= Number hbase scanner caching rows to read
> (default 50)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16379) [replication] Minor improvement to replication/copy_tables_desc.rb
[ https://issues.apache.org/jira/browse/HBASE-16379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416216#comment-15416216 ] Hudson commented on HBASE-16379: SUCCESS: Integrated in HBase-1.1-JDK8 #1849 (See [https://builds.apache.org/job/HBase-1.1-JDK8/1849/]) HBASE-16379 [replication] Minor improvement to (esteban: rev 3259f03a592b42abe05ae1b89540b15ee4245cbe) * bin/replication/copy_tables_desc.rb > [replication] Minor improvement to replication/copy_tables_desc.rb > -- > > Key: HBASE-16379 > URL: https://issues.apache.org/jira/browse/HBASE-16379 > Project: HBase > Issue Type: Improvement > Components: Replication, shell >Affects Versions: 1.3.0, 1.2.2 >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez >Priority: Trivial > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3, 1.1.7 > > Attachments: > 0001-HBASE-16379-replication-Minor-improvement-to-replica.patch, > 0001-HBASE-16379-replication-Minor-improvement-to-replica.patch, > 0001-HBASE-16379-replication-Minor-improvement-to-replica.patch, > HBASE-16379.patch > > > copy_tables_desc.rb is helpful for quickly setting up a table remotely based > on an existing schema. However it does copy by default all tables. Now you > can pass a list of tables as an optional third argument and it will also > display what table descriptors where copied. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16213) A new HFileBlock structure for fast random get
[ https://issues.apache.org/jira/browse/HBASE-16213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416213#comment-15416213 ] stack commented on HBASE-16213: --- Why not turn this feature on always? If keys of 16bytes in value only add 5% to the size -- this is worst case -- the benefit far outweighs this extra size. When you say seek, you are seeking cached data? Nice numbers [~aoxiang] > A new HFileBlock structure for fast random get > -- > > Key: HBASE-16213 > URL: https://issues.apache.org/jira/browse/HBASE-16213 > Project: HBase > Issue Type: New Feature > Components: Performance >Reporter: binlijin >Assignee: binlijin > Attachments: HBASE-16213-master_v1.patch, HBASE-16213.patch, > HBASE-16213_branch1_v3.patch, HBASE-16213_v2.patch, hfile-cpu.png, > hfile_block_performance.pptx, hfile_block_performance2.pptx, > new-hfile-block.xlsx > > > HFileBlock store cells sequential, current when to get a row from the block, > it scan from the first cell until the row's cell. > The new structure store every row's start offset with data, so it can find > the exact row with binarySearch. > I use EncodedSeekPerformanceTest test the performance. > First use ycsb write 100w data, every row have only one qualifier, and > valueLength=16B/64/256B/1k. > Then use EncodedSeekPerformanceTest to test random read 1w or 100w row, and > also record HFileBlock's dataSize/dataWithMetaSize in the encoding. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16213) A new HFileBlock structure for fast random get
[ https://issues.apache.org/jira/browse/HBASE-16213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416205#comment-15416205 ] stack commented on HBASE-16213: --- I would imagine it makes a lot of difference in cpu usage when random reads; this linear seek is one of our main cpu consumers random reading. > A new HFileBlock structure for fast random get > -- > > Key: HBASE-16213 > URL: https://issues.apache.org/jira/browse/HBASE-16213 > Project: HBase > Issue Type: New Feature > Components: Performance >Reporter: binlijin >Assignee: binlijin > Attachments: HBASE-16213-master_v1.patch, HBASE-16213.patch, > HBASE-16213_branch1_v3.patch, HBASE-16213_v2.patch, hfile-cpu.png, > hfile_block_performance.pptx, hfile_block_performance2.pptx, > new-hfile-block.xlsx > > > HFileBlock store cells sequential, current when to get a row from the block, > it scan from the first cell until the row's cell. > The new structure store every row's start offset with data, so it can find > the exact row with binarySearch. > I use EncodedSeekPerformanceTest test the performance. > First use ycsb write 100w data, every row have only one qualifier, and > valueLength=16B/64/256B/1k. > Then use EncodedSeekPerformanceTest to test random read 1w or 100w row, and > also record HFileBlock's dataSize/dataWithMetaSize in the encoding. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14450) HBase Backup/Restore Phase 3: Multiwal support
[
https://issues.apache.org/jira/browse/HBASE-14450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416189#comment-15416189
]
Ted Yu edited comment on HBASE-14450 at 8/10/16 11:27 PM:
--
Added more debug logging.
{code}
2016-08-10 15:46:16,306 DEBUG [sync.3] wal.FSHLog$SyncRunner(1275): syncing
writer
hdfs://localhost:56218/user/tyu/test-data/f650797e-abdc-4669-aac9-39b68914fcf9/WALs/10.22.16.34,56228,1470869104167/10.22.16.34%2C56228%2C1470869104167.regiongroup-2.1470869132540
...
2016-08-10 15:46:16,629 DEBUG [main] backup.TestIncrementalBackup(75): written
199 rows to ns1:test-1470869129051
2016-08-10 15:46:16,633 DEBUG [sync.3] wal.FSHLog$SyncRunner(1275): syncing
writer
hdfs://localhost:56218/user/tyu/test-data/f650797e-abdc-4669-aac9-39b68914fcf9/WALs/10.22.16.34,56228,1470869104167/10.22.16.34%2C56228%2C1470869104167.regiongroup-3.1470869134197
...
2016-08-10 15:46:16,797 DEBUG [sync.2] wal.FSHLog$SyncRunner(1275): syncing
writer
hdfs://localhost:56218/user/tyu/test-data/f650797e-abdc-4669-aac9-39b68914fcf9/WALs/10.22.16.34,56228,1470869104167/10.22.16.34%2C56228%2C1470869104167.regiongroup-1.1470869110496
...
2016-08-10 15:46:16,824 DEBUG
[rs(10.22.16.34,56228,1470869104167)-backup-pool30-thread-1]
regionserver.LogRollBackupSubprocedure$RSRollLogTask(72): ++ DRPC started:
10.22.16.34,56228,1470869104167
2016-08-10 15:46:16,824 DEBUG [member: '10.22.16.34,56228,1470869104167'
subprocedure-pool2-thread-1] regionserver.LogRollBackupSubprocedurePool(84):
Waiting for backup procedure to finish.
2016-08-10 15:46:16,824 INFO
[rs(10.22.16.34,56228,1470869104167)-backup-pool30-thread-1]
regionserver.LogRollBackupSubprocedure$RSRollLogTask(77): Trying to roll log in
backup subprocedure, current log number: 1470869138221
2016-08-10 15:46:16,824 DEBUG [main-EventThread]
procedure.ZKProcedureUtil(263): |rolllog
2016-08-10 15:46:16,825 DEBUG [main-EventThread]
procedure.ZKProcedureUtil(263): |---10.22.16.34,56228,1470869104167
2016-08-10 15:46:16,826 DEBUG [main-EventThread]
procedure.ZKProcedureUtil(263): |---10.22.16.34,56226,1470869103454
2016-08-10 15:46:16,826 DEBUG [main-EventThread]
procedure.ZKProcedureUtil(263): |-abort
2016-08-10 15:46:16,826 DEBUG [main-EventThread]
procedure.ZKProcedureUtil(263): |-reached
2016-08-10 15:46:16,827 DEBUG
[rs(10.22.16.34,56226,1470869103454)-backup-pool29-thread-1] wal.FSHLog(665):
syncing writer
hdfs://localhost:56218/user/tyu/test-data/f650797e-abdc-4669-aac9-39b68914fcf9/WALs/10.22.16.34,56226,1470869103454/10.22.16.34%2C56226%2C1470869103454.regiongroup-0.1470869176824
2016-08-10 15:46:16,827 DEBUG [main-EventThread]
procedure.ZKProcedureCoordinatorRpcs$1(238): Ignoring created notification for
node:/1/rolllog-proc/reached/rolllog
2016-08-10 15:46:16,829 DEBUG
[rs(10.22.16.34,56228,1470869104167)-backup-pool30-thread-1] wal.FSHLog(665):
syncing writer
hdfs://localhost:56218/user/tyu/test-data/f650797e-abdc-4669-aac9-39b68914fcf9/WALs/10.22.16.34,56228,1470869104167/10.22.16.34%2C56228%2C1470869104167.regiongroup-0.1470869176825
2016-08-10 15:46:16,833 DEBUG [sync.2] wal.FSHLog$SyncRunner(1275): syncing
writer
hdfs://localhost:56218/user/tyu/test-data/f650797e-abdc-4669-aac9-39b68914fcf9/WALs/10.22.16.34,56226,1470869103454/10.22.16.34%2C56226%2C1470869103454.regiongroup-0.1470869138221
2016-08-10 15:46:16,833 DEBUG [sync.3] wal.FSHLog$SyncRunner(1275): syncing
writer
hdfs://localhost:56218/user/tyu/test-data/f650797e-abdc-4669-aac9-39b68914fcf9/WALs/10.22.16.34,56228,1470869104167/10.22.16.34%2C56228%2C1470869104167.regiongroup-0.1470869138221
2016-08-10 15:46:17,249 DEBUG [sync.3] wal.FSHLog$SyncRunner(1275): syncing
writer
hdfs://localhost:56218/user/tyu/test-data/f650797e-abdc-4669-aac9-39b68914fcf9/WALs/10.22.16.34,56228,1470869104167/10.22.16.34%2C56228%2C1470869104167.regiongroup-1.1470869110496
2016-08-10 15:46:17,250 DEBUG [member: '10.22.16.34,56228,1470869104167'
subprocedure-pool2-thread-1] procedure.Subprocedure(188): Subprocedure
'rolllog' locally completed
2016-08-10 15:46:17,250 DEBUG [member: '10.22.16.34,56228,1470869104167'
subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(269): Marking
procedure 'rolllog' completed for member '10.22.16. 34,56228,1470869104167'
in zk
2016-08-10 15:46:17,250 DEBUG
[rs(10.22.16.34,56226,1470869103454)-backup-pool29-thread-1]
impl.BackupSystemTable(254): write region server last roll log result to
hbase:backup
2016-08-10 15:46:17,398 DEBUG [ProcedureExecutor-5] backup.BackupInfo(313):
setting incr backup file list
2016-08-10 15:46:17,398 DEBUG [ProcedureExecutor-5] backup.BackupInfo(315):
hdfs://localhost:56218/user/tyu/test-data/f650797e-abdc-4669-aac9-39b68914fcf9/WALs/10.22.16.34,56226,1470869103454/10.22.16.
34%2C56226%2C1470869103454.regiongroup-1.1470869108161
2016-08-10 15:46:17,398 DEBUG [ProcedureExecutor-5] backup.BackupInfo(315):
[jira] [Commented] (HBASE-14450) HBase Backup/Restore Phase 3: Multiwal support
[
https://issues.apache.org/jira/browse/HBASE-14450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416189#comment-15416189
]
Ted Yu commented on HBASE-14450:
Added more debug logging.
{code}
2016-08-10 15:46:16,306 DEBUG [sync.3] wal.FSHLog$SyncRunner(1275): syncing
writer
hdfs://localhost:56218/user/tyu/test-data/f650797e-abdc-4669-aac9-39b68914fcf9/WALs/10.22.16.34,56228,1470869104167/10.22.16.34%2C56228%2C1470869104167.regiongroup-2.1470869132540
2016-08-10 15:46:16,629 DEBUG [main] backup.TestIncrementalBackup(75): written
199 rows to ns1:test-1470869129051
2016-08-10 15:46:16,633 DEBUG [sync.3] wal.FSHLog$SyncRunner(1275): syncing
writer
hdfs://localhost:56218/user/tyu/test-data/f650797e-abdc-4669-aac9-39b68914fcf9/WALs/10.22.16.34,56228,1470869104167/10.22.16.34%2C56228%2C1470869104167.regiongroup-3.1470869134197
2016-08-10 15:46:16,797 DEBUG [sync.2] wal.FSHLog$SyncRunner(1275): syncing
writer
hdfs://localhost:56218/user/tyu/test-data/f650797e-abdc-4669-aac9-39b68914fcf9/WALs/10.22.16.34,56228,1470869104167/10.22.16.34%2C56228%2C1470869104167.regiongroup-1.1470869110496
2016-08-10 15:46:16,824 DEBUG
[rs(10.22.16.34,56228,1470869104167)-backup-pool30-thread-1]
regionserver.LogRollBackupSubprocedure$RSRollLogTask(72): ++ DRPC started:
10.22.16.34,56228,1470869104167
2016-08-10 15:46:16,824 DEBUG [member: '10.22.16.34,56228,1470869104167'
subprocedure-pool2-thread-1] regionserver.LogRollBackupSubprocedurePool(84):
Waiting for backup procedure to finish.
2016-08-10 15:46:16,824 INFO
[rs(10.22.16.34,56228,1470869104167)-backup-pool30-thread-1]
regionserver.LogRollBackupSubprocedure$RSRollLogTask(77): Trying to roll log in
backup subprocedure, current log number: 1470869138221
2016-08-10 15:46:16,824 DEBUG [main-EventThread]
procedure.ZKProcedureUtil(263): |rolllog
2016-08-10 15:46:16,825 DEBUG [main-EventThread]
procedure.ZKProcedureUtil(263): |---10.22.16.34,56228,1470869104167
2016-08-10 15:46:16,826 DEBUG [main-EventThread]
procedure.ZKProcedureUtil(263): |---10.22.16.34,56226,1470869103454
2016-08-10 15:46:16,826 DEBUG [main-EventThread]
procedure.ZKProcedureUtil(263): |-abort
2016-08-10 15:46:16,826 DEBUG [main-EventThread]
procedure.ZKProcedureUtil(263): |-reached
2016-08-10 15:46:16,827 DEBUG
[rs(10.22.16.34,56226,1470869103454)-backup-pool29-thread-1] wal.FSHLog(665):
syncing writer
hdfs://localhost:56218/user/tyu/test-data/f650797e-abdc-4669-aac9-39b68914fcf9/WALs/10.22.16.34,56226,1470869103454/10.22.16.34%2C56226%2C1470869103454.regiongroup-0.1470869176824
2016-08-10 15:46:16,827 DEBUG [main-EventThread]
procedure.ZKProcedureCoordinatorRpcs$1(238): Ignoring created notification for
node:/1/rolllog-proc/reached/rolllog
2016-08-10 15:46:16,829 DEBUG
[rs(10.22.16.34,56228,1470869104167)-backup-pool30-thread-1] wal.FSHLog(665):
syncing writer
hdfs://localhost:56218/user/tyu/test-data/f650797e-abdc-4669-aac9-39b68914fcf9/WALs/10.22.16.34,56228,1470869104167/10.22.16.34%2C56228%2C1470869104167.regiongroup-0.1470869176825
2016-08-10 15:46:16,833 DEBUG [sync.2] wal.FSHLog$SyncRunner(1275): syncing
writer
hdfs://localhost:56218/user/tyu/test-data/f650797e-abdc-4669-aac9-39b68914fcf9/WALs/10.22.16.34,56226,1470869103454/10.22.16.34%2C56226%2C1470869103454.regiongroup-0.1470869138221
2016-08-10 15:46:16,833 DEBUG [sync.3] wal.FSHLog$SyncRunner(1275): syncing
writer
hdfs://localhost:56218/user/tyu/test-data/f650797e-abdc-4669-aac9-39b68914fcf9/WALs/10.22.16.34,56228,1470869104167/10.22.16.34%2C56228%2C1470869104167.regiongroup-0.1470869138221
2016-08-10 15:46:17,249 DEBUG [sync.3] wal.FSHLog$SyncRunner(1275): syncing
writer
hdfs://localhost:56218/user/tyu/test-data/f650797e-abdc-4669-aac9-39b68914fcf9/WALs/10.22.16.34,56228,1470869104167/10.22.16.34%2C56228%2C1470869104167.regiongroup-1.1470869110496
2016-08-10 15:46:17,250 DEBUG [member: '10.22.16.34,56228,1470869104167'
subprocedure-pool2-thread-1] procedure.Subprocedure(188): Subprocedure
'rolllog' locally completed
2016-08-10 15:46:17,250 DEBUG [member: '10.22.16.34,56228,1470869104167'
subprocedure-pool2-thread-1] procedure.ZKProcedureMemberRpcs(269): Marking
procedure 'rolllog' completed for member '10.22.16. 34,56228,1470869104167'
in zk
2016-08-10 15:46:17,250 DEBUG
[rs(10.22.16.34,56226,1470869103454)-backup-pool29-thread-1]
impl.BackupSystemTable(254): write region server last roll log result to
hbase:backup
2016-08-10 15:46:17,398 DEBUG [ProcedureExecutor-5] backup.BackupInfo(313):
setting incr backup file list
2016-08-10 15:46:17,398 DEBUG [ProcedureExecutor-5] backup.BackupInfo(315):
hdfs://localhost:56218/user/tyu/test-data/f650797e-abdc-4669-aac9-39b68914fcf9/WALs/10.22.16.34,56226,1470869103454/10.22.16.
34%2C56226%2C1470869103454.regiongroup-1.1470869108161
2016-08-10 15:46:17,398 DEBUG [ProcedureExecutor-5] backup.BackupInfo(315):
[jira] [Updated] (HBASE-16385) Have hbase-rest pull hbase.rest.port from Constants.java
[
https://issues.apache.org/jira/browse/HBASE-16385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-16385:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 1.2.3
1.3.0
Status: Resolved (was: Patch Available)
Pushed to branch-1.2+
> Have hbase-rest pull hbase.rest.port from Constants.java
>
>
> Key: HBASE-16385
> URL: https://issues.apache.org/jira/browse/HBASE-16385
> Project: HBase
> Issue Type: Improvement
> Components: REST
>Reporter: Dima Spivak
>Assignee: Yi Liang
>Priority: Minor
> Labels: beginner
> Fix For: 2.0.0, 1.3.0, 1.2.3
>
> Attachments: HBASE-16385-V1.patch
>
>
> [A while back|https://issues.apache.org/jira/browse/HBASE-3272], we
> [hardcoded|https://github.com/apache/hbase/blob/master/hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/RESTServer.java#L266]
> the {{hbase.rest.port}} instead of getting it from
> [Constants.java|https://github.com/apache/hbase/blob/master/hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/Constants.java#L36].
> We should fix that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16392) Backup delete fault tolerance
[ https://issues.apache.org/jira/browse/HBASE-16392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16392: -- Summary: Backup delete fault tolerance (was: Backup delete fault tolearnce) > Backup delete fault tolerance > - > > Key: HBASE-16392 > URL: https://issues.apache.org/jira/browse/HBASE-16392 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov > > Backup delete modified file system and backup system table. We have to make > sure that operation is atomic, durable and isolated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16379) [replication] Minor improvement to replication/copy_tables_desc.rb
[ https://issues.apache.org/jira/browse/HBASE-16379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416135#comment-15416135 ] Hudson commented on HBASE-16379: FAILURE: Integrated in HBase-Trunk_matrix #1392 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1392/]) HBASE-16379 [replication] Minor improvement to (esteban: rev b3888eadf8ac081729385b434f8da626d02d031a) * bin/replication/copy_tables_desc.rb > [replication] Minor improvement to replication/copy_tables_desc.rb > -- > > Key: HBASE-16379 > URL: https://issues.apache.org/jira/browse/HBASE-16379 > Project: HBase > Issue Type: Improvement > Components: Replication, shell >Affects Versions: 1.3.0, 1.2.2 >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez >Priority: Trivial > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3, 1.1.7 > > Attachments: > 0001-HBASE-16379-replication-Minor-improvement-to-replica.patch, > 0001-HBASE-16379-replication-Minor-improvement-to-replica.patch, > 0001-HBASE-16379-replication-Minor-improvement-to-replica.patch, > HBASE-16379.patch > > > copy_tables_desc.rb is helpful for quickly setting up a table remotely based > on an existing schema. However it does copy by default all tables. Now you > can pass a list of tables as an optional third argument and it will also > display what table descriptors where copied. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16197) Enchance backup delete command
[ https://issues.apache.org/jira/browse/HBASE-16197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416133#comment-15416133 ] Vladimir Rodionov commented on HBASE-16197: --- Review board https://reviews.apache.org/r/50968/ > Enchance backup delete command > -- > > Key: HBASE-16197 > URL: https://issues.apache.org/jira/browse/HBASE-16197 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-16197-v1.patch, HBASE-16197-v2.patch, > HBASE-16197-v4.patch > > > Currently, backup delete command deletes only physical files/directories in > backup destination. It does not update backup system table (incremental > backup table set) and it does not delete related backup images (previous > ones), thus creating a hole in dependency chain. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16385) Have hbase-rest pull hbase.rest.port from Constants.java
[
https://issues.apache.org/jira/browse/HBASE-16385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416126#comment-15416126
]
Dima Spivak commented on HBASE-16385:
-
Was just about to ping you, [~stack]. :) findbugs warning is unrelated to the
patch and tests pass, so if it's cool with you, I'm +1 on it.
> Have hbase-rest pull hbase.rest.port from Constants.java
>
>
> Key: HBASE-16385
> URL: https://issues.apache.org/jira/browse/HBASE-16385
> Project: HBase
> Issue Type: Improvement
> Components: REST
>Reporter: Dima Spivak
>Assignee: Yi Liang
>Priority: Minor
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-16385-V1.patch
>
>
> [A while back|https://issues.apache.org/jira/browse/HBASE-3272], we
> [hardcoded|https://github.com/apache/hbase/blob/master/hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/RESTServer.java#L266]
> the {{hbase.rest.port}} instead of getting it from
> [Constants.java|https://github.com/apache/hbase/blob/master/hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/Constants.java#L36].
> We should fix that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16197) Enchance backup delete command
[ https://issues.apache.org/jira/browse/HBASE-16197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416124#comment-15416124 ] Vladimir Rodionov commented on HBASE-16197: --- Created HBASE-16392 to take care of fault tolerance. > Enchance backup delete command > -- > > Key: HBASE-16197 > URL: https://issues.apache.org/jira/browse/HBASE-16197 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-16197-v1.patch, HBASE-16197-v2.patch, > HBASE-16197-v4.patch > > > Currently, backup delete command deletes only physical files/directories in > backup destination. It does not update backup system table (incremental > backup table set) and it does not delete related backup images (previous > ones), thus creating a hole in dependency chain. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16392) Backup delete fault tolearnce
Vladimir Rodionov created HBASE-16392: - Summary: Backup delete fault tolearnce Key: HBASE-16392 URL: https://issues.apache.org/jira/browse/HBASE-16392 Project: HBase Issue Type: Task Reporter: Vladimir Rodionov Backup delete modified file system and backup system table. We have to make sure that operation is atomic, durable and isolated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16267) Remove commons-httpclient dependency from hbase-rest module
[
https://issues.apache.org/jira/browse/HBASE-16267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16267:
---
Release Note:
This issue upgrades httpclient to 4.5.2 and httpcore to 4.4.4 which are the
versions used by hadoop-2.
This is to handle the following CVE's.
https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-5262 :
http/conn/ssl/SSLConnectionSocketFactory.java in Apache HttpComponents
HttpClient before 4.3.6 ignores the http.socket.timeout configuration setting
during an SSL handshake, which allows remote attackers to cause a denial of
service (HTTPS call hang) via unspecified vectors.
https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-6153
https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-5783
Apache Commons HttpClient 3.x, as used in Amazon Flexible Payments Service
(FPS) merchant Java SDK and other products, does not verify that the server
hostname matches a domain name in the subject's Common Name (CN) or
subjectAltName field of the X.509 certificate, which allows man-in-the-middle
attackers to spoof SSL servers via an arbitrary valid certificate.
Downstream users who are exposed to commons-httpclient via the HBase classpath
will have to similarly update their dependency.
was:
This issue upgrades httpclient to 4.5.2 and httpcore to 4.4.4 which are the
versions used by hadoop-2.
https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-5262 :
http/conn/ssl/SSLConnectionSocketFactory.java in Apache HttpComponents
HttpClient before 4.3.6 ignores the http.socket.timeout configuration setting
during an SSL handshake, which allows remote attackers to cause a denial of
service (HTTPS call hang) via unspecified vectors.
https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-6153
https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-5783
Apache Commons HttpClient 3.x, as used in Amazon Flexible Payments Service
(FPS) merchant Java SDK and other products, does not verify that the server
hostname matches a domain name in the subject's Common Name (CN) or
subjectAltName field of the X.509 certificate, which allows man-in-the-middle
attackers to spoof SSL servers via an arbitrary valid certificate.
Downstream users who are exposed to commons-httpclient via the HBase classpath
will have to similarly update their dependency.
> Remove commons-httpclient dependency from hbase-rest module
> ---
>
> Key: HBASE-16267
> URL: https://issues.apache.org/jira/browse/HBASE-16267
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 16267.v10.txt, 16267.v11.txt, 16267.v12.txt,
> 16267.v13.txt, 16267.v14.txt, 16267.v15.txt, 16267.v2.txt, 16267.v4.txt,
> 16267.v6.txt, 16267.v8.txt, 16267.v9.txt
>
>
> hbase-rest module still has imports from org.apache.commons.httpclient .
> There is more work to be done after HBASE-15767 was integrated.
> In master branch, there seems to be transitive dependency which allows the
> code to compile:
> {code}
> [INFO] +- org.apache.hadoop:hadoop-common:jar:2.7.1:compile
> [INFO] | +- org.apache.hadoop:hadoop-annotations:jar:2.7.1:compile
> [INFO] | +- commons-cli:commons-cli:jar:1.2:compile
> [INFO] | +- org.apache.commons:commons-math3:jar:3.1.1:compile
> [INFO] | +- xmlenc:xmlenc:jar:0.52:compile
> [INFO] | +- commons-httpclient:commons-httpclient:jar:3.1:compile
> {code}
> HADOOP-12767
> to move the uses of httpclient HADOOP-10105
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-5262 :
> http/conn/ssl/SSLConnectionSocketFactory.java in Apache HttpComponents
> HttpClient before 4.3.6 ignores the http.socket.timeout configuration setting
> during an SSL handshake, which allows remote attackers to cause a denial of
> service (HTTPS call hang) via unspecified vectors.
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-6153
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-5783
> Apache Commons HttpClient 3.x, as used in Amazon Flexible Payments
> Service (FPS) merchant Java SDK and other products, does not verify that the
> server hostname matches a domain name in the subject's Common Name (CN) or
> subjectAltName field of the X.509 certificate, which allows man-in-the-middle
> attackers to spoof SSL servers via an arbitrary valid certificate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-13559) Region stuck in PENDING_OPEN on repeated master failure
[ https://issues.apache.org/jira/browse/HBASE-13559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark resolved HBASE-13559. --- Resolution: Cannot Reproduce > Region stuck in PENDING_OPEN on repeated master failure > --- > > Key: HBASE-13559 > URL: https://issues.apache.org/jira/browse/HBASE-13559 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark > > We had some regions stay un-assigned. They were in pending open on a server > that was dead. The cluster was restarted as a whole a few times and then the > regions were stuck until a manual assign was performed on them. > I know timing things out has been a pain. Could we have a timeout on servers > that are not alive? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16384) Update report-flakies.py script to allow specifying a list of build ids to be excluded
[ https://issues.apache.org/jira/browse/HBASE-16384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416090#comment-15416090 ] stack commented on HBASE-16384: --- [~appy] [~dimaspivak] said pretty please. You going to refuse him? > Update report-flakies.py script to allow specifying a list of build ids to be > excluded > -- > > Key: HBASE-16384 > URL: https://issues.apache.org/jira/browse/HBASE-16384 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: HBASE-16384.master.001.patch, > HBASE-16384.master.002.patch > > > Sometimes, builds fail mysteriously and leave lots of tests hanging. This > makes [flaky > list|https://builds.apache.org/job/HBase-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html] > go crazy. > This patch adds that feature to specify build ids to exclude in > report-flakies.py. > If we find that a build screwed up, we can exclude it using "exclude=" option > in --urls param and rerun the job to fix the flaky list. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16267) Remove commons-httpclient dependency from hbase-rest module
[
https://issues.apache.org/jira/browse/HBASE-16267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16267:
---
Release Note:
This issue upgrades httpclient to 4.5.2 and httpcore to 4.4.4 which are the
versions used by hadoop-2.
https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-5262 :
http/conn/ssl/SSLConnectionSocketFactory.java in Apache HttpComponents
HttpClient before 4.3.6 ignores the http.socket.timeout configuration setting
during an SSL handshake, which allows remote attackers to cause a denial of
service (HTTPS call hang) via unspecified vectors.
https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-6153
https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-5783
Apache Commons HttpClient 3.x, as used in Amazon Flexible Payments Service
(FPS) merchant Java SDK and other products, does not verify that the server
hostname matches a domain name in the subject's Common Name (CN) or
subjectAltName field of the X.509 certificate, which allows man-in-the-middle
attackers to spoof SSL servers via an arbitrary valid certificate.
Downstream users who are exposed to commons-httpclient via the HBase classpath
will have to similarly update their dependency.
was:
This issue upgrades httpclient to 4.5.2 and httpcore to 4.4.4 which are the
versions used by hadoop-2.
Downstream users who are exposed to it via the HBase classpath will have to
similarly update their dependency.
> Remove commons-httpclient dependency from hbase-rest module
> ---
>
> Key: HBASE-16267
> URL: https://issues.apache.org/jira/browse/HBASE-16267
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 16267.v10.txt, 16267.v11.txt, 16267.v12.txt,
> 16267.v13.txt, 16267.v14.txt, 16267.v15.txt, 16267.v2.txt, 16267.v4.txt,
> 16267.v6.txt, 16267.v8.txt, 16267.v9.txt
>
>
> hbase-rest module still has imports from org.apache.commons.httpclient .
> There is more work to be done after HBASE-15767 was integrated.
> In master branch, there seems to be transitive dependency which allows the
> code to compile:
> {code}
> [INFO] +- org.apache.hadoop:hadoop-common:jar:2.7.1:compile
> [INFO] | +- org.apache.hadoop:hadoop-annotations:jar:2.7.1:compile
> [INFO] | +- commons-cli:commons-cli:jar:1.2:compile
> [INFO] | +- org.apache.commons:commons-math3:jar:3.1.1:compile
> [INFO] | +- xmlenc:xmlenc:jar:0.52:compile
> [INFO] | +- commons-httpclient:commons-httpclient:jar:3.1:compile
> {code}
> HADOOP-12767
> to move the uses of httpclient HADOOP-10105
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-5262 :
> http/conn/ssl/SSLConnectionSocketFactory.java in Apache HttpComponents
> HttpClient before 4.3.6 ignores the http.socket.timeout configuration setting
> during an SSL handshake, which allows remote attackers to cause a denial of
> service (HTTPS call hang) via unspecified vectors.
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-6153
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-5783
> Apache Commons HttpClient 3.x, as used in Amazon Flexible Payments
> Service (FPS) merchant Java SDK and other products, does not verify that the
> server hostname matches a domain name in the subject's Common Name (CN) or
> subjectAltName field of the X.509 certificate, which allows man-in-the-middle
> attackers to spoof SSL servers via an arbitrary valid certificate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16385) Have hbase-rest pull hbase.rest.port from Constants.java
[
https://issues.apache.org/jira/browse/HBASE-16385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416076#comment-15416076
]
stack commented on HBASE-16385:
---
You good w/ the patch [~dimaspivak]?
> Have hbase-rest pull hbase.rest.port from Constants.java
>
>
> Key: HBASE-16385
> URL: https://issues.apache.org/jira/browse/HBASE-16385
> Project: HBase
> Issue Type: Improvement
> Components: REST
>Reporter: Dima Spivak
>Assignee: Yi Liang
>Priority: Minor
> Labels: beginner
> Fix For: 2.0.0
>
> Attachments: HBASE-16385-V1.patch
>
>
> [A while back|https://issues.apache.org/jira/browse/HBASE-3272], we
> [hardcoded|https://github.com/apache/hbase/blob/master/hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/RESTServer.java#L266]
> the {{hbase.rest.port}} instead of getting it from
> [Constants.java|https://github.com/apache/hbase/blob/master/hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/Constants.java#L36].
> We should fix that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16267) Remove commons-httpclient dependency from hbase-rest module
[
https://issues.apache.org/jira/browse/HBASE-16267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416067#comment-15416067
]
stack commented on HBASE-16267:
---
+1 on patch. Fix the release note. Doesn't make mention of why and doesn't make
sense as written (what is the 'it' referred to). You want to hoist the CVE up
into the release note? That'd help folks trying to figure why this issue.
Thanks.
> Remove commons-httpclient dependency from hbase-rest module
> ---
>
> Key: HBASE-16267
> URL: https://issues.apache.org/jira/browse/HBASE-16267
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 16267.v10.txt, 16267.v11.txt, 16267.v12.txt,
> 16267.v13.txt, 16267.v14.txt, 16267.v15.txt, 16267.v2.txt, 16267.v4.txt,
> 16267.v6.txt, 16267.v8.txt, 16267.v9.txt
>
>
> hbase-rest module still has imports from org.apache.commons.httpclient .
> There is more work to be done after HBASE-15767 was integrated.
> In master branch, there seems to be transitive dependency which allows the
> code to compile:
> {code}
> [INFO] +- org.apache.hadoop:hadoop-common:jar:2.7.1:compile
> [INFO] | +- org.apache.hadoop:hadoop-annotations:jar:2.7.1:compile
> [INFO] | +- commons-cli:commons-cli:jar:1.2:compile
> [INFO] | +- org.apache.commons:commons-math3:jar:3.1.1:compile
> [INFO] | +- xmlenc:xmlenc:jar:0.52:compile
> [INFO] | +- commons-httpclient:commons-httpclient:jar:3.1:compile
> {code}
> HADOOP-12767
> to move the uses of httpclient HADOOP-10105
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-5262 :
> http/conn/ssl/SSLConnectionSocketFactory.java in Apache HttpComponents
> HttpClient before 4.3.6 ignores the http.socket.timeout configuration setting
> during an SSL handshake, which allows remote attackers to cause a denial of
> service (HTTPS call hang) via unspecified vectors.
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-6153
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-5783
> Apache Commons HttpClient 3.x, as used in Amazon Flexible Payments
> Service (FPS) merchant Java SDK and other products, does not verify that the
> server hostname matches a domain name in the subject's Common Name (CN) or
> subjectAltName field of the X.509 certificate, which allows man-in-the-middle
> attackers to spoof SSL servers via an arbitrary valid certificate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16267) Remove commons-httpclient dependency from hbase-rest module
[
https://issues.apache.org/jira/browse/HBASE-16267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16267:
---
Attachment: 16267.v15.txt
Patch v15 with the change to TestLogLevel suggested by Stack.
I checked dependency tree which doesn't show dependency on commons-httpclient
in shaded module.
> Remove commons-httpclient dependency from hbase-rest module
> ---
>
> Key: HBASE-16267
> URL: https://issues.apache.org/jira/browse/HBASE-16267
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 16267.v10.txt, 16267.v11.txt, 16267.v12.txt,
> 16267.v13.txt, 16267.v14.txt, 16267.v15.txt, 16267.v2.txt, 16267.v4.txt,
> 16267.v6.txt, 16267.v8.txt, 16267.v9.txt
>
>
> hbase-rest module still has imports from org.apache.commons.httpclient .
> There is more work to be done after HBASE-15767 was integrated.
> In master branch, there seems to be transitive dependency which allows the
> code to compile:
> {code}
> [INFO] +- org.apache.hadoop:hadoop-common:jar:2.7.1:compile
> [INFO] | +- org.apache.hadoop:hadoop-annotations:jar:2.7.1:compile
> [INFO] | +- commons-cli:commons-cli:jar:1.2:compile
> [INFO] | +- org.apache.commons:commons-math3:jar:3.1.1:compile
> [INFO] | +- xmlenc:xmlenc:jar:0.52:compile
> [INFO] | +- commons-httpclient:commons-httpclient:jar:3.1:compile
> {code}
> HADOOP-12767
> to move the uses of httpclient HADOOP-10105
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-5262 :
> http/conn/ssl/SSLConnectionSocketFactory.java in Apache HttpComponents
> HttpClient before 4.3.6 ignores the http.socket.timeout configuration setting
> during an SSL handshake, which allows remote attackers to cause a denial of
> service (HTTPS call hang) via unspecified vectors.
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-6153
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-5783
> Apache Commons HttpClient 3.x, as used in Amazon Flexible Payments
> Service (FPS) merchant Java SDK and other products, does not verify that the
> server hostname matches a domain name in the subject's Common Name (CN) or
> subjectAltName field of the X.509 certificate, which allows man-in-the-middle
> attackers to spoof SSL servers via an arbitrary valid certificate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16391) Multiple backup/restore sessions support
Vladimir Rodionov created HBASE-16391: - Summary: Multiple backup/restore sessions support Key: HBASE-16391 URL: https://issues.apache.org/jira/browse/HBASE-16391 Project: HBase Issue Type: New Feature Affects Versions: HBASE-7912 Reporter: Vladimir Rodionov Fix For: HBASE-7912 Multiple simultaneous sessions support for backup/restore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14138) HBase Backup/Restore Phase 3: Security
[ https://issues.apache.org/jira/browse/HBASE-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-14138: -- Description: Security is not supported. Only authorized user (GLOBAL ADMIN) must be allowed to perform backup/restore. See: HBASE-7367 for good discussion on snapshot security model. * Backup between two secured (Kerberos) clusters (Cross-realm authentication is required to use distcp/export snapshot between two secured cluster?) * Backup between secured (Kerberos) and secured non-Kerberos cluster (AWS) * Backup between secured and unsecured cluster * Restore between two Kerberos clusters * Restore from non-Kerberos (AWS) to Kerberos * Restore from unsecured to secured (Kerberos) * Users must be able to run backup/restore for table if they have admin privileges for a table (?) Some relevant JIRAs https://issues.apache.org/jira/browse/HADOOP-8828 https://issues.apache.org/jira/browse/HDFS-6776 Links: http://henning.kropponline.de/2015/10/04/distcp-between-kerberized-and-none-kerberized-cluster/ http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_Sys_Admin_Guides/content/distcp_and_security_settings.html https://discuss.zendesk.com/hc/en-us/articles/203176207-Setting-up-a-kerberos-cross-realm-trust-for-distcp http://www.cloudera.com/documentation/enterprise/5-5-x/topics/cdh_admin_distcp_secure_insecure.html https://www.cloudera.com/documentation/enterprise/5-4-x/topics/cdh_admin_distcp_data_cluster_migrate.html https://www.cloudera.com/documentation/enterprise/5-7-x/topics/cdh_admin_distcp_data_cluster_migrate.html was: Security is not supported. Only authorized user (GLOBAL ADMIN) must be allowed to perform backup/restore. See: HBASE-7367 for good discussion on snapshot security model. Some relevant JIRAs https://issues.apache.org/jira/browse/HADOOP-8828 > HBase Backup/Restore Phase 3: Security > -- > > Key: HBASE-14138 > URL: https://issues.apache.org/jira/browse/HBASE-14138 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu > Labels: backup > Fix For: 2.0.0 > > > Security is not supported. Only authorized user (GLOBAL ADMIN) must be > allowed to perform backup/restore. See: HBASE-7367 for good discussion on > snapshot security model. > * Backup between two secured (Kerberos) clusters (Cross-realm authentication > is required to use distcp/export snapshot between two secured cluster?) > * Backup between secured (Kerberos) and secured non-Kerberos cluster (AWS) > * Backup between secured and unsecured cluster > * Restore between two Kerberos clusters > * Restore from non-Kerberos (AWS) to Kerberos > * Restore from unsecured to secured (Kerberos) > * Users must be able to run backup/restore for table if they have admin > privileges for a table (?) > Some relevant JIRAs > https://issues.apache.org/jira/browse/HADOOP-8828 > https://issues.apache.org/jira/browse/HDFS-6776 > Links: > http://henning.kropponline.de/2015/10/04/distcp-between-kerberized-and-none-kerberized-cluster/ > http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_Sys_Admin_Guides/content/distcp_and_security_settings.html > https://discuss.zendesk.com/hc/en-us/articles/203176207-Setting-up-a-kerberos-cross-realm-trust-for-distcp > http://www.cloudera.com/documentation/enterprise/5-5-x/topics/cdh_admin_distcp_secure_insecure.html > https://www.cloudera.com/documentation/enterprise/5-4-x/topics/cdh_admin_distcp_data_cluster_migrate.html > https://www.cloudera.com/documentation/enterprise/5-7-x/topics/cdh_admin_distcp_data_cluster_migrate.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16267) Remove commons-httpclient dependency from hbase-rest module
[
https://issues.apache.org/jira/browse/HBASE-16267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416012#comment-15416012
]
stack commented on HBASE-16267:
---
To remove dependency on commons-httpclient (and HttpURLConnection) by test
TestLogLevel, do the following change:
{code}
diff --git
a/hbase-server/src/test/java/org/apache/hadoop/hbase/http/log/TestLogLevel.java
b/hbase-server/src/test/java/org/apache/hadoop/hbase/http/log/TestLogLevel.java
index d7942d1..e14e3b4 100644
---
a/hbase-server/src/test/java/org/apache/hadoop/hbase/http/log/TestLogLevel.java
+++
b/hbase-server/src/test/java/org/apache/hadoop/hbase/http/log/TestLogLevel.java
@@ -61,18 +61,12 @@ public class TestLogLevel {
.getConnectorAddress(0));
//servlet
-URL url = new URL("http://; + authority + "/logLevel?log=" + logName
-+ "=" + Level.ERROR);
+URL url =
+new URL("http://; + authority + "/logLevel?log=" + logName +
"=" + Level.ERROR);
out.println("*** Connecting to " + url);
-HttpURLConnection connection = (HttpURLConnection)url.openConnection();
-connection.connect();
-
-BufferedReader in = new BufferedReader(new InputStreamReader(
-connection.getInputStream()));
-for(String line; (line = in.readLine()) != null; out.println(line));
-in.close();
-connection.disconnect();
-
+try (BufferedReader in = new BufferedReader(new
InputStreamReader(url.openStream( {
+ for(String line; (line = in.readLine()) != null; out.println(line));
+}
log.debug("log.debug2");
log.info("log.info2");
log.error("log.error2");
{code}
It looks like there is a reference to commons httpclient in shaded module too
that should be removed if purging commons httpclient.
> Remove commons-httpclient dependency from hbase-rest module
> ---
>
> Key: HBASE-16267
> URL: https://issues.apache.org/jira/browse/HBASE-16267
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 16267.v10.txt, 16267.v11.txt, 16267.v12.txt,
> 16267.v13.txt, 16267.v14.txt, 16267.v2.txt, 16267.v4.txt, 16267.v6.txt,
> 16267.v8.txt, 16267.v9.txt
>
>
> hbase-rest module still has imports from org.apache.commons.httpclient .
> There is more work to be done after HBASE-15767 was integrated.
> In master branch, there seems to be transitive dependency which allows the
> code to compile:
> {code}
> [INFO] +- org.apache.hadoop:hadoop-common:jar:2.7.1:compile
> [INFO] | +- org.apache.hadoop:hadoop-annotations:jar:2.7.1:compile
> [INFO] | +- commons-cli:commons-cli:jar:1.2:compile
> [INFO] | +- org.apache.commons:commons-math3:jar:3.1.1:compile
> [INFO] | +- xmlenc:xmlenc:jar:0.52:compile
> [INFO] | +- commons-httpclient:commons-httpclient:jar:3.1:compile
> {code}
> HADOOP-12767
> to move the uses of httpclient HADOOP-10105
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-5262 :
> http/conn/ssl/SSLConnectionSocketFactory.java in Apache HttpComponents
> HttpClient before 4.3.6 ignores the http.socket.timeout configuration setting
> during an SSL handshake, which allows remote attackers to cause a denial of
> service (HTTPS call hang) via unspecified vectors.
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-6153
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-5783
> Apache Commons HttpClient 3.x, as used in Amazon Flexible Payments
> Service (FPS) merchant Java SDK and other products, does not verify that the
> server hostname matches a domain name in the subject's Common Name (CN) or
> subjectAltName field of the X.509 certificate, which allows man-in-the-middle
> attackers to spoof SSL servers via an arbitrary valid certificate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16098) Chaos Monkey shouldn't run during ITBLL's RandomInputGenerator phase
[ https://issues.apache.org/jira/browse/HBASE-16098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415992#comment-15415992 ] Andrew Purtell commented on HBASE-16098: I just had an ITBLL job fail after a long generation step because chaos added a junk family and then removed it while the load job was starting. The table input format wasn't happy about the suddenly missing family while calculating splits and bailed. > Chaos Monkey shouldn't run during ITBLL's RandomInputGenerator phase > > > Key: HBASE-16098 > URL: https://issues.apache.org/jira/browse/HBASE-16098 > Project: HBase > Issue Type: Bug > Components: integration tests >Reporter: Dima Spivak >Priority: Minor > > Noticed that we get a bunch of exceptions (though none that stop the run) as > a result of chaos monkey running while generating the random inputs to be > used by ITBLL generator. Mostly stuff related to the fact that we try moving > things around on a table that doesn't exist yet. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14138) HBase Backup/Restore Phase 3: Security
[ https://issues.apache.org/jira/browse/HBASE-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-14138: -- Description: Security is not supported. Only authorized user (GLOBAL ADMIN) must be allowed to perform backup/restore. See: HBASE-7367 for good discussion on snapshot security model. Some relevant JIRAs https://issues.apache.org/jira/browse/HADOOP-8828 was:Security is not supported. Only authorized user (GLOBAL ADMIN) must be allowed to perform backup/restore. See: HBASE-7367 for good discussion on snapshot security model. > HBase Backup/Restore Phase 3: Security > -- > > Key: HBASE-14138 > URL: https://issues.apache.org/jira/browse/HBASE-14138 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu > Labels: backup > Fix For: 2.0.0 > > > Security is not supported. Only authorized user (GLOBAL ADMIN) must be > allowed to perform backup/restore. See: HBASE-7367 for good discussion on > snapshot security model. > Some relevant JIRAs > https://issues.apache.org/jira/browse/HADOOP-8828 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16267) Remove commons-httpclient dependency from hbase-rest module
[
https://issues.apache.org/jira/browse/HBASE-16267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415967#comment-15415967
]
Ted Yu commented on HBASE-16267:
bq. It seems like HttpURLConnection is using httpclient as a provider. Can you
change that by setting a config on the JVM?
{code}
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
{code}
HttpURLConnection is in java.net.
Not sure how the provider can be changed at JVM level.
> Remove commons-httpclient dependency from hbase-rest module
> ---
>
> Key: HBASE-16267
> URL: https://issues.apache.org/jira/browse/HBASE-16267
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 16267.v10.txt, 16267.v11.txt, 16267.v12.txt,
> 16267.v13.txt, 16267.v14.txt, 16267.v2.txt, 16267.v4.txt, 16267.v6.txt,
> 16267.v8.txt, 16267.v9.txt
>
>
> hbase-rest module still has imports from org.apache.commons.httpclient .
> There is more work to be done after HBASE-15767 was integrated.
> In master branch, there seems to be transitive dependency which allows the
> code to compile:
> {code}
> [INFO] +- org.apache.hadoop:hadoop-common:jar:2.7.1:compile
> [INFO] | +- org.apache.hadoop:hadoop-annotations:jar:2.7.1:compile
> [INFO] | +- commons-cli:commons-cli:jar:1.2:compile
> [INFO] | +- org.apache.commons:commons-math3:jar:3.1.1:compile
> [INFO] | +- xmlenc:xmlenc:jar:0.52:compile
> [INFO] | +- commons-httpclient:commons-httpclient:jar:3.1:compile
> {code}
> HADOOP-12767
> to move the uses of httpclient HADOOP-10105
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-5262 :
> http/conn/ssl/SSLConnectionSocketFactory.java in Apache HttpComponents
> HttpClient before 4.3.6 ignores the http.socket.timeout configuration setting
> during an SSL handshake, which allows remote attackers to cause a denial of
> service (HTTPS call hang) via unspecified vectors.
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-6153
> https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2012-5783
> Apache Commons HttpClient 3.x, as used in Amazon Flexible Payments
> Service (FPS) merchant Java SDK and other products, does not verify that the
> server hostname matches a domain name in the subject's Common Name (CN) or
> subjectAltName field of the X.509 certificate, which allows man-in-the-middle
> attackers to spoof SSL servers via an arbitrary valid certificate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16379) [replication] Minor improvement to replication/copy_tables_desc.rb
[ https://issues.apache.org/jira/browse/HBASE-16379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415948#comment-15415948 ] Hudson commented on HBASE-16379: FAILURE: Integrated in HBase-1.3 #814 (See [https://builds.apache.org/job/HBase-1.3/814/]) HBASE-16379 [replication] Minor improvement to (esteban: rev 66c23e37003a97778840f12cb25f65f51ce677fe) * bin/replication/copy_tables_desc.rb > [replication] Minor improvement to replication/copy_tables_desc.rb > -- > > Key: HBASE-16379 > URL: https://issues.apache.org/jira/browse/HBASE-16379 > Project: HBase > Issue Type: Improvement > Components: Replication, shell >Affects Versions: 1.3.0, 1.2.2 >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez >Priority: Trivial > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3, 1.1.7 > > Attachments: > 0001-HBASE-16379-replication-Minor-improvement-to-replica.patch, > 0001-HBASE-16379-replication-Minor-improvement-to-replica.patch, > 0001-HBASE-16379-replication-Minor-improvement-to-replica.patch, > HBASE-16379.patch > > > copy_tables_desc.rb is helpful for quickly setting up a table remotely based > on an existing schema. However it does copy by default all tables. Now you > can pass a list of tables as an optional third argument and it will also > display what table descriptors where copied. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14345) Consolidate printUsage in IntegrationTestLoadAndVerify
[
https://issues.apache.org/jira/browse/HBASE-14345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415947#comment-15415947
]
Hudson commented on HBASE-14345:
FAILURE: Integrated in HBase-1.3 #814 (See
[https://builds.apache.org/job/HBase-1.3/814/])
HBASE-14345 Consolidate printUsage in IntegrationTestLoadAndVerify (Reid (appy:
rev 2759b4a4a36b62f0d5595785ab5e91bd7a863a83)
*
hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestLoadAndVerify.java
*
hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestWithCellVisibilityLoadAndVerify.java
> Consolidate printUsage in IntegrationTestLoadAndVerify
> --
>
> Key: HBASE-14345
> URL: https://issues.apache.org/jira/browse/HBASE-14345
> Project: HBase
> Issue Type: Improvement
> Components: integration tests
>Reporter: Nick Dimiduk
>Assignee: Reid Chan
>Priority: Trivial
> Labels: beginner
> Fix For: 2.0.0, 1.3.0, 1.0.4, 1.4.0, 1.1.6, 1.2.3
>
> Attachments: HBASE-14345.002.patch, HBASE-14345.003.patch,
> HBASE-14345.004.patch, HBASE-14345.master.001.patch,
> HBASE-14345.master.002.patch, HBASE-14345.patch, itlav-2016-07-07.png,
> itlv.png
>
>
> Investigating the use of {{itlav}} is a little screwy. Subclasses are not
> overriding the {{printUsage()}} methods correctly, so you have to pass
> {{--help}} to get some info and no arguments to get the rest.
> {noformat}
> [hbase@ndimiduk-112rc2-7 ~]$ hbase
> org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify --help
> usage: bin/hbase org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify
>
> Options:
> -h,--help Show usage
> -m,--monkey Which chaos monkey to run
> -monkeyProps The properties file for specifying chaos monkey
> properties.
> -ncc,--noClusterCleanUp Don't clean up the cluster at the end
> [hbase@ndimiduk-112rc2-7 ~]$ hbase
> org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify
> IntegrationTestLoadAndVerify [-Doptions]
> Loads a table with row dependencies and verifies the dependency chains
> Options
> -Dloadmapper.table=Table to write/verify (default autogen)
> -Dloadmapper.backrefs=Number of backreferences per row (default
> 50)
> -Dloadmapper.num_to_write=Number of rows per mapper (default 100,000
> per mapper)
> -Dloadmapper.deleteAfter= Delete after a successful verify (default
> true)
> -Dloadmapper.numPresplits=Number of presplit regions to start with
> (default 40)
> -Dloadmapper.map.tasks= Number of map tasks for load (default 200)
> -Dverify.reduce.tasks=Number of reduce tasks for verify (default
> 35)
> -Dverify.scannercaching= Number hbase scanner caching rows to read
> (default 50)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
