[jira] [Updated] (HBASE-10656) high-scale-lib's Counter depends on Oracle (Sun) JRE, and also has some bug
[ https://issues.apache.org/jira/browse/HBASE-10656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hiroshi Ikeda updated HBASE-10656: -- Attachment: MyCounter2.java Added an alternative counter class, using some logic from LongAdder in Java8. The counter reuse resources when extending the internal hash table, so that it avoids complexity of race condition caused by the extending, and it avoid to hold old unused resources and their sum cache for performance. Instead, the counter has to create an instance per each value holder. I think this may be overhead, but each instance can be independently placed in memory so that this may help to reduce cache line contention. Like LongAdder, each value holder is surrounded by 2 of fixed size pads. Each of the pads consists of 7 long values (64bit * 7). That means, this expects the size of cache line is fixed to about 64bit * 16. This counter simply uses Thread.getId() as hashcode. The reversed bit of the id is well spreaded, but cache line contention is not avoidable before every extending the hash table if the estimate size of cache line is small. With starting small hash size, this may come frequently in the early stages, and anyone don't know whether the extendings will be stopped at a good point or not. Unlike LongAdder, when this counter creates the internal hash table (that is an array of references to value holders), this counter fills it with value holder instances. Creating instances on demand is too complex to keep consistency. In source code of LongAdder I referred, I didn't find out what ensures happens-before relation between putting value holders on demand to the internal array and referring them from the array in order to sum up the values. Besides the above concerns, in my environment, the added counter seems to have enough performance to (a bit faster than) high-scale-lib even without mixing read operations. high-scale-lib's Counter depends on Oracle (Sun) JRE, and also has some bug Key: HBASE-10656 URL: https://issues.apache.org/jira/browse/HBASE-10656 Project: HBase Issue Type: Bug Reporter: Hiroshi Ikeda Priority: Minor Attachments: MyCounter.java, MyCounter2.java, MyCounterTest.java, MyCounterTest.java Cliff's high-scale-lib's Counter is used in important classes (for example, HRegion) in HBase, but Counter uses sun.misc.Unsafe, that is implementation detail of the Java standard library and belongs to Oracle (Sun). That consequently makes HBase depend on the specific JRE Implementation. To make matters worse, Counter has a bug and you may get wrong result if you mix a reading method into your logic calling writing methods. In more detail, I think the bug is caused by reading an internal array field without resolving memory caching, which is intentional the comment says, but storing the read result into a volatile field. That field may be not changed after you can see the true values of the array field, and also may be not changed after updating the next CAT instance's values in some race condition when extending CAT instance chain. Anyway, it is possible that you create a new alternative class which only depends on the standard library. I know Java8 provides its alternative, but HBase should support Java6 and Java7 for some time. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-10679) Both clients operating on a same region will get wrong scan results if the first scanner expires and the second scanner is created with the same scannerId
Feng Honghua created HBASE-10679: Summary: Both clients operating on a same region will get wrong scan results if the first scanner expires and the second scanner is created with the same scannerId Key: HBASE-10679 URL: https://issues.apache.org/jira/browse/HBASE-10679 Project: HBase Issue Type: Bug Components: regionserver Reporter: Feng Honghua Assignee: Feng Honghua Priority: Critical The scenario is as below (both Client A and Client B scan against Region R) # A opens a scanner SA on R, the scannerId is N, it successfully get its first row a # SA's lease expires and it's removed from scanners # B opens a scanner SB on R, the scannerId is N too. it successfully get its first row m # A issues its second scan request with scannerId N, regionserver finds N is valid scannerId and the region matches too. (since the region is always online on this regionserver and both two scanners are against it), so it executes scan request on SB, returns n to A -- wrong! (get data from other scanner, A expects row something like b that follows a) # B issues its second scan request with scannerId N, regionserver also thinks it's valid, and executes scan on SB, return o to B -- wrong! (should return n but n has been scanned out by A just now) The consequence is both clients get wrong scan results: # A gets data from scanner created by other client, its own scanner has expired and removed # B misses data which should be gotten but has been wrongly scanned out by A The root cause is scannerId generated by regionserver can't be guaranteed unique within regionserver's whole lifecycle, *there is only guarantee that scannerIds of scanners that are currently still valid (not expired) are unique*, so a same scannerId can present in scanners again after a former scanner with this scannerId expires and has been removed from scanners. And if the second scanner is against the same region, the bug arises. Theoretically, the possibility of above scenario should be very rare(two consecutive scans on a same region from two different clients get a same scannerId, and the first expires before the second is created), but it does can happen, and once it happens, the consequence is severe(all clients involved get wrong data), and should be extremely hard to diagnose/debug -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10679) Both clients operating on a same region will get wrong scan results if the first scanner expires and the second scanner is created with the same scannerId
[ https://issues.apache.org/jira/browse/HBASE-10679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Feng Honghua updated HBASE-10679: - Attachment: HBASE-10679-trunk_v1.patch Patch attached Both clients operating on a same region will get wrong scan results if the first scanner expires and the second scanner is created with the same scannerId -- Key: HBASE-10679 URL: https://issues.apache.org/jira/browse/HBASE-10679 Project: HBase Issue Type: Bug Components: regionserver Reporter: Feng Honghua Assignee: Feng Honghua Priority: Critical Attachments: HBASE-10679-trunk_v1.patch The scenario is as below (both Client A and Client B scan against Region R) # A opens a scanner SA on R, the scannerId is N, it successfully get its first row a # SA's lease expires and it's removed from scanners # B opens a scanner SB on R, the scannerId is N too. it successfully get its first row m # A issues its second scan request with scannerId N, regionserver finds N is valid scannerId and the region matches too. (since the region is always online on this regionserver and both two scanners are against it), so it executes scan request on SB, returns n to A -- wrong! (get data from other scanner, A expects row something like b that follows a) # B issues its second scan request with scannerId N, regionserver also thinks it's valid, and executes scan on SB, return o to B -- wrong! (should return n but n has been scanned out by A just now) The consequence is both clients get wrong scan results: # A gets data from scanner created by other client, its own scanner has expired and removed # B misses data which should be gotten but has been wrongly scanned out by A The root cause is scannerId generated by regionserver can't be guaranteed unique within regionserver's whole lifecycle, *there is only guarantee that scannerIds of scanners that are currently still valid (not expired) are unique*, so a same scannerId can present in scanners again after a former scanner with this scannerId expires and has been removed from scanners. And if the second scanner is against the same region, the bug arises. Theoretically, the possibility of above scenario should be very rare(two consecutive scans on a same region from two different clients get a same scannerId, and the first expires before the second is created), but it does can happen, and once it happens, the consequence is severe(all clients involved get wrong data), and should be extremely hard to diagnose/debug -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-8604) improve reporting of incorrect peer address in replication
[ https://issues.apache.org/jira/browse/HBASE-8604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rekha Joshi updated HBASE-8604: --- Attachment: HBASE-8604.1.patch Patch available. improve reporting of incorrect peer address in replication -- Key: HBASE-8604 URL: https://issues.apache.org/jira/browse/HBASE-8604 Project: HBase Issue Type: Improvement Components: Replication Affects Versions: 0.94.6 Reporter: Roman Shaposhnik Assignee: Rekha Joshi Priority: Minor Fix For: 0.94.19 Attachments: HBASE-8604.1.patch I was running some replication code that incorrectly advertised the peer address for replication. HBase complained that the format of the record was NOT what it was expecting but it didn't include what it saw in the exception message. Including that string would help cutting down the time it takes to debug issues like that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-8604) improve reporting of incorrect peer address in replication
[ https://issues.apache.org/jira/browse/HBASE-8604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rekha Joshi updated HBASE-8604: --- Fix Version/s: 0.94.19 Assignee: Rekha Joshi Status: Patch Available (was: Open) Patch available. improve reporting of incorrect peer address in replication -- Key: HBASE-8604 URL: https://issues.apache.org/jira/browse/HBASE-8604 Project: HBase Issue Type: Improvement Components: Replication Affects Versions: 0.94.6 Reporter: Roman Shaposhnik Assignee: Rekha Joshi Priority: Minor Fix For: 0.94.19 Attachments: HBASE-8604.1.patch I was running some replication code that incorrectly advertised the peer address for replication. HBase complained that the format of the record was NOT what it was expecting but it didn't include what it saw in the exception message. Including that string would help cutting down the time it takes to debug issues like that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-10680) Check if the block keys, index keys can be used as Cells instead of byte[]
ramkrishna.s.vasudevan created HBASE-10680: -- Summary: Check if the block keys, index keys can be used as Cells instead of byte[] Key: HBASE-10680 URL: https://issues.apache.org/jira/browse/HBASE-10680 Project: HBase Issue Type: Sub-task Affects Versions: 0.99.0 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan While doing HBASE-10531 we can observe that most of the comparison happens between the key part of the KVs. If suppose we need to make use of Cells and get rid of Kv.getBuffer() then the Keys with which the comparison are done like the block keys, index keys etc should also be changed to Cells and internally do the comparison as cell. This issue would help to brain storm this and implement the feasible soln. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10678) Make verifyrep tool implement toolrunner
[ https://issues.apache.org/jira/browse/HBASE-10678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13920677#comment-13920677 ] Matteo Bertozzi commented on HBASE-10678: - +1 Make verifyrep tool implement toolrunner Key: HBASE-10678 URL: https://issues.apache.org/jira/browse/HBASE-10678 Project: HBase Issue Type: Improvement Components: Replication Affects Versions: 0.99.0 Reporter: bharath v Assignee: bharath v Priority: Minor Attachments: HBASE-10678-trunk-v1.patch Currently verifyrep tool doesn't take -D args since it doesn't implement the toolrunner interface. So we need to make changes to config files on the client everytime we need run it with custom arguments. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10594) Speed up TestRestoreSnapshotFromClient a bit
[ https://issues.apache.org/jira/browse/HBASE-10594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13920678#comment-13920678 ] Matteo Bertozzi commented on HBASE-10594: - +1 looks good to me, thanks! Speed up TestRestoreSnapshotFromClient a bit Key: HBASE-10594 URL: https://issues.apache.org/jira/browse/HBASE-10594 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Priority: Minor Attachments: 10594-0.94-v2.txt, 10594-0.94.txt, 10594-trunk-v2.txt, 10594-trunk.txt Looking through the longest running test in 0.94 I noticed that TestRestoreSnapshotFromClient runs for over 10 minutes on the jenkins boxes (264s on my local box). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10018) Change the location prefetch
[ https://issues.apache.org/jira/browse/HBASE-10018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13920682#comment-13920682 ] Nicolas Liochon commented on HBASE-10018: - Any opinion? By default I would go for the patch as it is. Change the location prefetch Key: HBASE-10018 URL: https://issues.apache.org/jira/browse/HBASE-10018 Project: HBase Issue Type: Bug Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.99.0 Attachments: 10018.v1.patch, 10018.v2.patch, 10018.v4.patch, 10018v3.patch Issues with prefetching are: - we do two calls to meta: one for the exact row, one for the prefetch - it's done in a lock - we take the next 10 regions. Why 10, why the 10 next? - is it useful if the table has 100K regions? Options are: - just remove it - replace it with a reverse scan: this would save a call -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10355) Failover RPC's from client using region replicas
[ https://issues.apache.org/jira/browse/HBASE-10355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13920683#comment-13920683 ] Nicolas Liochon commented on HBASE-10355: - This was reviewed before. I changes some stuff in the HBASE-10356 since the previous +1. Any issue if I commit this patch as it is in the branch 10070? Failover RPC's from client using region replicas Key: HBASE-10355 URL: https://issues.apache.org/jira/browse/HBASE-10355 Project: HBase Issue Type: Sub-task Components: Client Reporter: Enis Soztutar Assignee: Nicolas Liochon Fix For: 0.99.0 Attachments: 10355.v1.patch, 10355.v2.patch, 10355.v3.patch, 10355.v4.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10665) TestCompaction and TestCompactionWithCoprocessor run too long
[
https://issues.apache.org/jira/browse/HBASE-10665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13920684#comment-13920684
]
Hadoop QA commented on HBASE-10665:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12632787/10665.patch
against trunk revision .
ATTACHMENT ID: 12632787
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 20 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8895//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8895//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8895//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8895//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8895//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8895//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8895//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8895//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8895//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8895//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8895//console
This message is automatically generated.
TestCompaction and TestCompactionWithCoprocessor run too long
-
Key: HBASE-10665
URL: https://issues.apache.org/jira/browse/HBASE-10665
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.0
Reporter: Andrew Purtell
Fix For: 0.98.1, 0.99.0
Attachments: 10665.patch
584 seconds each
TestCompaction:
{noformat}
Forking command line: /bin/sh -c cd /data/src/hbase/hbase-server
/usr/lib/jvm/java-1.7.0.45-oracle-amd64/jre/bin/java -enableassertions
-Xmx1900m -XX:MaxPermSize=100m -Djava.security.egd=file:/dev/./urandom
-Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -jar
/data/src/hbase/hbase-server/target/surefire/surefirebooter5980733570856201818.jar
/data/src/hbase/hbase-server/target/surefire/surefire4520171250819563114tmp
/data/src/hbase/hbase-server/target/surefire/surefire_2794381603824180144412tmp
Running org.apache.hadoop.hbase.regionserver.TestCompaction
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 584.609 sec
{noformat}
TestCompactionWithCoprocessor:
{noformat}
Forking command line: /bin/sh -c cd /data/src/hbase/hbase-server
/usr/lib/jvm/java-1.7.0.45-oracle-amd64/jre/bin/java -enableassertions
-Xmx1900m -XX:MaxPermSize=100m -Djava.security.egd=file:/dev/./urandom
-Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -jar
/data/src/hbase/hbase-server/target/surefire/surefirebooter7194368346045889527.jar
/data/src/hbase/hbase-server/target/surefire/surefire9025480282422315585tmp
/data/src/hbase/hbase-server/target/surefire/surefire_2815590620956840351617tmp
Running org.apache.hadoop.hbase.regionserver.TestCompactionWithCoprocessor
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 584.399 sec
{noformat}
Slim down or split up.
[jira] [Created] (HBASE-10681) Allow Hbase artifacts to deploy to remote repo other than apache
Guo Ruijing created HBASE-10681:
---
Summary: Allow Hbase artifacts to deploy to remote repo other than
apache
Key: HBASE-10681
URL: https://issues.apache.org/jira/browse/HBASE-10681
Project: HBase
Issue Type: Improvement
Components: build
Reporter: Guo Ruijing
pom.xml in hbase write as
distributionManagement
site
idhbase.apache.org/id
nameHBase Website at hbase.apache.org/name
!-- On why this is the tmp dir and not hbase.apache.org, see
https://issues.apache.org/jira/browse/HBASE-7593?focusedCommentId=13555866page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13555866
--
urlfile:///tmp/url
/site
We expect pom.xml in hbase like hadoop can be:
distributionManagement
repository
id${distMgmtStagingId}/id
name${distMgmtStagingName}/name
url${distMgmtStagingUrl}/url
/repository
snapshotRepository
id${distMgmtSnapshotsId}/id
name${distMgmtSnapshotsName}/name
url${distMgmtSnapshotsUrl}/url
/snapshotRepository
site
idhbase.apache.org/id
nameHBase Website at hbase.apache.org/name
!-- On why this is the tmp dir and not hbase.apache.org, see
https://issues.apache.org/jira/browse/HBASE-7593?focusedCommentId=13555866page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13555866
--
urlfile:///tmp/url
/site
/distributionManagement
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10677) boundaries check in hbck throwing IllegalArgumentException
[
https://issues.apache.org/jira/browse/HBASE-10677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13920693#comment-13920693
]
Hadoop QA commented on HBASE-10677:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12632788/HBASE-10677.patch
against trunk revision .
ATTACHMENT ID: 12632788
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8896//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8896//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8896//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8896//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8896//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8896//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8896//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8896//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8896//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8896//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8896//console
This message is automatically generated.
boundaries check in hbck throwing IllegalArgumentException
--
Key: HBASE-10677
URL: https://issues.apache.org/jira/browse/HBASE-10677
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 0.98.0
Reporter: rajeshbabu
Assignee: rajeshbabu
Fix For: 0.96.2, 0.98.1, 0.99.0
Attachments: HBASE-10677.patch, HBASE-10677_v1.patch
This is the exception
{code}
java.lang.IllegalArgumentException: Pathname
/hbase/hbase:labels/438f84dea5d0e24e390de927deb8a84e from
hdfs://10.18.40.29:54310/hbase/hbase:labels/438f84dea5d0e24e390de927deb8a84e
is not a valid DFS filename.
at
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:184)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:637)
at
org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:92)
at
org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:702)
at
org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:698)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:698)
at
org.apache.hadoop.hbase.util.HBaseFsck.checkRegionBoundaries(HBaseFsck.java:537)
at
org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:487)
at
[jira] [Assigned] (HBASE-9270) [0.94] FSTableDescriptors caching is racy
[
https://issues.apache.org/jira/browse/HBASE-9270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rekha Joshi reassigned HBASE-9270:
--
Assignee: Rekha Joshi
[0.94] FSTableDescriptors caching is racy
-
Key: HBASE-9270
URL: https://issues.apache.org/jira/browse/HBASE-9270
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.11
Reporter: Andrew Purtell
Assignee: Rekha Joshi
Priority: Minor
Attachments: HBASE-9270.1.patch
An occasionally failing test in 0.92 branch that concurrently executes master
operations on a single table found this problem in FSTableDescriptors:
{code}
diff --git src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
index e882621..b0042cd 100644
--- src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
+++ src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
@@ -221,8 +221,15 @@ public class FSTableDescriptors implements
TableDescriptors {
if
(HConstants.HBASE_NON_USER_TABLE_DIRS.contains(htd.getNameAsString())) {
throw new NotImplementedException();
}
-if (!this.fsreadonly) updateHTableDescriptor(this.fs, this.rootdir, htd);
-long modtime = getTableInfoModtime(this.fs, this.rootdir,
htd.getNameAsString());
+if (fsreadonly) {
+ // Cannot cache here.
+ // We can't know if a modtime from the most recent file found in a
+ // directory listing at some arbitrary point in time still corresponds
+ // to the latest, nor that our htd is the latest.
+ return;
+}
+// Cache with the modtime of the descriptor we wrote
+long modtime = updateHTableDescriptor(this.fs, this.rootdir,
htd).getModificationTime();
this.cache.put(htd.getNameAsString(), new
TableDescriptorModtime(modtime, htd));
}
{code}
After HBASE-7305 master operations are serialized by a write lock on the
table.
However, 0.94 has code with the same issue:
{code}
@Override
public void add(HTableDescriptor htd) throws IOException {
if (Bytes.equals(HConstants.ROOT_TABLE_NAME, htd.getName())) {
throw new NotImplementedException();
}
if (Bytes.equals(HConstants.META_TABLE_NAME, htd.getName())) {
throw new NotImplementedException();
}
if (HConstants.HBASE_NON_USER_TABLE_DIRS.contains(htd.getNameAsString()))
{
throw new NotImplementedException();
}
if (!this.fsreadonly) updateHTableDescriptor(this.fs, this.rootdir, htd);
String tableName = htd.getNameAsString();
long modtime = getTableInfoModtime(this.fs, this.rootdir, tableName);
long dirmodtime = getTableDirModtime(this.fs, this.rootdir, tableName);
this.cache.put(tableName, new TableDescriptorModtime(modtime, dirmodtime,
htd));
}
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Assigned] (HBASE-7193) [89-fb] Print the detail exceptions info from the RetriesExhaustedException
[ https://issues.apache.org/jira/browse/HBASE-7193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rekha Joshi reassigned HBASE-7193: -- Assignee: Rekha Joshi [89-fb] Print the detail exceptions info from the RetriesExhaustedException --- Key: HBASE-7193 URL: https://issues.apache.org/jira/browse/HBASE-7193 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Rekha Joshi Priority: Minor The hbase client only prints the name of exception for the RetriesExhaustedException logging purpose, which failed to provide any useful debug information. So this jira is to enhance the logging to print the entire stack track of the exception to help on issue investigation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-10682) region_mover.rb throws can't convert nil into String for regions moved
Matteo Bertozzi created HBASE-10682: --- Summary: region_mover.rb throws can't convert nil into String for regions moved Key: HBASE-10682 URL: https://issues.apache.org/jira/browse/HBASE-10682 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 0.94.17, 0.96.1.1, 0.98.0, 0.99.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-7193) [89-fb] Print the detail exceptions info from the RetriesExhaustedException
[ https://issues.apache.org/jira/browse/HBASE-7193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13920733#comment-13920733 ] Rekha Joshi commented on HBASE-7193: just a thought, would not RetriesExhaustedWithDetailsException and its methods suffice for your use-case? [89-fb] Print the detail exceptions info from the RetriesExhaustedException --- Key: HBASE-7193 URL: https://issues.apache.org/jira/browse/HBASE-7193 Project: HBase Issue Type: Improvement Reporter: Liyin Tang Assignee: Rekha Joshi Priority: Minor The hbase client only prints the name of exception for the RetriesExhaustedException logging purpose, which failed to provide any useful debug information. So this jira is to enhance the logging to print the entire stack track of the exception to help on issue investigation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10682) region_mover.rb throws can't convert nil into String for regions moved
[ https://issues.apache.org/jira/browse/HBASE-10682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-10682: Attachment: HBASE-10682-v0.patch HBASE-10682-0.94-v0.patch region_mover.rb throws can't convert nil into String for regions moved Key: HBASE-10682 URL: https://issues.apache.org/jira/browse/HBASE-10682 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 0.98.0, 0.99.0, 0.96.1.1, 0.94.17 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Attachments: HBASE-10682-0.94-v0.patch, HBASE-10682-v0.patch if we call unload to generate the region file and then one of the region in the splits or we end up with a null pointer exception during one of the print. if we skip that NPE then we end up with the split/moved error message, which is the expected one. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10682) region_mover.rb throws can't convert nil into String for regions moved
[ https://issues.apache.org/jira/browse/HBASE-10682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-10682: Description: if we call unload to generate the region file and then one of the region in the splits or we end up with a null pointer exception during one of the print. if we skip that NPE then we end up with the split/moved error message, which is the expected one. Release Note: (was: if we call unload to generate the region file and then one of the region in the splits or we end up with a null pointer exception during one of the print. if we skip that NPE then we end up with the split/moved error message, which is the expected one.) region_mover.rb throws can't convert nil into String for regions moved Key: HBASE-10682 URL: https://issues.apache.org/jira/browse/HBASE-10682 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 0.98.0, 0.99.0, 0.96.1.1, 0.94.17 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Attachments: HBASE-10682-0.94-v0.patch, HBASE-10682-v0.patch if we call unload to generate the region file and then one of the region in the splits or we end up with a null pointer exception during one of the print. if we skip that NPE then we end up with the split/moved error message, which is the expected one. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HBASE-10668) TestExportSnapshot runs too long
[
https://issues.apache.org/jira/browse/HBASE-10668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rekha Joshi reassigned HBASE-10668:
---
Assignee: Rekha Joshi
TestExportSnapshot runs too long
Key: HBASE-10668
URL: https://issues.apache.org/jira/browse/HBASE-10668
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.0
Reporter: Andrew Purtell
Assignee: Rekha Joshi
Fix For: 0.98.1, 0.99.0
332 seconds
{noformat}
Forking command line: /bin/sh -c cd /data/src/hbase/hbase-server
/usr/lib/jvm/java-1.7.0.45-oracle-amd64/jre/bin/java -enableassertions
-Xmx1900m -XX:MaxPermSize=100m -Djava.security.egd=file:/dev/./urandom
-Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -jar
/data/src/hbase/hbase-server/target/surefire/surefirebooter1668068702110669265.jar
/data/src/hbase/hbase-server/target/surefire/surefire5744357307851892501tmp
/data/src/hbase/hbase-server/target/surefire/surefire_3661340119563945183029tmp
Running org.apache.hadoop.hbase.snapshot.TestExportSnapshot
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 332.421 sec
{noformat}
Slim down or split up.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-10682) region_mover.rb throws can't convert nil into String for regions moved
[ https://issues.apache.org/jira/browse/HBASE-10682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-10682: Fix Version/s: 0.94.18 0.99.0 0.98.1 0.96.2 Status: Patch Available (was: Open) region_mover.rb throws can't convert nil into String for regions moved Key: HBASE-10682 URL: https://issues.apache.org/jira/browse/HBASE-10682 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 0.94.17, 0.96.1.1, 0.98.0, 0.99.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.18 Attachments: HBASE-10682-0.94-v0.patch, HBASE-10682-v0.patch if we call unload to generate the region file and then one of the region in the splits or we end up with a null pointer exception during one of the print. if we skip that NPE then we end up with the split/moved error message, which is the expected one. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10678) Make verifyrep tool implement toolrunner
[
https://issues.apache.org/jira/browse/HBASE-10678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13920737#comment-13920737
]
Hadoop QA commented on HBASE-10678:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12632792/HBASE-10678-trunk-v1.patch
against trunk revision .
ATTACHMENT ID: 12632792
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8897//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8897//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8897//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8897//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8897//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8897//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8897//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8897//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8897//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8897//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8897//console
This message is automatically generated.
Make verifyrep tool implement toolrunner
Key: HBASE-10678
URL: https://issues.apache.org/jira/browse/HBASE-10678
Project: HBase
Issue Type: Improvement
Components: Replication
Affects Versions: 0.99.0
Reporter: bharath v
Assignee: bharath v
Priority: Minor
Attachments: HBASE-10678-trunk-v1.patch
Currently verifyrep tool doesn't take -D args since it doesn't implement the
toolrunner interface. So we need to make changes to config files on the
client everytime we need run it with custom arguments.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Assigned] (HBASE-10664) TestImportExport runs too long
[
https://issues.apache.org/jira/browse/HBASE-10664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rekha Joshi reassigned HBASE-10664:
---
Assignee: Rekha Joshi
TestImportExport runs too long
--
Key: HBASE-10664
URL: https://issues.apache.org/jira/browse/HBASE-10664
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.0
Reporter: Andrew Purtell
Assignee: Rekha Joshi
Fix For: 0.98.1, 0.99.0
Debugging with -Dsurefire.firstPartForkMode=always
-Dsurefire.secondPartForkMode=always looking for a hanging test.
388 seconds.
{noformat}
Forking command line: /bin/sh -c cd /data/src/hbase/hbase-server
/usr/lib/jvm/java-1.7.0.45-oracle-amd64/jre/bin/java -enableassertions
-Xmx1900m -XX:MaxPermSize=100m -Djava.security.egd=file:/dev/./urandom
-Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -jar
/data/src/hbase/hbase-server/target/surefire/surefirebooter7637958208277391169.jar
/data/src/hbase/hbase-server/target/surefire/surefire6877889026110956843tmp
/data/src/hbase/hbase-server/target/surefire/surefire_1907837210788480451831tmp
Running org.apache.hadoop.hbase.mapreduce.TestImportExport
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 388.246 sec
{noformat}
Slim down or break it up.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Assigned] (HBASE-10667) TestEncodedSeekers runs too long
[
https://issues.apache.org/jira/browse/HBASE-10667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rekha Joshi reassigned HBASE-10667:
---
Assignee: Rekha Joshi
TestEncodedSeekers runs too long
Key: HBASE-10667
URL: https://issues.apache.org/jira/browse/HBASE-10667
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.0
Reporter: Andrew Purtell
Assignee: Rekha Joshi
Fix For: 0.98.1, 0.99.0
214 seconds, borderline
{noformat}
Forking command line: /bin/sh -c cd /data/src/hbase/hbase-server
/usr/lib/jvm/java-1.7.0.45-oracle-amd64/jre/bin/java -enableassertions
-Xmx1900m -XX:MaxPermSize=100m -Djava.security.egd=file:/dev/./urandom
-Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -jar
/data/src/hbase/hbase-server/target/surefire/surefirebooter309171614766381235.jar
/data/src/hbase/hbase-server/target/surefire/surefire1759919541562435761tmp
/data/src/hbase/hbase-server/target/surefire/surefire_274515185028609451271tmp
Running org.apache.hadoop.hbase.io.encoding.TestEncodedSeekers
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 214.105 sec
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-8604) improve reporting of incorrect peer address in replication
[
https://issues.apache.org/jira/browse/HBASE-8604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13920744#comment-13920744
]
Hadoop QA commented on HBASE-8604:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12632798/HBASE-8604.1.patch
against trunk revision .
ATTACHMENT ID: 12632798
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8899//console
This message is automatically generated.
improve reporting of incorrect peer address in replication
--
Key: HBASE-8604
URL: https://issues.apache.org/jira/browse/HBASE-8604
Project: HBase
Issue Type: Improvement
Components: Replication
Affects Versions: 0.94.6
Reporter: Roman Shaposhnik
Assignee: Rekha Joshi
Priority: Minor
Fix For: 0.94.19
Attachments: HBASE-8604.1.patch
I was running some replication code that incorrectly advertised the peer
address for replication. HBase complained that the format of the record was
NOT what it was expecting but it didn't include what it saw in the exception
message. Including that string would help cutting down the time it takes to
debug issues like that.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-10531) Revisit how the key byte[] is passed to HFileScanner.seekTo and reseekTo
[ https://issues.apache.org/jira/browse/HBASE-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-10531: --- Status: Open (was: Patch Available) Revisit how the key byte[] is passed to HFileScanner.seekTo and reseekTo Key: HBASE-10531 URL: https://issues.apache.org/jira/browse/HBASE-10531 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0 Attachments: HBASE-10531.patch, HBASE-10531_1.patch Currently the byte[] key passed to HFileScanner.seekTo and HFileScanner.reseekTo, is a combination of row, cf, qual, type and ts. And the caller forms this by using kv.getBuffer, which is actually deprecated. So see how this can be achieved considering kv.getBuffer is removed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HBASE-10653) Incorrect table status in HBase shell Describe
[ https://issues.apache.org/jira/browse/HBASE-10653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rekha Joshi reassigned HBASE-10653: --- Assignee: Rekha Joshi Incorrect table status in HBase shell Describe -- Key: HBASE-10653 URL: https://issues.apache.org/jira/browse/HBASE-10653 Project: HBase Issue Type: Bug Components: shell Reporter: Biju Nair Assignee: Rekha Joshi Labels: HbaseShell, describe Describe output of table which is disabled shows as enabled. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10677) boundaries check in hbck throwing IllegalArgumentException
[
https://issues.apache.org/jira/browse/HBASE-10677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13920761#comment-13920761
]
Hadoop QA commented on HBASE-10677:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12632793/HBASE-10677_v1.patch
against trunk revision .
ATTACHMENT ID: 12632793
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8898//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8898//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8898//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8898//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8898//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8898//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8898//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8898//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8898//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8898//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8898//console
This message is automatically generated.
boundaries check in hbck throwing IllegalArgumentException
--
Key: HBASE-10677
URL: https://issues.apache.org/jira/browse/HBASE-10677
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 0.98.0
Reporter: rajeshbabu
Assignee: rajeshbabu
Fix For: 0.96.2, 0.98.1, 0.99.0
Attachments: HBASE-10677.patch, HBASE-10677_v1.patch
This is the exception
{code}
java.lang.IllegalArgumentException: Pathname
/hbase/hbase:labels/438f84dea5d0e24e390de927deb8a84e from
hdfs://10.18.40.29:54310/hbase/hbase:labels/438f84dea5d0e24e390de927deb8a84e
is not a valid DFS filename.
at
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:184)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:637)
at
org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:92)
at
org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:702)
at
org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:698)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:698)
at
org.apache.hadoop.hbase.util.HBaseFsck.checkRegionBoundaries(HBaseFsck.java:537)
at
org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:487)
at
[jira] [Updated] (HBASE-10531) Revisit how the key byte[] is passed to HFileScanner.seekTo and reseekTo
[ https://issues.apache.org/jira/browse/HBASE-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-10531: --- Attachment: (was: HBASE-10532_2.patch) Revisit how the key byte[] is passed to HFileScanner.seekTo and reseekTo Key: HBASE-10531 URL: https://issues.apache.org/jira/browse/HBASE-10531 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0 Attachments: HBASE-10531.patch, HBASE-10531_1.patch, HBASE-10531_2.patch Currently the byte[] key passed to HFileScanner.seekTo and HFileScanner.reseekTo, is a combination of row, cf, qual, type and ts. And the caller forms this by using kv.getBuffer, which is actually deprecated. So see how this can be achieved considering kv.getBuffer is removed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10531) Revisit how the key byte[] is passed to HFileScanner.seekTo and reseekTo
[ https://issues.apache.org/jira/browse/HBASE-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-10531: --- Attachment: HBASE-10531_2.patch Revisit how the key byte[] is passed to HFileScanner.seekTo and reseekTo Key: HBASE-10531 URL: https://issues.apache.org/jira/browse/HBASE-10531 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0 Attachments: HBASE-10531.patch, HBASE-10531_1.patch, HBASE-10531_2.patch Currently the byte[] key passed to HFileScanner.seekTo and HFileScanner.reseekTo, is a combination of row, cf, qual, type and ts. And the caller forms this by using kv.getBuffer, which is actually deprecated. So see how this can be achieved considering kv.getBuffer is removed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10531) Revisit how the key byte[] is passed to HFileScanner.seekTo and reseekTo
[ https://issues.apache.org/jira/browse/HBASE-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-10531: --- Attachment: HBASE-10532_2.patch I have uploaded a patch that uses an inner class KeyAloneKeyValue that does not copy the bytes. I have not yet removed the duplicate code. Let me know what you think on this. Revisit how the key byte[] is passed to HFileScanner.seekTo and reseekTo Key: HBASE-10531 URL: https://issues.apache.org/jira/browse/HBASE-10531 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0 Attachments: HBASE-10531.patch, HBASE-10531_1.patch, HBASE-10531_2.patch Currently the byte[] key passed to HFileScanner.seekTo and HFileScanner.reseekTo, is a combination of row, cf, qual, type and ts. And the caller forms this by using kv.getBuffer, which is actually deprecated. So see how this can be achieved considering kv.getBuffer is removed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10531) Revisit how the key byte[] is passed to HFileScanner.seekTo and reseekTo
[ https://issues.apache.org/jira/browse/HBASE-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13920772#comment-13920772 ] ramkrishna.s.vasudevan commented on HBASE-10531: https://reviews.apache.org/r/18570/ Revisit how the key byte[] is passed to HFileScanner.seekTo and reseekTo Key: HBASE-10531 URL: https://issues.apache.org/jira/browse/HBASE-10531 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0 Attachments: HBASE-10531.patch, HBASE-10531_1.patch, HBASE-10531_2.patch Currently the byte[] key passed to HFileScanner.seekTo and HFileScanner.reseekTo, is a combination of row, cf, qual, type and ts. And the caller forms this by using kv.getBuffer, which is actually deprecated. So see how this can be achieved considering kv.getBuffer is removed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bharath v reassigned HBASE-8963: Assignee: bharath v Add configuration option to skip HFile archiving Key: HBASE-8963 URL: https://issues.apache.org/jira/browse/HBASE-8963 Project: HBase Issue Type: Improvement Reporter: Ted Yu Assignee: bharath v Currently HFileArchiver is always called when a table is dropped. A configuration option (either global or per table) should be provided so that archiving can be skipped when table is deleted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10653) Incorrect table status in HBase shell Describe
[ https://issues.apache.org/jira/browse/HBASE-10653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rekha Joshi updated HBASE-10653: Fix Version/s: 0.94.17 Affects Version/s: 0.94.17 Status: Patch Available (was: In Progress) Attached quick patch to have ENABLED set in separate line to avoid confusion. Incorrect table status in HBase shell Describe -- Key: HBASE-10653 URL: https://issues.apache.org/jira/browse/HBASE-10653 Project: HBase Issue Type: Bug Components: shell Affects Versions: 0.94.17 Reporter: Biju Nair Assignee: Rekha Joshi Labels: HbaseShell, describe Fix For: 0.94.17 Attachments: HBASE-10653.1.patch Describe output of table which is disabled shows as enabled. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10653) Incorrect table status in HBase shell Describe
[
https://issues.apache.org/jira/browse/HBASE-10653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rekha Joshi updated HBASE-10653:
Attachment: HBASE-10653.1.patch
Verified and enabled(true,here) is not clear.
{noformat}
describe 'ripple_dnb_test'
DESCRIPTION
ENABLED
{NAME = 'ripple_dnb_test', FAMILIES = [{NAME = 'master',
DATA_BLOCK_ENCODING = 'NONE', BLOOMFILTER = 'NONE', REPLICATION true
_SCOPE = '0', VERSIONS = '3', COMPRESSION = 'NONE', MIN_VERSIONS = '0',
TTL = '2147483647', KEEP_DELETED_CELLS = 'false
', BLOCKSIZE = '65536', IN_MEMORY = 'false', ENCODE_ON_DISK = 'true',
BLOCKCACHE = 'true'}]}
1 row(s) in 0.2660 seconds
{noformat}
Attached quick patch to have ENABLED set in separate line to avoid confusion.
Incorrect table status in HBase shell Describe
--
Key: HBASE-10653
URL: https://issues.apache.org/jira/browse/HBASE-10653
Project: HBase
Issue Type: Bug
Components: shell
Affects Versions: 0.94.17
Reporter: Biju Nair
Assignee: Rekha Joshi
Labels: HbaseShell, describe
Fix For: 0.94.17
Attachments: HBASE-10653.1.patch
Describe output of table which is disabled shows as enabled.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-10677) boundaries check in hbck throwing IllegalArgumentException
[
https://issues.apache.org/jira/browse/HBASE-10677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
rajeshbabu updated HBASE-10677:
---
Attachment: HBASE-10677_v2.patch
Added test case.
boundaries check in hbck throwing IllegalArgumentException
--
Key: HBASE-10677
URL: https://issues.apache.org/jira/browse/HBASE-10677
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 0.98.0
Reporter: rajeshbabu
Assignee: rajeshbabu
Fix For: 0.96.2, 0.98.1, 0.99.0
Attachments: HBASE-10677.patch, HBASE-10677_v1.patch,
HBASE-10677_v2.patch
This is the exception
{code}
java.lang.IllegalArgumentException: Pathname
/hbase/hbase:labels/438f84dea5d0e24e390de927deb8a84e from
hdfs://10.18.40.29:54310/hbase/hbase:labels/438f84dea5d0e24e390de927deb8a84e
is not a valid DFS filename.
at
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:184)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:637)
at
org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:92)
at
org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:702)
at
org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:698)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:698)
at
org.apache.hadoop.hbase.util.HBaseFsck.checkRegionBoundaries(HBaseFsck.java:537)
at
org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:487)
at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:4028)
at
org.apache.hadoop.hbase.util.HBaseFsck$HBaseFsckTool.run(HBaseFsck.java:3837)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3825)
{code}
We are pointing to wrong table directory. This is the code causing the
problem.
{code}
// For each region, get the start and stop key from the META and
compare them to the
// same information from the Stores.
Path path = new Path(getConf().get(HConstants.HBASE_DIR) + /
+ Bytes.toString(regionInfo.getTable().getName()) + /
+ regionInfo.getEncodedName() + /);
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Work started] (HBASE-10653) Incorrect table status in HBase shell Describe
[ https://issues.apache.org/jira/browse/HBASE-10653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-10653 started by Rekha Joshi. Incorrect table status in HBase shell Describe -- Key: HBASE-10653 URL: https://issues.apache.org/jira/browse/HBASE-10653 Project: HBase Issue Type: Bug Components: shell Affects Versions: 0.94.17 Reporter: Biju Nair Assignee: Rekha Joshi Labels: HbaseShell, describe Fix For: 0.94.17 Attachments: HBASE-10653.1.patch Describe output of table which is disabled shows as enabled. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10653) Incorrect table status in HBase shell Describe
[ https://issues.apache.org/jira/browse/HBASE-10653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13920783#comment-13920783 ] Rekha Joshi commented on HBASE-10653: - Towards format changes, think we may introduce –noformat option on shell (hbase noformat RETURN or ./bin/hbase shell –noformat option) 1.Towards that approach, stipulate changes mainly in shell.rb/or couple files and passed effect.But more testing needed to check if –noformat has no adverse effect on multiple cmds. 2.Or Should we think to have options like RDBMS(SQLWorld), provide user to set column format, line size etc (sqlcolumn column_name format a30 ; sqlset line size 300, etc) What do you think? Incorrect table status in HBase shell Describe -- Key: HBASE-10653 URL: https://issues.apache.org/jira/browse/HBASE-10653 Project: HBase Issue Type: Bug Components: shell Affects Versions: 0.94.17 Reporter: Biju Nair Assignee: Rekha Joshi Labels: HbaseShell, describe Fix For: 0.94.17 Attachments: HBASE-10653.1.patch Describe output of table which is disabled shows as enabled. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10531) Revisit how the key byte[] is passed to HFileScanner.seekTo and reseekTo
[ https://issues.apache.org/jira/browse/HBASE-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13920799#comment-13920799 ] ramkrishna.s.vasudevan commented on HBASE-10531: One interesting thing is running PerformanceEvaluation with 3 threads randomReads and sequentialReads the code with copy performs faster. (!!) Revisit how the key byte[] is passed to HFileScanner.seekTo and reseekTo Key: HBASE-10531 URL: https://issues.apache.org/jira/browse/HBASE-10531 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0 Attachments: HBASE-10531.patch, HBASE-10531_1.patch, HBASE-10531_2.patch Currently the byte[] key passed to HFileScanner.seekTo and HFileScanner.reseekTo, is a combination of row, cf, qual, type and ts. And the caller forms this by using kv.getBuffer, which is actually deprecated. So see how this can be achieved considering kv.getBuffer is removed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10682) region_mover.rb throws can't convert nil into String for regions moved
[
https://issues.apache.org/jira/browse/HBASE-10682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13920812#comment-13920812
]
Hadoop QA commented on HBASE-10682:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12632806/HBASE-10682-v0.patch
against trunk revision .
ATTACHMENT ID: 12632806
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8900//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8900//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8900//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8900//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8900//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8900//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8900//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8900//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8900//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8900//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8900//console
This message is automatically generated.
region_mover.rb throws can't convert nil into String for regions moved
Key: HBASE-10682
URL: https://issues.apache.org/jira/browse/HBASE-10682
Project: HBase
Issue Type: Bug
Components: scripts
Affects Versions: 0.98.0, 0.99.0, 0.96.1.1, 0.94.17
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.18
Attachments: HBASE-10682-0.94-v0.patch, HBASE-10682-v0.patch
if we call unload to generate the region file and then one of the region in
the splits or we end up with a null pointer exception during one of the print.
if we skip that NPE then we end up with the split/moved error message,
which is the expected one.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10682) region_mover.rb throws can't convert nil into String for regions moved
[ https://issues.apache.org/jira/browse/HBASE-10682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13920837#comment-13920837 ] Jean-Marc Spaggiari commented on HBASE-10682: - Are we sure that currentServer.to_s will not throw an exception if currentServer is null? It will try to call the to_string to it, no? So we might fail with the same issue? We might want to put a if before the log and log 2 different messages in case currentServer or/and servername is/are null? region_mover.rb throws can't convert nil into String for regions moved Key: HBASE-10682 URL: https://issues.apache.org/jira/browse/HBASE-10682 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 0.98.0, 0.99.0, 0.96.1.1, 0.94.17 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.18 Attachments: HBASE-10682-0.94-v0.patch, HBASE-10682-v0.patch if we call unload to generate the region file and then one of the region in the splits or we end up with a null pointer exception during one of the print. if we skip that NPE then we end up with the split/moved error message, which is the expected one. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10677) boundaries check in hbck throwing IllegalArgumentException
[
https://issues.apache.org/jira/browse/HBASE-10677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13920850#comment-13920850
]
Jean-Marc Spaggiari commented on HBASE-10677:
-
Thanks for fixing that.
+1 on V2.
boundaries check in hbck throwing IllegalArgumentException
--
Key: HBASE-10677
URL: https://issues.apache.org/jira/browse/HBASE-10677
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 0.98.0
Reporter: rajeshbabu
Assignee: rajeshbabu
Fix For: 0.96.2, 0.98.1, 0.99.0
Attachments: HBASE-10677.patch, HBASE-10677_v1.patch,
HBASE-10677_v2.patch
This is the exception
{code}
java.lang.IllegalArgumentException: Pathname
/hbase/hbase:labels/438f84dea5d0e24e390de927deb8a84e from
hdfs://10.18.40.29:54310/hbase/hbase:labels/438f84dea5d0e24e390de927deb8a84e
is not a valid DFS filename.
at
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:184)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:637)
at
org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:92)
at
org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:702)
at
org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:698)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:698)
at
org.apache.hadoop.hbase.util.HBaseFsck.checkRegionBoundaries(HBaseFsck.java:537)
at
org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:487)
at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:4028)
at
org.apache.hadoop.hbase.util.HBaseFsck$HBaseFsckTool.run(HBaseFsck.java:3837)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3825)
{code}
We are pointing to wrong table directory. This is the code causing the
problem.
{code}
// For each region, get the start and stop key from the META and
compare them to the
// same information from the Stores.
Path path = new Path(getConf().get(HConstants.HBASE_DIR) + /
+ Bytes.toString(regionInfo.getTable().getName()) + /
+ regionInfo.getEncodedName() + /);
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10676) Removing ThreadLocal of PrefetchedHeader in HFileBlock.FSReaderV2 make higher perforamce of scan
[
https://issues.apache.org/jira/browse/HBASE-10676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13920853#comment-13920853
]
Jean-Marc Spaggiari commented on HBASE-10676:
-
Nice, any impact on the other operations? Like get?
I can run it on PE for a day if you want.
Removing ThreadLocal of PrefetchedHeader in HFileBlock.FSReaderV2 make higher
perforamce of scan
Key: HBASE-10676
URL: https://issues.apache.org/jira/browse/HBASE-10676
Project: HBase
Issue Type: Improvement
Affects Versions: 0.98.0
Reporter: zhaojianbo
Attachments: HBASE-10676-0.98-branch.patch
PrefetchedHeader variable in HFileBlock.FSReaderV2 is used for avoiding
backward seek operation as the comment said:
{quote}
we will not incur a backward seek operation if we have already read this
block's header as part of the previous read's look-ahead. And we also want to
skip reading the header again if it has already been read.
{quote}
But that is not the case. In the code of 0.98, prefetchedHeader is
threadlocal for one storefile reader, and in the RegionScanner
lifecycle，different rpc handlers will serve scan requests of the same
scanner. Even though one handler of previous scan call prefetched the next
block header, the other handlers of current scan call will still trigger a
backward seek operation. The process is like this:
# rs handler1 serves the scan call, reads block1 and prefetches the header of
block2
# rs handler2 serves the same scanner's next scan call, because rs handler2
doesn't know the header of block2 already prefetched by rs handler1, triggers
a backward seek and reads block2, and prefetches the header of block3.
It is not the sequential read. So I think that the threadlocal is useless,
and should be abandoned. I did the work, and evaluated the performance of one
client, two client and four client scanning the same region with one
storefile. The test environment is
# A hdfs cluster with a namenode, a secondary namenode , a datanode in a
machine
# A hbase cluster with a zk, a master, a regionserver in the same machine
# clients are also in the same machine.
So all the data is local. The storefile is about 22.7GB from our online data,
18995949 kvs. Caching is set 1000.
With the improvement, the client total scan time decreases 21% for the one
client case, 11% for the two clients case. But the four clients case is
almost the same. The details tests' data is the following:
||case||client||time(ms)||
| original | 1 | 306222 |
| new | 1 | 241313 |
| original | 2 | 416390 |
| new | 2 | 369064 |
| original | 4 | 555986 |
| new | 4 | 562152 |
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10653) Incorrect table status in HBase shell Describe
[
https://issues.apache.org/jira/browse/HBASE-10653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13920855#comment-13920855
]
Hadoop QA commented on HBASE-10653:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12632815/HBASE-10653.1.patch
against trunk revision .
ATTACHMENT ID: 12632815
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8902//console
This message is automatically generated.
Incorrect table status in HBase shell Describe
--
Key: HBASE-10653
URL: https://issues.apache.org/jira/browse/HBASE-10653
Project: HBase
Issue Type: Bug
Components: shell
Affects Versions: 0.94.17
Reporter: Biju Nair
Assignee: Rekha Joshi
Labels: HbaseShell, describe
Fix For: 0.94.17
Attachments: HBASE-10653.1.patch
Describe output of table which is disabled shows as enabled.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10682) region_mover.rb throws can't convert nil into String for regions moved
[ https://issues.apache.org/jira/browse/HBASE-10682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13920873#comment-13920873 ] Matteo Bertozzi commented on HBASE-10682: - the patch was tested on the server that was failing.. so yeah, I'm sure that will not throw the exception. region_mover.rb throws can't convert nil into String for regions moved Key: HBASE-10682 URL: https://issues.apache.org/jira/browse/HBASE-10682 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 0.98.0, 0.99.0, 0.96.1.1, 0.94.17 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.18 Attachments: HBASE-10682-0.94-v0.patch, HBASE-10682-v0.patch if we call unload to generate the region file and then one of the region in the splits or we end up with a null pointer exception during one of the print. if we skip that NPE then we end up with the split/moved error message, which is the expected one. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10682) region_mover.rb throws can't convert nil into String for regions moved
[ https://issues.apache.org/jira/browse/HBASE-10682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13920893#comment-13920893 ] Jonathan Hsieh commented on HBASE-10682: +1 region_mover.rb throws can't convert nil into String for regions moved Key: HBASE-10682 URL: https://issues.apache.org/jira/browse/HBASE-10682 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 0.98.0, 0.99.0, 0.96.1.1, 0.94.17 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.18 Attachments: HBASE-10682-0.94-v0.patch, HBASE-10682-v0.patch if we call unload to generate the region file and then one of the region in the splits or we end up with a null pointer exception during one of the print. if we skip that NPE then we end up with the split/moved error message, which is the expected one. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10682) region_mover.rb throws can't convert nil into String for regions moved
[ https://issues.apache.org/jira/browse/HBASE-10682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13920896#comment-13920896 ] Jean-Marc Spaggiari commented on HBASE-10682: - Thanks for testing and fixing that Matteo. I'm +1 too. I will open another JIRA for the never ending issue. region_mover.rb throws can't convert nil into String for regions moved Key: HBASE-10682 URL: https://issues.apache.org/jira/browse/HBASE-10682 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 0.98.0, 0.99.0, 0.96.1.1, 0.94.17 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.18 Attachments: HBASE-10682-0.94-v0.patch, HBASE-10682-v0.patch if we call unload to generate the region file and then one of the region in the splits or we end up with a null pointer exception during one of the print. if we skip that NPE then we end up with the split/moved error message, which is the expected one. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10677) boundaries check in hbck throwing IllegalArgumentException
[
https://issues.apache.org/jira/browse/HBASE-10677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13920898#comment-13920898
]
Hadoop QA commented on HBASE-10677:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12632814/HBASE-10677_v2.patch
against trunk revision .
ATTACHMENT ID: 12632814
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8901//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8901//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8901//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8901//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8901//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8901//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8901//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8901//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8901//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8901//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8901//console
This message is automatically generated.
boundaries check in hbck throwing IllegalArgumentException
--
Key: HBASE-10677
URL: https://issues.apache.org/jira/browse/HBASE-10677
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 0.98.0
Reporter: rajeshbabu
Assignee: rajeshbabu
Fix For: 0.96.2, 0.98.1, 0.99.0
Attachments: HBASE-10677.patch, HBASE-10677_v1.patch,
HBASE-10677_v2.patch
This is the exception
{code}
java.lang.IllegalArgumentException: Pathname
/hbase/hbase:labels/438f84dea5d0e24e390de927deb8a84e from
hdfs://10.18.40.29:54310/hbase/hbase:labels/438f84dea5d0e24e390de927deb8a84e
is not a valid DFS filename.
at
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:184)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:637)
at
org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:92)
at
org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:702)
at
org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:698)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:698)
at
org.apache.hadoop.hbase.util.HBaseFsck.checkRegionBoundaries(HBaseFsck.java:537)
at
org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:487)
at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:4028)
at
org.apache.hadoop.hbase.util.HBaseFsck$HBaseFsckTool.run(HBaseFsck.java:3837)
at
[jira] [Created] (HBASE-10683) Rolling restart never ending when a RS doesn't come back online.
Jean-Marc Spaggiari created HBASE-10683: --- Summary: Rolling restart never ending when a RS doesn't come back online. Key: HBASE-10683 URL: https://issues.apache.org/jira/browse/HBASE-10683 Project: HBase Issue Type: Bug Reporter: Jean-Marc Spaggiari When doing a rolling restart using rolling-restart.sh, if a region server is not able to come back online, the script will never stop and wait forever for the server. Instead of that we might want to wait for a certain period of time and exit with failure. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10677) boundaries check in hbck throwing IllegalArgumentException
[
https://issues.apache.org/jira/browse/HBASE-10677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13920910#comment-13920910
]
Jonathan Hsieh commented on HBASE-10677:
+1.
boundaries check in hbck throwing IllegalArgumentException
--
Key: HBASE-10677
URL: https://issues.apache.org/jira/browse/HBASE-10677
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 0.98.0
Reporter: rajeshbabu
Assignee: rajeshbabu
Fix For: 0.96.2, 0.98.1, 0.99.0
Attachments: HBASE-10677.patch, HBASE-10677_v1.patch,
HBASE-10677_v2.patch
This is the exception
{code}
java.lang.IllegalArgumentException: Pathname
/hbase/hbase:labels/438f84dea5d0e24e390de927deb8a84e from
hdfs://10.18.40.29:54310/hbase/hbase:labels/438f84dea5d0e24e390de927deb8a84e
is not a valid DFS filename.
at
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:184)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:637)
at
org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:92)
at
org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:702)
at
org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:698)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:698)
at
org.apache.hadoop.hbase.util.HBaseFsck.checkRegionBoundaries(HBaseFsck.java:537)
at
org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:487)
at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:4028)
at
org.apache.hadoop.hbase.util.HBaseFsck$HBaseFsckTool.run(HBaseFsck.java:3837)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3825)
{code}
We are pointing to wrong table directory. This is the code causing the
problem.
{code}
// For each region, get the start and stop key from the META and
compare them to the
// same information from the Stores.
Path path = new Path(getConf().get(HConstants.HBASE_DIR) + /
+ Bytes.toString(regionInfo.getTable().getName()) + /
+ regionInfo.getEncodedName() + /);
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-9117) Remove HTablePool and all HConnection pooling related APIs
[ https://issues.apache.org/jira/browse/HBASE-9117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13920998#comment-13920998 ] Nick Dimiduk commented on HBASE-9117: - Here's a pile of commits, unwound from the above patches: https://github.com/ndimiduk/hbase/commits/HBASE-9117 It's not perfect, there's still more tests to clean up, but it's a start. Remove HTablePool and all HConnection pooling related APIs -- Key: HBASE-9117 URL: https://issues.apache.org/jira/browse/HBASE-9117 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Nick Dimiduk Priority: Critical Fix For: 0.99.0 Attachments: HBASE-9117.00.patch, HBASE-9117.01.patch, HBASE-9117.02.patch, HBASE-9117.03.patch, HBASE-9117.04.patch, HBASE-9117.05.patch, HBASE-9117.06.patch The recommended way is now: # Create an HConnection: HConnectionManager.createConnection(...) # Create a light HTable: HConnection.getTable(...) # table.close() # connection.close() All other API and pooling will be removed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10670) HBaseFsck#connect() should use new connection
[
https://issues.apache.org/jira/browse/HBASE-10670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921003#comment-13921003
]
Jonathan Hsieh commented on HBASE-10670:
The story you mentioned is helpful but doesn't provide enough context to do a
review without looking having to hunt for things in the code. If you explain
things then others can be aware of the anti-pattern in other places.
* Where is the edge.incrementTime call? (not clear that it was in TestHBaseFsck)
* How does this race happen?
* Why does the connection time out in the old implementation instead of the new?
HBaseFsck#connect() should use new connection
-
Key: HBASE-10670
URL: https://issues.apache.org/jira/browse/HBASE-10670
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Attachments: 10670-TestHBaseFsck.testCheckTableLocks.html,
10670-v1.txt
When investigating TestHBaseFsck#testCheckTableLocks failure, I noticed the
following:
{code}
2014-03-03 04:26:04,981 WARN [Thread-1180]
client.ConnectionManager$HConnectionImplementation(1998): Checking master
connection
com.google.protobuf.ServiceException: java.io.IOException: Call to
c59-s15.cs1cloud.internal/172.18.145.15:52272 failed on local exception:
org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call id=1282,
waitTime=1, rpcTimeout=0
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1699)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1740)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:40216)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionManager.java:1358)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionManager.java:1991)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1710)
at
org.apache.hadoop.hbase.client.HBaseAdmin$MasterCallable.prepare(HBaseAdmin.java:3199)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:120)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:97)
at
org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3226)
at
org.apache.hadoop.hbase.client.HBaseAdmin.getClusterStatus(HBaseAdmin.java:2158)
at org.apache.hadoop.hbase.util.HBaseFsck.connect(HBaseFsck.java:308)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:52)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:43)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:38)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testCheckTableLocks(TestHBaseFsck.java:2100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: java.io.IOException: Call to
c59-s15.cs1cloud.internal/172.18.145.15:52272 failed on local exception:
org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call id=1282,
waitTime=1, rpcTimeout=0
at
org.apache.hadoop.hbase.ipc.RpcClient.wrapException(RpcClient.java:1516)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1486)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1684)
... 24 more
Caused by: org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call
id=1282, waitTime=1, rpcTimeout=0
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.cleanupCalls(RpcClient.java:1214)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.cleanupCalls(RpcClient.java:1205)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.close(RpcClient.java:1006)
[jira] [Commented] (HBASE-10670) HBaseFsck#connect() should use new connection
[
https://issues.apache.org/jira/browse/HBASE-10670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921045#comment-13921045
]
Ted Yu commented on HBASE-10670:
The edge.incrementTime() call is around line 2097:
{code}
hbck = doFsck(conf, false);
assertNoErrors(hbck); // should not have expired, no problems
edge.incrementTime(conf.getLong(TableLockManager.TABLE_LOCK_EXPIRE_TIMEOUT,
TableLockManager.DEFAULT_TABLE_LOCK_EXPIRE_TIMEOUT_MS)); // let table
lock expire
hbck = doFsck(conf, false);
{code}
The connection held by the first doFsck() call timed out after incrementTime().
Without patch, this connection would be returned by HBaseFsck#connect(),
leading to RpcClient$CallTimeoutException and failing the test.
With patch, a new connection is established which is not affected by
incrementTime() call.
This new connection can serve the second doFsck() call.
HBaseFsck#connect() should use new connection
-
Key: HBASE-10670
URL: https://issues.apache.org/jira/browse/HBASE-10670
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Attachments: 10670-TestHBaseFsck.testCheckTableLocks.html,
10670-v1.txt
When investigating TestHBaseFsck#testCheckTableLocks failure, I noticed the
following:
{code}
2014-03-03 04:26:04,981 WARN [Thread-1180]
client.ConnectionManager$HConnectionImplementation(1998): Checking master
connection
com.google.protobuf.ServiceException: java.io.IOException: Call to
c59-s15.cs1cloud.internal/172.18.145.15:52272 failed on local exception:
org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call id=1282,
waitTime=1, rpcTimeout=0
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1699)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1740)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:40216)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionManager.java:1358)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionManager.java:1991)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1710)
at
org.apache.hadoop.hbase.client.HBaseAdmin$MasterCallable.prepare(HBaseAdmin.java:3199)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:120)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:97)
at
org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3226)
at
org.apache.hadoop.hbase.client.HBaseAdmin.getClusterStatus(HBaseAdmin.java:2158)
at org.apache.hadoop.hbase.util.HBaseFsck.connect(HBaseFsck.java:308)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:52)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:43)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:38)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testCheckTableLocks(TestHBaseFsck.java:2100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: java.io.IOException: Call to
c59-s15.cs1cloud.internal/172.18.145.15:52272 failed on local exception:
org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call id=1282,
waitTime=1, rpcTimeout=0
at
org.apache.hadoop.hbase.ipc.RpcClient.wrapException(RpcClient.java:1516)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1486)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1684)
... 24 more
Caused by: org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call
id=1282, waitTime=1,
[jira] [Commented] (HBASE-10670) HBaseFsck#connect() should use new connection
[
https://issues.apache.org/jira/browse/HBASE-10670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921050#comment-13921050
]
Nicolas Liochon commented on HBASE-10670:
-
This seems to fix the issue as well (by beeing sure the timeout won't happen
during the test)
{code}
diff --git
hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
index 8cddae8..5a83cb5 100644
--- hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
+++ hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
@@ -131,6 +131,8 @@ public class TestHBaseFsck {
public static void setUpBeforeClass() throws Exception {
TEST_UTIL.getConfiguration().setInt(hbase.regionserver.handler.count, 2);
TEST_UTIL.getConfiguration().setInt(hbase.regionserver.metahandler.count, 2);
+TEST_UTIL.getConfiguration().setInt(
+hbase.ipc.client.connection.minIdleTimeBeforeClose, 20 * 60 * 1000);
TEST_UTIL.startMiniCluster(3);
executorService = new ThreadPoolExecutor(1, Integer.MAX_VALUE, 60,
TimeUnit.SECONDS,
{code}
Then is it a test issue only or a Fsck one? For myself I don't know.
HBaseFsck#connect() should use new connection
-
Key: HBASE-10670
URL: https://issues.apache.org/jira/browse/HBASE-10670
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Attachments: 10670-TestHBaseFsck.testCheckTableLocks.html,
10670-v1.txt
When investigating TestHBaseFsck#testCheckTableLocks failure, I noticed the
following:
{code}
2014-03-03 04:26:04,981 WARN [Thread-1180]
client.ConnectionManager$HConnectionImplementation(1998): Checking master
connection
com.google.protobuf.ServiceException: java.io.IOException: Call to
c59-s15.cs1cloud.internal/172.18.145.15:52272 failed on local exception:
org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call id=1282,
waitTime=1, rpcTimeout=0
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1699)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1740)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:40216)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionManager.java:1358)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionManager.java:1991)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1710)
at
org.apache.hadoop.hbase.client.HBaseAdmin$MasterCallable.prepare(HBaseAdmin.java:3199)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:120)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:97)
at
org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3226)
at
org.apache.hadoop.hbase.client.HBaseAdmin.getClusterStatus(HBaseAdmin.java:2158)
at org.apache.hadoop.hbase.util.HBaseFsck.connect(HBaseFsck.java:308)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:52)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:43)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:38)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testCheckTableLocks(TestHBaseFsck.java:2100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: java.io.IOException: Call to
c59-s15.cs1cloud.internal/172.18.145.15:52272 failed on local exception:
org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call id=1282,
waitTime=1, rpcTimeout=0
at
[jira] [Commented] (HBASE-10670) HBaseFsck#connect() should use new connection
[
https://issues.apache.org/jira/browse/HBASE-10670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921058#comment-13921058
]
Ted Yu commented on HBASE-10670:
Each doFsck() call in TestHBaseFsck simulates new invocation of hbck.
In production, new connection would be established to the cluster.
My patch reflects the above scenario.
Retrieving used connection from previous doFsck() call is susceptible to
variation of timeout parameters, thus prone to test failure.
HBaseFsck#connect() should use new connection
-
Key: HBASE-10670
URL: https://issues.apache.org/jira/browse/HBASE-10670
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Attachments: 10670-TestHBaseFsck.testCheckTableLocks.html,
10670-v1.txt
When investigating TestHBaseFsck#testCheckTableLocks failure, I noticed the
following:
{code}
2014-03-03 04:26:04,981 WARN [Thread-1180]
client.ConnectionManager$HConnectionImplementation(1998): Checking master
connection
com.google.protobuf.ServiceException: java.io.IOException: Call to
c59-s15.cs1cloud.internal/172.18.145.15:52272 failed on local exception:
org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call id=1282,
waitTime=1, rpcTimeout=0
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1699)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1740)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:40216)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionManager.java:1358)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionManager.java:1991)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1710)
at
org.apache.hadoop.hbase.client.HBaseAdmin$MasterCallable.prepare(HBaseAdmin.java:3199)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:120)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:97)
at
org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3226)
at
org.apache.hadoop.hbase.client.HBaseAdmin.getClusterStatus(HBaseAdmin.java:2158)
at org.apache.hadoop.hbase.util.HBaseFsck.connect(HBaseFsck.java:308)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:52)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:43)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:38)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testCheckTableLocks(TestHBaseFsck.java:2100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: java.io.IOException: Call to
c59-s15.cs1cloud.internal/172.18.145.15:52272 failed on local exception:
org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call id=1282,
waitTime=1, rpcTimeout=0
at
org.apache.hadoop.hbase.ipc.RpcClient.wrapException(RpcClient.java:1516)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1486)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1684)
... 24 more
Caused by: org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call
id=1282, waitTime=1, rpcTimeout=0
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.cleanupCalls(RpcClient.java:1214)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.cleanupCalls(RpcClient.java:1205)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.close(RpcClient.java:1006)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.run(RpcClient.java:749)
{code}
This ctor was used in HBaseFsck#connect():
[jira] [Commented] (HBASE-10670) HBaseFsck#connect() should use new connection
[
https://issues.apache.org/jira/browse/HBASE-10670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921062#comment-13921062
]
Jonathan Hsieh commented on HBASE-10670:
ok. that makes sense now. +1
HBaseFsck#connect() should use new connection
-
Key: HBASE-10670
URL: https://issues.apache.org/jira/browse/HBASE-10670
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Attachments: 10670-TestHBaseFsck.testCheckTableLocks.html,
10670-v1.txt
When investigating TestHBaseFsck#testCheckTableLocks failure, I noticed the
following:
{code}
2014-03-03 04:26:04,981 WARN [Thread-1180]
client.ConnectionManager$HConnectionImplementation(1998): Checking master
connection
com.google.protobuf.ServiceException: java.io.IOException: Call to
c59-s15.cs1cloud.internal/172.18.145.15:52272 failed on local exception:
org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call id=1282,
waitTime=1, rpcTimeout=0
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1699)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1740)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:40216)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionManager.java:1358)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionManager.java:1991)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1710)
at
org.apache.hadoop.hbase.client.HBaseAdmin$MasterCallable.prepare(HBaseAdmin.java:3199)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:120)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:97)
at
org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3226)
at
org.apache.hadoop.hbase.client.HBaseAdmin.getClusterStatus(HBaseAdmin.java:2158)
at org.apache.hadoop.hbase.util.HBaseFsck.connect(HBaseFsck.java:308)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:52)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:43)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:38)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testCheckTableLocks(TestHBaseFsck.java:2100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: java.io.IOException: Call to
c59-s15.cs1cloud.internal/172.18.145.15:52272 failed on local exception:
org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call id=1282,
waitTime=1, rpcTimeout=0
at
org.apache.hadoop.hbase.ipc.RpcClient.wrapException(RpcClient.java:1516)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1486)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1684)
... 24 more
Caused by: org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call
id=1282, waitTime=1, rpcTimeout=0
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.cleanupCalls(RpcClient.java:1214)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.cleanupCalls(RpcClient.java:1205)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.close(RpcClient.java:1006)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.run(RpcClient.java:749)
{code}
This ctor was used in HBaseFsck#connect():
{code}
public HBaseAdmin(Configuration c)
throws MasterNotRunningException, ZooKeeperConnectionException, IOException
{
// Will not leak connections, as the new implementation of the constructor
// does not throw exceptions anymore.
[jira] [Commented] (HBASE-10670) HBaseFsck#connect() should use new connection
[
https://issues.apache.org/jira/browse/HBASE-10670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921068#comment-13921068
]
Jonathan Hsieh commented on HBASE-10670:
should update comments about new connection upon connect()
HBaseFsck#connect() should use new connection
-
Key: HBASE-10670
URL: https://issues.apache.org/jira/browse/HBASE-10670
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Attachments: 10670-TestHBaseFsck.testCheckTableLocks.html,
10670-v1.txt
When investigating TestHBaseFsck#testCheckTableLocks failure, I noticed the
following:
{code}
2014-03-03 04:26:04,981 WARN [Thread-1180]
client.ConnectionManager$HConnectionImplementation(1998): Checking master
connection
com.google.protobuf.ServiceException: java.io.IOException: Call to
c59-s15.cs1cloud.internal/172.18.145.15:52272 failed on local exception:
org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call id=1282,
waitTime=1, rpcTimeout=0
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1699)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1740)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:40216)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionManager.java:1358)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionManager.java:1991)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1710)
at
org.apache.hadoop.hbase.client.HBaseAdmin$MasterCallable.prepare(HBaseAdmin.java:3199)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:120)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:97)
at
org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3226)
at
org.apache.hadoop.hbase.client.HBaseAdmin.getClusterStatus(HBaseAdmin.java:2158)
at org.apache.hadoop.hbase.util.HBaseFsck.connect(HBaseFsck.java:308)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:52)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:43)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:38)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testCheckTableLocks(TestHBaseFsck.java:2100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: java.io.IOException: Call to
c59-s15.cs1cloud.internal/172.18.145.15:52272 failed on local exception:
org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call id=1282,
waitTime=1, rpcTimeout=0
at
org.apache.hadoop.hbase.ipc.RpcClient.wrapException(RpcClient.java:1516)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1486)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1684)
... 24 more
Caused by: org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call
id=1282, waitTime=1, rpcTimeout=0
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.cleanupCalls(RpcClient.java:1214)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.cleanupCalls(RpcClient.java:1205)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.close(RpcClient.java:1006)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.run(RpcClient.java:749)
{code}
This ctor was used in HBaseFsck#connect():
{code}
public HBaseAdmin(Configuration c)
throws MasterNotRunningException, ZooKeeperConnectionException, IOException
{
// Will not leak connections, as the new implementation of the constructor
// does not throw
[jira] [Commented] (HBASE-10670) HBaseFsck#connect() should use new connection
[
https://issues.apache.org/jira/browse/HBASE-10670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921075#comment-13921075
]
Nicolas Liochon commented on HBASE-10670:
-
No real opinion, +0 from me.
And thank for working on this, Ted.
HBaseFsck#connect() should use new connection
-
Key: HBASE-10670
URL: https://issues.apache.org/jira/browse/HBASE-10670
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Attachments: 10670-TestHBaseFsck.testCheckTableLocks.html,
10670-v1.txt
When investigating TestHBaseFsck#testCheckTableLocks failure, I noticed the
following:
{code}
2014-03-03 04:26:04,981 WARN [Thread-1180]
client.ConnectionManager$HConnectionImplementation(1998): Checking master
connection
com.google.protobuf.ServiceException: java.io.IOException: Call to
c59-s15.cs1cloud.internal/172.18.145.15:52272 failed on local exception:
org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call id=1282,
waitTime=1, rpcTimeout=0
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1699)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1740)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:40216)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionManager.java:1358)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionManager.java:1991)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1710)
at
org.apache.hadoop.hbase.client.HBaseAdmin$MasterCallable.prepare(HBaseAdmin.java:3199)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:120)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:97)
at
org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3226)
at
org.apache.hadoop.hbase.client.HBaseAdmin.getClusterStatus(HBaseAdmin.java:2158)
at org.apache.hadoop.hbase.util.HBaseFsck.connect(HBaseFsck.java:308)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:52)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:43)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:38)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testCheckTableLocks(TestHBaseFsck.java:2100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: java.io.IOException: Call to
c59-s15.cs1cloud.internal/172.18.145.15:52272 failed on local exception:
org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call id=1282,
waitTime=1, rpcTimeout=0
at
org.apache.hadoop.hbase.ipc.RpcClient.wrapException(RpcClient.java:1516)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1486)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1684)
... 24 more
Caused by: org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call
id=1282, waitTime=1, rpcTimeout=0
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.cleanupCalls(RpcClient.java:1214)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.cleanupCalls(RpcClient.java:1205)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.close(RpcClient.java:1006)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.run(RpcClient.java:749)
{code}
This ctor was used in HBaseFsck#connect():
{code}
public HBaseAdmin(Configuration c)
throws MasterNotRunningException, ZooKeeperConnectionException, IOException
{
// Will not leak connections, as the new implementation of the constructor
// does not
[jira] [Updated] (HBASE-10670) HBaseFsck#connect() should use new connection
[
https://issues.apache.org/jira/browse/HBASE-10670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-10670:
---
Fix Version/s: 0.99.0
Hadoop Flags: Reviewed
HBaseFsck#connect() should use new connection
-
Key: HBASE-10670
URL: https://issues.apache.org/jira/browse/HBASE-10670
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 0.99.0
Attachments: 10670-TestHBaseFsck.testCheckTableLocks.html,
10670-v1.txt
When investigating TestHBaseFsck#testCheckTableLocks failure, I noticed the
following:
{code}
2014-03-03 04:26:04,981 WARN [Thread-1180]
client.ConnectionManager$HConnectionImplementation(1998): Checking master
connection
com.google.protobuf.ServiceException: java.io.IOException: Call to
c59-s15.cs1cloud.internal/172.18.145.15:52272 failed on local exception:
org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call id=1282,
waitTime=1, rpcTimeout=0
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1699)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1740)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:40216)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionManager.java:1358)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionManager.java:1991)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1710)
at
org.apache.hadoop.hbase.client.HBaseAdmin$MasterCallable.prepare(HBaseAdmin.java:3199)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:120)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:97)
at
org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3226)
at
org.apache.hadoop.hbase.client.HBaseAdmin.getClusterStatus(HBaseAdmin.java:2158)
at org.apache.hadoop.hbase.util.HBaseFsck.connect(HBaseFsck.java:308)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:52)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:43)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:38)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testCheckTableLocks(TestHBaseFsck.java:2100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: java.io.IOException: Call to
c59-s15.cs1cloud.internal/172.18.145.15:52272 failed on local exception:
org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call id=1282,
waitTime=1, rpcTimeout=0
at
org.apache.hadoop.hbase.ipc.RpcClient.wrapException(RpcClient.java:1516)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1486)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1684)
... 24 more
Caused by: org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call
id=1282, waitTime=1, rpcTimeout=0
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.cleanupCalls(RpcClient.java:1214)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.cleanupCalls(RpcClient.java:1205)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.close(RpcClient.java:1006)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.run(RpcClient.java:749)
{code}
This ctor was used in HBaseFsck#connect():
{code}
public HBaseAdmin(Configuration c)
throws MasterNotRunningException, ZooKeeperConnectionException, IOException
{
// Will not leak connections, as the new implementation of the constructor
// does not throw exceptions anymore.
[jira] [Updated] (HBASE-10677) boundaries check in hbck throwing IllegalArgumentException
[
https://issues.apache.org/jira/browse/HBASE-10677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
rajeshbabu updated HBASE-10677:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
committed to 0.96, 0.98 and trunk.
Thanks for review [~jmhsieh], [~jmspaggi]
boundaries check in hbck throwing IllegalArgumentException
--
Key: HBASE-10677
URL: https://issues.apache.org/jira/browse/HBASE-10677
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 0.98.0
Reporter: rajeshbabu
Assignee: rajeshbabu
Fix For: 0.96.2, 0.98.1, 0.99.0
Attachments: HBASE-10677.patch, HBASE-10677_v1.patch,
HBASE-10677_v2.patch
This is the exception
{code}
java.lang.IllegalArgumentException: Pathname
/hbase/hbase:labels/438f84dea5d0e24e390de927deb8a84e from
hdfs://10.18.40.29:54310/hbase/hbase:labels/438f84dea5d0e24e390de927deb8a84e
is not a valid DFS filename.
at
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:184)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:637)
at
org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:92)
at
org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:702)
at
org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:698)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:698)
at
org.apache.hadoop.hbase.util.HBaseFsck.checkRegionBoundaries(HBaseFsck.java:537)
at
org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:487)
at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:4028)
at
org.apache.hadoop.hbase.util.HBaseFsck$HBaseFsckTool.run(HBaseFsck.java:3837)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3825)
{code}
We are pointing to wrong table directory. This is the code causing the
problem.
{code}
// For each region, get the start and stop key from the META and
compare them to the
// same information from the Stores.
Path path = new Path(getConf().get(HConstants.HBASE_DIR) + /
+ Bytes.toString(regionInfo.getTable().getName()) + /
+ regionInfo.getEncodedName() + /);
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10668) TestExportSnapshot runs too long
[
https://issues.apache.org/jira/browse/HBASE-10668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921108#comment-13921108
]
Lars Hofhansl commented on HBASE-10668:
---
This would probably be sped up by my changes in HBASE-10594.
TestExportSnapshot runs too long
Key: HBASE-10668
URL: https://issues.apache.org/jira/browse/HBASE-10668
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.0
Reporter: Andrew Purtell
Assignee: Rekha Joshi
Fix For: 0.98.1, 0.99.0
332 seconds
{noformat}
Forking command line: /bin/sh -c cd /data/src/hbase/hbase-server
/usr/lib/jvm/java-1.7.0.45-oracle-amd64/jre/bin/java -enableassertions
-Xmx1900m -XX:MaxPermSize=100m -Djava.security.egd=file:/dev/./urandom
-Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -jar
/data/src/hbase/hbase-server/target/surefire/surefirebooter1668068702110669265.jar
/data/src/hbase/hbase-server/target/surefire/surefire5744357307851892501tmp
/data/src/hbase/hbase-server/target/surefire/surefire_3661340119563945183029tmp
Running org.apache.hadoop.hbase.snapshot.TestExportSnapshot
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 332.421 sec
{noformat}
Slim down or split up.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-10594) Speed up TestRestoreSnapshotFromClient
[ https://issues.apache.org/jira/browse/HBASE-10594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-10594: -- Summary: Speed up TestRestoreSnapshotFromClient (was: Speed up TestRestoreSnapshotFromClient a bit) Speed up TestRestoreSnapshotFromClient -- Key: HBASE-10594 URL: https://issues.apache.org/jira/browse/HBASE-10594 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Priority: Minor Attachments: 10594-0.94-v2.txt, 10594-0.94.txt, 10594-trunk-v2.txt, 10594-trunk.txt Looking through the longest running test in 0.94 I noticed that TestRestoreSnapshotFromClient runs for over 10 minutes on the jenkins boxes (264s on my local box). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10669) [hbck tool] Usage is wrong for hbck tool for -sidelineCorruptHfiles option
[
https://issues.apache.org/jira/browse/HBASE-10669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
rajeshbabu updated HBASE-10669:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
committed to all versions.
Thanks for patch [~deepakhuawei]. Nice catch.
Next time on wards create patch from HBase root directory.
[hbck tool] Usage is wrong for hbck tool for -sidelineCorruptHfiles option
--
Key: HBASE-10669
URL: https://issues.apache.org/jira/browse/HBASE-10669
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 0.94.11, 0.96.0
Reporter: Deepak Sharma
Priority: Minor
Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.18
Attachments: HBASE_10669_94.11.patch, Hbck_usage_issue.patch
Usage is wrong for hbck tool for -sidelineCorruptHfiles option:
it is like:
-sidelineCorruptHfiles Quarantine corrupted HFiles. implies
-checkCorruptHfiles
here in sidelineCorruptHfiles and checkCorruptHfiles small 'f' is used
but actually in code it is like
else if (cmd.equals(-checkCorruptHFiles)) {
checkCorruptHFiles = true;
} else if (cmd.equals(-sidelineCorruptHFiles)) {
sidelineCorruptHFiles = true;
}
so if we use sidelineCorruptHfiles option for hbck then it will give error
Unrecognized option:-sidelineCorruptHfiles
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10669) [hbck tool] Usage is wrong for hbck tool for -sidelineCorruptHfiles option
[
https://issues.apache.org/jira/browse/HBASE-10669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921166#comment-13921166
]
rajeshbabu commented on HBASE-10669:
I am not able to assign this issue to [~deepakhuawei].
Can any one assign to him?
Thanks.
[hbck tool] Usage is wrong for hbck tool for -sidelineCorruptHfiles option
--
Key: HBASE-10669
URL: https://issues.apache.org/jira/browse/HBASE-10669
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 0.94.11, 0.96.0
Reporter: Deepak Sharma
Priority: Minor
Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.18
Attachments: HBASE_10669_94.11.patch, Hbck_usage_issue.patch
Usage is wrong for hbck tool for -sidelineCorruptHfiles option:
it is like:
-sidelineCorruptHfiles Quarantine corrupted HFiles. implies
-checkCorruptHfiles
here in sidelineCorruptHfiles and checkCorruptHfiles small 'f' is used
but actually in code it is like
else if (cmd.equals(-checkCorruptHFiles)) {
checkCorruptHFiles = true;
} else if (cmd.equals(-sidelineCorruptHFiles)) {
sidelineCorruptHFiles = true;
}
so if we use sidelineCorruptHfiles option for hbck then it will give error
Unrecognized option:-sidelineCorruptHfiles
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-10594) Speed up TestRestoreSnapshotFromClient
[ https://issues.apache.org/jira/browse/HBASE-10594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-10594: -- Resolution: Fixed Assignee: Lars Hofhansl Status: Resolved (was: Patch Available) Committed to all branches again. Thanks for taking a look [~mbertozzi]. Speed up TestRestoreSnapshotFromClient -- Key: HBASE-10594 URL: https://issues.apache.org/jira/browse/HBASE-10594 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Minor Attachments: 10594-0.94-v2.txt, 10594-0.94.txt, 10594-trunk-v2.txt, 10594-trunk.txt Looking through the longest running test in 0.94 I noticed that TestRestoreSnapshotFromClient runs for over 10 minutes on the jenkins boxes (264s on my local box). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10594) Speed up TestRestoreSnapshotFromClient
[ https://issues.apache.org/jira/browse/HBASE-10594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-10594: -- Fix Version/s: 0.94.18 0.99.0 0.98.1 0.96.2 Speed up TestRestoreSnapshotFromClient -- Key: HBASE-10594 URL: https://issues.apache.org/jira/browse/HBASE-10594 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Minor Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.18 Attachments: 10594-0.94-v2.txt, 10594-0.94.txt, 10594-trunk-v2.txt, 10594-trunk.txt Looking through the longest running test in 0.94 I noticed that TestRestoreSnapshotFromClient runs for over 10 minutes on the jenkins boxes (264s on my local box). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10668) TestExportSnapshot runs too long
[
https://issues.apache.org/jira/browse/HBASE-10668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921217#comment-13921217
]
Lars Hofhansl commented on HBASE-10668:
---
HBASE-10594 is committed. On my local machine it sped the test up from ~90s to
~70s.
If the speed up is linear on slower machines we can expect this to go to ~240s
now.
TestExportSnapshot runs too long
Key: HBASE-10668
URL: https://issues.apache.org/jira/browse/HBASE-10668
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.0
Reporter: Andrew Purtell
Assignee: Rekha Joshi
Fix For: 0.98.1, 0.99.0
332 seconds
{noformat}
Forking command line: /bin/sh -c cd /data/src/hbase/hbase-server
/usr/lib/jvm/java-1.7.0.45-oracle-amd64/jre/bin/java -enableassertions
-Xmx1900m -XX:MaxPermSize=100m -Djava.security.egd=file:/dev/./urandom
-Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -jar
/data/src/hbase/hbase-server/target/surefire/surefirebooter1668068702110669265.jar
/data/src/hbase/hbase-server/target/surefire/surefire5744357307851892501tmp
/data/src/hbase/hbase-server/target/surefire/surefire_3661340119563945183029tmp
Running org.apache.hadoop.hbase.snapshot.TestExportSnapshot
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 332.421 sec
{noformat}
Slim down or split up.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-6849) Make StochasticLoadBalancer the default
[ https://issues.apache.org/jira/browse/HBASE-6849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921218#comment-13921218 ] Enis Soztutar commented on HBASE-6849: -- bq. Does this mean that by table load balancing is now broken and 0.95 and later releases? (Elliott Clark, stack) It should still work, but I have not tried it myself. The SLB will produce a better plan if it sees the global view, and it has cost functions for computing the skew for not-evenly distributed regions of a table. Are you using per-table LB? I was in favor of removing it actually. Make StochasticLoadBalancer the default --- Key: HBASE-6849 URL: https://issues.apache.org/jira/browse/HBASE-6849 Project: HBase Issue Type: Improvement Components: master Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.95.0 Attachments: HBASE-6849-0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-6849) Make StochasticLoadBalancer the default
[ https://issues.apache.org/jira/browse/HBASE-6849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921246#comment-13921246 ] Lars Hofhansl commented on HBASE-6849: -- We tend to do larger scans (not huge, maybe a few million rows or less) along a table in parallel, so table balancing is important for us. Although I suppose that when a table is large enough for this to matter it will naturally have been spread out over the RSs, so maybe it's not an issue. Make StochasticLoadBalancer the default --- Key: HBASE-6849 URL: https://issues.apache.org/jira/browse/HBASE-6849 Project: HBase Issue Type: Improvement Components: master Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.95.0 Attachments: HBASE-6849-0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-10684) Fixing the TestSimpleOperations' delete tests
Gaurav Menghani created HBASE-10684: --- Summary: Fixing the TestSimpleOperations' delete tests Key: HBASE-10684 URL: https://issues.apache.org/jira/browse/HBASE-10684 Project: HBase Issue Type: Bug Affects Versions: 0.89-fb Reporter: Gaurav Menghani Fix For: 0.89-fb -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10676) Removing ThreadLocal of PrefetchedHeader in HFileBlock.FSReaderV2 make higher perforamce of scan
[
https://issues.apache.org/jira/browse/HBASE-10676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921271#comment-13921271
]
Nick Dimiduk commented on HBASE-10676:
--
Does your box have SSD or spinning disk?
Removing ThreadLocal of PrefetchedHeader in HFileBlock.FSReaderV2 make higher
perforamce of scan
Key: HBASE-10676
URL: https://issues.apache.org/jira/browse/HBASE-10676
Project: HBase
Issue Type: Improvement
Affects Versions: 0.98.0
Reporter: zhaojianbo
Attachments: HBASE-10676-0.98-branch.patch
PrefetchedHeader variable in HFileBlock.FSReaderV2 is used for avoiding
backward seek operation as the comment said:
{quote}
we will not incur a backward seek operation if we have already read this
block's header as part of the previous read's look-ahead. And we also want to
skip reading the header again if it has already been read.
{quote}
But that is not the case. In the code of 0.98, prefetchedHeader is
threadlocal for one storefile reader, and in the RegionScanner
lifecycle，different rpc handlers will serve scan requests of the same
scanner. Even though one handler of previous scan call prefetched the next
block header, the other handlers of current scan call will still trigger a
backward seek operation. The process is like this:
# rs handler1 serves the scan call, reads block1 and prefetches the header of
block2
# rs handler2 serves the same scanner's next scan call, because rs handler2
doesn't know the header of block2 already prefetched by rs handler1, triggers
a backward seek and reads block2, and prefetches the header of block3.
It is not the sequential read. So I think that the threadlocal is useless,
and should be abandoned. I did the work, and evaluated the performance of one
client, two client and four client scanning the same region with one
storefile. The test environment is
# A hdfs cluster with a namenode, a secondary namenode , a datanode in a
machine
# A hbase cluster with a zk, a master, a regionserver in the same machine
# clients are also in the same machine.
So all the data is local. The storefile is about 22.7GB from our online data,
18995949 kvs. Caching is set 1000.
With the improvement, the client total scan time decreases 21% for the one
client case, 11% for the two clients case. But the four clients case is
almost the same. The details tests' data is the following:
||case||client||time(ms)||
| original | 1 | 306222 |
| new | 1 | 241313 |
| original | 2 | 416390 |
| new | 2 | 369064 |
| original | 4 | 555986 |
| new | 4 | 562152 |
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10678) Make verifyrep tool implement toolrunner
[ https://issues.apache.org/jira/browse/HBASE-10678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921285#comment-13921285 ] Nick Dimiduk commented on HBASE-10678: -- Patch looks good to me. You ran the tool with this change, [~bharathv]? It works for you? Make verifyrep tool implement toolrunner Key: HBASE-10678 URL: https://issues.apache.org/jira/browse/HBASE-10678 Project: HBase Issue Type: Improvement Components: Replication Affects Versions: 0.99.0 Reporter: bharath v Assignee: bharath v Priority: Minor Attachments: HBASE-10678-trunk-v1.patch Currently verifyrep tool doesn't take -D args since it doesn't implement the toolrunner interface. So we need to make changes to config files on the client everytime we need run it with custom arguments. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10672) Table snapshot should handle tables whose REGION_REPLICATION is greater than one
[ https://issues.apache.org/jira/browse/HBASE-10672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921301#comment-13921301 ] Devaraj Das commented on HBASE-10672: - Thanks, Ted. Will be good if someone takes a look before I commit to the branch. Table snapshot should handle tables whose REGION_REPLICATION is greater than one Key: HBASE-10672 URL: https://issues.apache.org/jira/browse/HBASE-10672 Project: HBase Issue Type: Sub-task Reporter: Devaraj Das Assignee: Devaraj Das Fix For: hbase-10070 Attachments: 10672-1.1.txt, 10672-1.1.txt If a table has more than one region replica, then snapshot utility - take-snapshot, clone-snapshot etc crashes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10676) Removing ThreadLocal of PrefetchedHeader in HFileBlock.FSReaderV2 make higher perforamce of scan
[
https://issues.apache.org/jira/browse/HBASE-10676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921318#comment-13921318
]
stack commented on HBASE-10676:
---
Patch lgtm
Removing ThreadLocal of PrefetchedHeader in HFileBlock.FSReaderV2 make higher
perforamce of scan
Key: HBASE-10676
URL: https://issues.apache.org/jira/browse/HBASE-10676
Project: HBase
Issue Type: Improvement
Affects Versions: 0.98.0
Reporter: zhaojianbo
Attachments: HBASE-10676-0.98-branch.patch
PrefetchedHeader variable in HFileBlock.FSReaderV2 is used for avoiding
backward seek operation as the comment said:
{quote}
we will not incur a backward seek operation if we have already read this
block's header as part of the previous read's look-ahead. And we also want to
skip reading the header again if it has already been read.
{quote}
But that is not the case. In the code of 0.98, prefetchedHeader is
threadlocal for one storefile reader, and in the RegionScanner
lifecycle，different rpc handlers will serve scan requests of the same
scanner. Even though one handler of previous scan call prefetched the next
block header, the other handlers of current scan call will still trigger a
backward seek operation. The process is like this:
# rs handler1 serves the scan call, reads block1 and prefetches the header of
block2
# rs handler2 serves the same scanner's next scan call, because rs handler2
doesn't know the header of block2 already prefetched by rs handler1, triggers
a backward seek and reads block2, and prefetches the header of block3.
It is not the sequential read. So I think that the threadlocal is useless,
and should be abandoned. I did the work, and evaluated the performance of one
client, two client and four client scanning the same region with one
storefile. The test environment is
# A hdfs cluster with a namenode, a secondary namenode , a datanode in a
machine
# A hbase cluster with a zk, a master, a regionserver in the same machine
# clients are also in the same machine.
So all the data is local. The storefile is about 22.7GB from our online data,
18995949 kvs. Caching is set 1000.
With the improvement, the client total scan time decreases 21% for the one
client case, 11% for the two clients case. But the four clients case is
almost the same. The details tests' data is the following:
||case||client||time(ms)||
| original | 1 | 306222 |
| new | 1 | 241313 |
| original | 2 | 416390 |
| new | 2 | 369064 |
| original | 4 | 555986 |
| new | 4 | 562152 |
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10677) boundaries check in hbck throwing IllegalArgumentException
[
https://issues.apache.org/jira/browse/HBASE-10677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921362#comment-13921362
]
Hudson commented on HBASE-10677:
SUCCESS: Integrated in hbase-0.96 #328 (See
[https://builds.apache.org/job/hbase-0.96/328/])
HBASE-10677 boundaries check in hbck throwing IllegalArgumentException(Rajesh)
(rajeshbabu: rev 1574587)
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
boundaries check in hbck throwing IllegalArgumentException
--
Key: HBASE-10677
URL: https://issues.apache.org/jira/browse/HBASE-10677
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 0.98.0
Reporter: rajeshbabu
Assignee: rajeshbabu
Fix For: 0.96.2, 0.98.1, 0.99.0
Attachments: HBASE-10677.patch, HBASE-10677_v1.patch,
HBASE-10677_v2.patch
This is the exception
{code}
java.lang.IllegalArgumentException: Pathname
/hbase/hbase:labels/438f84dea5d0e24e390de927deb8a84e from
hdfs://10.18.40.29:54310/hbase/hbase:labels/438f84dea5d0e24e390de927deb8a84e
is not a valid DFS filename.
at
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:184)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:637)
at
org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:92)
at
org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:702)
at
org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:698)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:698)
at
org.apache.hadoop.hbase.util.HBaseFsck.checkRegionBoundaries(HBaseFsck.java:537)
at
org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:487)
at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:4028)
at
org.apache.hadoop.hbase.util.HBaseFsck$HBaseFsckTool.run(HBaseFsck.java:3837)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3825)
{code}
We are pointing to wrong table directory. This is the code causing the
problem.
{code}
// For each region, get the start and stop key from the META and
compare them to the
// same information from the Stores.
Path path = new Path(getConf().get(HConstants.HBASE_DIR) + /
+ Bytes.toString(regionInfo.getTable().getName()) + /
+ regionInfo.getEncodedName() + /);
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-10670) HBaseFsck#connect() should use new connection
[
https://issues.apache.org/jira/browse/HBASE-10670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-10670:
---
Fix Version/s: 0.98.1
HBaseFsck#connect() should use new connection
-
Key: HBASE-10670
URL: https://issues.apache.org/jira/browse/HBASE-10670
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 0.98.1, 0.99.0
Attachments: 10670-TestHBaseFsck.testCheckTableLocks.html,
10670-v1.txt
When investigating TestHBaseFsck#testCheckTableLocks failure, I noticed the
following:
{code}
2014-03-03 04:26:04,981 WARN [Thread-1180]
client.ConnectionManager$HConnectionImplementation(1998): Checking master
connection
com.google.protobuf.ServiceException: java.io.IOException: Call to
c59-s15.cs1cloud.internal/172.18.145.15:52272 failed on local exception:
org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call id=1282,
waitTime=1, rpcTimeout=0
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1699)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1740)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:40216)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionManager.java:1358)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionManager.java:1991)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1710)
at
org.apache.hadoop.hbase.client.HBaseAdmin$MasterCallable.prepare(HBaseAdmin.java:3199)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:120)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:97)
at
org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3226)
at
org.apache.hadoop.hbase.client.HBaseAdmin.getClusterStatus(HBaseAdmin.java:2158)
at org.apache.hadoop.hbase.util.HBaseFsck.connect(HBaseFsck.java:308)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:52)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:43)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:38)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testCheckTableLocks(TestHBaseFsck.java:2100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: java.io.IOException: Call to
c59-s15.cs1cloud.internal/172.18.145.15:52272 failed on local exception:
org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call id=1282,
waitTime=1, rpcTimeout=0
at
org.apache.hadoop.hbase.ipc.RpcClient.wrapException(RpcClient.java:1516)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1486)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1684)
... 24 more
Caused by: org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call
id=1282, waitTime=1, rpcTimeout=0
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.cleanupCalls(RpcClient.java:1214)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.cleanupCalls(RpcClient.java:1205)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.close(RpcClient.java:1006)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.run(RpcClient.java:749)
{code}
This ctor was used in HBaseFsck#connect():
{code}
public HBaseAdmin(Configuration c)
throws MasterNotRunningException, ZooKeeperConnectionException, IOException
{
// Will not leak connections, as the new implementation of the constructor
// does not throw exceptions anymore.
[jira] [Assigned] (HBASE-10669) [hbck tool] Usage is wrong for hbck tool for -sidelineCorruptHfiles option
[
https://issues.apache.org/jira/browse/HBASE-10669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack reassigned HBASE-10669:
-
Assignee: Deepak Sharma
Added Deepak as a contributor and assigned him the issue ([~rajesh23] I made
you an administrator of hbase project in JIRA so you should be able to add
folks going forward -- go easy)
[hbck tool] Usage is wrong for hbck tool for -sidelineCorruptHfiles option
--
Key: HBASE-10669
URL: https://issues.apache.org/jira/browse/HBASE-10669
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 0.94.11, 0.96.0
Reporter: Deepak Sharma
Assignee: Deepak Sharma
Priority: Minor
Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.18
Attachments: HBASE_10669_94.11.patch, Hbck_usage_issue.patch
Usage is wrong for hbck tool for -sidelineCorruptHfiles option:
it is like:
-sidelineCorruptHfiles Quarantine corrupted HFiles. implies
-checkCorruptHfiles
here in sidelineCorruptHfiles and checkCorruptHfiles small 'f' is used
but actually in code it is like
else if (cmd.equals(-checkCorruptHFiles)) {
checkCorruptHFiles = true;
} else if (cmd.equals(-sidelineCorruptHFiles)) {
sidelineCorruptHFiles = true;
}
so if we use sidelineCorruptHfiles option for hbck then it will give error
Unrecognized option:-sidelineCorruptHfiles
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10676) Removing ThreadLocal of PrefetchedHeader in HFileBlock.FSReaderV2 make higher perforamce of scan
[
https://issues.apache.org/jira/browse/HBASE-10676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921396#comment-13921396
]
Jean-Marc Spaggiari commented on HBASE-10676:
-
{quote}
Does your box have SSD or spinning disk?
{quote}
SSD on master only. RS are spinning disks.
Removing ThreadLocal of PrefetchedHeader in HFileBlock.FSReaderV2 make higher
perforamce of scan
Key: HBASE-10676
URL: https://issues.apache.org/jira/browse/HBASE-10676
Project: HBase
Issue Type: Improvement
Affects Versions: 0.98.0
Reporter: zhaojianbo
Attachments: HBASE-10676-0.98-branch.patch
PrefetchedHeader variable in HFileBlock.FSReaderV2 is used for avoiding
backward seek operation as the comment said:
{quote}
we will not incur a backward seek operation if we have already read this
block's header as part of the previous read's look-ahead. And we also want to
skip reading the header again if it has already been read.
{quote}
But that is not the case. In the code of 0.98, prefetchedHeader is
threadlocal for one storefile reader, and in the RegionScanner
lifecycle，different rpc handlers will serve scan requests of the same
scanner. Even though one handler of previous scan call prefetched the next
block header, the other handlers of current scan call will still trigger a
backward seek operation. The process is like this:
# rs handler1 serves the scan call, reads block1 and prefetches the header of
block2
# rs handler2 serves the same scanner's next scan call, because rs handler2
doesn't know the header of block2 already prefetched by rs handler1, triggers
a backward seek and reads block2, and prefetches the header of block3.
It is not the sequential read. So I think that the threadlocal is useless,
and should be abandoned. I did the work, and evaluated the performance of one
client, two client and four client scanning the same region with one
storefile. The test environment is
# A hdfs cluster with a namenode, a secondary namenode , a datanode in a
machine
# A hbase cluster with a zk, a master, a regionserver in the same machine
# clients are also in the same machine.
So all the data is local. The storefile is about 22.7GB from our online data,
18995949 kvs. Caching is set 1000.
With the improvement, the client total scan time decreases 21% for the one
client case, 11% for the two clients case. But the four clients case is
almost the same. The details tests' data is the following:
||case||client||time(ms)||
| original | 1 | 306222 |
| new | 1 | 241313 |
| original | 2 | 416390 |
| new | 2 | 369064 |
| original | 4 | 555986 |
| new | 4 | 562152 |
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10669) [hbck tool] Usage is wrong for hbck tool for -sidelineCorruptHfiles option
[
https://issues.apache.org/jira/browse/HBASE-10669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921401#comment-13921401
]
Hudson commented on HBASE-10669:
SUCCESS: Integrated in HBase-TRUNK #4979 (See
[https://builds.apache.org/job/HBase-TRUNK/4979/])
HBASE-10669 [hbck tool] Usage is wrong for hbck tool for -sidelineCorruptHfiles
option(Deepak Sharma) (rajeshbabu: rev 1574593)
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
[hbck tool] Usage is wrong for hbck tool for -sidelineCorruptHfiles option
--
Key: HBASE-10669
URL: https://issues.apache.org/jira/browse/HBASE-10669
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 0.94.11, 0.96.0
Reporter: Deepak Sharma
Assignee: Deepak Sharma
Priority: Minor
Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.18
Attachments: HBASE_10669_94.11.patch, Hbck_usage_issue.patch
Usage is wrong for hbck tool for -sidelineCorruptHfiles option:
it is like:
-sidelineCorruptHfiles Quarantine corrupted HFiles. implies
-checkCorruptHfiles
here in sidelineCorruptHfiles and checkCorruptHfiles small 'f' is used
but actually in code it is like
else if (cmd.equals(-checkCorruptHFiles)) {
checkCorruptHFiles = true;
} else if (cmd.equals(-sidelineCorruptHFiles)) {
sidelineCorruptHFiles = true;
}
so if we use sidelineCorruptHfiles option for hbck then it will give error
Unrecognized option:-sidelineCorruptHfiles
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10670) HBaseFsck#connect() should use new connection
[
https://issues.apache.org/jira/browse/HBASE-10670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921403#comment-13921403
]
Hudson commented on HBASE-10670:
SUCCESS: Integrated in HBase-TRUNK #4979 (See
[https://builds.apache.org/job/HBase-TRUNK/4979/])
HBASE-10670 HBaseFsck#connect() should use new connection (tedyu: rev 1574588)
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
HBaseFsck#connect() should use new connection
-
Key: HBASE-10670
URL: https://issues.apache.org/jira/browse/HBASE-10670
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 0.98.1, 0.99.0
Attachments: 10670-TestHBaseFsck.testCheckTableLocks.html,
10670-v1.txt
When investigating TestHBaseFsck#testCheckTableLocks failure, I noticed the
following:
{code}
2014-03-03 04:26:04,981 WARN [Thread-1180]
client.ConnectionManager$HConnectionImplementation(1998): Checking master
connection
com.google.protobuf.ServiceException: java.io.IOException: Call to
c59-s15.cs1cloud.internal/172.18.145.15:52272 failed on local exception:
org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call id=1282,
waitTime=1, rpcTimeout=0
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1699)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1740)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:40216)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionManager.java:1358)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionManager.java:1991)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1710)
at
org.apache.hadoop.hbase.client.HBaseAdmin$MasterCallable.prepare(HBaseAdmin.java:3199)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:120)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:97)
at
org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3226)
at
org.apache.hadoop.hbase.client.HBaseAdmin.getClusterStatus(HBaseAdmin.java:2158)
at org.apache.hadoop.hbase.util.HBaseFsck.connect(HBaseFsck.java:308)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:52)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:43)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:38)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testCheckTableLocks(TestHBaseFsck.java:2100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: java.io.IOException: Call to
c59-s15.cs1cloud.internal/172.18.145.15:52272 failed on local exception:
org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call id=1282,
waitTime=1, rpcTimeout=0
at
org.apache.hadoop.hbase.ipc.RpcClient.wrapException(RpcClient.java:1516)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1486)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1684)
... 24 more
Caused by: org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call
id=1282, waitTime=1, rpcTimeout=0
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.cleanupCalls(RpcClient.java:1214)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.cleanupCalls(RpcClient.java:1205)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.close(RpcClient.java:1006)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.run(RpcClient.java:749)
{code}
This ctor was used in HBaseFsck#connect():
{code}
[jira] [Commented] (HBASE-10677) boundaries check in hbck throwing IllegalArgumentException
[
https://issues.apache.org/jira/browse/HBASE-10677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921402#comment-13921402
]
Hudson commented on HBASE-10677:
SUCCESS: Integrated in HBase-TRUNK #4979 (See
[https://builds.apache.org/job/HBase-TRUNK/4979/])
HBASE-10677 boundaries check in hbck throwing IllegalArgumentException(Rajesh)
(rajeshbabu: rev 1574583)
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
boundaries check in hbck throwing IllegalArgumentException
--
Key: HBASE-10677
URL: https://issues.apache.org/jira/browse/HBASE-10677
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 0.98.0
Reporter: rajeshbabu
Assignee: rajeshbabu
Fix For: 0.96.2, 0.98.1, 0.99.0
Attachments: HBASE-10677.patch, HBASE-10677_v1.patch,
HBASE-10677_v2.patch
This is the exception
{code}
java.lang.IllegalArgumentException: Pathname
/hbase/hbase:labels/438f84dea5d0e24e390de927deb8a84e from
hdfs://10.18.40.29:54310/hbase/hbase:labels/438f84dea5d0e24e390de927deb8a84e
is not a valid DFS filename.
at
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:184)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:637)
at
org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:92)
at
org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:702)
at
org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:698)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:698)
at
org.apache.hadoop.hbase.util.HBaseFsck.checkRegionBoundaries(HBaseFsck.java:537)
at
org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:487)
at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:4028)
at
org.apache.hadoop.hbase.util.HBaseFsck$HBaseFsckTool.run(HBaseFsck.java:3837)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3825)
{code}
We are pointing to wrong table directory. This is the code causing the
problem.
{code}
// For each region, get the start and stop key from the META and
compare them to the
// same information from the Stores.
Path path = new Path(getConf().get(HConstants.HBASE_DIR) + /
+ Bytes.toString(regionInfo.getTable().getName()) + /
+ regionInfo.getEncodedName() + /);
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10676) Removing ThreadLocal of PrefetchedHeader in HFileBlock.FSReaderV2 make higher perforamce of scan
[
https://issues.apache.org/jira/browse/HBASE-10676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921410#comment-13921410
]
Lars Hofhansl commented on HBASE-10676:
---
We should also test the scenario when most data is filtered at the server (such
as in Phoenix).
Removing ThreadLocal of PrefetchedHeader in HFileBlock.FSReaderV2 make higher
perforamce of scan
Key: HBASE-10676
URL: https://issues.apache.org/jira/browse/HBASE-10676
Project: HBase
Issue Type: Improvement
Affects Versions: 0.98.0
Reporter: zhaojianbo
Attachments: HBASE-10676-0.98-branch.patch
PrefetchedHeader variable in HFileBlock.FSReaderV2 is used for avoiding
backward seek operation as the comment said:
{quote}
we will not incur a backward seek operation if we have already read this
block's header as part of the previous read's look-ahead. And we also want to
skip reading the header again if it has already been read.
{quote}
But that is not the case. In the code of 0.98, prefetchedHeader is
threadlocal for one storefile reader, and in the RegionScanner
lifecycle，different rpc handlers will serve scan requests of the same
scanner. Even though one handler of previous scan call prefetched the next
block header, the other handlers of current scan call will still trigger a
backward seek operation. The process is like this:
# rs handler1 serves the scan call, reads block1 and prefetches the header of
block2
# rs handler2 serves the same scanner's next scan call, because rs handler2
doesn't know the header of block2 already prefetched by rs handler1, triggers
a backward seek and reads block2, and prefetches the header of block3.
It is not the sequential read. So I think that the threadlocal is useless,
and should be abandoned. I did the work, and evaluated the performance of one
client, two client and four client scanning the same region with one
storefile. The test environment is
# A hdfs cluster with a namenode, a secondary namenode , a datanode in a
machine
# A hbase cluster with a zk, a master, a regionserver in the same machine
# clients are also in the same machine.
So all the data is local. The storefile is about 22.7GB from our online data,
18995949 kvs. Caching is set 1000.
With the improvement, the client total scan time decreases 21% for the one
client case, 11% for the two clients case. But the four clients case is
almost the same. The details tests' data is the following:
||case||client||time(ms)||
| original | 1 | 306222 |
| new | 1 | 241313 |
| original | 2 | 416390 |
| new | 2 | 369064 |
| original | 4 | 555986 |
| new | 4 | 562152 |
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10669) [hbck tool] Usage is wrong for hbck tool for -sidelineCorruptHfiles option
[
https://issues.apache.org/jira/browse/HBASE-10669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921425#comment-13921425
]
Hudson commented on HBASE-10669:
FAILURE: Integrated in HBase-0.94-security #431 (See
[https://builds.apache.org/job/HBase-0.94-security/431/])
HBASE-10669 [hbck tool] Usage is wrong for hbck tool for -sidelineCorruptHfiles
option(Deepak Sharma) (rajeshbabu: rev 1574606)
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
[hbck tool] Usage is wrong for hbck tool for -sidelineCorruptHfiles option
--
Key: HBASE-10669
URL: https://issues.apache.org/jira/browse/HBASE-10669
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 0.94.11, 0.96.0
Reporter: Deepak Sharma
Assignee: Deepak Sharma
Priority: Minor
Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.18
Attachments: HBASE_10669_94.11.patch, Hbck_usage_issue.patch
Usage is wrong for hbck tool for -sidelineCorruptHfiles option:
it is like:
-sidelineCorruptHfiles Quarantine corrupted HFiles. implies
-checkCorruptHfiles
here in sidelineCorruptHfiles and checkCorruptHfiles small 'f' is used
but actually in code it is like
else if (cmd.equals(-checkCorruptHFiles)) {
checkCorruptHFiles = true;
} else if (cmd.equals(-sidelineCorruptHFiles)) {
sidelineCorruptHFiles = true;
}
so if we use sidelineCorruptHfiles option for hbck then it will give error
Unrecognized option:-sidelineCorruptHfiles
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10594) Speed up TestRestoreSnapshotFromClient
[ https://issues.apache.org/jira/browse/HBASE-10594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921426#comment-13921426 ] Hudson commented on HBASE-10594: FAILURE: Integrated in HBase-0.94-security #431 (See [https://builds.apache.org/job/HBase-0.94-security/431/]) HBASE-10594 Speed up TestRestoreSnapshotFromClient. (larsh: rev 1574599) * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestRestoreSnapshotFromClient.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/snapshot/SnapshotTestingUtils.java Speed up TestRestoreSnapshotFromClient -- Key: HBASE-10594 URL: https://issues.apache.org/jira/browse/HBASE-10594 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Minor Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.18 Attachments: 10594-0.94-v2.txt, 10594-0.94.txt, 10594-trunk-v2.txt, 10594-trunk.txt Looking through the longest running test in 0.94 I noticed that TestRestoreSnapshotFromClient runs for over 10 minutes on the jenkins boxes (264s on my local box). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-6849) Make StochasticLoadBalancer the default
[ https://issues.apache.org/jira/browse/HBASE-6849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921442#comment-13921442 ] Enis Soztutar commented on HBASE-6849: -- bq. Although I suppose that when a table is large enough for this to matter it will naturally have been spread out over the RSs, so maybe it's not an issue. I agree that balancing of a single table's regions is important, but we can improve on the SLB if it does produce skewed plans. Make StochasticLoadBalancer the default --- Key: HBASE-6849 URL: https://issues.apache.org/jira/browse/HBASE-6849 Project: HBase Issue Type: Improvement Components: master Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.95.0 Attachments: HBASE-6849-0.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10622) Improve log and Exceptions in Export Snapshot
[ https://issues.apache.org/jira/browse/HBASE-10622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921459#comment-13921459 ] Hudson commented on HBASE-10622: SUCCESS: Integrated in HBase-0.98 #200 (See [https://builds.apache.org/job/HBase-0.98/200/]) HBASE-10622 Improve log and Exceptions in Export Snapshot (mbertozzi: rev 1574032) * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java Improve log and Exceptions in Export Snapshot -- Key: HBASE-10622 URL: https://issues.apache.org/jira/browse/HBASE-10622 Project: HBase Issue Type: Bug Components: snapshots Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.18 Attachments: HBASE-10622-v0.patch, HBASE-10622-v1.patch, HBASE-10622-v2.patch, HBASE-10622-v3.patch, HBASE-10622-v4.patch from the logs of export snapshot is not really clear what's going on, adding some extra information useful to debug, and in some places the real exception can be thrown -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10598) Written data can not be read out because MemStore#timeRangeTracker might be updated concurrently
[
https://issues.apache.org/jira/browse/HBASE-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921454#comment-13921454
]
Hudson commented on HBASE-10598:
SUCCESS: Integrated in HBase-0.98 #200 (See
[https://builds.apache.org/job/HBase-0.98/200/])
HBASE-10624 Fix 2 new findbugs warnings introduced by HBASE-10598 (tedyu: rev
1574148)
*
/hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
*
/hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/TimeRangeTracker.java
Written data can not be read out because MemStore#timeRangeTracker might be
updated concurrently
Key: HBASE-10598
URL: https://issues.apache.org/jira/browse/HBASE-10598
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.16
Reporter: cuijianwei
Assignee: cuijianwei
Priority: Critical
Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.18
Attachments: HBASE-10598-0.94-v2.patch, HBASE-10598-0.94.v1.patch,
HBASE-10598-trunk-v1.patch
In our test environment, we find written data can't be read out occasionally.
After debugging, we find that maximumTimestamp/minimumTimestamp of
MemStore#timeRangeTracker might decrease/increase when
MemStore#timeRangeTracker is updated concurrently, which might make the
MemStore/StoreFile to be filtered incorrectly when reading data out. Let's
see how the concurrent updating of timeRangeTracker#maximumTimestamp cause
this problem.
Imagining there are two threads T1 and T2 putting two KeyValues kv1 and kv2.
kv1 and kv2 belong to the same Store(so belong to the same region), but
contain different rowkeys. Consequently, kv1 and kv2 could be updated
concurrently. When we see the implementation of HRegionServer#multi, kv1 and
kv2 will be add to MemStore by HRegion#applyFamilyMapToMemstore in
HRegion#doMiniBatchMutation. Then, MemStore#internalAdd will be invoked and
MemStore#timeRangeTracker will be updated by
TimeRangeTracker#includeTimestamp as follows:
{code}
private void includeTimestamp(final long timestamp) {
...
else if (maximumTimestamp timestamp) {
maximumTimestamp = timestamp;
}
return;
}
{code}
Imagining the current maximumTimestamp of TimeRangeTracker is t0 before
includeTimestamp(...) invoked, kv1.timestamp=t1, kv2.timestamp=t2, t1 and t2
are both set by user(then, user knows the timestamps of kv1 and kv2), and t1
t2 t0. T1 and T2 will be executed concurrently, therefore, the two
threads might both find the current maximumTimestamp is less than the
timestamp of its kv. After that, T1 and T2 will both set maximumTimestamp to
timestamp of its kv. If T1 set maximumTimestamp before T2 doing that, the
maximumTimestamp will be set to t2. Then, before any new update with bigger
timestamp has been applied to the MemStore, if we try to read out kv1 by
HTable#get and set the timestamp of 'Get' to t1, the StoreScanner will decide
whether the MemStoreScanner(imagining kv1 has not been flushed) should be
selected as candidate scanner by MemStoreScanner#shouldUseScanner. Then, the
MemStore won't be selected in MemStoreScanner#shouldUseScanner because
maximumTimestamp of the MemStore has been set to t2 (t2 t1). Consequently,
the written kv1 can't be read out and kv1 is lost from user's perspective.
If the above analysis is right, after maximumTimestamp of
MemStore#timeRangeTracker has been set to t2, user will experience data lass
in the following situations:
1. Before any new write with kv.timestamp t1 has been add to the MemStore,
read request of kv1 with timestamp=t1 can not read kv1 out.
2. Before any new write with kv.timestamp t1 has been add to the MemStore,
if a flush happened, the data of MemStore will be flushed to StoreFile with
StoreFile#maximumTimestamp set to t2. After that, any read request with
timestamp=t1 can not read kv1 before next compaction(Actually, kv1.timestamp
might not be included in timeRange of the StoreFile even after compaction).
The second situation is much more serious because the incorrect timeRange of
MemStore has been persisted to the file.
Similarly, the concurrent update of TimeRangeTracker#minimumTimestamp may
also cause this problem.
As a simple way to fix the problem, we could add synchronized to
TimeRangeTracker#includeTimestamp so that this method won't be invoked
concurrently.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10624) Fix 2 new findbugs warnings introduced by HBASE-10598
[
https://issues.apache.org/jira/browse/HBASE-10624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921456#comment-13921456
]
Hudson commented on HBASE-10624:
SUCCESS: Integrated in HBase-0.98 #200 (See
[https://builds.apache.org/job/HBase-0.98/200/])
HBASE-10624 Fix 2 new findbugs warnings introduced by HBASE-10598 (tedyu: rev
1574148)
*
/hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
*
/hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/TimeRangeTracker.java
Fix 2 new findbugs warnings introduced by HBASE-10598
-
Key: HBASE-10624
URL: https://issues.apache.org/jira/browse/HBASE-10624
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.18
Attachments: 10624-0.94.txt, 10624-v1.txt, 10624-v2.txt, 10624-v4.txt
Inconsistent synchronization of
org.apache.hadoop.hbase.regionserver.TimeRangeTracker.maximumTimestamp;
locked 66% of time
{code}
In class org.apache.hadoop.hbase.regionserver.TimeRangeTracker
Field org.apache.hadoop.hbase.regionserver.TimeRangeTracker.maximumTimestamp
Synchronized 66% of the time
{code}
Inconsistent synchronization of
org.apache.hadoop.hbase.regionserver.TimeRangeTracker.minimumTimestamp;
locked 62% of time
{code}
In class org.apache.hadoop.hbase.regionserver.TimeRangeTracker
Field org.apache.hadoop.hbase.regionserver.TimeRangeTracker.minimumTimestamp
Synchronized 62% of the time
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10669) [hbck tool] Usage is wrong for hbck tool for -sidelineCorruptHfiles option
[
https://issues.apache.org/jira/browse/HBASE-10669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921453#comment-13921453
]
Hudson commented on HBASE-10669:
SUCCESS: Integrated in HBase-0.98 #200 (See
[https://builds.apache.org/job/HBase-0.98/200/])
HBASE-10669 [hbck tool] Usage is wrong for hbck tool for -sidelineCorruptHfiles
option(Deepak Sharma) (rajeshbabu: rev 1574596)
*
/hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
[hbck tool] Usage is wrong for hbck tool for -sidelineCorruptHfiles option
--
Key: HBASE-10669
URL: https://issues.apache.org/jira/browse/HBASE-10669
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 0.94.11, 0.96.0
Reporter: Deepak Sharma
Assignee: Deepak Sharma
Priority: Minor
Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.18
Attachments: HBASE_10669_94.11.patch, Hbck_usage_issue.patch
Usage is wrong for hbck tool for -sidelineCorruptHfiles option:
it is like:
-sidelineCorruptHfiles Quarantine corrupted HFiles. implies
-checkCorruptHfiles
here in sidelineCorruptHfiles and checkCorruptHfiles small 'f' is used
but actually in code it is like
else if (cmd.equals(-checkCorruptHFiles)) {
checkCorruptHFiles = true;
} else if (cmd.equals(-sidelineCorruptHFiles)) {
sidelineCorruptHFiles = true;
}
so if we use sidelineCorruptHfiles option for hbck then it will give error
Unrecognized option:-sidelineCorruptHfiles
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10677) boundaries check in hbck throwing IllegalArgumentException
[
https://issues.apache.org/jira/browse/HBASE-10677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921455#comment-13921455
]
Hudson commented on HBASE-10677:
SUCCESS: Integrated in HBase-0.98 #200 (See
[https://builds.apache.org/job/HBase-0.98/200/])
HBASE-10677 boundaries check in hbck throwing IllegalArgumentException(Rajesh)
(rajeshbabu: rev 1574585)
*
/hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
*
/hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
boundaries check in hbck throwing IllegalArgumentException
--
Key: HBASE-10677
URL: https://issues.apache.org/jira/browse/HBASE-10677
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 0.98.0
Reporter: rajeshbabu
Assignee: rajeshbabu
Fix For: 0.96.2, 0.98.1, 0.99.0
Attachments: HBASE-10677.patch, HBASE-10677_v1.patch,
HBASE-10677_v2.patch
This is the exception
{code}
java.lang.IllegalArgumentException: Pathname
/hbase/hbase:labels/438f84dea5d0e24e390de927deb8a84e from
hdfs://10.18.40.29:54310/hbase/hbase:labels/438f84dea5d0e24e390de927deb8a84e
is not a valid DFS filename.
at
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:184)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:637)
at
org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:92)
at
org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:702)
at
org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:698)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:698)
at
org.apache.hadoop.hbase.util.HBaseFsck.checkRegionBoundaries(HBaseFsck.java:537)
at
org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:487)
at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:4028)
at
org.apache.hadoop.hbase.util.HBaseFsck$HBaseFsckTool.run(HBaseFsck.java:3837)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3825)
{code}
We are pointing to wrong table directory. This is the code causing the
problem.
{code}
// For each region, get the start and stop key from the META and
compare them to the
// same information from the Stores.
Path path = new Path(getConf().get(HConstants.HBASE_DIR) + /
+ Bytes.toString(regionInfo.getTable().getName()) + /
+ regionInfo.getEncodedName() + /);
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10594) Speed up TestRestoreSnapshotFromClient
[ https://issues.apache.org/jira/browse/HBASE-10594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921457#comment-13921457 ] Hudson commented on HBASE-10594: SUCCESS: Integrated in HBase-0.98 #200 (See [https://builds.apache.org/job/HBase-0.98/200/]) HBASE-10594 Speed up TestRestoreSnapshotFromClient. (larsh: rev 1574601) * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestRestoreSnapshotFromClient.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/SnapshotTestingUtils.java Speed up TestRestoreSnapshotFromClient -- Key: HBASE-10594 URL: https://issues.apache.org/jira/browse/HBASE-10594 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Minor Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.18 Attachments: 10594-0.94-v2.txt, 10594-0.94.txt, 10594-trunk-v2.txt, 10594-trunk.txt Looking through the longest running test in 0.94 I noticed that TestRestoreSnapshotFromClient runs for over 10 minutes on the jenkins boxes (264s on my local box). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10676) Removing ThreadLocal of PrefetchedHeader in HFileBlock.FSReaderV2 make higher perforamce of scan
[
https://issues.apache.org/jira/browse/HBASE-10676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921461#comment-13921461
]
Nick Dimiduk commented on HBASE-10676:
--
bq. Does your box have SSD or spinning disk?
That was a question for [~zhaojianbo] re: the perf numbers posted.
Removing ThreadLocal of PrefetchedHeader in HFileBlock.FSReaderV2 make higher
perforamce of scan
Key: HBASE-10676
URL: https://issues.apache.org/jira/browse/HBASE-10676
Project: HBase
Issue Type: Improvement
Affects Versions: 0.98.0
Reporter: zhaojianbo
Attachments: HBASE-10676-0.98-branch.patch
PrefetchedHeader variable in HFileBlock.FSReaderV2 is used for avoiding
backward seek operation as the comment said:
{quote}
we will not incur a backward seek operation if we have already read this
block's header as part of the previous read's look-ahead. And we also want to
skip reading the header again if it has already been read.
{quote}
But that is not the case. In the code of 0.98, prefetchedHeader is
threadlocal for one storefile reader, and in the RegionScanner
lifecycle，different rpc handlers will serve scan requests of the same
scanner. Even though one handler of previous scan call prefetched the next
block header, the other handlers of current scan call will still trigger a
backward seek operation. The process is like this:
# rs handler1 serves the scan call, reads block1 and prefetches the header of
block2
# rs handler2 serves the same scanner's next scan call, because rs handler2
doesn't know the header of block2 already prefetched by rs handler1, triggers
a backward seek and reads block2, and prefetches the header of block3.
It is not the sequential read. So I think that the threadlocal is useless,
and should be abandoned. I did the work, and evaluated the performance of one
client, two client and four client scanning the same region with one
storefile. The test environment is
# A hdfs cluster with a namenode, a secondary namenode , a datanode in a
machine
# A hbase cluster with a zk, a master, a regionserver in the same machine
# clients are also in the same machine.
So all the data is local. The storefile is about 22.7GB from our online data,
18995949 kvs. Caching is set 1000.
With the improvement, the client total scan time decreases 21% for the one
client case, 11% for the two clients case. But the four clients case is
almost the same. The details tests' data is the following:
||case||client||time(ms)||
| original | 1 | 306222 |
| new | 1 | 241313 |
| original | 2 | 416390 |
| new | 2 | 369064 |
| original | 4 | 555986 |
| new | 4 | 562152 |
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-9708) Improve Snapshot Name Error Message
[
https://issues.apache.org/jira/browse/HBASE-9708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921458#comment-13921458
]
Hudson commented on HBASE-9708:
---
SUCCESS: Integrated in HBase-0.98 #200 (See
[https://builds.apache.org/job/HBase-0.98/200/])
HBASE-9708 Improve Snapshot Name Error Message (Esteban Gutierrez) (mbertozzi:
rev 1573948)
*
/hbase/branches/0.98/hbase-client/src/main/java/org/apache/hadoop/hbase/snapshot/ClientSnapshotDescriptionUtils.java
*
/hbase/branches/0.98/hbase-common/src/main/java/org/apache/hadoop/hbase/TableName.java
Improve Snapshot Name Error Message
---
Key: HBASE-9708
URL: https://issues.apache.org/jira/browse/HBASE-9708
Project: HBase
Issue Type: Bug
Components: snapshots
Affects Versions: 0.94.2
Reporter: Jesse Anderson
Assignee: Matteo Bertozzi
Priority: Minor
Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.18
Attachments: HBASE-9708.1.patch, HBASE-9708.1.patch
The output for snapshots when you enter an invalid snapshot name talks about
User-space table names instead of Snapshot names. The error message
should say Snapshot names can only contain
Here is an example of the output:
{noformat}
hbase(main):001:0 snapshot 'user', 'asdf asdf'
ERROR: java.lang.IllegalArgumentException: Illegal character 32 at 4.
User-space table names can only contain 'word characters': i.e.
[a-zA-Z_0-9-.]: asdf asdf
Here is some help for this command:
Take a snapshot of specified table. Examples:
hbase snapshot 'sourceTable', 'snapshotName'
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10669) [hbck tool] Usage is wrong for hbck tool for -sidelineCorruptHfiles option
[
https://issues.apache.org/jira/browse/HBASE-10669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921474#comment-13921474
]
Hudson commented on HBASE-10669:
FAILURE: Integrated in HBase-0.94-JDK7 #74 (See
[https://builds.apache.org/job/HBase-0.94-JDK7/74/])
HBASE-10669 [hbck tool] Usage is wrong for hbck tool for -sidelineCorruptHfiles
option(Deepak Sharma) (rajeshbabu: rev 1574606)
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
[hbck tool] Usage is wrong for hbck tool for -sidelineCorruptHfiles option
--
Key: HBASE-10669
URL: https://issues.apache.org/jira/browse/HBASE-10669
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 0.94.11, 0.96.0
Reporter: Deepak Sharma
Assignee: Deepak Sharma
Priority: Minor
Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.18
Attachments: HBASE_10669_94.11.patch, Hbck_usage_issue.patch
Usage is wrong for hbck tool for -sidelineCorruptHfiles option:
it is like:
-sidelineCorruptHfiles Quarantine corrupted HFiles. implies
-checkCorruptHfiles
here in sidelineCorruptHfiles and checkCorruptHfiles small 'f' is used
but actually in code it is like
else if (cmd.equals(-checkCorruptHFiles)) {
checkCorruptHFiles = true;
} else if (cmd.equals(-sidelineCorruptHFiles)) {
sidelineCorruptHFiles = true;
}
so if we use sidelineCorruptHfiles option for hbck then it will give error
Unrecognized option:-sidelineCorruptHfiles
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10594) Speed up TestRestoreSnapshotFromClient
[ https://issues.apache.org/jira/browse/HBASE-10594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921475#comment-13921475 ] Hudson commented on HBASE-10594: FAILURE: Integrated in HBase-0.94-JDK7 #74 (See [https://builds.apache.org/job/HBase-0.94-JDK7/74/]) HBASE-10594 Speed up TestRestoreSnapshotFromClient. (larsh: rev 1574599) * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestRestoreSnapshotFromClient.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/snapshot/SnapshotTestingUtils.java Speed up TestRestoreSnapshotFromClient -- Key: HBASE-10594 URL: https://issues.apache.org/jira/browse/HBASE-10594 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Minor Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.18 Attachments: 10594-0.94-v2.txt, 10594-0.94.txt, 10594-trunk-v2.txt, 10594-trunk.txt Looking through the longest running test in 0.94 I noticed that TestRestoreSnapshotFromClient runs for over 10 minutes on the jenkins boxes (264s on my local box). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10649) TestMasterMetrics fails occasionally
[
https://issues.apache.org/jira/browse/HBASE-10649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921482#comment-13921482
]
Ted Yu commented on HBASE-10649:
lgtm
TestMasterMetrics fails occasionally
Key: HBASE-10649
URL: https://issues.apache.org/jira/browse/HBASE-10649
Project: HBase
Issue Type: Test
Reporter: Ted Yu
Assignee: Jimmy Xiang
Attachments: hbase-10649.patch
Latest occurrence was in https://builds.apache.org/job/HBase-TRUNK/4970
{code}
java.io.IOException: Shutting down
at
org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:231)
at
org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:93)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:875)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:839)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:756)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:727)
at
org.apache.hadoop.hbase.master.TestMasterMetrics.startCluster(TestMasterMetrics.java:56)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runners.Suite.runChild(Suite.java:127)
at org.junit.runners.Suite.runChild(Suite.java:26)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.lang.RuntimeException: Master not initialized after 20ms
seconds
at
org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:221)
at
org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:425)
at
org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:224)
... 25 more
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-10670) HBaseFsck#connect() should use new connection
[
https://issues.apache.org/jira/browse/HBASE-10670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-10670:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
HBaseFsck#connect() should use new connection
-
Key: HBASE-10670
URL: https://issues.apache.org/jira/browse/HBASE-10670
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 0.98.1, 0.99.0
Attachments: 10670-TestHBaseFsck.testCheckTableLocks.html,
10670-v1.txt
When investigating TestHBaseFsck#testCheckTableLocks failure, I noticed the
following:
{code}
2014-03-03 04:26:04,981 WARN [Thread-1180]
client.ConnectionManager$HConnectionImplementation(1998): Checking master
connection
com.google.protobuf.ServiceException: java.io.IOException: Call to
c59-s15.cs1cloud.internal/172.18.145.15:52272 failed on local exception:
org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call id=1282,
waitTime=1, rpcTimeout=0
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1699)
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1740)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:40216)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionManager.java:1358)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionManager.java:1991)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1710)
at
org.apache.hadoop.hbase.client.HBaseAdmin$MasterCallable.prepare(HBaseAdmin.java:3199)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:120)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:97)
at
org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3226)
at
org.apache.hadoop.hbase.client.HBaseAdmin.getClusterStatus(HBaseAdmin.java:2158)
at org.apache.hadoop.hbase.util.HBaseFsck.connect(HBaseFsck.java:308)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:52)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:43)
at
org.apache.hadoop.hbase.util.hbck.HbckTestingUtil.doFsck(HbckTestingUtil.java:38)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testCheckTableLocks(TestHBaseFsck.java:2100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: java.io.IOException: Call to
c59-s15.cs1cloud.internal/172.18.145.15:52272 failed on local exception:
org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call id=1282,
waitTime=1, rpcTimeout=0
at
org.apache.hadoop.hbase.ipc.RpcClient.wrapException(RpcClient.java:1516)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1486)
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1684)
... 24 more
Caused by: org.apache.hadoop.hbase.ipc.RpcClient$CallTimeoutException: Call
id=1282, waitTime=1, rpcTimeout=0
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.cleanupCalls(RpcClient.java:1214)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.cleanupCalls(RpcClient.java:1205)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.close(RpcClient.java:1006)
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.run(RpcClient.java:749)
{code}
This ctor was used in HBaseFsck#connect():
{code}
public HBaseAdmin(Configuration c)
throws MasterNotRunningException, ZooKeeperConnectionException, IOException
{
// Will not leak connections, as the new implementation of the constructor
// does not throw exceptions anymore.
[jira] [Created] (HBASE-10685) [WINDOWS] TestKeyStoreKeyProvider fails on windows
Enis Soztutar created HBASE-10685:
-
Summary: [WINDOWS] TestKeyStoreKeyProvider fails on windows
Key: HBASE-10685
URL: https://issues.apache.org/jira/browse/HBASE-10685
Project: HBase
Issue Type: Bug
Components: test
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Priority: Minor
Fix For: 0.98.1, 0.99.0
Fails because of lack of URI escaping:
{code}
java.lang.RuntimeException: java.net.URISyntaxException: Illegal character in
authority at index 8:
jceks://D:\w\hbbk\hbase-common\target\test-data\78c21f3c-80a6-4cd7-9706-0098a1d33496\keystore.jks?passwordFile=D%3A%5Cw%5Chbbk%5Chbase-common%5Ctarget%5Ctest-data%5C78c21f3c-80a6-4cd7-9706-0098a1d33496%5Ckeystore.pw
at
org.apache.hadoop.hbase.io.crypto.KeyStoreKeyProvider.init(KeyStoreKeyProvider.java:149)
at
org.apache.hadoop.hbase.io.crypto.TestKeyStoreKeyProvider.testKeyStoreKeyProviderWithPasswordFile(TestKeyStoreKeyProvider.java:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: java.net.URISyntaxException: Illegal character in authority at index
8:
jceks://D:\w\hbbk\hbase-common\target\test-data\78c21f3c-80a6-4cd7-9706-0098a1d33496\keystore.jks?passwordFile=D%3A%5Cw%5Chbbk%5Chbase-common%5Ctarget%5Ctest-data%5C78c21f3c-80a6-4cd7-9706-0098a1d33496%5Ckeystore.pw
at java.net.URI$Parser.fail(URI.java:2810)
at java.net.URI$Parser.parseAuthority(URI.java:3148)
at java.net.URI$Parser.parseHierarchical(URI.java:3059)
at java.net.URI$Parser.parse(URI.java:3015)
at java.net.URI.init(URI.java:577)
at
org.apache.hadoop.hbase.io.crypto.KeyStoreKeyProvider.init(KeyStoreKeyProvider.java:139)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-10685) [WINDOWS] TestKeyStoreKeyProvider fails on windows
[
https://issues.apache.org/jira/browse/HBASE-10685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-10685:
--
Status: Patch Available (was: Open)
[WINDOWS] TestKeyStoreKeyProvider fails on windows
--
Key: HBASE-10685
URL: https://issues.apache.org/jira/browse/HBASE-10685
Project: HBase
Issue Type: Bug
Components: test
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Priority: Minor
Fix For: 0.98.1, 0.99.0
Attachments: hbase-10685_v1.patch
Fails because of lack of URI escaping:
{code}
java.lang.RuntimeException: java.net.URISyntaxException: Illegal character in
authority at index 8:
jceks://D:\w\hbbk\hbase-common\target\test-data\78c21f3c-80a6-4cd7-9706-0098a1d33496\keystore.jks?passwordFile=D%3A%5Cw%5Chbbk%5Chbase-common%5Ctarget%5Ctest-data%5C78c21f3c-80a6-4cd7-9706-0098a1d33496%5Ckeystore.pw
at
org.apache.hadoop.hbase.io.crypto.KeyStoreKeyProvider.init(KeyStoreKeyProvider.java:149)
at
org.apache.hadoop.hbase.io.crypto.TestKeyStoreKeyProvider.testKeyStoreKeyProviderWithPasswordFile(TestKeyStoreKeyProvider.java:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: java.net.URISyntaxException: Illegal character in authority at
index 8:
jceks://D:\w\hbbk\hbase-common\target\test-data\78c21f3c-80a6-4cd7-9706-0098a1d33496\keystore.jks?passwordFile=D%3A%5Cw%5Chbbk%5Chbase-common%5Ctarget%5Ctest-data%5C78c21f3c-80a6-4cd7-9706-0098a1d33496%5Ckeystore.pw
at java.net.URI$Parser.fail(URI.java:2810)
at java.net.URI$Parser.parseAuthority(URI.java:3148)
at java.net.URI$Parser.parseHierarchical(URI.java:3059)
at java.net.URI$Parser.parse(URI.java:3015)
at java.net.URI.init(URI.java:577)
at
org.apache.hadoop.hbase.io.crypto.KeyStoreKeyProvider.init(KeyStoreKeyProvider.java:139)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10669) [hbck tool] Usage is wrong for hbck tool for -sidelineCorruptHfiles option
[
https://issues.apache.org/jira/browse/HBASE-10669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13921490#comment-13921490
]
Hudson commented on HBASE-10669:
FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #188 (See
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/188/])
HBASE-10669 [hbck tool] Usage is wrong for hbck tool for -sidelineCorruptHfiles
option(Deepak Sharma) (rajeshbabu: rev 1574596)
*
/hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
[hbck tool] Usage is wrong for hbck tool for -sidelineCorruptHfiles option
--
Key: HBASE-10669
URL: https://issues.apache.org/jira/browse/HBASE-10669
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 0.94.11, 0.96.0
Reporter: Deepak Sharma
Assignee: Deepak Sharma
Priority: Minor
Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.18
Attachments: HBASE_10669_94.11.patch, Hbck_usage_issue.patch
Usage is wrong for hbck tool for -sidelineCorruptHfiles option:
it is like:
-sidelineCorruptHfiles Quarantine corrupted HFiles. implies
-checkCorruptHfiles
here in sidelineCorruptHfiles and checkCorruptHfiles small 'f' is used
but actually in code it is like
else if (cmd.equals(-checkCorruptHFiles)) {
checkCorruptHFiles = true;
} else if (cmd.equals(-sidelineCorruptHFiles)) {
sidelineCorruptHFiles = true;
}
so if we use sidelineCorruptHfiles option for hbck then it will give error
Unrecognized option:-sidelineCorruptHfiles
--
This message was sent by Atlassian JIRA
(v6.2#6252)
