[jira] [Updated] (HBASE-11802) Scan copy constructor doesn't copy reversed member variable
[ https://issues.apache.org/jira/browse/HBASE-11802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-11802: --- Attachment: HBASE-11802.patch Simple patch. [~jamestaylor]? Scan copy constructor doesn't copy reversed member variable --- Key: HBASE-11802 URL: https://issues.apache.org/jira/browse/HBASE-11802 Project: HBase Issue Type: Bug Affects Versions: 0.98.5 Reporter: James Taylor Priority: Minor Attachments: HBASE-11802.patch The Scan copy constructor doesn't copy reversed member variable. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11802) Scan copy constructor doesn't copy reversed member variable
[ https://issues.apache.org/jira/browse/HBASE-11802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-11802: --- Status: Patch Available (was: Open) Scan copy constructor doesn't copy reversed member variable --- Key: HBASE-11802 URL: https://issues.apache.org/jira/browse/HBASE-11802 Project: HBase Issue Type: Bug Affects Versions: 0.98.5 Reporter: James Taylor Priority: Minor Attachments: HBASE-11802.patch The Scan copy constructor doesn't copy reversed member variable. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11802) Scan copy constructor doesn't copy reversed member variable
[ https://issues.apache.org/jira/browse/HBASE-11802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-11802: --- Fix Version/s: 0.98.6 2.0.0 0.99.0 Scan copy constructor doesn't copy reversed member variable --- Key: HBASE-11802 URL: https://issues.apache.org/jira/browse/HBASE-11802 Project: HBase Issue Type: Bug Affects Versions: 0.98.5 Reporter: James Taylor Priority: Minor Fix For: 0.99.0, 2.0.0, 0.98.6 Attachments: HBASE-11802.patch The Scan copy constructor doesn't copy reversed member variable. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load COMMIT events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106539#comment-14106539 ] Alex Newman commented on HBASE-11567: - +1 overall (although my vote doesn't count). Sorry for not getting this done sooner, I have been distracted trying to make the build cleaner. On your comments 1) agreed 2) That makes sense, I did to avoid making fields public. I am down either way - Overall it looks good however i have some concerns. - It seems as though the formatting is wrong in some places. Please double check that you autoindent everything - I am curious if we should add a unit test (as opposed to regression or acceptance test) so that the wal entry is only written if everything succeeds. It would verify that your change is successful. Write bulk load COMMIT events to WAL Key: HBASE-11567 URL: https://issues.apache.org/jira/browse/HBASE-11567 Project: HBase Issue Type: Sub-task Reporter: Enis Soztutar Assignee: Alex Newman Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch, hbase-11567-v3.patch Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and region open/close (HBASE-11512) , we should persist bulk load events to WAL. This is especially important for secondary region replicas, since we can use this information to pick up primary regions' files from secondary replicas. A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11802) Scan copy constructor doesn't copy reversed member variable
[ https://issues.apache.org/jira/browse/HBASE-11802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106561#comment-14106561 ] James Taylor commented on HBASE-11802: -- Thanks, [~ramkrishna]. Scan copy constructor doesn't copy reversed member variable --- Key: HBASE-11802 URL: https://issues.apache.org/jira/browse/HBASE-11802 Project: HBase Issue Type: Bug Affects Versions: 0.98.5 Reporter: James Taylor Priority: Minor Fix For: 0.99.0, 2.0.0, 0.98.6 Attachments: HBASE-11802.patch The Scan copy constructor doesn't copy reversed member variable. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load COMMIT events to WAL
[
https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106568#comment-14106568
]
Hadoop QA commented on HBASE-11567:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12663565/hbase-11567-v3.patch
against trunk revision .
ATTACHMENT ID: 12663565
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 6
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ new java.lang.String[] { TableName, EncodedRegionName,
Stores, BulkloadSeqNum, });
+ * @param assignSeqId Force a flush, get it's sequenceId to preserve
the guarantee that all the
+ * edits lower than the highest sequential ID from
all the HFiles are flushed
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/10531//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10531//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10531//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10531//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10531//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10531//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10531//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10531//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10531//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10531//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10531//console
This message is automatically generated.
Write bulk load COMMIT events to WAL
Key: HBASE-11567
URL: https://issues.apache.org/jira/browse/HBASE-11567
Project: HBase
Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Alex Newman
Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch,
hbase-11567-v3.patch
Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and
region open/close (HBASE-11512) , we should persist bulk load events to WAL.
This is especially important for secondary region replicas, since we can use
this information to pick up primary regions' files from secondary replicas.
A design doc for secondary replica replication can be found at HBASE-11183.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11802) Scan copy constructor doesn't copy reversed member variable
[ https://issues.apache.org/jira/browse/HBASE-11802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106570#comment-14106570 ] ramkrishna.s.vasudevan commented on HBASE-11802: Will commit this later today. Thanks [~giacomotaylor]] for the review. Scan copy constructor doesn't copy reversed member variable --- Key: HBASE-11802 URL: https://issues.apache.org/jira/browse/HBASE-11802 Project: HBase Issue Type: Bug Affects Versions: 0.98.5 Reporter: James Taylor Priority: Minor Fix For: 0.99.0, 2.0.0, 0.98.6 Attachments: HBASE-11802.patch The Scan copy constructor doesn't copy reversed member variable. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11339) HBase MOB
[ https://issues.apache.org/jira/browse/HBASE-11339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Jiajia updated HBASE-11339: -- Attachment: (was: MOB user guide .docx) HBase MOB - Key: HBASE-11339 URL: https://issues.apache.org/jira/browse/HBASE-11339 Project: HBase Issue Type: Umbrella Components: regionserver, Scanners Reporter: Jingcheng Du Assignee: Jingcheng Du Attachments: HBase MOB Design-v2.pdf, HBase MOB Design-v3.pdf, HBase MOB Design-v4.pdf, HBase MOB Design.pdf, hbase-11339-in-dev.patch It's quite useful to save the medium binary data like images, documents into Apache HBase. Unfortunately directly saving the binary MOB(medium object) to HBase leads to a worse performance since the frequent split and compaction. In this design, the MOB data are stored in an more efficient way, which keeps a high write/read performance and guarantees the data consistency in Apache HBase. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11339) HBase MOB
[ https://issues.apache.org/jira/browse/HBASE-11339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Jiajia updated HBASE-11339: -- Attachment: MOB user guide.docx update the mob user guide. HBase MOB - Key: HBASE-11339 URL: https://issues.apache.org/jira/browse/HBASE-11339 Project: HBase Issue Type: Umbrella Components: regionserver, Scanners Reporter: Jingcheng Du Assignee: Jingcheng Du Attachments: HBase MOB Design-v2.pdf, HBase MOB Design-v3.pdf, HBase MOB Design-v4.pdf, HBase MOB Design.pdf, MOB user guide.docx, hbase-11339-in-dev.patch It's quite useful to save the medium binary data like images, documents into Apache HBase. Unfortunately directly saving the binary MOB(medium object) to HBase leads to a worse performance since the frequent split and compaction. In this design, the MOB data are stored in an more efficient way, which keeps a high write/read performance and guarantees the data consistency in Apache HBase. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11610) Enhance remote meta updates
[ https://issues.apache.org/jira/browse/HBASE-11610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virag Kothari updated HBASE-11610: -- Attachment: HBASE-11610_2.patch Updated patch removes ThreadLocalHTable and addresses other comments. Also added a small change to avoid reflection call (in most cases) to instantiate RpcRetryingCallerFactory during creation of Htable instance. Enhance remote meta updates --- Key: HBASE-11610 URL: https://issues.apache.org/jira/browse/HBASE-11610 Project: HBase Issue Type: Sub-task Reporter: Jimmy Xiang Assignee: Virag Kothari Attachments: HBASE-11610.patch, HBASE-11610_2.patch Currently, if the meta region is on a regionserver instead of the master, meta update is synchronized on one HTable instance. We should be able to do better. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11610) Enhance remote meta updates
[ https://issues.apache.org/jira/browse/HBASE-11610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106597#comment-14106597 ] Virag Kothari commented on HBASE-11610: --- [~nkeywal] I will create a JIRA to further the discussion around undeprecation of processBatch(). Thanks for your input Enhance remote meta updates --- Key: HBASE-11610 URL: https://issues.apache.org/jira/browse/HBASE-11610 Project: HBase Issue Type: Sub-task Reporter: Jimmy Xiang Assignee: Virag Kothari Attachments: HBASE-11610.patch, HBASE-11610_2.patch Currently, if the meta region is on a regionserver instead of the master, meta update is synchronized on one HTable instance. We should be able to do better. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11794) StripeStoreFlusher causes NullPointerException and Region down
[ https://issues.apache.org/jira/browse/HBASE-11794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jeongmin kim updated HBASE-11794: - Attachment: HBASE_11794.patch StripeStoreFlusher causes NullPointerException and Region down -- Key: HBASE-11794 URL: https://issues.apache.org/jira/browse/HBASE-11794 Project: HBase Issue Type: Bug Components: Compaction Affects Versions: 0.98.3, 0.98.4, 0.98.5 Reporter: jeongmin kim Priority: Critical Attachments: HBASE_11794.patch StoreFlusher.flushSnapshot() mustn't return null value. But StripeStoreFlusher.flushSnapshot() does. It cause NullPointerException at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:802) and this makes regions dead after exhaustive retries and no recovery available from it. the code (StripeStoreFlusher:64) has to be changed === from ListPath result = null to ListPath result = new ArrayListPath(); === to return a empty list not null value. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11794) StripeStoreFlusher causes NullPointerException and Region down
[ https://issues.apache.org/jira/browse/HBASE-11794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106647#comment-14106647 ] jeongmin kim commented on HBASE-11794: -- HBASE_11794.patch file attached. null checking codes also removed as Jean-Marc Spaggiari recommended. flushCache seems to be the only caller of flushSnapshot() and this's stackTrace of the exception: --- java.lang.NullPointerException at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:802) at org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:1949) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1777) at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1659) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1118) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1084) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:147) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) --- thanx StripeStoreFlusher causes NullPointerException and Region down -- Key: HBASE-11794 URL: https://issues.apache.org/jira/browse/HBASE-11794 Project: HBase Issue Type: Bug Components: Compaction Affects Versions: 0.98.3, 0.98.4, 0.98.5 Reporter: jeongmin kim Priority: Critical Attachments: HBASE_11794.patch StoreFlusher.flushSnapshot() mustn't return null value. But StripeStoreFlusher.flushSnapshot() does. It cause NullPointerException at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:802) and this makes regions dead after exhaustive retries and no recovery available from it. the code (StripeStoreFlusher:64) has to be changed === from ListPath result = null to ListPath result = new ArrayListPath(); === to return a empty list not null value. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11803) Programming model for reverse scan is confusing
[ https://issues.apache.org/jira/browse/HBASE-11803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106657#comment-14106657 ] chunhui shen commented on HBASE-11803: -- Now, we have the InclusiveStopFilter to make the stop row inclusive. If add a new ExclusiveStartFilter, I think your problem could be fixed. Programming model for reverse scan is confusing --- Key: HBASE-11803 URL: https://issues.apache.org/jira/browse/HBASE-11803 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.1 Reporter: James Taylor The reverse scan is a very nice feature in HBase. We leverage it in Apache Phoenix 4.1 when possible and see a huge boost in performance over re-ordering the result set ourselves. However, the way in which you have to adjust the start/stop key is confusing. Our use case is that we have a scan that needs to be done and we've calculated an inclusive start row and an exclusive stop row. This is the way region boundaries are which is convenient as they can easily be intersected against the scan stop/start row. When we use a reverse scan, we are forced to switch the start and stop row values of the scan *and* adjust the byte values from inclusive to exclusive and from exclusive to inclusive. The former is not too bad, as you can just add a zero byte, but the latter is problematic. You can decrease the last byte by one, but you need to add an indeterminate 0xFF bytes to ensure you're not including a row unintentionally. IMHO, it would be much cleaner to just keep the start/stop row as is and just set call the Scan.setReversed(true) method. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11405) Multiple invocations of hbck in parallel disables balancer permanently
[
https://issues.apache.org/jira/browse/HBASE-11405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
bharath v updated HBASE-11405:
--
Attachment: HBASE-11405-trunk-rebased.patch
Rebased the patch to trunk.
Multiple invocations of hbck in parallel disables balancer permanently
---
Key: HBASE-11405
URL: https://issues.apache.org/jira/browse/HBASE-11405
Project: HBase
Issue Type: Bug
Components: Balancer, hbck
Affects Versions: 0.99.0
Reporter: bharath v
Assignee: bharath v
Attachments: HBASE-11405-trunk-rebased.patch,
HBASE-11405-trunk.patch, HBASE-11405-trunk.patch.1
This is because of the following piece of code in hbck
{code:borderStyle=solid}
boolean oldBalancer = admin.setBalancerRunning(false, true);
try {
onlineConsistencyRepair();
}
finally {
admin.setBalancerRunning(oldBalancer, false);
}
{code}
Newer invocations set oldBalancer to false as it was disabled by previous
invocations and this disables balancer permanently unless its manually turned
on by the user. Easy to reproduce, just run hbck 100 times in a loop in 2
different sessions and you can see that balancer is set to false in the
HMaster logs.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11794) StripeStoreFlusher causes NullPointerException and Region down
[ https://issues.apache.org/jira/browse/HBASE-11794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-11794: --- Status: Patch Available (was: Open) StripeStoreFlusher causes NullPointerException and Region down -- Key: HBASE-11794 URL: https://issues.apache.org/jira/browse/HBASE-11794 Project: HBase Issue Type: Bug Components: Compaction Affects Versions: 0.98.5, 0.98.4, 0.98.3 Reporter: jeongmin kim Priority: Critical Attachments: HBASE_11794.patch StoreFlusher.flushSnapshot() mustn't return null value. But StripeStoreFlusher.flushSnapshot() does. It cause NullPointerException at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:802) and this makes regions dead after exhaustive retries and no recovery available from it. the code (StripeStoreFlusher:64) has to be changed === from ListPath result = null to ListPath result = new ArrayListPath(); === to return a empty list not null value. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11326) Use an InputFormat for ExportSnapshot
[ https://issues.apache.org/jira/browse/HBASE-11326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-11326: Fix Version/s: 0.98.6 Use an InputFormat for ExportSnapshot - Key: HBASE-11326 URL: https://issues.apache.org/jira/browse/HBASE-11326 Project: HBase Issue Type: Improvement Components: snapshots Affects Versions: 0.99.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 0.99.0, 0.98.6 Attachments: HBASE-11326-v0.patch Use an InputFormat instead of uploading a set of input files to have a progress based on the file size -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11326) Use an InputFormat for ExportSnapshot
[ https://issues.apache.org/jira/browse/HBASE-11326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106674#comment-14106674 ] Matteo Bertozzi commented on HBASE-11326: - backported to 98 now that applies cleanly Use an InputFormat for ExportSnapshot - Key: HBASE-11326 URL: https://issues.apache.org/jira/browse/HBASE-11326 Project: HBase Issue Type: Improvement Components: snapshots Affects Versions: 0.99.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 0.99.0, 0.98.6 Attachments: HBASE-11326-v0.patch Use an InputFormat instead of uploading a set of input files to have a progress based on the file size -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11643) Read and write MOB in HBase
[ https://issues.apache.org/jira/browse/HBASE-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-11643: - Attachment: HBASE-11643-V9.diff Refine the code according to the comments in review board. And re-attach it to pass the Hadoop QA testing. Read and write MOB in HBase --- Key: HBASE-11643 URL: https://issues.apache.org/jira/browse/HBASE-11643 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Jingcheng Du Assignee: Jingcheng Du Attachments: HBASE-11643-V2.diff, HBASE-11643-V3.diff, HBASE-11643-V4.diff, HBASE-11643-V5.diff, HBASE-11643-V6.diff, HBASE-11643-V7.diff, HBASE-11643-V8.diff, HBASE-11643-V9.diff, HBase-11643.diff The read/write MOB in HBase are implemented in this JIRA. Normally, the Cells are saved in the MemStore, and flushed to the HFiles when necessary. For MOB, the Cells are saved in the MemStore as well, but they're flushed to elsewhere out of HBase in the format of HFiles. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-7767) Get rid of ZKTable, and table enable/disable state in ZK
[ https://issues.apache.org/jira/browse/HBASE-7767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Stepachev updated HBASE-7767: Status: Patch Available (was: Open) Get rid of ZKTable, and table enable/disable state in ZK - Key: HBASE-7767 URL: https://issues.apache.org/jira/browse/HBASE-7767 Project: HBase Issue Type: Sub-task Components: Zookeeper Affects Versions: 2.0.0 Reporter: Enis Soztutar Assignee: Andrey Stepachev Attachments: HBASE-7767.patch, HBASE-7767.patch As discussed table state in zookeeper for enable/disable state breaks our zookeeper contract. It is also very intrusive, used from the client side, master and region servers. We should get rid of it. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-7767) Get rid of ZKTable, and table enable/disable state in ZK
[ https://issues.apache.org/jira/browse/HBASE-7767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Stepachev updated HBASE-7767: Status: Open (was: Patch Available) Get rid of ZKTable, and table enable/disable state in ZK - Key: HBASE-7767 URL: https://issues.apache.org/jira/browse/HBASE-7767 Project: HBase Issue Type: Sub-task Components: Zookeeper Affects Versions: 2.0.0 Reporter: Enis Soztutar Assignee: Andrey Stepachev Attachments: HBASE-7767.patch, HBASE-7767.patch As discussed table state in zookeeper for enable/disable state breaks our zookeeper contract. It is also very intrusive, used from the client side, master and region servers. We should get rid of it. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11450) Improve file size info in SnapshotInfo tool
[
https://issues.apache.org/jira/browse/HBASE-11450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106757#comment-14106757
]
Hudson commented on HBASE-11450:
SUCCESS: Integrated in HBase-0.98 #464 (See
[https://builds.apache.org/job/HBase-0.98/464/])
HBASE-11450 Improve file size info in SnapshotInfo tool (addendum)
(matteo.bertozzi: rev 488afeb45cceaa51bc9302814c6661ec148b9d96)
* hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java
Improve file size info in SnapshotInfo tool
---
Key: HBASE-11450
URL: https://issues.apache.org/jira/browse/HBASE-11450
Project: HBase
Issue Type: Improvement
Components: snapshots
Affects Versions: 0.99.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
Fix For: 0.99.0, 0.98.4, 0.94.22, 2.0.0
Attachments: HBASE-11450-v0.patch, HBASE-11450-v1.patch
Add a -size-in-bytes flag to print the file size in byte instead of the
human readable format.
and add a check on the file length between the manifest and the hfile,
marking as CORRUPTED files with length that don't match.
{noformat}
Snapshot Files
4839b
testtb/a81219be11ade1d0d2913267caeeb3fe/cf/bec5567b2cb04cd1a76c2f4106991de7
-
testtb/f261b913c44a61a03550cb74e3a1/cf/8b02813a4f564957bd820c88fccf376a
(NOT FOUND)
4967b
testtb/0cab854a3877697e726a73187fe21643/cf/7afb8fe1e2f141eb9b8e17d1f68cd576
(archive)
12b
testtb/35074c28fd4dc304930f261fa8e0ce9c/cf/fedd9b9044b74768a6631003695c2f32
(CORRUPTED)
4839b
testtb/bb2ac9c8efc5ac9077084268c60dd8da/cf/50a3144088b049d98007af331821abd7
4839b
testtb/cda2a3ea0f5630d19018916cbe73264e/cf/da0a1008656c403cb17f37a061d04120
4905b
testtb/5b6e6b804e075778185e2fb1a27bae90/cf/1177a58d798a46ec952a0bc19f902711
(archive)
**
BAD SNAPSHOT: 1 hfile(s) and 0 log(s) missing.
1 hfile(s) corrupted.
**
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11326) Use an InputFormat for ExportSnapshot
[ https://issues.apache.org/jira/browse/HBASE-11326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106756#comment-14106756 ] Hudson commented on HBASE-11326: SUCCESS: Integrated in HBase-0.98 #464 (See [https://builds.apache.org/job/HBase-0.98/464/]) HBASE-11326 Use an InputFormat for ExportSnapshot (matteo.bertozzi: rev 333ea48ae2fef96a9d55ba5f920e4db2274614f7) * hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java Use an InputFormat for ExportSnapshot - Key: HBASE-11326 URL: https://issues.apache.org/jira/browse/HBASE-11326 Project: HBase Issue Type: Improvement Components: snapshots Affects Versions: 0.99.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 0.99.0, 0.98.6 Attachments: HBASE-11326-v0.patch Use an InputFormat instead of uploading a set of input files to have a progress based on the file size -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11326) Use an InputFormat for ExportSnapshot
[ https://issues.apache.org/jira/browse/HBASE-11326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106776#comment-14106776 ] Hudson commented on HBASE-11326: SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #437 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/437/]) HBASE-11326 Use an InputFormat for ExportSnapshot (matteo.bertozzi: rev 333ea48ae2fef96a9d55ba5f920e4db2274614f7) * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java * hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java Use an InputFormat for ExportSnapshot - Key: HBASE-11326 URL: https://issues.apache.org/jira/browse/HBASE-11326 Project: HBase Issue Type: Improvement Components: snapshots Affects Versions: 0.99.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 0.99.0, 0.98.6 Attachments: HBASE-11326-v0.patch Use an InputFormat instead of uploading a set of input files to have a progress based on the file size -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11450) Improve file size info in SnapshotInfo tool
[
https://issues.apache.org/jira/browse/HBASE-11450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106777#comment-14106777
]
Hudson commented on HBASE-11450:
SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #437 (See
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/437/])
HBASE-11450 Improve file size info in SnapshotInfo tool (addendum)
(matteo.bertozzi: rev 488afeb45cceaa51bc9302814c6661ec148b9d96)
* hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java
Improve file size info in SnapshotInfo tool
---
Key: HBASE-11450
URL: https://issues.apache.org/jira/browse/HBASE-11450
Project: HBase
Issue Type: Improvement
Components: snapshots
Affects Versions: 0.99.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
Fix For: 0.99.0, 0.98.4, 0.94.22, 2.0.0
Attachments: HBASE-11450-v0.patch, HBASE-11450-v1.patch
Add a -size-in-bytes flag to print the file size in byte instead of the
human readable format.
and add a check on the file length between the manifest and the hfile,
marking as CORRUPTED files with length that don't match.
{noformat}
Snapshot Files
4839b
testtb/a81219be11ade1d0d2913267caeeb3fe/cf/bec5567b2cb04cd1a76c2f4106991de7
-
testtb/f261b913c44a61a03550cb74e3a1/cf/8b02813a4f564957bd820c88fccf376a
(NOT FOUND)
4967b
testtb/0cab854a3877697e726a73187fe21643/cf/7afb8fe1e2f141eb9b8e17d1f68cd576
(archive)
12b
testtb/35074c28fd4dc304930f261fa8e0ce9c/cf/fedd9b9044b74768a6631003695c2f32
(CORRUPTED)
4839b
testtb/bb2ac9c8efc5ac9077084268c60dd8da/cf/50a3144088b049d98007af331821abd7
4839b
testtb/cda2a3ea0f5630d19018916cbe73264e/cf/da0a1008656c403cb17f37a061d04120
4905b
testtb/5b6e6b804e075778185e2fb1a27bae90/cf/1177a58d798a46ec952a0bc19f902711
(archive)
**
BAD SNAPSHOT: 1 hfile(s) and 0 log(s) missing.
1 hfile(s) corrupted.
**
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11794) StripeStoreFlusher causes NullPointerException and Region down
[
https://issues.apache.org/jira/browse/HBASE-11794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106794#comment-14106794
]
Jean-Marc Spaggiari commented on HBASE-11794:
-
What's about something like this?
{code}
@Override
public ListPath flushSnapshot(MemStoreSnapshot snapshot, long
cacheFlushSeqNum,
MonitoredTask status) throws IOException {
int cellsCount = snapshot.getCellsCount();
if (cellsCount == 0) return new ArrayListPath(); // don't flush if there
are no entries
long smallestReadPoint = store.getSmallestReadPoint();
InternalScanner scanner = createScanner(snapshot.getScanner(),
smallestReadPoint);
if (scanner == null) {
return new ArrayListPath();; // NULL scanner returned from coprocessor
hooks means skip normal processing
}
// Let policy select flush method.
StripeFlushRequest req = this.policy.selectFlush(this.stripes, cellsCount);
boolean success = false;
StripeMultiFileWriter mw = null;
ListPath result = null;
try {
mw = req.createWriter(); // Writer according to the policy.
StripeMultiFileWriter.WriterFactory factory = createWriterFactory(
snapshot.getTimeRangeTracker(), cellsCount);
StoreScanner storeScanner = (scanner instanceof StoreScanner) ?
(StoreScanner)scanner : null;
mw.init(storeScanner, factory, store.getComparator());
synchronized (flushLock) {
performFlush(scanner, mw, smallestReadPoint);
result = mw.commitWriters(cacheFlushSeqNum, false);
success = true;
}
} finally {
if (!success (mw != null)) {
if (result != null) {
result.clear();
}
for (Path leftoverFile : mw.abortWriters()) {
try {
store.getFileSystem().delete(leftoverFile, false);
} catch (Exception e) {
LOG.error(Failed to delete a file after failed flush: + e);
}
}
}
try {
scanner.close();
} catch (IOException ex) {
LOG.warn(Failed to close flush scanner, ignoring, ex);
}
}
return result;
}
{code}
Idea is, if we don't need to return the empty list, then we create an ArrayList
that we loose after when we assign result with mw.commitWriters. I don't know
how often this method is called but this will save one object creation.
StripeStoreFlusher causes NullPointerException and Region down
--
Key: HBASE-11794
URL: https://issues.apache.org/jira/browse/HBASE-11794
Project: HBase
Issue Type: Bug
Components: Compaction
Affects Versions: 0.98.3, 0.98.4, 0.98.5
Reporter: jeongmin kim
Priority: Critical
Attachments: HBASE_11794.patch
StoreFlusher.flushSnapshot() mustn't return null value.
But StripeStoreFlusher.flushSnapshot() does.
It cause NullPointerException at
org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:802)
and this makes regions dead after exhaustive retries and no recovery
available from it.
the code (StripeStoreFlusher:64) has to be changed
===
from
ListPath result = null
to
ListPath result = new ArrayListPath();
===
to return a empty list not null value.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11405) Multiple invocations of hbck in parallel disables balancer permanently
[
https://issues.apache.org/jira/browse/HBASE-11405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106869#comment-14106869
]
Hadoop QA commented on HBASE-11405:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12663629/HBASE-11405-trunk-rebased.patch
against trunk revision .
ATTACHMENT ID: 12663629
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10533//console
This message is automatically generated.
Multiple invocations of hbck in parallel disables balancer permanently
---
Key: HBASE-11405
URL: https://issues.apache.org/jira/browse/HBASE-11405
Project: HBase
Issue Type: Bug
Components: Balancer, hbck
Affects Versions: 0.99.0
Reporter: bharath v
Assignee: bharath v
Attachments: HBASE-11405-trunk-rebased.patch,
HBASE-11405-trunk.patch, HBASE-11405-trunk.patch.1
This is because of the following piece of code in hbck
{code:borderStyle=solid}
boolean oldBalancer = admin.setBalancerRunning(false, true);
try {
onlineConsistencyRepair();
}
finally {
admin.setBalancerRunning(oldBalancer, false);
}
{code}
Newer invocations set oldBalancer to false as it was disabled by previous
invocations and this disables balancer permanently unless its manually turned
on by the user. Easy to reproduce, just run hbck 100 times in a loop in 2
different sessions and you can see that balancer is set to false in the
HMaster logs.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11799) AsyncProcess can get locked when running PE
[
https://issues.apache.org/jira/browse/HBASE-11799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106945#comment-14106945
]
stack commented on HBASE-11799:
---
You want to look at this [~nkeywal]?
AsyncProcess can get locked when running PE
---
Key: HBASE-11799
URL: https://issues.apache.org/jira/browse/HBASE-11799
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.98.7
Reporter: Elliott Clark
Attachments: jstack
On a clean checkout of 0.98.
{code}
mvn clean package -DskipTests
bin/start-hbase.sh
bin/hbase pe --nomapred randomWrite 10 1000{code}
Everything runs fine for a while then the client hangs.
The region server thinks that all requests have completed successfully. The
PE process is all stuck on the final flush.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Assigned] (HBASE-11802) Scan copy constructor doesn't copy reversed member variable
[ https://issues.apache.org/jira/browse/HBASE-11802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan reassigned HBASE-11802: -- Assignee: ramkrishna.s.vasudevan Scan copy constructor doesn't copy reversed member variable --- Key: HBASE-11802 URL: https://issues.apache.org/jira/browse/HBASE-11802 Project: HBase Issue Type: Bug Affects Versions: 0.98.5 Reporter: James Taylor Assignee: ramkrishna.s.vasudevan Priority: Minor Fix For: 0.99.0, 2.0.0, 0.98.6 Attachments: HBASE-11802.patch The Scan copy constructor doesn't copy reversed member variable. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11803) Programming model for reverse scan is confusing
[ https://issues.apache.org/jira/browse/HBASE-11803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107060#comment-14107060 ] Nick Dimiduk commented on HBASE-11803: -- I think adding the ExclusiveStartFilter and configuring the scan to use them both when setReversed(true) would alleviate James's woes. Programming model for reverse scan is confusing --- Key: HBASE-11803 URL: https://issues.apache.org/jira/browse/HBASE-11803 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.1 Reporter: James Taylor The reverse scan is a very nice feature in HBase. We leverage it in Apache Phoenix 4.1 when possible and see a huge boost in performance over re-ordering the result set ourselves. However, the way in which you have to adjust the start/stop key is confusing. Our use case is that we have a scan that needs to be done and we've calculated an inclusive start row and an exclusive stop row. This is the way region boundaries are which is convenient as they can easily be intersected against the scan stop/start row. When we use a reverse scan, we are forced to switch the start and stop row values of the scan *and* adjust the byte values from inclusive to exclusive and from exclusive to inclusive. The former is not too bad, as you can just add a zero byte, but the latter is problematic. You can decrease the last byte by one, but you need to add an indeterminate 0xFF bytes to ensure you're not including a row unintentionally. IMHO, it would be much cleaner to just keep the start/stop row as is and just set call the Scan.setReversed(true) method. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11802) Scan copy constructor doesn't copy reversed member variable
[ https://issues.apache.org/jira/browse/HBASE-11802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-11802: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master, branch-1 and 0.98. Thanks for the review. Scan copy constructor doesn't copy reversed member variable --- Key: HBASE-11802 URL: https://issues.apache.org/jira/browse/HBASE-11802 Project: HBase Issue Type: Bug Affects Versions: 0.98.5 Reporter: James Taylor Assignee: ramkrishna.s.vasudevan Priority: Minor Fix For: 0.99.0, 2.0.0, 0.98.6 Attachments: HBASE-11802.patch The Scan copy constructor doesn't copy reversed member variable. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11779) Document the new requirement to set JAVA_HOME before starting HBase
[ https://issues.apache.org/jira/browse/HBASE-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107072#comment-14107072 ] Nick Dimiduk commented on HBASE-11779: -- Huh. I didn't know this JAVA_HOME thing was a requirement :) +1, I'll get this committed. As for refreshing site, I can't actually build the site. Upgrading my java version is on the list, but I don't think i'll get it done today. Document the new requirement to set JAVA_HOME before starting HBase --- Key: HBASE-11779 URL: https://issues.apache.org/jira/browse/HBASE-11779 Project: HBase Issue Type: Sub-task Components: documentation Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: HBASE-11534.patch, HBASE-11779.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11779) Document the new requirement to set JAVA_HOME before starting HBase
[ https://issues.apache.org/jira/browse/HBASE-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107073#comment-14107073 ] Nick Dimiduk commented on HBASE-11779: -- Rather, mvn site works, but javadoc:aggregate does not. Document the new requirement to set JAVA_HOME before starting HBase --- Key: HBASE-11779 URL: https://issues.apache.org/jira/browse/HBASE-11779 Project: HBase Issue Type: Sub-task Components: documentation Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: HBASE-11534.patch, HBASE-11779.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11738) Document improvements to LoadTestTool and PerformanceEvaluation
[ https://issues.apache.org/jira/browse/HBASE-11738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107077#comment-14107077 ] Nick Dimiduk commented on HBASE-11738: -- Yes, appendix for our various tools would be very helpful. Better still if it could be auto-generated from the actual help output the tools produce, so the docs are never out of sync with the tools themselves. Document improvements to LoadTestTool and PerformanceEvaluation --- Key: HBASE-11738 URL: https://issues.apache.org/jira/browse/HBASE-11738 Project: HBase Issue Type: Sub-task Components: documentation Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Priority: Minor Fix For: 0.99.0, 0.98.4 Attachments: HBABASE-11738.patch, HBASE-11738.patch, HBASE-11738.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11803) Programming model for reverse scan is confusing
[ https://issues.apache.org/jira/browse/HBASE-11803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107108#comment-14107108 ] James Taylor commented on HBASE-11803: -- Thanks for the workaround - sounds like it might be a bit more expensive to have a couple of extra filters rather than calculating the row key up front. Am I the only one who thinks this is more complicated than it needs to be? That's really my main point. Programming model for reverse scan is confusing --- Key: HBASE-11803 URL: https://issues.apache.org/jira/browse/HBASE-11803 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.1 Reporter: James Taylor The reverse scan is a very nice feature in HBase. We leverage it in Apache Phoenix 4.1 when possible and see a huge boost in performance over re-ordering the result set ourselves. However, the way in which you have to adjust the start/stop key is confusing. Our use case is that we have a scan that needs to be done and we've calculated an inclusive start row and an exclusive stop row. This is the way region boundaries are which is convenient as they can easily be intersected against the scan stop/start row. When we use a reverse scan, we are forced to switch the start and stop row values of the scan *and* adjust the byte values from inclusive to exclusive and from exclusive to inclusive. The former is not too bad, as you can just add a zero byte, but the latter is problematic. You can decrease the last byte by one, but you need to add an indeterminate 0xFF bytes to ensure you're not including a row unintentionally. IMHO, it would be much cleaner to just keep the start/stop row as is and just set call the Scan.setReversed(true) method. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11788) hbase is not deleting the cell when a Put with a KeyValue, KeyValue.Type.Delete is submitted
[
https://issues.apache.org/jira/browse/HBASE-11788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107135#comment-14107135
]
Cristian Armaselu commented on HBASE-11788:
---
checkAndMutate would be nice, but we also use hive to insert data. For that we
apply a coprocessor (prePut) to turn the empty values inserted from hive into
cell deletes (DeleteColumn). Since hive generates Puts and not Mutations, we
will have still have the same issue. The old behavior was working really well.
Was it changed on purpose?
hbase is not deleting the cell when a Put with a KeyValue,
KeyValue.Type.Delete is submitted
Key: HBASE-11788
URL: https://issues.apache.org/jira/browse/HBASE-11788
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0, 0.96.1.1, 0.98.5, 2.0.0
Environment: Cloudera CDH 5.1.x
Reporter: Cristian Armaselu
Assignee: Srikanth Srungarapu
Attachments: TestPutWithDelete.java
Code executed:
{code}
@Test
public void testHbasePutDeleteCell() throws Exception {
TableName tableName = TableName.valueOf(my_test);
Configuration configuration = HBaseConfiguration.create();
HTableInterface table = new HTable(configuration, tableName);
final String rowKey = 12345;
final byte[] familly = Bytes.toBytes(default);
// put one row
Put put = new Put(Bytes.toBytes(rowKey));
put.add(familly, Bytes.toBytes(A), Bytes.toBytes(a));
put.add(familly, Bytes.toBytes(B), Bytes.toBytes(b));
put.add(familly, Bytes.toBytes(C), Bytes.toBytes(c));
table.put(put);
// get row back and assert the values
Get get = new Get(Bytes.toBytes(rowKey));
Result result = table.get(get);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(A))).equals(a), Column A value should be a);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(B))).equals(b), Column B value should be b);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(C))).equals(c), Column C value should be c);
// put the same row again with C column deleted
put = new Put(Bytes.toBytes(rowKey));
put.add(familly, Bytes.toBytes(A), Bytes.toBytes(a));
put.add(familly, Bytes.toBytes(B), Bytes.toBytes(b));
put.add(new KeyValue(Bytes.toBytes(rowKey), familly,
Bytes.toBytes(C), HConstants.LATEST_TIMESTAMP, KeyValue.Type.DeleteColumn));
table.put(put);
// get row back and assert the values
get = new Get(Bytes.toBytes(rowKey));
result = table.get(get);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(A))).equals(a), Column A value should be a);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(B))).equals(b), Column A value should be b);
Assert.isTrue(result.getValue(familly, Bytes.toBytes(C)) == null,
Column C should not exists);
}
{code}
This assertion fails, the cell is not deleted but rather the value is empty:
{code}
hbase(main):029:0 scan 'my_test'
ROW COLUMN+CELL
12345column=default:A,
timestamp=1408473082290, value=a
12345column=default:B,
timestamp=1408473082290, value=b
12345column=default:C,
timestamp=1408473082290, value=
{code}
This behavior is different than previous 4.8.x Cloudera version and is
currently corrupting all hive queries involving is null or is not null
operators on the columns mapped to hbase
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11779) Document the new requirement to set JAVA_HOME before starting HBase
[ https://issues.apache.org/jira/browse/HBASE-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-11779: - Resolution: Fixed Fix Version/s: 2.0.0 Status: Resolved (was: Patch Available) Committed to master. Thanks for another fix [~misty]! Document the new requirement to set JAVA_HOME before starting HBase --- Key: HBASE-11779 URL: https://issues.apache.org/jira/browse/HBASE-11779 Project: HBase Issue Type: Sub-task Components: documentation Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Fix For: 2.0.0 Attachments: HBASE-11534.patch, HBASE-11779.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11804) Raise default heap size if unspecified
Elliott Clark created HBASE-11804: - Summary: Raise default heap size if unspecified Key: HBASE-11804 URL: https://issues.apache.org/jira/browse/HBASE-11804 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 2.0.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0 On master running pe randomWrite with ten threads and the default heap size crashes the system. This works on 0.98 and before. It's been a long time that 1000mb has been our default. Maybe we should start looking at raising that limit on master. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11804) Raise default heap size if unspecified
[ https://issues.apache.org/jira/browse/HBASE-11804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107202#comment-14107202 ] Nick Dimiduk commented on HBASE-11804: -- I experience this as well. Looks like long GC pauses. I only have 4g of ram on my usual dev machine, so I'd prefer to track down where all the extra garbage is coming from. Raise default heap size if unspecified -- Key: HBASE-11804 URL: https://issues.apache.org/jira/browse/HBASE-11804 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 2.0.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0 On master running pe randomWrite with ten threads and the default heap size crashes the system. This works on 0.98 and before. It's been a long time that 1000mb has been our default. Maybe we should start looking at raising that limit on master. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11800) Coprocessor service methods in HTableInterface should be annotated public
[
https://issues.apache.org/jira/browse/HBASE-11800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107217#comment-14107217
]
Gary Helmling commented on HBASE-11800:
---
Committed to master.
[~enis] What do you think about including this in branch-1 for 1.0? I think it
would be good to get it in and clarify the API for 1.0.
Coprocessor service methods in HTableInterface should be annotated public
-
Key: HBASE-11800
URL: https://issues.apache.org/jira/browse/HBASE-11800
Project: HBase
Issue Type: Task
Components: Client
Affects Versions: 0.98.0, 0.96.0
Reporter: Gary Helmling
Assignee: Gary Helmling
Attachments: HBASE-11800.patch, HBASE-11800_2.patch
The {{HTableInterface.coprocessorService(...)}} and
{{HTableInterface.batchCoprocessorService(...)}} methods were made private in
HBASE-9529, when the coprocessor APIs were seen as unstable and evolving.
However, these methods represent a standard way for clients to use custom
APIs exposed via coprocessors. In that sense, they are targeted at general
HBase users (who may run but not develop coprocessors), as opposed to
coprocessor developers who want to extend HBase.
The coprocessor endpoint API has also remained much more stable than the
coprocessor Observer interfaces, which tend to change along with HBase
internals. So there should not be much difficulty in supporting these
methods as part of the public API.
I think we should drop the {{@InterfaceAudience.Private}} annotation on these
methods and support them as part of the public {{HTableInterface}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11794) StripeStoreFlusher causes NullPointerException and Region down
[
https://issues.apache.org/jira/browse/HBASE-11794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107223#comment-14107223
]
Sean Busbey commented on HBASE-11794:
-
Looks good [~eomiks]. Would you mind adding a test?
Since {{result}} doesn't get passed to anyone, you can also avoid the extra
object creation by initializing it to {{Collections.emptyList()}}.
StripeStoreFlusher causes NullPointerException and Region down
--
Key: HBASE-11794
URL: https://issues.apache.org/jira/browse/HBASE-11794
Project: HBase
Issue Type: Bug
Components: Compaction
Affects Versions: 0.98.3, 0.98.4, 0.98.5
Reporter: jeongmin kim
Priority: Critical
Attachments: HBASE_11794.patch
StoreFlusher.flushSnapshot() mustn't return null value.
But StripeStoreFlusher.flushSnapshot() does.
It cause NullPointerException at
org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:802)
and this makes regions dead after exhaustive retries and no recovery
available from it.
the code (StripeStoreFlusher:64) has to be changed
===
from
ListPath result = null
to
ListPath result = new ArrayListPath();
===
to return a empty list not null value.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11802) Scan copy constructor doesn't copy reversed member variable
[ https://issues.apache.org/jira/browse/HBASE-11802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107235#comment-14107235 ] Hudson commented on HBASE-11802: SUCCESS: Integrated in HBase-TRUNK #5420 (See [https://builds.apache.org/job/HBase-TRUNK/5420/]) HBASE-11802 Scan copy constructor doesn't copy reversed member variable (ramkrishna: rev 6f00f859a75d617924158b4b40aaa8618dd1a6ae) * hbase-client/src/main/java/org/apache/hadoop/hbase/client/Scan.java Scan copy constructor doesn't copy reversed member variable --- Key: HBASE-11802 URL: https://issues.apache.org/jira/browse/HBASE-11802 Project: HBase Issue Type: Bug Affects Versions: 0.98.5 Reporter: James Taylor Assignee: ramkrishna.s.vasudevan Priority: Minor Fix For: 0.99.0, 2.0.0, 0.98.6 Attachments: HBASE-11802.patch The Scan copy constructor doesn't copy reversed member variable. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11326) Use an InputFormat for ExportSnapshot
[ https://issues.apache.org/jira/browse/HBASE-11326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107246#comment-14107246 ] Jerry He commented on HBASE-11326: -- Hi, [~mbertozzi] Thanks for back-porting it to 0.98. Use an InputFormat for ExportSnapshot - Key: HBASE-11326 URL: https://issues.apache.org/jira/browse/HBASE-11326 Project: HBase Issue Type: Improvement Components: snapshots Affects Versions: 0.99.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 0.99.0, 0.98.6 Attachments: HBASE-11326-v0.patch Use an InputFormat instead of uploading a set of input files to have a progress based on the file size -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11805) KeyValue to Cell Convert in WALEdit APIs
Anoop Sam John created HBASE-11805: -- Summary: KeyValue to Cell Convert in WALEdit APIs Key: HBASE-11805 URL: https://issues.apache.org/jira/browse/HBASE-11805 Project: HBase Issue Type: Improvement Components: wal Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0, 2.0.0, 0.98.6 In almost all other main interface class/APIs we have changed KeyValue to Cell. But missing in WALEdit. This is public marked for Replication (Well it should be for CP also) These 2 APIs deal with KVs add(KeyValue kv) ArrayListKeyValue getKeyValues() Suggest deprecate them and add for 0.98 add(Cell kv) ListCell getCells() And just replace from 1.0 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11804) Raise default heap size if unspecified
[ https://issues.apache.org/jira/browse/HBASE-11804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107253#comment-14107253 ] Elliott Clark commented on HBASE-11804: --- Tracking down the extra garbage is a fine goal as well. I don't think they have to mutually exclusive. 1gig at the time HBase started was a pretty decent amount of heap. Now most boxes that really run HBase are running with 10 gig heaps and 90 gigs or ram. It's hard to really call our default something that's even close to what should be a good value. Everytime we have a default that users have to tweak before they can really play well with HBase we are creating a hurdle to adoption. If users like [~ndimiduk] want to run on a machine with less ram then they just have to have HBASE_HEAPSIZE in their env. So to me that seems like more users will have a better into experience at the cost of more experienced users with much older hardware. Raise default heap size if unspecified -- Key: HBASE-11804 URL: https://issues.apache.org/jira/browse/HBASE-11804 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 2.0.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0 On master running pe randomWrite with ten threads and the default heap size crashes the system. This works on 0.98 and before. It's been a long time that 1000mb has been our default. Maybe we should start looking at raising that limit on master. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-8541) implement flush-into-stripes in stripe compactions
[ https://issues.apache.org/jira/browse/HBASE-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-8541: --- Fix Version/s: (was: 0.98.3) (was: 1.0.0) 0.98.0 0.99.0 implement flush-into-stripes in stripe compactions -- Key: HBASE-8541 URL: https://issues.apache.org/jira/browse/HBASE-8541 Project: HBase Issue Type: Sub-task Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.98.0, 0.99.0, 2.0.0 Attachments: HBASE-8541-latest-with-dependencies.patch, HBASE-8541-latest-with-dependencies.patch, HBASE-8541-latest-with-dependencies.patch, HBASE-8541-latest-with-dependencies.patch, HBASE-8541-v0.patch, HBASE-8541-v1.patch, HBASE-8541-v2.patch, HBASE-8541-v3.patch, HBASE-8541-v4.patch, HBASE-8541-v5.patch Flush will be able to flush into multiple files under this design, avoiding L0 I/O amplification. I have the patch which is missing just one feature - support for concurrent flushes and stripe changes. This can be done via extensive try-locking of stripe changes and flushes, or advisory flags without blocking flushes, dumping conflicting flushes into L0 in case of (very rare) collisions. For file loading for the latter, a set-cover-like problem needs to be solved to determine optimal stripes. That will also address Jimmy's concern of getting rid of metadata, btw. However currently I don't have time for that. I plan to attach the try-locking patch first, but this won't happen for a couple weeks probably and should not block main reviews. Hopefully this will be added on top of main reviews. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-8298) desc tablename shorthand for describe tablename, similar to how databases have
[ https://issues.apache.org/jira/browse/HBASE-8298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107272#comment-14107272 ] Sean Busbey commented on HBASE-8298: QA build failure looks like a jenkins issue unrelated to the patch. desc tablename shorthand for describe tablename, similar to how databases have -- Key: HBASE-8298 URL: https://issues.apache.org/jira/browse/HBASE-8298 Project: HBase Issue Type: Improvement Components: shell Affects Versions: 0.94.2 Reporter: Hari Sekhon Assignee: Sean Busbey Priority: Trivial Attachments: HBASE-8298-v3.patch, desc.patch, desc2.patch It would be nice if you could type desc tablename in hbase shell as shorthand for describe tablename similar to how you can in traditional databases. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11804) Raise default heap size if unspecified
[ https://issues.apache.org/jira/browse/HBASE-11804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107283#comment-14107283 ] Nick Dimiduk commented on HBASE-11804: -- Fair enough. Under this reasoning, I would recommend 16g as the new default. Really though, this is a value that everyone changes in production because everyone's environment is different. Having a decent default for development purposes seems more reasonable to me. Raise default heap size if unspecified -- Key: HBASE-11804 URL: https://issues.apache.org/jira/browse/HBASE-11804 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 2.0.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0 On master running pe randomWrite with ten threads and the default heap size crashes the system. This works on 0.98 and before. It's been a long time that 1000mb has been our default. Maybe we should start looking at raising that limit on master. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11802) Scan copy constructor doesn't copy reversed member variable
[ https://issues.apache.org/jira/browse/HBASE-11802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107281#comment-14107281 ] Hudson commented on HBASE-11802: FAILURE: Integrated in HBase-1.0 #118 (See [https://builds.apache.org/job/HBase-1.0/118/]) HBASE-11802 Scan copy constructor doesn't copy reversed member variable (ramkrishna: rev 6aba7cf40ee57e13dec68773bb5565183332b448) * hbase-client/src/main/java/org/apache/hadoop/hbase/client/Scan.java Scan copy constructor doesn't copy reversed member variable --- Key: HBASE-11802 URL: https://issues.apache.org/jira/browse/HBASE-11802 Project: HBase Issue Type: Bug Affects Versions: 0.98.5 Reporter: James Taylor Assignee: ramkrishna.s.vasudevan Priority: Minor Fix For: 0.99.0, 2.0.0, 0.98.6 Attachments: HBASE-11802.patch The Scan copy constructor doesn't copy reversed member variable. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11794) StripeStoreFlusher causes NullPointerException and Region down
[ https://issues.apache.org/jira/browse/HBASE-11794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107298#comment-14107298 ] Enis Soztutar commented on HBASE-11794: --- [~sershe] FYI. StripeStoreFlusher causes NullPointerException and Region down -- Key: HBASE-11794 URL: https://issues.apache.org/jira/browse/HBASE-11794 Project: HBase Issue Type: Bug Components: Compaction Affects Versions: 0.98.3, 0.98.4, 0.98.5 Reporter: jeongmin kim Priority: Critical Attachments: HBASE_11794.patch StoreFlusher.flushSnapshot() mustn't return null value. But StripeStoreFlusher.flushSnapshot() does. It cause NullPointerException at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:802) and this makes regions dead after exhaustive retries and no recovery available from it. the code (StripeStoreFlusher:64) has to be changed === from ListPath result = null to ListPath result = new ArrayListPath(); === to return a empty list not null value. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11794) StripeStoreFlusher causes NullPointerException and Region down
[
https://issues.apache.org/jira/browse/HBASE-11794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107305#comment-14107305
]
Jean-Marc Spaggiari commented on HBASE-11794:
-
+1 for {{Collections.emptyList();}}
StripeStoreFlusher causes NullPointerException and Region down
--
Key: HBASE-11794
URL: https://issues.apache.org/jira/browse/HBASE-11794
Project: HBase
Issue Type: Bug
Components: Compaction
Affects Versions: 0.98.3, 0.98.4, 0.98.5
Reporter: jeongmin kim
Priority: Critical
Attachments: HBASE_11794.patch
StoreFlusher.flushSnapshot() mustn't return null value.
But StripeStoreFlusher.flushSnapshot() does.
It cause NullPointerException at
org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:802)
and this makes regions dead after exhaustive retries and no recovery
available from it.
the code (StripeStoreFlusher:64) has to be changed
===
from
ListPath result = null
to
ListPath result = new ArrayListPath();
===
to return a empty list not null value.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11804) Raise default heap size if unspecified
[ https://issues.apache.org/jira/browse/HBASE-11804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107309#comment-14107309 ] Jean-Marc Spaggiari commented on HBASE-11804: - Wow. I found that pretty high. Many newcomers try HBase locally and might not have this amount of memory. Then they will see it failing and that might create issues. I will more recommend something like 4GB max. Anyway on all production clusters ops will adjust based on the available memory... My 2¢... Raise default heap size if unspecified -- Key: HBASE-11804 URL: https://issues.apache.org/jira/browse/HBASE-11804 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 2.0.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0 On master running pe randomWrite with ten threads and the default heap size crashes the system. This works on 0.98 and before. It's been a long time that 1000mb has been our default. Maybe we should start looking at raising that limit on master. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11802) Scan copy constructor doesn't copy reversed member variable
[ https://issues.apache.org/jira/browse/HBASE-11802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107317#comment-14107317 ] Hudson commented on HBASE-11802: SUCCESS: Integrated in HBase-0.98 #465 (See [https://builds.apache.org/job/HBase-0.98/465/]) HBASE-11802 - Scan copy constructor doesn't copy reversed member variable (ramkrishna: rev b21dcbf3216b69c192b499a090ee71552226a3f8) * hbase-client/src/main/java/org/apache/hadoop/hbase/client/Scan.java Scan copy constructor doesn't copy reversed member variable --- Key: HBASE-11802 URL: https://issues.apache.org/jira/browse/HBASE-11802 Project: HBase Issue Type: Bug Affects Versions: 0.98.5 Reporter: James Taylor Assignee: ramkrishna.s.vasudevan Priority: Minor Fix For: 0.99.0, 2.0.0, 0.98.6 Attachments: HBASE-11802.patch The Scan copy constructor doesn't copy reversed member variable. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HBASE-11721) jdiff script no longer works as usage instructions indicate
[ https://issues.apache.org/jira/browse/HBASE-11721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dima Spivak reassigned HBASE-11721: --- Assignee: Dima Spivak jdiff script no longer works as usage instructions indicate --- Key: HBASE-11721 URL: https://issues.apache.org/jira/browse/HBASE-11721 Project: HBase Issue Type: Bug Components: scripts Reporter: Misty Stanley-Jones Assignee: Dima Spivak I pasted the command from the usage instructions embedded in the script, but it fails as follows: [misty@cheezel dev-support](master)$ bash ./jdiffHBasePublicAPI.sh https://github.com/apache/hbase.git 0.94 https://github.com/MY_REPO/hbase.git 0.94 JDiff evaluation beginning: Determining if this is a local directory or a git repo. Looks like https://github.com/apache/hbase.git is a git repo Determining if this is a local directory or a git repo. Looks like https://github.com/MY_REPO/hbase.git is a git repo We are going to compare source 1 which is a git_repo and source 2, which is a git_repo 0.94 0.94 JDIFF_WORKING_DIRECTORY not set. That's not an issue. We will default it to /tmp/jdiff. % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 183 100 1830 0447 0 --:--:-- --:--:-- --:--:-- 448 Archive: jdiff-1.1.1-with-incompatible-option.zip End-of-central-directory signature not found. Either this file is not a zipfile, or it constitutes one disk of a multi-part archive. In the latter case the central directory and zipfile comment will be found on the last disk(s) of this archive. unzip: cannot find zipfile directory in one of jdiff-1.1.1-with-incompatible-option.zip or jdiff-1.1.1-with-incompatible-option.zip.zip, and cannot find jdiff-1.1.1-with-incompatible-option.zip.ZIP, period. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11804) Raise default heap size if unspecified
[ https://issues.apache.org/jira/browse/HBASE-11804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107322#comment-14107322 ] Elliott Clark commented on HBASE-11804: --- I agree and I would think that 1g's lower than most will have for testing something that usually occupies a full system. I was thinking something along the lines of 2g. Most laptops have 4 to 32 gigs. I don't think that taking 1/2 of that as a default for testing is too onerous. Raise default heap size if unspecified -- Key: HBASE-11804 URL: https://issues.apache.org/jira/browse/HBASE-11804 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 2.0.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0 On master running pe randomWrite with ten threads and the default heap size crashes the system. This works on 0.98 and before. It's been a long time that 1000mb has been our default. Maybe we should start looking at raising that limit on master. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11800) Coprocessor service methods in HTableInterface should be annotated public
[
https://issues.apache.org/jira/browse/HBASE-11800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107358#comment-14107358
]
Hudson commented on HBASE-11800:
SUCCESS: Integrated in HBase-TRUNK #5421 (See
[https://builds.apache.org/job/HBase-TRUNK/5421/])
HBASE-11800 Make HTableInterface coprocessorService methods public (garyh: rev
db520b94cbf961408e606040763da8b45616c70f)
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/Batch.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/CoprocessorRpcChannel.java
Coprocessor service methods in HTableInterface should be annotated public
-
Key: HBASE-11800
URL: https://issues.apache.org/jira/browse/HBASE-11800
Project: HBase
Issue Type: Task
Components: Client
Affects Versions: 0.98.0, 0.96.0
Reporter: Gary Helmling
Assignee: Gary Helmling
Attachments: HBASE-11800.patch, HBASE-11800_2.patch
The {{HTableInterface.coprocessorService(...)}} and
{{HTableInterface.batchCoprocessorService(...)}} methods were made private in
HBASE-9529, when the coprocessor APIs were seen as unstable and evolving.
However, these methods represent a standard way for clients to use custom
APIs exposed via coprocessors. In that sense, they are targeted at general
HBase users (who may run but not develop coprocessors), as opposed to
coprocessor developers who want to extend HBase.
The coprocessor endpoint API has also remained much more stable than the
coprocessor Observer interfaces, which tend to change along with HBase
internals. So there should not be much difficulty in supporting these
methods as part of the public API.
I think we should drop the {{@InterfaceAudience.Private}} annotation on these
methods and support them as part of the public {{HTableInterface}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11779) Document the new requirement to set JAVA_HOME before starting HBase
[ https://issues.apache.org/jira/browse/HBASE-11779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107359#comment-14107359 ] Hudson commented on HBASE-11779: SUCCESS: Integrated in HBase-TRUNK #5421 (See [https://builds.apache.org/job/HBase-TRUNK/5421/]) HBASE-11779 Document the new requirement to set JAVA_HOME before starting HBase (Misty Stanley-Jones) (ndimiduk: rev a2fc3efebfb277d2e57712a4c5b210a01fd7d5c8) * src/main/docbkx/getting_started.xml * src/main/docbkx/configuration.xml Document the new requirement to set JAVA_HOME before starting HBase --- Key: HBASE-11779 URL: https://issues.apache.org/jira/browse/HBASE-11779 Project: HBase Issue Type: Sub-task Components: documentation Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Fix For: 2.0.0 Attachments: HBASE-11534.patch, HBASE-11779.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11802) Scan copy constructor doesn't copy reversed member variable
[ https://issues.apache.org/jira/browse/HBASE-11802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107366#comment-14107366 ] Hudson commented on HBASE-11802: SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #438 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/438/]) HBASE-11802 - Scan copy constructor doesn't copy reversed member variable (ramkrishna: rev b21dcbf3216b69c192b499a090ee71552226a3f8) * hbase-client/src/main/java/org/apache/hadoop/hbase/client/Scan.java Scan copy constructor doesn't copy reversed member variable --- Key: HBASE-11802 URL: https://issues.apache.org/jira/browse/HBASE-11802 Project: HBase Issue Type: Bug Affects Versions: 0.98.5 Reporter: James Taylor Assignee: ramkrishna.s.vasudevan Priority: Minor Fix For: 0.99.0, 2.0.0, 0.98.6 Attachments: HBASE-11802.patch The Scan copy constructor doesn't copy reversed member variable. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11331) [blockcache] lazy block decompression
[
https://issues.apache.org/jira/browse/HBASE-11331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107368#comment-14107368
]
Nick Dimiduk commented on HBASE-11331:
--
As best as I can tell, both of these configurations stay entirely in
BlockCache. Verified by looking at the RS BlockCache stats and confirmed by the
low iowait stat being basically flat for them both. Looks like enabling this
feature when it's not needed is quite expensive.
|| ||=false, 11g||=true, 40g||delta|
|hbase.regionserver.server.Get_num_ops|15.15 k|6.07 k|{color:red}-60%{color}|
|hbase.regionserver.server.Get_mean|0.00 ns| 0.00 ns|{color:green}0%{color}|
|hbase.regionserver.server.Get_99th_percentile|1.00 ms|22.65
ms|{color:red}2165%{color}|
|hbase.regionserver.jvmmetrics.GcTimeMillis|48.89 ms|441.33
ms|{color:red}802%{color}|
|proc.loadavg.1min|0.56|3.25|{color:red}480%{color}|
|proc.stat.cpu.percpu{type=iowait}|3.55|3.47|{color:green}-2%{color}|
|hbase.regionserver.server.blockCacheCount|181.75 k|666.44
k|{color:green}266%{color}|
[blockcache] lazy block decompression
-
Key: HBASE-11331
URL: https://issues.apache.org/jira/browse/HBASE-11331
Project: HBase
Issue Type: Improvement
Components: regionserver
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
Attachments: HBASE-11331.00.patch, HBASE-11331.01.patch,
HBASE-11331.02.patch, HBASE-11331.03.patch, HBASE-11331.04.patch,
HBASE-11331.05.patch, HBASE-11331LazyBlockDecompressperfcompare.pdf,
lazy-decompress.02.0.pdf, lazy-decompress.02.1.json,
lazy-decompress.02.1.pdf, v03-20g-045g-false.pdf, v03-20g-045g-true-16h.pdf,
v03-20g-045g-true.pdf
Maintaining data in its compressed form in the block cache will greatly
increase our effective blockcache size and should show a meaning improvement
in cache hit rates in well designed applications. The idea here is to lazily
decompress/decrypt blocks when they're consumed, rather than as soon as
they're pulled off of disk.
This is related to but less invasive than HBASE-8894.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11738) Document improvements to LoadTestTool and PerformanceEvaluation
[
https://issues.apache.org/jira/browse/HBASE-11738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107427#comment-14107427
]
Hadoop QA commented on HBASE-11738:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12663569/HBASE-11738.patch
against trunk revision .
ATTACHMENT ID: 12663569
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 7
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 2 zombie test(s):
at
org.apache.hadoop.hbase.util.TestDrainBarrier.testStopIsBlockedByOps(TestDrainBarrier.java:95)
at
org.apache.camel.test.junit4.CamelTestSupport.doStopCamelContext(CamelTestSupport.java:450)
at
org.apache.camel.test.junit4.CamelTestSupport.tearDown(CamelTestSupport.java:351)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/10539//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10539//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10539//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10539//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10539//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10539//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10539//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10539//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10539//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10539//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10539//console
This message is automatically generated.
Document improvements to LoadTestTool and PerformanceEvaluation
---
Key: HBASE-11738
URL: https://issues.apache.org/jira/browse/HBASE-11738
Project: HBase
Issue Type: Sub-task
Components: documentation
Reporter: Misty Stanley-Jones
Assignee: Misty Stanley-Jones
Priority: Minor
Fix For: 0.99.0, 0.98.4
Attachments: HBABASE-11738.patch, HBASE-11738.patch, HBASE-11738.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11794) StripeStoreFlusher causes NullPointerException and Region down
[
https://issues.apache.org/jira/browse/HBASE-11794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107426#comment-14107426
]
Hadoop QA commented on HBASE-11794:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12663625/HBASE_11794.patch
against trunk revision .
ATTACHMENT ID: 12663625
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 7
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testStarted(ResourceCheckerJUnitListener.java:179)
at
org.apache.camel.test.junit4.CamelTestSupport.doStopCamelContext(CamelTestSupport.java:450)
at
org.apache.camel.test.junit4.CamelTestSupport.tearDown(CamelTestSupport.java:351)
at
org.apache.camel.test.spring.CamelSpringTestSupport.tearDown(CamelSpringTestSupport.java:115)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/10538//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10538//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10538//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10538//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10538//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10538//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10538//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10538//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10538//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10538//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10538//console
This message is automatically generated.
StripeStoreFlusher causes NullPointerException and Region down
--
Key: HBASE-11794
URL: https://issues.apache.org/jira/browse/HBASE-11794
Project: HBase
Issue Type: Bug
Components: Compaction
Affects Versions: 0.98.3, 0.98.4, 0.98.5
Reporter: jeongmin kim
Priority: Critical
Attachments: HBASE_11794.patch
StoreFlusher.flushSnapshot() mustn't return null value.
But StripeStoreFlusher.flushSnapshot() does.
It cause NullPointerException at
org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:802)
and this makes regions dead after exhaustive retries and no recovery
available from it.
the code (StripeStoreFlusher:64) has to be changed
===
from
ListPath result = null
to
ListPath result = new ArrayListPath();
===
to return a empty list not null value.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-7767) Get rid of ZKTable, and table enable/disable state in ZK
[
https://issues.apache.org/jira/browse/HBASE-7767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107428#comment-14107428
]
Hadoop QA commented on HBASE-7767:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12663486/HBASE-7767.patch
against trunk revision .
ATTACHMENT ID: 12663486
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 28 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 7
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.camel.test.junit4.CamelTestSupport.doStopCamelContext(CamelTestSupport.java:450)
at
org.apache.camel.test.junit4.CamelTestSupport.tearDown(CamelTestSupport.java:351)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/10537//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10537//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10537//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10537//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10537//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10537//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10537//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10537//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10537//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10537//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10537//console
This message is automatically generated.
Get rid of ZKTable, and table enable/disable state in ZK
-
Key: HBASE-7767
URL: https://issues.apache.org/jira/browse/HBASE-7767
Project: HBase
Issue Type: Sub-task
Components: Zookeeper
Affects Versions: 2.0.0
Reporter: Enis Soztutar
Assignee: Andrey Stepachev
Attachments: HBASE-7767.patch, HBASE-7767.patch
As discussed table state in zookeeper for enable/disable state breaks our
zookeeper contract. It is also very intrusive, used from the client side,
master and region servers. We should get rid of it.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11546) Backport HBASE-11059 (ZK-less region assignment) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virag Kothari updated HBASE-11546: -- Attachment: HBASE-11531-0.98.patch HBASE-11659-0.98.patch HBASE-11725-0.98.patch HBASE-11687-0.98.patch HBASE-11059-0.98.patch The patches are ready. To cleanly apply, maintain the following order: HBASE-11059, HBASE-11687, HBASE-11725, HBASE-11659, HBASE-11531. Have run unit tests and integration tests. HBASE-11610 and HBASE-11689 can be ported later once they are in trunk. Backport HBASE-11059 (ZK-less region assignment) to 0.98 Key: HBASE-11546 URL: https://issues.apache.org/jira/browse/HBASE-11546 Project: HBase Issue Type: Sub-task Reporter: Virag Kothari Assignee: Virag Kothari Fix For: 0.98.6 Attachments: HBASE-11059-0.98.patch, HBASE-11531-0.98.patch, HBASE-11546_98.patch, HBASE-11659-0.98.patch, HBASE-11687-0.98.patch, HBASE-11725-0.98.patch Discuss concerns about backporting HBASE-11059 to 0.98 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11546) Backport HBASE-11059 (ZK-less region assignment) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virag Kothari updated HBASE-11546: -- Attachment: (was: HBASE-11546_98.patch) Backport HBASE-11059 (ZK-less region assignment) to 0.98 Key: HBASE-11546 URL: https://issues.apache.org/jira/browse/HBASE-11546 Project: HBase Issue Type: Sub-task Reporter: Virag Kothari Assignee: Virag Kothari Fix For: 0.98.6 Attachments: HBASE-11059-0.98.patch, HBASE-11531-0.98.patch, HBASE-11659-0.98.patch, HBASE-11687-0.98.patch, HBASE-11725-0.98.patch Discuss concerns about backporting HBASE-11059 to 0.98 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11806) Reasses test categories
Alex Newman created HBASE-11806: --- Summary: Reasses test categories Key: HBASE-11806 URL: https://issues.apache.org/jira/browse/HBASE-11806 Project: HBase Issue Type: Bug Reporter: Alex Newman HBase has an impressive set of tests. It remains a great investment and we have done a lot to make them comprehensive. However I feel like we could use some improvements on test categorization. From http://hbase.apache.org/book/hbase.tests.html 18.8.2.1. Small Tests Small tests are executed in a shared JVM. We put in this category all the tests that can be executed quickly in a shared JVM. The maximum execution time for a small test is 15 seconds, and small tests should not use a (mini)cluster. 18.8.2.2. Medium Tests Medium tests represent tests that must be executed before proposing a patch. They are designed to run in less than 30 minutes altogether, and are quite stable in their results. They are designed to last less than 50 seconds individually. They can use a cluster, and each of them is executed in a separate JVM. 18.8.2.3. Large Tests Large tests are everything else. They are typically large-scale tests, regression tests for specific bugs, timeout tests, performance tests. They are executed before a commit on the pre-integration machines. They can be run on the developer machine as well. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11806) Reasses test categories
[ https://issues.apache.org/jira/browse/HBASE-11806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107443#comment-14107443 ] Alex Newman commented on HBASE-11806: - So I can tell you that small tests are around 22 minutes to run on circleci Some ones which take more than 20 seconds (15s is probably a bit short) Running org.apache.hadoop.hbase.thrift.TestThriftServerCmdLine Tests run: 32, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 49.727 sec - in org.apache.hadoop.hbase.thrift.TestThriftServerCmdLine Running org.apache.hadoop.hbase.thrift2.TestHTablePool$TestHTableReusablePool Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.688 sec - in org.apache.hadoop.hbase.thrift2.TestHTablePool$TestHTableReusablePool Running org.apache.hadoop.hbase.thrift.TestThriftServer Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 41.913 sec - in org.apache.hadoop.hbase.thrift.TestThriftServer Reasses test categories --- Key: HBASE-11806 URL: https://issues.apache.org/jira/browse/HBASE-11806 Project: HBase Issue Type: Bug Reporter: Alex Newman HBase has an impressive set of tests. It remains a great investment and we have done a lot to make them comprehensive. However I feel like we could use some improvements on test categorization. From http://hbase.apache.org/book/hbase.tests.html 18.8.2.1. Small Tests Small tests are executed in a shared JVM. We put in this category all the tests that can be executed quickly in a shared JVM. The maximum execution time for a small test is 15 seconds, and small tests should not use a (mini)cluster. 18.8.2.2. Medium Tests Medium tests represent tests that must be executed before proposing a patch. They are designed to run in less than 30 minutes altogether, and are quite stable in their results. They are designed to last less than 50 seconds individually. They can use a cluster, and each of them is executed in a separate JVM. 18.8.2.3. Large Tests Large tests are everything else. They are typically large-scale tests, regression tests for specific bugs, timeout tests, performance tests. They are executed before a commit on the pre-integration machines. They can be run on the developer machine as well. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11806) Reasses test categories
[ https://issues.apache.org/jira/browse/HBASE-11806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107452#comment-14107452 ] Alex Newman commented on HBASE-11806: - I realized there's something wrong about my set setup please ignore the above comment. Just because you pass -P runtestTest doesn't guarantee that it's only going to run those. Reasses test categories --- Key: HBASE-11806 URL: https://issues.apache.org/jira/browse/HBASE-11806 Project: HBase Issue Type: Bug Reporter: Alex Newman HBase has an impressive set of tests. It remains a great investment and we have done a lot to make them comprehensive. However I feel like we could use some improvements on test categorization. From http://hbase.apache.org/book/hbase.tests.html 18.8.2.1. Small Tests Small tests are executed in a shared JVM. We put in this category all the tests that can be executed quickly in a shared JVM. The maximum execution time for a small test is 15 seconds, and small tests should not use a (mini)cluster. 18.8.2.2. Medium Tests Medium tests represent tests that must be executed before proposing a patch. They are designed to run in less than 30 minutes altogether, and are quite stable in their results. They are designed to last less than 50 seconds individually. They can use a cluster, and each of them is executed in a separate JVM. 18.8.2.3. Large Tests Large tests are everything else. They are typically large-scale tests, regression tests for specific bugs, timeout tests, performance tests. They are executed before a commit on the pre-integration machines. They can be run on the developer machine as well. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11807) Verify that Running mvn test -P runSmallTests will execute small tests only, using a single JVM., same for medium and large
Alex Newman created HBASE-11807: --- Summary: Verify that Running mvn test -P runSmallTests will execute small tests only, using a single JVM., same for medium and large Key: HBASE-11807 URL: https://issues.apache.org/jira/browse/HBASE-11807 Project: HBase Issue Type: Sub-task Reporter: Alex Newman -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11689) Track meta in transition
[ https://issues.apache.org/jira/browse/HBASE-11689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Stepachev updated HBASE-11689: - Attachment: HBASE-11689.patch reattached patch Track meta in transition Key: HBASE-11689 URL: https://issues.apache.org/jira/browse/HBASE-11689 Project: HBase Issue Type: Bug Components: Region Assignment Affects Versions: 2.0.0 Reporter: Jimmy Xiang Assignee: Andrey Stepachev Attachments: HBASE-11689.patch With ZK-less region assignment, user regions in transition are tracked in meta. We need a way to track meta in transition too. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11467) New impl of Registry interface not using ZK + new RPCs on master protocol
[ https://issues.apache.org/jira/browse/HBASE-11467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107465#comment-14107465 ] Mikhail Antonov commented on HBASE-11467: - Thanks for review [~ryanobjc]! I'll post updated patch soon with those fixed + modified handling of registries on server-side. New impl of Registry interface not using ZK + new RPCs on master protocol - Key: HBASE-11467 URL: https://issues.apache.org/jira/browse/HBASE-11467 Project: HBase Issue Type: Sub-task Components: Client, Consensus, Zookeeper Affects Versions: 2.0.0 Reporter: Mikhail Antonov Assignee: Mikhail Antonov Fix For: 2.0.0 Attachments: HBASE-11467.patch, HBASE-11467.patch, HBASE-11467.patch Currently there' only one implementation of Registry interface, which is using ZK to get info about meta. Need to create implementation which will be using RPC calls to master the client is connected to. Review of early version of patch is here: https://reviews.apache.org/r/24296/ -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11689) Track meta in transition
[ https://issues.apache.org/jira/browse/HBASE-11689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107468#comment-14107468 ] Andrey Stepachev commented on HBASE-11689: -- https://reviews.apache.org/r/24996/ Track meta in transition Key: HBASE-11689 URL: https://issues.apache.org/jira/browse/HBASE-11689 Project: HBase Issue Type: Bug Components: Region Assignment Affects Versions: 2.0.0 Reporter: Jimmy Xiang Assignee: Andrey Stepachev Attachments: HBASE-11689.patch With ZK-less region assignment, user regions in transition are tracked in meta. We need a way to track meta in transition too. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11721) jdiff script no longer works as usage instructions indicate
[ https://issues.apache.org/jira/browse/HBASE-11721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dima Spivak updated HBASE-11721: Attachment: HBASE-11721.patch Fixes problems that prevent the script from running. Will file a separate JIRA to do a more thorough revision of the script to improve usability and readability. jdiff script no longer works as usage instructions indicate --- Key: HBASE-11721 URL: https://issues.apache.org/jira/browse/HBASE-11721 Project: HBase Issue Type: Bug Components: scripts Reporter: Misty Stanley-Jones Assignee: Dima Spivak Attachments: HBASE-11721.patch I pasted the command from the usage instructions embedded in the script, but it fails as follows: [misty@cheezel dev-support](master)$ bash ./jdiffHBasePublicAPI.sh https://github.com/apache/hbase.git 0.94 https://github.com/MY_REPO/hbase.git 0.94 JDiff evaluation beginning: Determining if this is a local directory or a git repo. Looks like https://github.com/apache/hbase.git is a git repo Determining if this is a local directory or a git repo. Looks like https://github.com/MY_REPO/hbase.git is a git repo We are going to compare source 1 which is a git_repo and source 2, which is a git_repo 0.94 0.94 JDIFF_WORKING_DIRECTORY not set. That's not an issue. We will default it to /tmp/jdiff. % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 183 100 1830 0447 0 --:--:-- --:--:-- --:--:-- 448 Archive: jdiff-1.1.1-with-incompatible-option.zip End-of-central-directory signature not found. Either this file is not a zipfile, or it constitutes one disk of a multi-part archive. In the latter case the central directory and zipfile comment will be found on the last disk(s) of this archive. unzip: cannot find zipfile directory in one of jdiff-1.1.1-with-incompatible-option.zip or jdiff-1.1.1-with-incompatible-option.zip.zip, and cannot find jdiff-1.1.1-with-incompatible-option.zip.ZIP, period. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Work started] (HBASE-11721) jdiff script no longer works as usage instructions indicate
[ https://issues.apache.org/jira/browse/HBASE-11721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-11721 started by Dima Spivak. jdiff script no longer works as usage instructions indicate --- Key: HBASE-11721 URL: https://issues.apache.org/jira/browse/HBASE-11721 Project: HBase Issue Type: Bug Components: scripts Reporter: Misty Stanley-Jones Assignee: Dima Spivak Attachments: HBASE-11721.patch I pasted the command from the usage instructions embedded in the script, but it fails as follows: [misty@cheezel dev-support](master)$ bash ./jdiffHBasePublicAPI.sh https://github.com/apache/hbase.git 0.94 https://github.com/MY_REPO/hbase.git 0.94 JDiff evaluation beginning: Determining if this is a local directory or a git repo. Looks like https://github.com/apache/hbase.git is a git repo Determining if this is a local directory or a git repo. Looks like https://github.com/MY_REPO/hbase.git is a git repo We are going to compare source 1 which is a git_repo and source 2, which is a git_repo 0.94 0.94 JDIFF_WORKING_DIRECTORY not set. That's not an issue. We will default it to /tmp/jdiff. % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 183 100 1830 0447 0 --:--:-- --:--:-- --:--:-- 448 Archive: jdiff-1.1.1-with-incompatible-option.zip End-of-central-directory signature not found. Either this file is not a zipfile, or it constitutes one disk of a multi-part archive. In the latter case the central directory and zipfile comment will be found on the last disk(s) of this archive. unzip: cannot find zipfile directory in one of jdiff-1.1.1-with-incompatible-option.zip or jdiff-1.1.1-with-incompatible-option.zip.zip, and cannot find jdiff-1.1.1-with-incompatible-option.zip.ZIP, period. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11721) jdiff script no longer works as usage instructions indicate
[ https://issues.apache.org/jira/browse/HBASE-11721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dima Spivak updated HBASE-11721: Status: Patch Available (was: In Progress) jdiff script no longer works as usage instructions indicate --- Key: HBASE-11721 URL: https://issues.apache.org/jira/browse/HBASE-11721 Project: HBase Issue Type: Bug Components: scripts Reporter: Misty Stanley-Jones Assignee: Dima Spivak Attachments: HBASE-11721.patch I pasted the command from the usage instructions embedded in the script, but it fails as follows: [misty@cheezel dev-support](master)$ bash ./jdiffHBasePublicAPI.sh https://github.com/apache/hbase.git 0.94 https://github.com/MY_REPO/hbase.git 0.94 JDiff evaluation beginning: Determining if this is a local directory or a git repo. Looks like https://github.com/apache/hbase.git is a git repo Determining if this is a local directory or a git repo. Looks like https://github.com/MY_REPO/hbase.git is a git repo We are going to compare source 1 which is a git_repo and source 2, which is a git_repo 0.94 0.94 JDIFF_WORKING_DIRECTORY not set. That's not an issue. We will default it to /tmp/jdiff. % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 183 100 1830 0447 0 --:--:-- --:--:-- --:--:-- 448 Archive: jdiff-1.1.1-with-incompatible-option.zip End-of-central-directory signature not found. Either this file is not a zipfile, or it constitutes one disk of a multi-part archive. In the latter case the central directory and zipfile comment will be found on the last disk(s) of this archive. unzip: cannot find zipfile directory in one of jdiff-1.1.1-with-incompatible-option.zip or jdiff-1.1.1-with-incompatible-option.zip.zip, and cannot find jdiff-1.1.1-with-incompatible-option.zip.ZIP, period. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11794) StripeStoreFlusher causes NullPointerException and Region down
[
https://issues.apache.org/jira/browse/HBASE-11794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107496#comment-14107496
]
Sergey Shelukhin commented on HBASE-11794:
--
Does
{noformat}
-if (result != null) {
- result.clear();
-}
{noformat}
need to be removed? Empty list still needs to be returned in case of error.
StripeStoreFlusher causes NullPointerException and Region down
--
Key: HBASE-11794
URL: https://issues.apache.org/jira/browse/HBASE-11794
Project: HBase
Issue Type: Bug
Components: Compaction
Affects Versions: 0.98.3, 0.98.4, 0.98.5
Reporter: jeongmin kim
Priority: Critical
Attachments: HBASE_11794.patch
StoreFlusher.flushSnapshot() mustn't return null value.
But StripeStoreFlusher.flushSnapshot() does.
It cause NullPointerException at
org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:802)
and this makes regions dead after exhaustive retries and no recovery
available from it.
the code (StripeStoreFlusher:64) has to be changed
===
from
ListPath result = null
to
ListPath result = new ArrayListPath();
===
to return a empty list not null value.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HBASE-11808) Update jdiff script to improve usability and readability
Dima Spivak created HBASE-11808: --- Summary: Update jdiff script to improve usability and readability Key: HBASE-11808 URL: https://issues.apache.org/jira/browse/HBASE-11808 Project: HBase Issue Type: Improvement Reporter: Dima Spivak Assignee: Dima Spivak We should update the jdiffHBasePublicAPI script in hbase/dev-support to be a bit more user-friendly. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11794) StripeStoreFlusher causes NullPointerException and Region down
[ https://issues.apache.org/jira/browse/HBASE-11794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107500#comment-14107500 ] Jean-Marc Spaggiari commented on HBASE-11794: - List is initialized empty. So even on failure, it will be returned empty, no? StripeStoreFlusher causes NullPointerException and Region down -- Key: HBASE-11794 URL: https://issues.apache.org/jira/browse/HBASE-11794 Project: HBase Issue Type: Bug Components: Compaction Affects Versions: 0.98.3, 0.98.4, 0.98.5 Reporter: jeongmin kim Priority: Critical Attachments: HBASE_11794.patch StoreFlusher.flushSnapshot() mustn't return null value. But StripeStoreFlusher.flushSnapshot() does. It cause NullPointerException at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:802) and this makes regions dead after exhaustive retries and no recovery available from it. the code (StripeStoreFlusher:64) has to be changed === from ListPath result = null to ListPath result = new ArrayListPath(); === to return a empty list not null value. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HBASE-11807) Verify that Running mvn test -P runSmallTests will execute small tests only, using a single JVM., same for medium and large
[ https://issues.apache.org/jira/browse/HBASE-11807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Newman reassigned HBASE-11807: --- Assignee: Alex Newman Verify that Running mvn test -P runSmallTests will execute small tests only, using a single JVM., same for medium and large - Key: HBASE-11807 URL: https://issues.apache.org/jira/browse/HBASE-11807 Project: HBase Issue Type: Sub-task Reporter: Alex Newman Assignee: Alex Newman -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11794) StripeStoreFlusher causes NullPointerException and Region down
[ https://issues.apache.org/jira/browse/HBASE-11794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107510#comment-14107510 ] Sergey Shelukhin commented on HBASE-11794: -- nm, it must be a leftover from earlier implementation where failure could happen with some files already in the list. LGTM StripeStoreFlusher causes NullPointerException and Region down -- Key: HBASE-11794 URL: https://issues.apache.org/jira/browse/HBASE-11794 Project: HBase Issue Type: Bug Components: Compaction Affects Versions: 0.98.3, 0.98.4, 0.98.5 Reporter: jeongmin kim Priority: Critical Attachments: HBASE_11794.patch StoreFlusher.flushSnapshot() mustn't return null value. But StripeStoreFlusher.flushSnapshot() does. It cause NullPointerException at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:802) and this makes regions dead after exhaustive retries and no recovery available from it. the code (StripeStoreFlusher:64) has to be changed === from ListPath result = null to ListPath result = new ArrayListPath(); === to return a empty list not null value. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11789) LoadIncrementalHFiles is not picking up the -D option
[ https://issues.apache.org/jira/browse/HBASE-11789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] deepankar updated HBASE-11789: -- Attachment: (was: HBASE-11789-0.94.patch) LoadIncrementalHFiles is not picking up the -D option -- Key: HBASE-11789 URL: https://issues.apache.org/jira/browse/HBASE-11789 Project: HBase Issue Type: Bug Affects Versions: 0.99.0, 0.98.5, 2.0.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 0.99.0, 2.0.0, 0.98.6 Attachments: HBASE-11789-v0.patch LoadIncrementalHFiles is not using the Tool class correctly, preventing to use the -D options. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11643) Read and write MOB in HBase
[
https://issues.apache.org/jira/browse/HBASE-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107511#comment-14107511
]
Hadoop QA commented on HBASE-11643:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12663641/HBASE-11643-V9.diff
against trunk revision .
ATTACHMENT ID: 12663641
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 18 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 7
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestRegionRebalancing
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at org.apache.oozie.test.MiniHCatServer$1.run(MiniHCatServer.java:136)
at
org.apache.oozie.test.XTestCase$MiniClusterShutdownMonitor.run(XTestCase.java:1041)
at
org.apache.oozie.service.TestPartitionDependencyManagerEhcache.testMemoryUsageAndSpeed(TestPartitionDependencyManagerEhcache.java:54)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/10535//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10535//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10535//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10535//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10535//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10535//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10535//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10535//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10535//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10535//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10535//console
This message is automatically generated.
Read and write MOB in HBase
---
Key: HBASE-11643
URL: https://issues.apache.org/jira/browse/HBASE-11643
Project: HBase
Issue Type: Sub-task
Components: regionserver, Scanners
Reporter: Jingcheng Du
Assignee: Jingcheng Du
Attachments: HBASE-11643-V2.diff, HBASE-11643-V3.diff,
HBASE-11643-V4.diff, HBASE-11643-V5.diff, HBASE-11643-V6.diff,
HBASE-11643-V7.diff, HBASE-11643-V8.diff, HBASE-11643-V9.diff,
HBase-11643.diff
The read/write MOB in HBase are implemented in this JIRA. Normally, the Cells
are saved in the MemStore, and flushed to the HFiles when necessary. For MOB,
the Cells are saved in the MemStore as well, but they're flushed to elsewhere
out of HBase in the format of HFiles.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11789) LoadIncrementalHFiles is not picking up the -D option
[ https://issues.apache.org/jira/browse/HBASE-11789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] deepankar updated HBASE-11789: -- Attachment: HBASE-11789-0.94-v1.patch There was small bug in the previous one, so attaching the fixed one LoadIncrementalHFiles is not picking up the -D option -- Key: HBASE-11789 URL: https://issues.apache.org/jira/browse/HBASE-11789 Project: HBase Issue Type: Bug Affects Versions: 0.99.0, 0.98.5, 2.0.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 0.99.0, 2.0.0, 0.98.6 Attachments: HBASE-11789-0.94-v1.patch, HBASE-11789-v0.patch LoadIncrementalHFiles is not using the Tool class correctly, preventing to use the -D options. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11610) Enhance remote meta updates
[ https://issues.apache.org/jira/browse/HBASE-11610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virag Kothari updated HBASE-11610: -- Status: Open (was: Patch Available) Enhance remote meta updates --- Key: HBASE-11610 URL: https://issues.apache.org/jira/browse/HBASE-11610 Project: HBase Issue Type: Sub-task Reporter: Jimmy Xiang Assignee: Virag Kothari Attachments: HBASE-11610.patch, HBASE-11610_2.patch Currently, if the meta region is on a regionserver instead of the master, meta update is synchronized on one HTable instance. We should be able to do better. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Work started] (HBASE-11808) Update jdiff script to improve usability and readability
[ https://issues.apache.org/jira/browse/HBASE-11808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-11808 started by Dima Spivak. Update jdiff script to improve usability and readability Key: HBASE-11808 URL: https://issues.apache.org/jira/browse/HBASE-11808 Project: HBase Issue Type: Improvement Reporter: Dima Spivak Assignee: Dima Spivak We should update the jdiffHBasePublicAPI script in hbase/dev-support to be a bit more user-friendly. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10977) TestHBaseFsck.testQuarantineMissingHFile fails missing files test
[
https://issues.apache.org/jira/browse/HBASE-10977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-10977:
--
Affects Version/s: 0.94.15
TestHBaseFsck.testQuarantineMissingHFile fails missing files test
-
Key: HBASE-10977
URL: https://issues.apache.org/jira/browse/HBASE-10977
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.15
Reporter: stack
Priority: Minor
On our internal rig, ran into this failure:
{code}
java.lang.AssertionError: expected:2 but was:1
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.doQuarantineTest(TestHBaseFsck.java:1737)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testQuarantineMissingHFile(TestHBaseFsck.java:1781)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}
This is what is failing:
assertEquals(hfcc.getMissing().size(), missing);
The file remove has not run yet or, else, this complaint is related:
{code}
2014-04-12 23:24:57,638 WARN [IPC Server handler 4 on 50919]
security.UserGroupInformation(1551): PriviledgedActionException as:jenkins
(auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist:
/user/jenkins/hbase/data/default/testQuarantineMissingHFile/63cdcba1fc55ae6463463ae16f4e454e/fam/57a07eaac97e4acd8dc04e08d1950adc
{code}
Below is full log. Will come back and add logging
{code}
Regression
org.apache.hadoop.hbase.util.TestHBaseFsck.testQuarantineMissingHFile
Failing for the past 1 build (Since Failed#3 )
Took 10 ms.
add description
Error Message
expected:2 but was:1
Stacktrace
java.lang.AssertionError: expected:2 but was:1
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.doQuarantineTest(TestHBaseFsck.java:1737)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testQuarantineMissingHFile(TestHBaseFsck.java:1781)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Standard Output
Allow checking/fixes for table: testQuarantineMissingHFile
Checked 4 hfile for corruption
HFiles corrupted: 0
HFiles successfully quarantined: 0
HFiles failed quarantine:0
HFiles moved while checking: 2
hdfs://localhost:50919/user/jenkins/hbase/data/default/testQuarantineMissingHFile/63cdcba1fc55ae6463463ae16f4e454e/fam/57a07eaac97e4acd8dc04e08d1950adc
hdfs://localhost:50919/user/jenkins/hbase/data/default/testQuarantineMissingHFile/d383980be98665b638fd56bfac97a351/fam/9dd79f30f29e4cfeaa46f6e20b32e078
Summary: OK = OK
Version: 0.96.1.1-cdh5.0.1-SNAPSHOT
Table 'testQuarantineMissingHFile': region split map
: [ { meta = null, hdfs =
hdfs://localhost:50919/user/jenkins/hbase/data/default/testQuarantineMissingHFile/63cdcba1fc55ae6463463ae16f4e454e,
deployed = }, A]
[jira] [Updated] (HBASE-11331) [blockcache] lazy block decompression
[ https://issues.apache.org/jira/browse/HBASE-11331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-11331: - Attachment: hbase-hbase-master-hor17n36.gq1.ygridcore.net.log re: hbase-hbase-master-hor17n36.gq1.ygridcore.net.log bq. I've tripped my assertions around blockcache count metrics. False alarm. There is a case when the reported metric is greater than 5% different from the actual map size, but it happened while executing the shutdown hooks, not while the job was running. Attaching the log file in case anyone is curious. [blockcache] lazy block decompression - Key: HBASE-11331 URL: https://issues.apache.org/jira/browse/HBASE-11331 Project: HBase Issue Type: Improvement Components: regionserver Reporter: Nick Dimiduk Assignee: Nick Dimiduk Attachments: HBASE-11331.00.patch, HBASE-11331.01.patch, HBASE-11331.02.patch, HBASE-11331.03.patch, HBASE-11331.04.patch, HBASE-11331.05.patch, HBASE-11331LazyBlockDecompressperfcompare.pdf, hbase-hbase-master-hor17n36.gq1.ygridcore.net.log, lazy-decompress.02.0.pdf, lazy-decompress.02.1.json, lazy-decompress.02.1.pdf, v03-20g-045g-false.pdf, v03-20g-045g-true-16h.pdf, v03-20g-045g-true.pdf Maintaining data in its compressed form in the block cache will greatly increase our effective blockcache size and should show a meaning improvement in cache hit rates in well designed applications. The idea here is to lazily decompress/decrypt blocks when they're consumed, rather than as soon as they're pulled off of disk. This is related to but less invasive than HBASE-8894. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11642) EOL 0.96
[ https://issues.apache.org/jira/browse/HBASE-11642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107613#comment-14107613 ] stack commented on HBASE-11642: --- Sent mail to list just now with subject: ANN: 0.96 is being EOL'd Added note to refguide (though we should start up an EOL section I'd say both what is EOL'd and how to... will wait on the how to till we've done a few). Updated the downloads HEADER.html to note 0.96 has been EOL'd. An interesting consequence of EOL that may have been plain to others but that I just realized is that now it is EOL'd, patches can go into 0.94 and 0.98 w/o having to go into 0.96. EOL 0.96 Key: HBASE-11642 URL: https://issues.apache.org/jira/browse/HBASE-11642 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Do the work to EOL 0.96. + No more patches on 0.96 + Remove 0.96 from downloads. + If user has issue with 0.96 and needs fix, fix it in 0.98 and have the user upgrade to get the fix. + Write email to user list stating 0.96 has been EOL'd September 1st? And add notice to refguide. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-11642) EOL 0.96
[ https://issues.apache.org/jira/browse/HBASE-11642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-11642. --- Resolution: Fixed Resolving as done. If objection on mailing list or later, can reopen. EOL 0.96 Key: HBASE-11642 URL: https://issues.apache.org/jira/browse/HBASE-11642 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Do the work to EOL 0.96. + No more patches on 0.96 + Remove 0.96 from downloads. + If user has issue with 0.96 and needs fix, fix it in 0.98 and have the user upgrade to get the fix. + Write email to user list stating 0.96 has been EOL'd September 1st? And add notice to refguide. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11797) Create Table interface to replace HTableInterface
[
https://issues.apache.org/jira/browse/HBASE-11797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107621#comment-14107621
]
stack commented on HBASE-11797:
---
Sounds good. Would be coolio having this in before 1.0.
Create Table interface to replace HTableInterface
-
Key: HBASE-11797
URL: https://issues.apache.org/jira/browse/HBASE-11797
Project: HBase
Issue Type: Bug
Reporter: Carter
Assignee: Carter
Basically doing this:
{code}
interface Table {
// get, put related stuff
}
@Deprecated
interface HTableInterface extends Table {
// users are encouraged to use the new Table interface
}
class HTable extends Table {
// all HTable constructors are deprecated
// Users are not encouraged to see this class
}
{code}
I'm proposing that in this JIRA I move everything from HTableInterface to
Table except the following:
* Anything deprecated
* Anything @InterfaceAudience.Private ({{coprocessorService(...)}} and
{{batchCoprocessorService(...)}})
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11072) Abstract WAL splitting from ZK
[ https://issues.apache.org/jira/browse/HBASE-11072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107633#comment-14107633 ] stack commented on HBASE-11072: --- [~sergey.soldatov] Does v10 address my rb comments? (It seems to on cursory review) Abstract WAL splitting from ZK -- Key: HBASE-11072 URL: https://issues.apache.org/jira/browse/HBASE-11072 Project: HBase Issue Type: Sub-task Components: Consensus, Zookeeper Affects Versions: 0.99.0 Reporter: Mikhail Antonov Assignee: Sergey Soldatov Attachments: HBASE-11072-1_v2.patch, HBASE-11072-1_v3.patch, HBASE-11072-1_v4.patch, HBASE-11072-2_v2.patch, HBASE-11072-v1.patch, HBASE-11072-v10.patch, HBASE-11072-v2.patch, HBASE-11072-v3.patch, HBASE-11072-v4.patch, HBASE-11072-v5.patch, HBASE-11072-v6.patch, HBASE-11072-v7.patch, HBASE-11072-v8.patch, HBASE-11072-v8.patch, HBASE-11072-v9.patch, HBASE_11072-1.patch HM side: - SplitLogManager RS side: - SplitLogWorker - HLogSplitter and a few handler classes. This jira may need to be split further apart into smaller ones. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11072) Abstract WAL splitting from ZK
[ https://issues.apache.org/jira/browse/HBASE-11072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107634#comment-14107634 ] stack commented on HBASE-11072: --- Is the javadoc failure yours? Abstract WAL splitting from ZK -- Key: HBASE-11072 URL: https://issues.apache.org/jira/browse/HBASE-11072 Project: HBase Issue Type: Sub-task Components: Consensus, Zookeeper Affects Versions: 0.99.0 Reporter: Mikhail Antonov Assignee: Sergey Soldatov Attachments: HBASE-11072-1_v2.patch, HBASE-11072-1_v3.patch, HBASE-11072-1_v4.patch, HBASE-11072-2_v2.patch, HBASE-11072-v1.patch, HBASE-11072-v10.patch, HBASE-11072-v10.patch, HBASE-11072-v2.patch, HBASE-11072-v3.patch, HBASE-11072-v4.patch, HBASE-11072-v5.patch, HBASE-11072-v6.patch, HBASE-11072-v7.patch, HBASE-11072-v8.patch, HBASE-11072-v8.patch, HBASE-11072-v9.patch, HBASE_11072-1.patch HM side: - SplitLogManager RS side: - SplitLogWorker - HLogSplitter and a few handler classes. This jira may need to be split further apart into smaller ones. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11072) Abstract WAL splitting from ZK
[ https://issues.apache.org/jira/browse/HBASE-11072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-11072: -- Attachment: HBASE-11072-v10.patch Retrying v10 though hadoopqa is on holiday these times. Abstract WAL splitting from ZK -- Key: HBASE-11072 URL: https://issues.apache.org/jira/browse/HBASE-11072 Project: HBase Issue Type: Sub-task Components: Consensus, Zookeeper Affects Versions: 0.99.0 Reporter: Mikhail Antonov Assignee: Sergey Soldatov Attachments: HBASE-11072-1_v2.patch, HBASE-11072-1_v3.patch, HBASE-11072-1_v4.patch, HBASE-11072-2_v2.patch, HBASE-11072-v1.patch, HBASE-11072-v10.patch, HBASE-11072-v10.patch, HBASE-11072-v2.patch, HBASE-11072-v3.patch, HBASE-11072-v4.patch, HBASE-11072-v5.patch, HBASE-11072-v6.patch, HBASE-11072-v7.patch, HBASE-11072-v8.patch, HBASE-11072-v8.patch, HBASE-11072-v9.patch, HBASE_11072-1.patch HM side: - SplitLogManager RS side: - SplitLogWorker - HLogSplitter and a few handler classes. This jira may need to be split further apart into smaller ones. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11809) Make sure that tests are properly categorized
Alex Newman created HBASE-11809: --- Summary: Make sure that tests are properly categorized Key: HBASE-11809 URL: https://issues.apache.org/jira/browse/HBASE-11809 Project: HBase Issue Type: Sub-task Reporter: Alex Newman Assignee: Alex Newman -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11567) Write bulk load COMMIT events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Zhong updated HBASE-11567: -- Attachment: hbase-11567-v4.patch Thanks [~posix4e] for the comments. The v4 tries to address your concerns. Thanks. Write bulk load COMMIT events to WAL Key: HBASE-11567 URL: https://issues.apache.org/jira/browse/HBASE-11567 Project: HBase Issue Type: Sub-task Reporter: Enis Soztutar Assignee: Alex Newman Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch, hbase-11567-v3.patch, hbase-11567-v4.patch Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and region open/close (HBASE-11512) , we should persist bulk load events to WAL. This is especially important for secondary region replicas, since we can use this information to pick up primary regions' files from secondary replicas. A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11806) Make the build more Blue
[ https://issues.apache.org/jira/browse/HBASE-11806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Newman updated HBASE-11806: Summary: Make the build more Blue (was: Reasses test categories) Make the build more Blue Key: HBASE-11806 URL: https://issues.apache.org/jira/browse/HBASE-11806 Project: HBase Issue Type: Bug Reporter: Alex Newman HBase has an impressive set of tests. It remains a great investment and we have done a lot to make them comprehensive. However I feel like we could use some improvements on test categorization. From http://hbase.apache.org/book/hbase.tests.html 18.8.2.1. Small Tests Small tests are executed in a shared JVM. We put in this category all the tests that can be executed quickly in a shared JVM. The maximum execution time for a small test is 15 seconds, and small tests should not use a (mini)cluster. 18.8.2.2. Medium Tests Medium tests represent tests that must be executed before proposing a patch. They are designed to run in less than 30 minutes altogether, and are quite stable in their results. They are designed to last less than 50 seconds individually. They can use a cluster, and each of them is executed in a separate JVM. 18.8.2.3. Large Tests Large tests are everything else. They are typically large-scale tests, regression tests for specific bugs, timeout tests, performance tests. They are executed before a commit on the pre-integration machines. They can be run on the developer machine as well. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11567) Write bulk load COMMIT events to WAL
[ https://issues.apache.org/jira/browse/HBASE-11567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107648#comment-14107648 ] Alex Newman commented on HBASE-11567: - Sounds good, although I can't really +1 since I am not a committer. But upon brief glance, it seems as though you have. Good on you! Write bulk load COMMIT events to WAL Key: HBASE-11567 URL: https://issues.apache.org/jira/browse/HBASE-11567 Project: HBase Issue Type: Sub-task Reporter: Enis Soztutar Assignee: Alex Newman Attachments: HBASE-11567-v1.patch, HBASE-11567-v2.patch, hbase-11567-v3.patch, hbase-11567-v4.patch Similar to writing flush (HBASE-11511), compaction(HBASE-2231) to WAL and region open/close (HBASE-11512) , we should persist bulk load events to WAL. This is especially important for secondary region replicas, since we can use this information to pick up primary regions' files from secondary replicas. A design doc for secondary replica replication can be found at HBASE-11183. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11647) MOB integration testing
[
https://issues.apache.org/jira/browse/HBASE-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107716#comment-14107716
]
Jonathan Hsieh commented on HBASE-11647:
I like where these tests are going. Can you post some examples of how you kick
off the test in the jira?
As an aside, the documentation in the IntegrationTestIngest could be improved,
as could the docs in this the IntegrationTestIngestMOB.
Explicitly call out this his an extension of the IntegrationTestIngest. that
uses LoadTestTool to generate and writes mob sized data into hbase and verify
it.
{quote}
+/**
+ * Integration Test for MOB ingest.
+ */
+@Category(IntegrationTests.class)
+public class IntegrationTestIngestMOB extends IntegrationTestIngest {
{quote}
Please provide some way of getting usage instructions and what the
LoadTestDataGeneraetyorMob:x:y:z:w args are!
{quote}
+ public static void main(String[] args) throws Exception {
+Configuration conf = HBaseConfiguration.create();
+IntegrationTestingUtility.setUseDistributedCluster(conf);
+int ret = ToolRunner.run(conf, new IntegrationTestIngestMOB(), args);
+System.exit(ret);
+ }
{quote}
Add a comment here saying we add a another value generator that has different
cols data size bounds to expcicitly test the mobs.
{quote}
+/**
+ * A load test data generator for MOB
+ */
+public class LoadTestDataGeneratorMOB
+extends MultiThreadedAction.DefaultDataGenerator {
+
{quote}
This instanceof is a bad smell -- it breaks encapsulation -- can we do this in
a cleaner way? Maybe add in a String... or Object... arg so that we can
handle all of these and without having to do the instanceof? At the least,
please leave a TODO here to refactor so that we just use an interface and
inheritance properly to avoid the instanceof.
{quote}
}
+ } else if(dataGen instanceof LoadTestDataGeneratorMOB) {
+LOG.info(Using LoadTestDataGeneratorMOB);
+String mobCf = clazzAndArgs[1];
+int minMobDataSize = Integer.parseInt(clazzAndArgs[2]);
+int maxMobDataSize = Integer.parseInt(clazzAndArgs[3]);
+LoadTestDataGeneratorMOB mobDatGen = (LoadTestDataGeneratorMOB)dataGen;
+mobDatGen.configureMob(mobCf.getBytes(), minMobDataSize,
maxMobDataSize);
+args = clazzAndArgs.length==4? new String[0] :
Arrays.copyOfRange(clazzAndArgs, 4, clazzAndArgs.length);
} else {
{quote}
MOB integration testing
---
Key: HBASE-11647
URL: https://issues.apache.org/jira/browse/HBASE-11647
Project: HBase
Issue Type: Sub-task
Components: Performance, test
Reporter: Jingcheng Du
Assignee: Jingcheng Du
Attachments: HBASE-11647.diff
The integration testings include the integration function testing and
performance testing.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11617) incorrect AgeOfLastAppliedOp and AgeOfLastShippedOp in replication Metrics when no new replication OP
[
https://issues.apache.org/jira/browse/HBASE-11617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107741#comment-14107741
]
Demai Ni commented on HBASE-11617:
--
[~andrew.purt...@gmail.com], thanks for the ping
[~lhofhansl], what's your take on this? maybe we can commit the current fix,
and consider to remove .refreshAgeOfLastAppliedOp in a later refactoring
effort?
incorrect AgeOfLastAppliedOp and AgeOfLastShippedOp in replication Metrics
when no new replication OP
--
Key: HBASE-11617
URL: https://issues.apache.org/jira/browse/HBASE-11617
Project: HBase
Issue Type: Bug
Components: Replication
Affects Versions: 0.98.2
Reporter: Demai Ni
Assignee: Demai Ni
Priority: Minor
Fix For: 0.99.0, 2.0.0, 0.98.6
Attachments: HBASE-11617-master-v1.patch
AgeOfLastAppliedOp in MetricsSink.java is to indicate the time an edit sat in
the 'replication queue' before it got replicated(aka applied)
{code}
/**
* Set the age of the last applied operation
*
* @param timestamp The timestamp of the last operation applied.
* @return the age that was set
*/
public long setAgeOfLastAppliedOp(long timestamp) {
lastTimestampForAge = timestamp;
long age = System.currentTimeMillis() - lastTimestampForAge;
rms.setGauge(SINK_AGE_OF_LAST_APPLIED_OP, age);
return age;
}
{code}
In the following scenario:
1) at 7:00am a sink op is applied, and the SINK_AGE_OF_LAST_APPLIED_OP is
set for example 100ms;
2) and then NO new Sink op occur.
3) when a refreshAgeOfLastAppliedOp() is invoked at 8:00am. Instead of
return the 100ms, the AgeOfLastAppliedOp become 1hour + 100ms,
It was because that refreshAgeOfLastAppliedOp() get invoked periodically by
getStats().
proposed fix:
{code}
---
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSink.java
+++
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSink.java
@@ -35,6 +35,7 @@ public class MetricsSink {
private MetricsReplicationSource rms;
private long lastTimestampForAge = System.currentTimeMillis();
+ private long age = 0;
public MetricsSink() {
rms =
CompatibilitySingletonFactory.getInstance(MetricsReplicationSource.class);
@@ -47,8 +48,12 @@ public class MetricsSink {
* @return the age that was set
*/
public long setAgeOfLastAppliedOp(long timestamp) {
-lastTimestampForAge = timestamp;
-long age = System.currentTimeMillis() - lastTimestampForAge;
+if (lastTimestampForAge != timestamp) {
+ lastTimestampForAge = timestamp;
+ this.age = System.currentTimeMillis() - lastTimestampForAge;
+} else {
+ this.age = 0;
+}
rms.setGauge(SINK_AGE_OF_LAST_APPLIED_OP, age);
return age;
}
{code}
detail discussion in [dev@hbase |
http://mail-archives.apache.org/mod_mbox/hbase-dev/201407.mbox/%3CCAOEq2C5BKMXAM2Fv4LGVb_Ktek-Pm%3DhjOi33gSHX-2qHqAou6w%40mail.gmail.com%3E
]
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-9746) RegionServer can't start when replication tries to replicate to an unknown host
[
https://issues.apache.org/jira/browse/HBASE-9746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107746#comment-14107746
]
Lars Hofhansl commented on HBASE-9746:
--
Oh forgot to mention: This generally make RecoverableZookeeper resilient to DNS
blips, which is nice.
RegionServer can't start when replication tries to replicate to an unknown
host
---
Key: HBASE-9746
URL: https://issues.apache.org/jira/browse/HBASE-9746
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.12
Reporter: Lars Hofhansl
Priority: Minor
Fix For: 0.99.0, 2.0.0, 0.98.7, 0.94.24
Attachments: 9746-0.98.txt
Just ran into this:
{code}
13/10/11 00:37:02 [regionserver60020] WARN zookeeper.ZKConfig(204):
java.net.UnknownHostException: old-host: Name or service not known
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:894)
at
java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1286)
at java.net.InetAddress.getAllByName0(InetAddress.java:1239)
at java.net.InetAddress.getAllByName(InetAddress.java:1155)
at java.net.InetAddress.getAllByName(InetAddress.java:1091)
at java.net.InetAddress.getByName(InetAddress.java:1041)
at
org.apache.hadoop.hbase.zookeeper.ZKConfig.getZKQuorumServersString(ZKConfig.java:201)
at
org.apache.hadoop.hbase.zookeeper.ZKConfig.getZKQuorumServersString(ZKConfig.java:245)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.init(ZooKeeperWatcher.java:147)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.init(ZooKeeperWatcher.java:127)
at
org.apache.hadoop.hbase.replication.ReplicationPeer.reloadZkWatcher(ReplicationPeer.java:170)
at
org.apache.hadoop.hbase.replication.ReplicationPeer.init(ReplicationPeer.java:69)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.getPeer(ReplicationZookeeper.java:343)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.connectToPeer(ReplicationZookeeper.java:308)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.connectExistingPeers(ReplicationZookeeper.java:189)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.init(ReplicationZookeeper.java:156)
at
org.apache.hadoop.hbase.replication.regionserver.Replication.initialize(Replication.java:89)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.newReplicationInstance(HRegionServer.java:3986)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.createNewReplicationInstance(HRegionServer.java:3955)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.setupWALAndReplication(HRegionServer.java:1412)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1096)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:749)
at java.lang.Thread.run(Thread.java:722)
13/10/11 00:37:02 [regionserver60020] ERROR zookeeper.ZKConfig(210): no valid
quorum servers found in zoo.cfg
13/10/11 00:37:02 [regionserver60020] WARN regionserver.HRegionServer(1108):
Exception in region server :
java.io.IOException: Unable to determine ZooKeeper ensemble
at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:116)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.init(ZooKeeperWatcher.java:153)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.init(ZooKeeperWatcher.java:127)
at
org.apache.hadoop.hbase.replication.ReplicationPeer.reloadZkWatcher(ReplicationPeer.java:170)
at
org.apache.hadoop.hbase.replication.ReplicationPeer.init(ReplicationPeer.java:69)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.getPeer(ReplicationZookeeper.java:343)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.connectToPeer(ReplicationZookeeper.java:308)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.connectExistingPeers(ReplicationZookeeper.java:189)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.init(ReplicationZookeeper.java:156)
at
org.apache.hadoop.hbase.replication.regionserver.Replication.initialize(Replication.java:89)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.newReplicationInstance(HRegionServer.java:3986)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.createNewReplicationInstance(HRegionServer.java:3955)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.setupWALAndReplication(HRegionServer.java:1412)
at
[jira] [Updated] (HBASE-9746) RegionServer can't start when replication tries to replicate to an unknown host
[
https://issues.apache.org/jira/browse/HBASE-9746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-9746:
-
Attachment: 9746-0.98.txt
Here's a W.I.P patch.
It works by allowing RecoverableZookeeper to handle UnknownHost issues just
like any other connection issues and to retry.
With that I can bring the cluster up. It's might be a bit verbose until the
slave is brought up (but RecoverableZookeeper retries with a back off, so it's
not too bad).
Please have a look.
RegionServer can't start when replication tries to replicate to an unknown
host
---
Key: HBASE-9746
URL: https://issues.apache.org/jira/browse/HBASE-9746
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.12
Reporter: Lars Hofhansl
Priority: Minor
Fix For: 0.99.0, 2.0.0, 0.98.7, 0.94.24
Attachments: 9746-0.98.txt
Just ran into this:
{code}
13/10/11 00:37:02 [regionserver60020] WARN zookeeper.ZKConfig(204):
java.net.UnknownHostException: old-host: Name or service not known
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:894)
at
java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1286)
at java.net.InetAddress.getAllByName0(InetAddress.java:1239)
at java.net.InetAddress.getAllByName(InetAddress.java:1155)
at java.net.InetAddress.getAllByName(InetAddress.java:1091)
at java.net.InetAddress.getByName(InetAddress.java:1041)
at
org.apache.hadoop.hbase.zookeeper.ZKConfig.getZKQuorumServersString(ZKConfig.java:201)
at
org.apache.hadoop.hbase.zookeeper.ZKConfig.getZKQuorumServersString(ZKConfig.java:245)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.init(ZooKeeperWatcher.java:147)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.init(ZooKeeperWatcher.java:127)
at
org.apache.hadoop.hbase.replication.ReplicationPeer.reloadZkWatcher(ReplicationPeer.java:170)
at
org.apache.hadoop.hbase.replication.ReplicationPeer.init(ReplicationPeer.java:69)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.getPeer(ReplicationZookeeper.java:343)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.connectToPeer(ReplicationZookeeper.java:308)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.connectExistingPeers(ReplicationZookeeper.java:189)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.init(ReplicationZookeeper.java:156)
at
org.apache.hadoop.hbase.replication.regionserver.Replication.initialize(Replication.java:89)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.newReplicationInstance(HRegionServer.java:3986)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.createNewReplicationInstance(HRegionServer.java:3955)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.setupWALAndReplication(HRegionServer.java:1412)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1096)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:749)
at java.lang.Thread.run(Thread.java:722)
13/10/11 00:37:02 [regionserver60020] ERROR zookeeper.ZKConfig(210): no valid
quorum servers found in zoo.cfg
13/10/11 00:37:02 [regionserver60020] WARN regionserver.HRegionServer(1108):
Exception in region server :
java.io.IOException: Unable to determine ZooKeeper ensemble
at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:116)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.init(ZooKeeperWatcher.java:153)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.init(ZooKeeperWatcher.java:127)
at
org.apache.hadoop.hbase.replication.ReplicationPeer.reloadZkWatcher(ReplicationPeer.java:170)
at
org.apache.hadoop.hbase.replication.ReplicationPeer.init(ReplicationPeer.java:69)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.getPeer(ReplicationZookeeper.java:343)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.connectToPeer(ReplicationZookeeper.java:308)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.connectExistingPeers(ReplicationZookeeper.java:189)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.init(ReplicationZookeeper.java:156)
at
org.apache.hadoop.hbase.replication.regionserver.Replication.initialize(Replication.java:89)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.newReplicationInstance(HRegionServer.java:3986)
at
