[jira] [Commented] (HBASE-11788) hbase is not deleting the cell when a Put with a KeyValue, KeyValue.Type.Delete is submitted
[
https://issues.apache.org/jira/browse/HBASE-11788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105093#comment-14105093
]
Lars Hofhansl commented on HBASE-11788:
---
The problem would seem to be in ProtobufUtil.toMutation(...). We set the
KeyValue type according to type of Mutation passed.
This is not incorrect per se as this is not really the expected use of the
HBase client API. It's easy to break stuff in subtle ways doing this, from that
angle the new behavior is better.
That said, if it breaks Hive, let's fix it.
hbase is not deleting the cell when a Put with a KeyValue,
KeyValue.Type.Delete is submitted
Key: HBASE-11788
URL: https://issues.apache.org/jira/browse/HBASE-11788
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0, 0.96.1.1, 0.98.5, 2.0.0
Environment: Cloudera CDH 5.1.x
Reporter: Cristian Armaselu
Attachments: TestPutWithDelete.java
Code executed:
{code}
@Test
public void testHbasePutDeleteCell() throws Exception {
TableName tableName = TableName.valueOf(my_test);
Configuration configuration = HBaseConfiguration.create();
HTableInterface table = new HTable(configuration, tableName);
final String rowKey = 12345;
final byte[] familly = Bytes.toBytes(default);
// put one row
Put put = new Put(Bytes.toBytes(rowKey));
put.add(familly, Bytes.toBytes(A), Bytes.toBytes(a));
put.add(familly, Bytes.toBytes(B), Bytes.toBytes(b));
put.add(familly, Bytes.toBytes(C), Bytes.toBytes(c));
table.put(put);
// get row back and assert the values
Get get = new Get(Bytes.toBytes(rowKey));
Result result = table.get(get);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(A))).equals(a), Column A value should be a);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(B))).equals(b), Column B value should be b);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(C))).equals(c), Column C value should be c);
// put the same row again with C column deleted
put = new Put(Bytes.toBytes(rowKey));
put.add(familly, Bytes.toBytes(A), Bytes.toBytes(a));
put.add(familly, Bytes.toBytes(B), Bytes.toBytes(b));
put.add(new KeyValue(Bytes.toBytes(rowKey), familly,
Bytes.toBytes(C), HConstants.LATEST_TIMESTAMP, KeyValue.Type.DeleteColumn));
table.put(put);
// get row back and assert the values
get = new Get(Bytes.toBytes(rowKey));
result = table.get(get);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(A))).equals(a), Column A value should be a);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(B))).equals(b), Column A value should be b);
Assert.isTrue(result.getValue(familly, Bytes.toBytes(C)) == null,
Column C should not exists);
}
{code}
This assertion fails, the cell is not deleted but rather the value is empty:
{code}
hbase(main):029:0 scan 'my_test'
ROW COLUMN+CELL
12345column=default:A,
timestamp=1408473082290, value=a
12345column=default:B,
timestamp=1408473082290, value=b
12345column=default:C,
timestamp=1408473082290, value=
{code}
This behavior is different than previous 4.8.x Cloudera version and is
currently corrupting all hive queries involving is null or is not null
operators on the columns mapped to hbase
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11643) Read and write MOB in HBase
[ https://issues.apache.org/jira/browse/HBASE-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-11643: - Attachment: HBASE-11643-V8.diff Merge the latest master code and re-generate the patch to make it pass the Hadoop QA. Read and write MOB in HBase --- Key: HBASE-11643 URL: https://issues.apache.org/jira/browse/HBASE-11643 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Jingcheng Du Assignee: Jingcheng Du Attachments: HBASE-11643-V2.diff, HBASE-11643-V3.diff, HBASE-11643-V4.diff, HBASE-11643-V5.diff, HBASE-11643-V6.diff, HBASE-11643-V7.diff, HBASE-11643-V8.diff, HBase-11643.diff The read/write MOB in HBase are implemented in this JIRA. Normally, the Cells are saved in the MemStore, and flushed to the HFiles when necessary. For MOB, the Cells are saved in the MemStore as well, but they're flushed to elsewhere out of HBase in the format of HFiles. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11788) hbase is not deleting the cell when a Put with a KeyValue, KeyValue.Type.Delete is submitted
[
https://issues.apache.org/jira/browse/HBASE-11788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105101#comment-14105101
]
Lars Hofhansl commented on HBASE-11788:
---
toPut would also need to be changed to allow building a Put with delete-type
KeyValue in it. Possible, but really messy.
hbase is not deleting the cell when a Put with a KeyValue,
KeyValue.Type.Delete is submitted
Key: HBASE-11788
URL: https://issues.apache.org/jira/browse/HBASE-11788
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0, 0.96.1.1, 0.98.5, 2.0.0
Environment: Cloudera CDH 5.1.x
Reporter: Cristian Armaselu
Attachments: TestPutWithDelete.java
Code executed:
{code}
@Test
public void testHbasePutDeleteCell() throws Exception {
TableName tableName = TableName.valueOf(my_test);
Configuration configuration = HBaseConfiguration.create();
HTableInterface table = new HTable(configuration, tableName);
final String rowKey = 12345;
final byte[] familly = Bytes.toBytes(default);
// put one row
Put put = new Put(Bytes.toBytes(rowKey));
put.add(familly, Bytes.toBytes(A), Bytes.toBytes(a));
put.add(familly, Bytes.toBytes(B), Bytes.toBytes(b));
put.add(familly, Bytes.toBytes(C), Bytes.toBytes(c));
table.put(put);
// get row back and assert the values
Get get = new Get(Bytes.toBytes(rowKey));
Result result = table.get(get);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(A))).equals(a), Column A value should be a);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(B))).equals(b), Column B value should be b);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(C))).equals(c), Column C value should be c);
// put the same row again with C column deleted
put = new Put(Bytes.toBytes(rowKey));
put.add(familly, Bytes.toBytes(A), Bytes.toBytes(a));
put.add(familly, Bytes.toBytes(B), Bytes.toBytes(b));
put.add(new KeyValue(Bytes.toBytes(rowKey), familly,
Bytes.toBytes(C), HConstants.LATEST_TIMESTAMP, KeyValue.Type.DeleteColumn));
table.put(put);
// get row back and assert the values
get = new Get(Bytes.toBytes(rowKey));
result = table.get(get);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(A))).equals(a), Column A value should be a);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(B))).equals(b), Column A value should be b);
Assert.isTrue(result.getValue(familly, Bytes.toBytes(C)) == null,
Column C should not exists);
}
{code}
This assertion fails, the cell is not deleted but rather the value is empty:
{code}
hbase(main):029:0 scan 'my_test'
ROW COLUMN+CELL
12345column=default:A,
timestamp=1408473082290, value=a
12345column=default:B,
timestamp=1408473082290, value=b
12345column=default:C,
timestamp=1408473082290, value=
{code}
This behavior is different than previous 4.8.x Cloudera version and is
currently corrupting all hive queries involving is null or is not null
operators on the columns mapped to hbase
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11165) Scaling so cluster can host 1M regions and beyond (50M regions?)
[ https://issues.apache.org/jira/browse/HBASE-11165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105102#comment-14105102 ] Mikhail Antonov commented on HBASE-11165: - bq. Do we need to split this conversation into what to do on master and what to do with 0.98? That makes lot of sense to me. Scaling so cluster can host 1M regions and beyond (50M regions?) Key: HBASE-11165 URL: https://issues.apache.org/jira/browse/HBASE-11165 Project: HBase Issue Type: Brainstorming Reporter: stack Attachments: HBASE-11165.zip, Region Scalability test.pdf, zk_less_assignment_comparison_2.pdf This discussion issue comes out of Co-locate Meta And Master HBASE-10569 and comments on the doc posted there. A user -- our Francis Liu -- needs to be able to scale a cluster to do 1M regions maybe even 50M later. This issue is about discussing how we will do that (or if not 50M on a cluster, how otherwise we can attain same end). More detail to follow. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11794) StripeStoreFlusher causes NullPointerException and Region down
jeongmin kim created HBASE-11794: Summary: StripeStoreFlusher causes NullPointerException and Region down Key: HBASE-11794 URL: https://issues.apache.org/jira/browse/HBASE-11794 Project: HBase Issue Type: Bug Components: Compaction Affects Versions: 0.98.5 Reporter: jeongmin kim Priority: Critical StoreFlusher.flushSnapshot() mustn't return null value. But StripeStoreFlusher.flushSnapshot() does. It cause NullPointerException at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:802) and this makes regions dead after exhaustive retries and no recovery available from it. the code (StripeStoreFlusher:64) has to be changed === from ListPath result = null to ListPath result = new ArrayListPath(); === to return a empty list not null value. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11787) TestRegionLocations is not categorized
[ https://issues.apache.org/jira/browse/HBASE-11787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105180#comment-14105180 ] Nicolas Liochon commented on HBASE-11787: - bq. TestPutWriteToWAL It's empty and you're the only committer there. I haven't investigated further :-) bq. TestCheckTestClasses only checks for hbase-server tests? Likely, it was done before the modularization... I don't know if there is an easy fix for this... TestRegionLocations is not categorized -- Key: HBASE-11787 URL: https://issues.apache.org/jira/browse/HBASE-11787 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.99.0, 2.0.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Trivial Fix For: 0.99.0, 2.0.0 Attachments: 11787.v1.patch It's actually executed (likely with the large tests), but it's risky. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11591) Scanner fails to retrieve KV from bulk loaded file with highest sequence id than the cell's mvcc in a non-bulk loaded file
[
https://issues.apache.org/jira/browse/HBASE-11591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105185#comment-14105185
]
ramkrishna.s.vasudevan commented on HBASE-11591:
[~jeffreyz]
First of all thanks a lot for taking a look at this issue and providing a patch.
I debugged this issue with 2 cases - with my patch and with Jeffrey's patch.
Observed the following things
- The testcase that was added as part of this testcase is for same KVs in the
Store file and the bulk loaded Store file and it was specific for that issue.
- After Jeffrey's first patch few testcases failed and those were not having
this case of same KVs. All were different row keys. Things were working fine
because it was purely based on row key comparison and no mvcc would have even
come into it. (I mean even before the patch). I think that exposed some of the
bug that was inside.
- Another important observation is that when we are scanning the KVs in the
bulk loaded file (atleast those created new LoadIncrementalHFile cases) there
is no mvcc info added to the metadata also. So
{code}
return new StoreFileScanner(this,
getScanner(cacheBlocks, pread, isCompaction),
!isCompaction, reader.hasMVCCInfo(), readPt);
{code}
will say has mvccInfo as false and hence skipNewerThanReadPoint() would never
be called because
{code}
if (hasMVCCInfo)
skipKVsNewerThanReadpoint();
{code}
So before the patch too, the scenario in the failed test case
TestWALReplay.testCompactedBulkLoadedFiles() though our seqID for the bulk
loaded files were 5, and the read point for all the scanners created in the
test case was 4 - we were trying to read the bulk loaded file also. But we
were not able to skip the kvs in the bulk loaded file just because hasMvccInfo
was false. So the tests were passing.
Ok so what happens after Jeffrey's patch(the first patch without HREgion's
change) is that on seeing any bulk loaded file we just assign the file's seqid
to the KV's seqId. And so after compaction still the read pt is not modified
to the latest (ie 5) and hence all the KVs that were written to the compacted
file from the bulk loaded files were missing.
I think the change in HRegion.java to set the write Sequence number is a bug
fix? I still feel the patch would cause issue in the following scenario after
the above changes
- Assume a scan started and the read point is 20 at that time
- Bulk load is just getting completed and the scanner heap gets reset. The
new bulk loaded file with seqId 22 (for eg) gets added now to the scanner heap.
But remember that the read point is still 20.
- After this change we would just set the bulk load file's seqId to all its
KVs which is 22.
- Because there is no mvcc info in this bulk loaded file the scan would not be
able to skipTheKvsWithNewerReadPt() and hence the scan would still see the Kvs
with 22 as the seqId though the intention is to see only KVs with seqID 20.
I may be wrong. Am I missing something here? I may be wrong because for bulk
loaded files because there is no mvcc we are allowed to read anything in that
irrespective of the read pt?
Scanner fails to retrieve KV from bulk loaded file with highest sequence id
than the cell's mvcc in a non-bulk loaded file
---
Key: HBASE-11591
URL: https://issues.apache.org/jira/browse/HBASE-11591
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Critical
Fix For: 0.99.0
Attachments: HBASE-11591.patch, HBASE-11591_1.patch,
HBASE-11591_2.patch, HBASE-11591_3.patch, TestBulkload.java,
hbase-11591-03-02.patch, hbase-11591-03-jeff.patch
See discussion in HBASE-11339.
When we have a case where there are same KVs in two files one produced by
flush/compaction and the other thro the bulk load.
Both the files have some same kvs which matches even in timestamp.
Steps:
Add some rows with a specific timestamp and flush the same.
Bulk load a file with the same data.. Enusre that assign seqnum property is
set.
The bulk load should use HFileOutputFormat2 (or ensure that we write the
bulk_time_output key).
This would ensure that the bulk loaded file has the highest seq num.
Assume the cell in the flushed/compacted store file is
row1,cf,cq,ts1, value1 and the cell in the bulk loaded file is
row1,cf,cq,ts1,value2
(There are no parallel scans).
Issue a scan on the table in 0.96. The retrieved value is
row1,cf1,cq,ts1,value2
But the same in 0.98 will retrieve row1,cf1,cq,ts2,value1.
This is a behaviour change. This is because of this code
{code}
[jira] [Updated] (HBASE-11647) MOB integration testing
[ https://issues.apache.org/jira/browse/HBASE-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-11647: - Attachment: HBASE-11647.diff Add the integration testing for the mob. The testing reports will be uploaded after they are done. Thanks. MOB integration testing --- Key: HBASE-11647 URL: https://issues.apache.org/jira/browse/HBASE-11647 Project: HBase Issue Type: Sub-task Components: Performance, test Reporter: Jingcheng Du Assignee: Jingcheng Du Attachments: HBASE-11647.diff The integration testings include the integration function testing and performance testing. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-4955) Use the official versions of surefire junit
[ https://issues.apache.org/jira/browse/HBASE-4955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105190#comment-14105190 ] Nicolas Liochon commented on HBASE-4955: bq. I wonder if this should give up and fail if we try for too long? We could. At the end of the day, it's a surefire bug: it should stop properly the process. bq. Could you be a bit more detailed on the things which we should fix to get this through? On the previous test, we did not generate the final report (iirc it was the issue I got as well a few months ago). I don't know how to fix this. On the test you relaunched, it worked. I don't know if it's linked to this new surefire version, and what is the probability to have the issue. I'm going to commit the patch, we will see and we can always revert. Guys, don't hesitate to revert. I'm leaving the office at 6pm my time today, so if there is an issue during the bay area office hours I won't be able to handle it. Use the official versions of surefire junit - Key: HBASE-4955 URL: https://issues.apache.org/jira/browse/HBASE-4955 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0, 0.98.0, 0.96.0, 0.99.0 Environment: all Reporter: Nicolas Liochon Assignee: Alex Newman Priority: Critical Attachments: 4955.v1.patch, 4955.v2.patch, 4955.v2.patch, 4955.v2.patch, 4955.v2.patch, 4955.v3.patch, 4955.v3.patch, 4955.v3.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v5.patch, 4955.v6.patch, 4955.v7.patch, 4955.v7.patch, 4955.v8.patch, 4955.v9.patch, 8204.v4.patch, HBASE-4955-v10.patch, HBASE-4955-v10.patch, HBASE-4955-v10.patch We currently use private versions for Surefire JUnit since HBASE-4763. This JIRA traks what we need to move to official versions. Surefire 2.11 is just out, but, after some tests, it does not contain all what we need. JUnit. Could be for JUnit 4.11. Issue to monitor: https://github.com/KentBeck/junit/issues/359: fixed in our version, no feedback for an integration on trunk Surefire: Could be for Surefire 2.12. Issues to monitor are: 329 (category support): fixed, we use the official implementation from the trunk 786 (@Category with forkMode=always): fixed, we use the official implementation from the trunk 791 (incorrect elapsed time on test failure): fixed, we use the official implementation from the trunk 793 (incorrect time in the XML report): Not fixed (reopen) on trunk, fixed on our version. 760 (does not take into account the test method): fixed in trunk, not fixed in our version 798 (print immediately the test class name): not fixed in trunk, not fixed in our version 799 (Allow test parallelization when forkMode=always): not fixed in trunk, not fixed in our version 800 (redirectTestOutputToFile not taken into account): not yet fix on trunk, fixed on our version 800 793 are the more important to monitor, it's the only ones that are fixed in our version but not on trunk. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11339) HBase MOB
[ https://issues.apache.org/jira/browse/HBASE-11339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105191#comment-14105191 ] Jingcheng Du commented on HBASE-11339: -- Dear all, the patches for the sub tasks of HBase MOB have been uploaded. Please help review and comment. Thanks a lot! HBase MOB - Key: HBASE-11339 URL: https://issues.apache.org/jira/browse/HBASE-11339 Project: HBase Issue Type: Umbrella Components: regionserver, Scanners Reporter: Jingcheng Du Assignee: Jingcheng Du Attachments: HBase MOB Design-v2.pdf, HBase MOB Design-v3.pdf, HBase MOB Design-v4.pdf, HBase MOB Design.pdf, MOB user guide .docx, hbase-11339-in-dev.patch It's quite useful to save the medium binary data like images, documents into Apache HBase. Unfortunately directly saving the binary MOB(medium object) to HBase leads to a worse performance since the frequent split and compaction. In this design, the MOB data are stored in an more efficient way, which keeps a high write/read performance and guarantees the data consistency in Apache HBase. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-7767) Get rid of ZKTable, and table enable/disable state in ZK
[ https://issues.apache.org/jira/browse/HBASE-7767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Stepachev updated HBASE-7767: Affects Version/s: (was: 0.95.2) 2.0.0 Status: Patch Available (was: In Progress) Get rid of ZKTable, and table enable/disable state in ZK - Key: HBASE-7767 URL: https://issues.apache.org/jira/browse/HBASE-7767 Project: HBase Issue Type: Sub-task Components: Zookeeper Affects Versions: 2.0.0 Reporter: Enis Soztutar Assignee: Andrey Stepachev Attachments: HBASE-7767.patch As discussed table state in zookeeper for enable/disable state breaks our zookeeper contract. It is also very intrusive, used from the client side, master and region servers. We should get rid of it. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-7767) Get rid of ZKTable, and table enable/disable state in ZK
[ https://issues.apache.org/jira/browse/HBASE-7767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Stepachev updated HBASE-7767: Attachment: HBASE-7767.patch Patch is quite big, but what was done. 1. Table state stored in .tableinfo. HTableDescriptor now has TABLE_STATE parameter. 2. Added TableState and inner enum State. 3. Added MasterRPC getTableState (which supersedes isTableEnabled/isTableDisabled, now getTableState returns enum) 4. TableStateManager removed from public api and become helper private class (similar to TableNamespaceManager) 5. TableStateManager caches reads and updates cache on writes. 6. TableStateManager doesn’t use table locks for modifications, caller should ensure, that table locked. That is because TableLocks not reentrant. An option is to add boolean, but currently TableStateManager used in master only as helper class. 5. Table state also available through HTableDescriptor, but can’t be altered. enable/disable RPC call should be used. 6. Removed TableState from ZookeeperProtos. 7. Removed methods for table availability from Registry. 7. Removed TableStateManager from CoordinatedStateManager (now master is responsible for table state) Rationale for additional method: currently getTableDecriptor method wants fully initialised HMaster, but table state requests are used before master fully initialised. Added method waits only HMaster ’started’ status. Get rid of ZKTable, and table enable/disable state in ZK - Key: HBASE-7767 URL: https://issues.apache.org/jira/browse/HBASE-7767 Project: HBase Issue Type: Sub-task Components: Zookeeper Affects Versions: 2.0.0 Reporter: Enis Soztutar Assignee: Andrey Stepachev Attachments: HBASE-7767.patch As discussed table state in zookeeper for enable/disable state breaks our zookeeper contract. It is also very intrusive, used from the client side, master and region servers. We should get rid of it. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-4955) Use the official versions of surefire junit
[ https://issues.apache.org/jira/browse/HBASE-4955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-4955: --- Resolution: Fixed Fix Version/s: 2.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Use the official versions of surefire junit - Key: HBASE-4955 URL: https://issues.apache.org/jira/browse/HBASE-4955 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0, 0.98.0, 0.96.0, 0.99.0 Environment: all Reporter: Nicolas Liochon Assignee: Alex Newman Priority: Critical Fix For: 2.0.0 Attachments: 4955.v1.patch, 4955.v2.patch, 4955.v2.patch, 4955.v2.patch, 4955.v2.patch, 4955.v3.patch, 4955.v3.patch, 4955.v3.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v5.patch, 4955.v6.patch, 4955.v7.patch, 4955.v7.patch, 4955.v8.patch, 4955.v9.patch, 8204.v4.patch, HBASE-4955-v10.patch, HBASE-4955-v10.patch, HBASE-4955-v10.patch We currently use private versions for Surefire JUnit since HBASE-4763. This JIRA traks what we need to move to official versions. Surefire 2.11 is just out, but, after some tests, it does not contain all what we need. JUnit. Could be for JUnit 4.11. Issue to monitor: https://github.com/KentBeck/junit/issues/359: fixed in our version, no feedback for an integration on trunk Surefire: Could be for Surefire 2.12. Issues to monitor are: 329 (category support): fixed, we use the official implementation from the trunk 786 (@Category with forkMode=always): fixed, we use the official implementation from the trunk 791 (incorrect elapsed time on test failure): fixed, we use the official implementation from the trunk 793 (incorrect time in the XML report): Not fixed (reopen) on trunk, fixed on our version. 760 (does not take into account the test method): fixed in trunk, not fixed in our version 798 (print immediately the test class name): not fixed in trunk, not fixed in our version 799 (Allow test parallelization when forkMode=always): not fixed in trunk, not fixed in our version 800 (redirectTestOutputToFile not taken into account): not yet fix on trunk, fixed on our version 800 793 are the more important to monitor, it's the only ones that are fixed in our version but not on trunk. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11787) TestRegionLocations is not categorized
[ https://issues.apache.org/jira/browse/HBASE-11787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105218#comment-14105218 ] Nicolas Liochon commented on HBASE-11787: - It seems hadoop qa didn't run... Retrying. TestRegionLocations is not categorized -- Key: HBASE-11787 URL: https://issues.apache.org/jira/browse/HBASE-11787 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.99.0, 2.0.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Trivial Fix For: 0.99.0, 2.0.0 Attachments: 11787.v1.patch It's actually executed (likely with the large tests), but it's risky. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-8927) Use nano time instead of mili time everywhere
[ https://issues.apache.org/jira/browse/HBASE-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105225#comment-14105225 ] Nicolas Liochon commented on HBASE-8927: So I'm adding there the comment I wrote in a duplicate jira: Am I the only one who would appreciate some extra bits, to have a timestamps + a counter? Use nano time instead of mili time everywhere - Key: HBASE-8927 URL: https://issues.apache.org/jira/browse/HBASE-8927 Project: HBase Issue Type: Bug Reporter: stack Labels: phoenix Attachments: 8927.txt Less collisions and we are paying the price of a long anyways so might as well fill it. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-7767) Get rid of ZKTable, and table enable/disable state in ZK
[ https://issues.apache.org/jira/browse/HBASE-7767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105231#comment-14105231 ] Andrey Stepachev commented on HBASE-7767: - https://reviews.apache.org/r/24924/ Get rid of ZKTable, and table enable/disable state in ZK - Key: HBASE-7767 URL: https://issues.apache.org/jira/browse/HBASE-7767 Project: HBase Issue Type: Sub-task Components: Zookeeper Affects Versions: 2.0.0 Reporter: Enis Soztutar Assignee: Andrey Stepachev Attachments: HBASE-7767.patch As discussed table state in zookeeper for enable/disable state breaks our zookeeper contract. It is also very intrusive, used from the client side, master and region servers. We should get rid of it. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11610) Enhance remote meta updates
[ https://issues.apache.org/jira/browse/HBASE-11610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105239#comment-14105239 ] Nicolas Liochon commented on HBASE-11610: - bq. I think we should have an additional processBatch which takes care of that I agree. bq. Should we have a JIRA to discuss all this? As you like. It's not necessary imho, we can do that here. Cleaning up all stuff around thread pools would take some time. This jira is a good input for this work. Enhance remote meta updates --- Key: HBASE-11610 URL: https://issues.apache.org/jira/browse/HBASE-11610 Project: HBase Issue Type: Sub-task Reporter: Jimmy Xiang Assignee: Virag Kothari Attachments: HBASE-11610.patch Currently, if the meta region is on a regionserver instead of the master, meta update is synchronized on one HTable instance. We should be able to do better. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11763) Move TTL handling into ScanQueryMatcher
[
https://issues.apache.org/jira/browse/HBASE-11763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105274#comment-14105274
]
Hadoop QA commented on HBASE-11763:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12662786/HBASE-11763.patch
against trunk revision .
ATTACHMENT ID: 12662786
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 7
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestRollingRestart
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/10514//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10514//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10514//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10514//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10514//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10514//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10514//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10514//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10514//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10514//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10514//console
This message is automatically generated.
Move TTL handling into ScanQueryMatcher
---
Key: HBASE-11763
URL: https://issues.apache.org/jira/browse/HBASE-11763
Project: HBase
Issue Type: Sub-task
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Fix For: 0.99.0, 2.0.0, 0.98.6
Attachments: HBASE-11763.patch, HBASE-11763.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11591) Scanner fails to retrieve KV from bulk loaded file with highest sequence id than the cell's mvcc in a non-bulk loaded file
[
https://issues.apache.org/jira/browse/HBASE-11591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105286#comment-14105286
]
ramkrishna.s.vasudevan commented on HBASE-11591:
Another thing is that while doing flush as part of Bulk load if there is
nothing to be flushed should we still update the mvcc.
{code}
if (this.memstoreSize.get() = 0) {
// Presume that if there are still no edits in the memstore, then
there are no edits for
// this region out in the WAL/HLog subsystem so no need to do any
trickery clearing out
// edits in the WAL system. Up the sequence number so the resulting
flush id is for
// sure just beyond the last appended region edit (useful as a marker
when bulk loading,
// etc.)
// wal can be null replaying edits.
return wal != null?
new FlushResult(FlushResult.Result.CANNOT_FLUSH_MEMSTORE_EMPTY,
getNextSequenceId(wal), Nothing to flush):
new FlushResult(FlushResult.Result.CANNOT_FLUSH_MEMSTORE_EMPTY,
Nothing to flush);
}
{code}
Scanner fails to retrieve KV from bulk loaded file with highest sequence id
than the cell's mvcc in a non-bulk loaded file
---
Key: HBASE-11591
URL: https://issues.apache.org/jira/browse/HBASE-11591
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Critical
Fix For: 0.99.0
Attachments: HBASE-11591.patch, HBASE-11591_1.patch,
HBASE-11591_2.patch, HBASE-11591_3.patch, TestBulkload.java,
hbase-11591-03-02.patch, hbase-11591-03-jeff.patch
See discussion in HBASE-11339.
When we have a case where there are same KVs in two files one produced by
flush/compaction and the other thro the bulk load.
Both the files have some same kvs which matches even in timestamp.
Steps:
Add some rows with a specific timestamp and flush the same.
Bulk load a file with the same data.. Enusre that assign seqnum property is
set.
The bulk load should use HFileOutputFormat2 (or ensure that we write the
bulk_time_output key).
This would ensure that the bulk loaded file has the highest seq num.
Assume the cell in the flushed/compacted store file is
row1,cf,cq,ts1, value1 and the cell in the bulk loaded file is
row1,cf,cq,ts1,value2
(There are no parallel scans).
Issue a scan on the table in 0.96. The retrieved value is
row1,cf1,cq,ts1,value2
But the same in 0.98 will retrieve row1,cf1,cq,ts2,value1.
This is a behaviour change. This is because of this code
{code}
public int compare(KeyValueScanner left, KeyValueScanner right) {
int comparison = compare(left.peek(), right.peek());
if (comparison != 0) {
return comparison;
} else {
// Since both the keys are exactly the same, we break the tie in favor
// of the key which came latest.
long leftSequenceID = left.getSequenceID();
long rightSequenceID = right.getSequenceID();
if (leftSequenceID rightSequenceID) {
return -1;
} else if (leftSequenceID rightSequenceID) {
return 1;
} else {
return 0;
}
}
}
{code}
Here in 0.96 case the mvcc of the cell in both the files will have 0 and so
the comparison will happen from the else condition . Where the seq id of the
bulk loaded file is greater and would sort out first ensuring that the scan
happens from that bulk loaded file.
In case of 0.98+ as we are retaining the mvcc+seqid we are not making the
mvcc as 0 (remains a non zero positive value). Hence the compare() sorts out
the cell in the flushed/compacted file. Which means though we know the
lateset file is the bulk loaded file we don't scan the data.
Seems to be a behaviour change. Will check on other corner cases also but we
are trying to know the behaviour of bulk load because we are evaluating if it
can be used for MOB design.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11591) Scanner fails to retrieve KV from bulk loaded file with highest sequence id than the cell's mvcc in a non-bulk loaded file
[
https://issues.apache.org/jira/browse/HBASE-11591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-11591:
---
Status: Open (was: Patch Available)
Scanner fails to retrieve KV from bulk loaded file with highest sequence id
than the cell's mvcc in a non-bulk loaded file
---
Key: HBASE-11591
URL: https://issues.apache.org/jira/browse/HBASE-11591
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Critical
Fix For: 0.99.0
Attachments: HBASE-11591.patch, HBASE-11591_1.patch,
HBASE-11591_2.patch, HBASE-11591_3.patch, TestBulkload.java,
hbase-11591-03-02.patch, hbase-11591-03-jeff.patch
See discussion in HBASE-11339.
When we have a case where there are same KVs in two files one produced by
flush/compaction and the other thro the bulk load.
Both the files have some same kvs which matches even in timestamp.
Steps:
Add some rows with a specific timestamp and flush the same.
Bulk load a file with the same data.. Enusre that assign seqnum property is
set.
The bulk load should use HFileOutputFormat2 (or ensure that we write the
bulk_time_output key).
This would ensure that the bulk loaded file has the highest seq num.
Assume the cell in the flushed/compacted store file is
row1,cf,cq,ts1, value1 and the cell in the bulk loaded file is
row1,cf,cq,ts1,value2
(There are no parallel scans).
Issue a scan on the table in 0.96. The retrieved value is
row1,cf1,cq,ts1,value2
But the same in 0.98 will retrieve row1,cf1,cq,ts2,value1.
This is a behaviour change. This is because of this code
{code}
public int compare(KeyValueScanner left, KeyValueScanner right) {
int comparison = compare(left.peek(), right.peek());
if (comparison != 0) {
return comparison;
} else {
// Since both the keys are exactly the same, we break the tie in favor
// of the key which came latest.
long leftSequenceID = left.getSequenceID();
long rightSequenceID = right.getSequenceID();
if (leftSequenceID rightSequenceID) {
return -1;
} else if (leftSequenceID rightSequenceID) {
return 1;
} else {
return 0;
}
}
}
{code}
Here in 0.96 case the mvcc of the cell in both the files will have 0 and so
the comparison will happen from the else condition . Where the seq id of the
bulk loaded file is greater and would sort out first ensuring that the scan
happens from that bulk loaded file.
In case of 0.98+ as we are retaining the mvcc+seqid we are not making the
mvcc as 0 (remains a non zero positive value). Hence the compare() sorts out
the cell in the flushed/compacted file. Which means though we know the
lateset file is the bulk loaded file we don't scan the data.
Seems to be a behaviour change. Will check on other corner cases also but we
are trying to know the behaviour of bulk load because we are evaluating if it
can be used for MOB design.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11591) Scanner fails to retrieve KV from bulk loaded file with highest sequence id than the cell's mvcc in a non-bulk loaded file
[
https://issues.apache.org/jira/browse/HBASE-11591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-11591:
---
Attachment: HBASE-11591_5.patch
The patch tries to do skipKVsGreaterthanReadPt() even for bulk loaded kvs.
In the attached patch if we don't do the mvcc thing for no flush result then
the testBulkLoad will fail. That is because though the scanner is created
after bulk load the read pt is still lower than the seqID created as the seqId
is not added to the mvcc writeentry.
In the next case testBulkLoadWithParallelScan() the scanner is created before
bulk load. And so the expected KV should be the KV that is not from the bulk
loaded file, though the scanner heap is reset after the bulk load.
Scanner fails to retrieve KV from bulk loaded file with highest sequence id
than the cell's mvcc in a non-bulk loaded file
---
Key: HBASE-11591
URL: https://issues.apache.org/jira/browse/HBASE-11591
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Critical
Fix For: 0.99.0
Attachments: HBASE-11591.patch, HBASE-11591_1.patch,
HBASE-11591_2.patch, HBASE-11591_3.patch, HBASE-11591_5.patch,
TestBulkload.java, hbase-11591-03-02.patch, hbase-11591-03-jeff.patch
See discussion in HBASE-11339.
When we have a case where there are same KVs in two files one produced by
flush/compaction and the other thro the bulk load.
Both the files have some same kvs which matches even in timestamp.
Steps:
Add some rows with a specific timestamp and flush the same.
Bulk load a file with the same data.. Enusre that assign seqnum property is
set.
The bulk load should use HFileOutputFormat2 (or ensure that we write the
bulk_time_output key).
This would ensure that the bulk loaded file has the highest seq num.
Assume the cell in the flushed/compacted store file is
row1,cf,cq,ts1, value1 and the cell in the bulk loaded file is
row1,cf,cq,ts1,value2
(There are no parallel scans).
Issue a scan on the table in 0.96. The retrieved value is
row1,cf1,cq,ts1,value2
But the same in 0.98 will retrieve row1,cf1,cq,ts2,value1.
This is a behaviour change. This is because of this code
{code}
public int compare(KeyValueScanner left, KeyValueScanner right) {
int comparison = compare(left.peek(), right.peek());
if (comparison != 0) {
return comparison;
} else {
// Since both the keys are exactly the same, we break the tie in favor
// of the key which came latest.
long leftSequenceID = left.getSequenceID();
long rightSequenceID = right.getSequenceID();
if (leftSequenceID rightSequenceID) {
return -1;
} else if (leftSequenceID rightSequenceID) {
return 1;
} else {
return 0;
}
}
}
{code}
Here in 0.96 case the mvcc of the cell in both the files will have 0 and so
the comparison will happen from the else condition . Where the seq id of the
bulk loaded file is greater and would sort out first ensuring that the scan
happens from that bulk loaded file.
In case of 0.98+ as we are retaining the mvcc+seqid we are not making the
mvcc as 0 (remains a non zero positive value). Hence the compare() sorts out
the cell in the flushed/compacted file. Which means though we know the
lateset file is the bulk loaded file we don't scan the data.
Seems to be a behaviour change. Will check on other corner cases also but we
are trying to know the behaviour of bulk load because we are evaluating if it
can be used for MOB design.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11591) Scanner fails to retrieve KV from bulk loaded file with highest sequence id than the cell's mvcc in a non-bulk loaded file
[
https://issues.apache.org/jira/browse/HBASE-11591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-11591:
---
Status: Patch Available (was: Open)
Scanner fails to retrieve KV from bulk loaded file with highest sequence id
than the cell's mvcc in a non-bulk loaded file
---
Key: HBASE-11591
URL: https://issues.apache.org/jira/browse/HBASE-11591
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Critical
Fix For: 0.99.0
Attachments: HBASE-11591.patch, HBASE-11591_1.patch,
HBASE-11591_2.patch, HBASE-11591_3.patch, HBASE-11591_5.patch,
TestBulkload.java, hbase-11591-03-02.patch, hbase-11591-03-jeff.patch
See discussion in HBASE-11339.
When we have a case where there are same KVs in two files one produced by
flush/compaction and the other thro the bulk load.
Both the files have some same kvs which matches even in timestamp.
Steps:
Add some rows with a specific timestamp and flush the same.
Bulk load a file with the same data.. Enusre that assign seqnum property is
set.
The bulk load should use HFileOutputFormat2 (or ensure that we write the
bulk_time_output key).
This would ensure that the bulk loaded file has the highest seq num.
Assume the cell in the flushed/compacted store file is
row1,cf,cq,ts1, value1 and the cell in the bulk loaded file is
row1,cf,cq,ts1,value2
(There are no parallel scans).
Issue a scan on the table in 0.96. The retrieved value is
row1,cf1,cq,ts1,value2
But the same in 0.98 will retrieve row1,cf1,cq,ts2,value1.
This is a behaviour change. This is because of this code
{code}
public int compare(KeyValueScanner left, KeyValueScanner right) {
int comparison = compare(left.peek(), right.peek());
if (comparison != 0) {
return comparison;
} else {
// Since both the keys are exactly the same, we break the tie in favor
// of the key which came latest.
long leftSequenceID = left.getSequenceID();
long rightSequenceID = right.getSequenceID();
if (leftSequenceID rightSequenceID) {
return -1;
} else if (leftSequenceID rightSequenceID) {
return 1;
} else {
return 0;
}
}
}
{code}
Here in 0.96 case the mvcc of the cell in both the files will have 0 and so
the comparison will happen from the else condition . Where the seq id of the
bulk loaded file is greater and would sort out first ensuring that the scan
happens from that bulk loaded file.
In case of 0.98+ as we are retaining the mvcc+seqid we are not making the
mvcc as 0 (remains a non zero positive value). Hence the compare() sorts out
the cell in the flushed/compacted file. Which means though we know the
lateset file is the bulk loaded file we don't scan the data.
Seems to be a behaviour change. Will check on other corner cases also but we
are trying to know the behaviour of bulk load because we are evaluating if it
can be used for MOB design.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11777) Find a way to use KV.setSequenceId() on Cells on the server-side read path
[ https://issues.apache.org/jira/browse/HBASE-11777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105298#comment-14105298 ] ramkrishna.s.vasudevan commented on HBASE-11777: bq.If one or a few, could be anonymous? Ya. It would be few only. Anonymous is fine with me. bq.This change would immediately double the allocation rate. May be this happens only in case of bulk loaded files. Not in all cases. What do you think? Find a way to use KV.setSequenceId() on Cells on the server-side read path -- Key: HBASE-11777 URL: https://issues.apache.org/jira/browse/HBASE-11777 Project: HBase Issue Type: Improvement Affects Versions: 0.99.0 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Attachments: CellWithSequenceNumber.java Over in HBASE-11591 there was a need to set the sequenceId of the HFile to the bulk loaded KVs. Since we are trying to use the concept of Cells in the read path if we need to use setSequenceId(), then the Cell has to be converted to KV and only KeyValue impl has the accessor setSequenceId(). [~anoop.hbase] suggested if we can use a Server side impl of Cell and have these accessors in them. This JIRA aims to solve this and see the related code changes that needs to be carried out for this. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11643) Read and write MOB in HBase
[
https://issues.apache.org/jira/browse/HBASE-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105344#comment-14105344
]
Hadoop QA commented on HBASE-11643:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12663351/HBASE-11643-V8.diff
against trunk revision .
ATTACHMENT ID: 12663351
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 18 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 8
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster
org.apache.hadoop.hbase.replication.TestPerTableCFReplication
org.apache.hadoop.hbase.TestRegionRebalancing
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.regionserver.wal.TestLogRolling.testCompactionRecordDoesntBlockRolling(TestLogRolling.java:618)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/10515//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10515//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10515//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10515//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10515//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10515//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10515//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10515//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10515//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10515//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10515//console
This message is automatically generated.
Read and write MOB in HBase
---
Key: HBASE-11643
URL: https://issues.apache.org/jira/browse/HBASE-11643
Project: HBase
Issue Type: Sub-task
Components: regionserver, Scanners
Reporter: Jingcheng Du
Assignee: Jingcheng Du
Attachments: HBASE-11643-V2.diff, HBASE-11643-V3.diff,
HBASE-11643-V4.diff, HBASE-11643-V5.diff, HBASE-11643-V6.diff,
HBASE-11643-V7.diff, HBASE-11643-V8.diff, HBase-11643.diff
The read/write MOB in HBase are implemented in this JIRA. Normally, the Cells
are saved in the MemStore, and flushed to the HFiles when necessary. For MOB,
the Cells are saved in the MemStore as well, but they're flushed to elsewhere
out of HBase in the format of HFiles.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11729) Document HFile v3
[
https://issues.apache.org/jira/browse/HBASE-11729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105347#comment-14105347
]
Hadoop QA commented on HBASE-11729:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12663339/HBASE-11729-v2.patch
against trunk revision .
ATTACHMENT ID: 12663339
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 7
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+paraAs we will be discussing changes to the HFile format, it is
useful to give a short overview of the original (HFile version 1) format./para
+ footnoteparaImage courtesy of Lars George, link
xlink:href=http://www.larsgeorge.com/2009/10/hbase-architecture-101-storage.html;hbase-architecture-101-storage.html/link./para/footnote
+ paraThe number of entries in the block index is stored in the fixed file
trailer, and has to be passed in to the method that reads the block index. One
of the limitations of the block index in version 1 is that it does not provide
the compressed size of a block, which turns out to be necessary for
decompression. Therefore, the HFile reader has to infer this compressed size
from the offset difference between blocks. We fix this limitation in version 2,
where we store on-disk block size instead of uncompressed size, and get
uncompressed size from the block header./para
+ paraWe found it necessary to revise the HFile format after encountering
high memory usage and slow startup times caused by large Bloom filters and
block indexes in the region server. Bloom filters can get as large as 100 MB
per HFile, which adds up to 2 GB when aggregated over 20 regions. Block indexes
can grow as large as 6 GB in aggregate size over the same set of regions. A
region is not considered opened until all of its block index data is loaded.
Large Bloom filters produce a different performance problem: the first get
request that requires a Bloom filter lookup will incur the latency of loading
the entire Bloom filter bit array./para
+ paraTo speed up region server startup we break Bloom filters and block
indexes into multiple blocks and write those blocks out as they fill up, which
also reduces the HFile writer���s memory footprint. In the Bloom filter case,
���filling up a block��� means accumulating enough keys to efficiently utilize
a fixed-size bit array, and in the block index case we accumulate an ���index
block��� of the desired size. Bloom filter blocks and index blocks (we call
these ���inline blocks���) become interspersed with data blocks, and as a side
effect we can no longer rely on the difference between block offsets to
determine data block length, as it was done in version 1./para
+ paraHFile is a low-level file format by design, and it should not deal
with application-specific details such as Bloom filters, which are handled at
StoreFile level. Therefore, we call Bloom filter blocks in an HFile inline
blocks. We also supply HFile with an interface to write those inline blocks.
/para
+ paraAnother format modification aimed at reducing the region server
startup time is to use a contiguous ���load-on-open��� section that has to be
loaded in memory at the time an HFile is being opened. Currently, as an HFile
opens, there are separate seek operations to read the trailer, data/meta
indexes, and file info. To read the Bloom filter, there are two more seek
operations for its ���data��� and ���meta��� portions. In version 2, we seek
once to read the trailer and seek again to read everything else we need to open
the file from a contiguous block./para/section
+ paraThe version of HBase introducing the above features reads both
version 1 and 2 HFiles, but only writes version 2 HFiles. A version 2 HFile is
structured as follows:
+ para8 bytes: Block type, a sequence of bytes equivalent to version
1's magic records. Supported block types are: /para
+ INTERMEDIATE_INDEX ��� intermediate-level index blocks in
a multi-level blockindex
{color:green}+1 site{color}. The mvn site goal
[jira] [Commented] (HBASE-11729) Document HFile v3
[ https://issues.apache.org/jira/browse/HBASE-11729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105363#comment-14105363 ] Sean Busbey commented on HBASE-11729: - AFAICT, both the javadoc warnings and the test failures are unrelated to this patch. Document HFile v3 - Key: HBASE-11729 URL: https://issues.apache.org/jira/browse/HBASE-11729 Project: HBase Issue Type: Task Components: documentation, HFile Affects Versions: 0.98.0 Reporter: Sean Busbey Assignee: Sean Busbey Priority: Trivial Labels: beginner Attachments: HBASE-11729-v2.patch, HBASE-11729-v2.pdf, HBASE-11729.patch, HBASE-11729.pdf 0.98 added HFile v3. There are a couple of mentions of it in the book on the sections on cell tags, but there isn't an actual overview or design explanation like there is for [HFile v2|http://hbase.apache.org/book/hfilev2.html]. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HBASE-2739) Master should fail to start if it cannot successfully split logs
[ https://issues.apache.org/jira/browse/HBASE-2739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey reassigned HBASE-2739: -- Assignee: Sean Busbey Master should fail to start if it cannot successfully split logs Key: HBASE-2739 URL: https://issues.apache.org/jira/browse/HBASE-2739 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.20.4, 0.90.0 Reporter: Todd Lipcon Assignee: Sean Busbey Priority: Critical In trunk, in splitLogAfterStartup(), we log the error splitting, but don't shut down. Depending on configuration, we should probably shut down here rather than continue with dataloss. In 0.20, we print the stacktrace to stdout in verifyClusterState, but continue through and often fail to start up -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11787) TestRegionLocations is not categorized
[
https://issues.apache.org/jira/browse/HBASE-11787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105406#comment-14105406
]
Hadoop QA commented on HBASE-11787:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12663100/11787.v1.patch
against trunk revision .
ATTACHMENT ID: 12663100
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 7
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/10517//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10517//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10517//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10517//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10517//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10517//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10517//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10517//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10517//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10517//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10517//console
This message is automatically generated.
TestRegionLocations is not categorized
--
Key: HBASE-11787
URL: https://issues.apache.org/jira/browse/HBASE-11787
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.99.0, 2.0.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
Priority: Trivial
Fix For: 0.99.0, 2.0.0
Attachments: 11787.v1.patch
It's actually executed (likely with the large tests), but it's risky.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11679) Replace HTable with HTableInterface where backwards-compatible
[
https://issues.apache.org/jira/browse/HBASE-11679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105407#comment-14105407
]
Carter commented on HBASE-11679:
Enis, if you want, I'm happy to start work onTable interface as you've
described it in HBASE-10602.
Replace HTable with HTableInterface where backwards-compatible
--
Key: HBASE-11679
URL: https://issues.apache.org/jira/browse/HBASE-11679
Project: HBase
Issue Type: Improvement
Reporter: Carter
Assignee: Carter
Attachments: HBASE_11679.patch, HBASE_11679.patch,
HBASE_11679_v3.patch
This is a refactor to move more of the code towards using interfaces for
proper encapsulation of logic.
The amount of code touched is large, but it should be fairly easy to review.
It changes variable declarations from HTable to HTableInterface where the
following holds:
# The declaration being updated won't break assignment
# The declaration change does not break the compile (eg trying to access
non-interface methods)
The two main situations are to change something like this:
{code}
HTable h = new HTable(c, tn);
{code}
to
{code}
HTableInterface h = new HTable(c, tn);
{code}
and this:
{code}
public void doSomething(HTable h) { ... }
{code}
to this:
{code}
public void doSomething(HTableInterface h) { ... }
{code}
This gets most of the obvious cases out of the way and prepares for more
complicated interface refactors in the future. In method signatures, I
changed parameters, but did _not_ change any public or protected method
return values, since that would violate criteria #1 above and break
compatibility.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11591) Scanner fails to retrieve KV from bulk loaded file with highest sequence id than the cell's mvcc in a non-bulk loaded file
[
https://issues.apache.org/jira/browse/HBASE-11591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105428#comment-14105428
]
Ted Yu commented on HBASE-11591:
Minor:
{code}
-return !hasMVCCInfo ? true : skipKVsNewerThanReadpoint();
+if (!hasMVCCInfo this.reader.isBulkLoaded()) {
+ return skipKVsNewerThanReadpoint();
+} else {
+ return !hasMVCCInfo ? true : skipKVsNewerThanReadpoint();
{code}
The if condition above would be more readable if written this way:
{code}
+if (hasMVCCInfo || this.reader.isBulkLoaded()) {
{code}
Scanner fails to retrieve KV from bulk loaded file with highest sequence id
than the cell's mvcc in a non-bulk loaded file
---
Key: HBASE-11591
URL: https://issues.apache.org/jira/browse/HBASE-11591
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Critical
Fix For: 0.99.0
Attachments: HBASE-11591.patch, HBASE-11591_1.patch,
HBASE-11591_2.patch, HBASE-11591_3.patch, HBASE-11591_5.patch,
TestBulkload.java, hbase-11591-03-02.patch, hbase-11591-03-jeff.patch
See discussion in HBASE-11339.
When we have a case where there are same KVs in two files one produced by
flush/compaction and the other thro the bulk load.
Both the files have some same kvs which matches even in timestamp.
Steps:
Add some rows with a specific timestamp and flush the same.
Bulk load a file with the same data.. Enusre that assign seqnum property is
set.
The bulk load should use HFileOutputFormat2 (or ensure that we write the
bulk_time_output key).
This would ensure that the bulk loaded file has the highest seq num.
Assume the cell in the flushed/compacted store file is
row1,cf,cq,ts1, value1 and the cell in the bulk loaded file is
row1,cf,cq,ts1,value2
(There are no parallel scans).
Issue a scan on the table in 0.96. The retrieved value is
row1,cf1,cq,ts1,value2
But the same in 0.98 will retrieve row1,cf1,cq,ts2,value1.
This is a behaviour change. This is because of this code
{code}
public int compare(KeyValueScanner left, KeyValueScanner right) {
int comparison = compare(left.peek(), right.peek());
if (comparison != 0) {
return comparison;
} else {
// Since both the keys are exactly the same, we break the tie in favor
// of the key which came latest.
long leftSequenceID = left.getSequenceID();
long rightSequenceID = right.getSequenceID();
if (leftSequenceID rightSequenceID) {
return -1;
} else if (leftSequenceID rightSequenceID) {
return 1;
} else {
return 0;
}
}
}
{code}
Here in 0.96 case the mvcc of the cell in both the files will have 0 and so
the comparison will happen from the else condition . Where the seq id of the
bulk loaded file is greater and would sort out first ensuring that the scan
happens from that bulk loaded file.
In case of 0.98+ as we are retaining the mvcc+seqid we are not making the
mvcc as 0 (remains a non zero positive value). Hence the compare() sorts out
the cell in the flushed/compacted file. Which means though we know the
lateset file is the bulk loaded file we don't scan the data.
Seems to be a behaviour change. Will check on other corner cases also but we
are trying to know the behaviour of bulk load because we are evaluating if it
can be used for MOB design.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10602) Cleanup HTable public interface
[
https://issues.apache.org/jira/browse/HBASE-10602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105437#comment-14105437
]
Carter commented on HBASE-10602:
Wouldn't you want the async call to be the new signature, so it doesn't break
backwards compatibility? I guess it doesn't matter with put, but for
instance checkAndPut and other non-void methods:
{code}
public boolean checkAndPut(byte[], byte[], byte[], byte[], Put);
public FutureBoolean asyncCheckAndPut(byte[], byte[], byte[], byte[], Put);
{code}
This also means that you can grow/evolve the Table interface rather than
figuring out the future/async version in the first iteration.
Last point, following on Nick's assertion about object creation (where does the
~100k objects # come from btw?), you may not want to default the user to an
async call creates more garbage collection in the client, rather let them
decide if they need it.
Cleanup HTable public interface
---
Key: HBASE-10602
URL: https://issues.apache.org/jira/browse/HBASE-10602
Project: HBase
Issue Type: Improvement
Components: Client, Usability
Reporter: Nick Dimiduk
Assignee: Enis Soztutar
Priority: Blocker
Fix For: 0.99.0
Attachments: hbase-10602_v1.patch
HBASE-6580 replaced the preferred means of HTableInterface acquisition to the
HConnection#getTable factory methods. HBASE-9117 removes the HConnection
cache, placing the burden of responsible connection cleanup on whomever
acquires it.
The remaining HTable constructors use a Connection instance and manage their
own HConnection on the callers behalf. This is convenient but also a
surprising source of poor performance for anyone accustomed to the previous
connection caching behavior. I propose deprecating those remaining
constructors for 0.98/0.96 and removing them for 1.0.
While I'm at it, I suggest we pursue some API hygiene in general and convert
HTable into an interface. I'm sure there are method overloads for accepting
String/byte[]/TableName where just TableName is sufficient. Can that be done
for 1.0 as well?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-4955) Use the official versions of surefire junit
[ https://issues.apache.org/jira/browse/HBASE-4955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105518#comment-14105518 ] Hudson commented on HBASE-4955: --- FAILURE: Integrated in HBase-TRUNK #5417 (See [https://builds.apache.org/job/HBase-TRUNK/5417/]) HBASE-4955 Use the official versions of surefire (Alex Newman) (nkeywal: rev 018bbc09931d45afa349a76fcc72283013e1d3a2) * hbase-server/src/test/java/org/apache/hadoop/hbase/http/TestServletFilter.java * pom.xml Use the official versions of surefire junit - Key: HBASE-4955 URL: https://issues.apache.org/jira/browse/HBASE-4955 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0, 0.98.0, 0.96.0, 0.99.0 Environment: all Reporter: Nicolas Liochon Assignee: Alex Newman Priority: Critical Fix For: 2.0.0 Attachments: 4955.v1.patch, 4955.v2.patch, 4955.v2.patch, 4955.v2.patch, 4955.v2.patch, 4955.v3.patch, 4955.v3.patch, 4955.v3.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v5.patch, 4955.v6.patch, 4955.v7.patch, 4955.v7.patch, 4955.v8.patch, 4955.v9.patch, 8204.v4.patch, HBASE-4955-v10.patch, HBASE-4955-v10.patch, HBASE-4955-v10.patch We currently use private versions for Surefire JUnit since HBASE-4763. This JIRA traks what we need to move to official versions. Surefire 2.11 is just out, but, after some tests, it does not contain all what we need. JUnit. Could be for JUnit 4.11. Issue to monitor: https://github.com/KentBeck/junit/issues/359: fixed in our version, no feedback for an integration on trunk Surefire: Could be for Surefire 2.12. Issues to monitor are: 329 (category support): fixed, we use the official implementation from the trunk 786 (@Category with forkMode=always): fixed, we use the official implementation from the trunk 791 (incorrect elapsed time on test failure): fixed, we use the official implementation from the trunk 793 (incorrect time in the XML report): Not fixed (reopen) on trunk, fixed on our version. 760 (does not take into account the test method): fixed in trunk, not fixed in our version 798 (print immediately the test class name): not fixed in trunk, not fixed in our version 799 (Allow test parallelization when forkMode=always): not fixed in trunk, not fixed in our version 800 (redirectTestOutputToFile not taken into account): not yet fix on trunk, fixed on our version 800 793 are the more important to monitor, it's the only ones that are fixed in our version but not on trunk. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11760) Tighten up region state transition
[ https://issues.apache.org/jira/browse/HBASE-11760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-11760: Status: Open (was: Patch Available) Tighten up region state transition -- Key: HBASE-11760 URL: https://issues.apache.org/jira/browse/HBASE-11760 Project: HBase Issue Type: Improvement Components: Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 2.0.0 Attachments: hbase-11760.patch When a regionserver reports to master a region transition, we should check the current region state to be exactly what we expect. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11772) Bulk load mvcc and seqId issues with native hfiles
[
https://issues.apache.org/jira/browse/HBASE-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105591#comment-14105591
]
Jerry He commented on HBASE-11772:
--
The work being done at HBASE-11591 will be good for this JIRA as well for 0.99+
branches! I will work on top of that to avoid conflicts.
For what is being done there please refer to HBASE-11591's latest patches.
Bulk load mvcc and seqId issues with native hfiles
--
Key: HBASE-11772
URL: https://issues.apache.org/jira/browse/HBASE-11772
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.5
Reporter: Jerry He
Assignee: Jerry He
Priority: Critical
Fix For: 0.98.6
Attachments: HBASE-11772-0.98.patch
There are mvcc and seqId issues when bulk load native hfiles -- meaning
hfiles that are direct file copy-out from hbase, not from HFileOutputFormat
job.
There are differences between these two types of hfiles.
Native hfiles have possible non-zero MAX_MEMSTORE_TS_KEY value and non-zero
mvcc values in cells.
Native hfiles also have MAX_SEQ_ID_KEY.
Native hfiles do not have BULKLOAD_TIME_KEY.
Here are a couple of problems I observed when bulk load native hfiles.
1. Cells in newly bulk loaded hfiles can be invisible to scan.
It is easy to re-create.
Bulk load a native hfile that has a larger mvcc value in cells, e.g 10
If the current readpoint when initiating a scan is less than 10, the cells in
the new hfile are skipped, thus become invisible.
We don't reset the readpoint of a region after bulk load.
2. The current StoreFile.isBulkLoadResult() is implemented as:
{code}
return metadataMap.containsKey(BULKLOAD_TIME_KEY)
{code}
which does not detect bulkloaded native hfiles.
3. Another observed problem is possible data loss during log recovery.
It is similar to HBASE-10958 reported by [~jdcryans]. Borrow the re-create
steps from HBASE-10958.
1) Create an empty table
2) Put one row in it (let's say it gets seqid 1)
3) Bulk load one native hfile with large seqId ( e.g. 100). The native hfile
can be obtained by copying out from existing table.
4) Kill the region server that holds the table's region.
Scan the table once the region is made available again. The first row, at
seqid 1, will be missing since the HFile with seqid 100 makes us believe that
everything that came before it was flushed.
The problem 3 is probably related to 2. We will be ok if we get the appended
seqId during bulk load instead of 100 from inside the file.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11764) Support per cell TTLs
[ https://issues.apache.org/jira/browse/HBASE-11764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105609#comment-14105609 ] Andrew Purtell commented on HBASE-11764: bq. If we have to handle the case of per cell TTL cf TTL, we have to track the per cell TTLs and write down this meta in HFile so that file level early out optimizations can be continued I have more time to work on this. I will proceed a bit here and maybe put up a patch containing the additional changes needed for that, we can decide. Thanks for having a look. Support per cell TTLs - Key: HBASE-11764 URL: https://issues.apache.org/jira/browse/HBASE-11764 Project: HBase Issue Type: Sub-task Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.99.0, 0.98.6 Attachments: HBASE-11764.patch, HBASE-11764.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11795) Add TTL unit tests
Andrew Purtell created HBASE-11795: -- Summary: Add TTL unit tests Key: HBASE-11795 URL: https://issues.apache.org/jira/browse/HBASE-11795 Project: HBase Issue Type: Sub-task Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.99.0, 2.0.0, 0.98.6 Our coverage of TTLs in unit tests is scant. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11331) [blockcache] lazy block decompression
[
https://issues.apache.org/jira/browse/HBASE-11331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-11331:
-
Attachment: v03-20g-045g-true.pdf
v03-20g-045g-false.pdf
Sharing some more test results. These are running PerfEval randomRead test on a
single machine cluster deployment. The RS gets a 20g heap out of 24g available
on the box. Table is snappy compressed. The idea is to demonstrate a best-case
scenario for this patch: you have enough ram for the whole compressed data set,
but not after decompressed. The {{true}} run below shows some IO wait, so I may
have the math on the compression ratio slightly off.
Values in this table come from the average values reported on the attached
screen shots. I've chosen some of the more critical metrics, but it's all there
for reference. Let me know if there's a metric I'm missing, I can add it to the
report (if OpenTSDB collects it, that is).
||
||hbase.block.data.cachecompressed=false||hbase.block.data.cachecompressed=true||delta|
|hbase.regionserver.server.Get_num_ops|423|4.93k|{color:green}1065%{color}|
|hbase.regionserver.server.Get_mean|19.14 ms|1.00 ms|{color:green}-94%{color}|
|hbase.regionserver.server.Get_99th_percentile|182.58 ms|33.17
ms|{color:green}-81%{color}|
|hbase.regionserver.jvmmetrics.GcTimeMillis|27.73 ms|401.16
ms|{color:red}1346%{color}|
|proc.loadavg.1min|11.55|7.82|{color:green}-32%{color}|
|proc.stat.cpu.percpu{type=iowait}|358.43|211.83|{color:green}-40%{color}|
|hbase.regionserver.server.blockCacheCount|181.66k|722.55k|{color:green}297%{color}|
[blockcache] lazy block decompression
-
Key: HBASE-11331
URL: https://issues.apache.org/jira/browse/HBASE-11331
Project: HBase
Issue Type: Improvement
Components: regionserver
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
Attachments: HBASE-11331.00.patch, HBASE-11331.01.patch,
HBASE-11331.02.patch, HBASE-11331.03.patch,
HBASE-11331LazyBlockDecompressperfcompare.pdf, lazy-decompress.02.0.pdf,
lazy-decompress.02.1.json, lazy-decompress.02.1.pdf, v03-20g-045g-false.pdf,
v03-20g-045g-true.pdf
Maintaining data in its compressed form in the block cache will greatly
increase our effective blockcache size and should show a meaning improvement
in cache hit rates in well designed applications. The idea here is to lazily
decompress/decrypt blocks when they're consumed, rather than as soon as
they're pulled off of disk.
This is related to but less invasive than HBASE-8894.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11331) [blockcache] lazy block decompression
[ https://issues.apache.org/jira/browse/HBASE-11331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105622#comment-14105622 ] Nick Dimiduk commented on HBASE-11331: -- I'm now investigating those heart-beat looking GC spikes in the compressed=true graphs. [blockcache] lazy block decompression - Key: HBASE-11331 URL: https://issues.apache.org/jira/browse/HBASE-11331 Project: HBase Issue Type: Improvement Components: regionserver Reporter: Nick Dimiduk Assignee: Nick Dimiduk Attachments: HBASE-11331.00.patch, HBASE-11331.01.patch, HBASE-11331.02.patch, HBASE-11331.03.patch, HBASE-11331LazyBlockDecompressperfcompare.pdf, lazy-decompress.02.0.pdf, lazy-decompress.02.1.json, lazy-decompress.02.1.pdf, v03-20g-045g-false.pdf, v03-20g-045g-true.pdf Maintaining data in its compressed form in the block cache will greatly increase our effective blockcache size and should show a meaning improvement in cache hit rates in well designed applications. The idea here is to lazily decompress/decrypt blocks when they're consumed, rather than as soon as they're pulled off of disk. This is related to but less invasive than HBASE-8894. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11331) [blockcache] lazy block decompression
[ https://issues.apache.org/jira/browse/HBASE-11331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105617#comment-14105617 ] Nick Dimiduk commented on HBASE-11331: -- Previous comment is re: v03-20g-045g-false.pdf, v03-20g-045g-true.pdf [blockcache] lazy block decompression - Key: HBASE-11331 URL: https://issues.apache.org/jira/browse/HBASE-11331 Project: HBase Issue Type: Improvement Components: regionserver Reporter: Nick Dimiduk Assignee: Nick Dimiduk Attachments: HBASE-11331.00.patch, HBASE-11331.01.patch, HBASE-11331.02.patch, HBASE-11331.03.patch, HBASE-11331LazyBlockDecompressperfcompare.pdf, lazy-decompress.02.0.pdf, lazy-decompress.02.1.json, lazy-decompress.02.1.pdf, v03-20g-045g-false.pdf, v03-20g-045g-true.pdf Maintaining data in its compressed form in the block cache will greatly increase our effective blockcache size and should show a meaning improvement in cache hit rates in well designed applications. The idea here is to lazily decompress/decrypt blocks when they're consumed, rather than as soon as they're pulled off of disk. This is related to but less invasive than HBASE-8894. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11788) hbase is not deleting the cell when a Put with a KeyValue, KeyValue.Type.Delete is submitted
[
https://issues.apache.org/jira/browse/HBASE-11788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105625#comment-14105625
]
Andrew Purtell commented on HBASE-11788:
[~cristian_armaselu], please have a look at the attached unit test. The
supported way to create 'custom' delete markers is the commented out section
just before the final query:
{code}
// Uncomment this line to get the test to pass
// table.delete(new Delete(rowKey).addDeleteMarker(marker));
{code}
Can you point us to where Hive or other code is doing it differently? I think
we'd like to talk with the downstream project(s).
What is the impact to you if we don't support deletes-in-puts in the HBase API?
I.e. would you be able to patch the downstream client or successfully request
your vendor do so?
hbase is not deleting the cell when a Put with a KeyValue,
KeyValue.Type.Delete is submitted
Key: HBASE-11788
URL: https://issues.apache.org/jira/browse/HBASE-11788
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0, 0.96.1.1, 0.98.5, 2.0.0
Environment: Cloudera CDH 5.1.x
Reporter: Cristian Armaselu
Attachments: TestPutWithDelete.java
Code executed:
{code}
@Test
public void testHbasePutDeleteCell() throws Exception {
TableName tableName = TableName.valueOf(my_test);
Configuration configuration = HBaseConfiguration.create();
HTableInterface table = new HTable(configuration, tableName);
final String rowKey = 12345;
final byte[] familly = Bytes.toBytes(default);
// put one row
Put put = new Put(Bytes.toBytes(rowKey));
put.add(familly, Bytes.toBytes(A), Bytes.toBytes(a));
put.add(familly, Bytes.toBytes(B), Bytes.toBytes(b));
put.add(familly, Bytes.toBytes(C), Bytes.toBytes(c));
table.put(put);
// get row back and assert the values
Get get = new Get(Bytes.toBytes(rowKey));
Result result = table.get(get);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(A))).equals(a), Column A value should be a);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(B))).equals(b), Column B value should be b);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(C))).equals(c), Column C value should be c);
// put the same row again with C column deleted
put = new Put(Bytes.toBytes(rowKey));
put.add(familly, Bytes.toBytes(A), Bytes.toBytes(a));
put.add(familly, Bytes.toBytes(B), Bytes.toBytes(b));
put.add(new KeyValue(Bytes.toBytes(rowKey), familly,
Bytes.toBytes(C), HConstants.LATEST_TIMESTAMP, KeyValue.Type.DeleteColumn));
table.put(put);
// get row back and assert the values
get = new Get(Bytes.toBytes(rowKey));
result = table.get(get);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(A))).equals(a), Column A value should be a);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(B))).equals(b), Column A value should be b);
Assert.isTrue(result.getValue(familly, Bytes.toBytes(C)) == null,
Column C should not exists);
}
{code}
This assertion fails, the cell is not deleted but rather the value is empty:
{code}
hbase(main):029:0 scan 'my_test'
ROW COLUMN+CELL
12345column=default:A,
timestamp=1408473082290, value=a
12345column=default:B,
timestamp=1408473082290, value=b
12345column=default:C,
timestamp=1408473082290, value=
{code}
This behavior is different than previous 4.8.x Cloudera version and is
currently corrupting all hive queries involving is null or is not null
operators on the columns mapped to hbase
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11331) [blockcache] lazy block decompression
[ https://issues.apache.org/jira/browse/HBASE-11331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105635#comment-14105635 ] Vladimir Rodionov commented on HBASE-11331: --- The standard YCSB is very friendly to compressed block cache (especially, with zipfian data access pattern). Just to let you know, Nick. [blockcache] lazy block decompression - Key: HBASE-11331 URL: https://issues.apache.org/jira/browse/HBASE-11331 Project: HBase Issue Type: Improvement Components: regionserver Reporter: Nick Dimiduk Assignee: Nick Dimiduk Attachments: HBASE-11331.00.patch, HBASE-11331.01.patch, HBASE-11331.02.patch, HBASE-11331.03.patch, HBASE-11331LazyBlockDecompressperfcompare.pdf, lazy-decompress.02.0.pdf, lazy-decompress.02.1.json, lazy-decompress.02.1.pdf, v03-20g-045g-false.pdf, v03-20g-045g-true.pdf Maintaining data in its compressed form in the block cache will greatly increase our effective blockcache size and should show a meaning improvement in cache hit rates in well designed applications. The idea here is to lazily decompress/decrypt blocks when they're consumed, rather than as soon as they're pulled off of disk. This is related to but less invasive than HBASE-8894. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-4955) Use the official versions of surefire junit
[ https://issues.apache.org/jira/browse/HBASE-4955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105638#comment-14105638 ] Alex Newman commented on HBASE-4955: I am going to take a look at the zombie test, and see if i can make it better. Use the official versions of surefire junit - Key: HBASE-4955 URL: https://issues.apache.org/jira/browse/HBASE-4955 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0, 0.98.0, 0.96.0, 0.99.0 Environment: all Reporter: Nicolas Liochon Assignee: Alex Newman Priority: Critical Fix For: 2.0.0 Attachments: 4955.v1.patch, 4955.v2.patch, 4955.v2.patch, 4955.v2.patch, 4955.v2.patch, 4955.v3.patch, 4955.v3.patch, 4955.v3.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v5.patch, 4955.v6.patch, 4955.v7.patch, 4955.v7.patch, 4955.v8.patch, 4955.v9.patch, 8204.v4.patch, HBASE-4955-v10.patch, HBASE-4955-v10.patch, HBASE-4955-v10.patch We currently use private versions for Surefire JUnit since HBASE-4763. This JIRA traks what we need to move to official versions. Surefire 2.11 is just out, but, after some tests, it does not contain all what we need. JUnit. Could be for JUnit 4.11. Issue to monitor: https://github.com/KentBeck/junit/issues/359: fixed in our version, no feedback for an integration on trunk Surefire: Could be for Surefire 2.12. Issues to monitor are: 329 (category support): fixed, we use the official implementation from the trunk 786 (@Category with forkMode=always): fixed, we use the official implementation from the trunk 791 (incorrect elapsed time on test failure): fixed, we use the official implementation from the trunk 793 (incorrect time in the XML report): Not fixed (reopen) on trunk, fixed on our version. 760 (does not take into account the test method): fixed in trunk, not fixed in our version 798 (print immediately the test class name): not fixed in trunk, not fixed in our version 799 (Allow test parallelization when forkMode=always): not fixed in trunk, not fixed in our version 800 (redirectTestOutputToFile not taken into account): not yet fix on trunk, fixed on our version 800 793 are the more important to monitor, it's the only ones that are fixed in our version but not on trunk. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-8298) desc tablename shorthand for describe tablename, similar to how databases have
[ https://issues.apache.org/jira/browse/HBASE-8298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-8298: --- Attachment: HBASE-8298-v3.patch Attaching alternate implementation that adds a general approach to aliasing commands (something we can reuse when maintaining deprecated command names). desc tablename shorthand for describe tablename, similar to how databases have -- Key: HBASE-8298 URL: https://issues.apache.org/jira/browse/HBASE-8298 Project: HBase Issue Type: Improvement Components: shell Affects Versions: 0.94.2 Reporter: Hari Sekhon Assignee: Sean Busbey Priority: Trivial Attachments: HBASE-8298-v3.patch, desc.patch, desc2.patch It would be nice if you could type desc tablename in hbase shell as shorthand for describe tablename similar to how you can in traditional databases. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-8298) desc tablename shorthand for describe tablename, similar to how databases have
[ https://issues.apache.org/jira/browse/HBASE-8298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-8298: --- Status: Patch Available (was: Reopened) desc tablename shorthand for describe tablename, similar to how databases have -- Key: HBASE-8298 URL: https://issues.apache.org/jira/browse/HBASE-8298 Project: HBase Issue Type: Improvement Components: shell Affects Versions: 0.94.2 Reporter: Hari Sekhon Assignee: Sean Busbey Priority: Trivial Attachments: HBASE-8298-v3.patch, desc.patch, desc2.patch It would be nice if you could type desc tablename in hbase shell as shorthand for describe tablename similar to how you can in traditional databases. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-8298) desc tablename shorthand for describe tablename, similar to how databases have
[ https://issues.apache.org/jira/browse/HBASE-8298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105651#comment-14105651 ] Sean Busbey commented on HBASE-8298: Leaves describing aliases to the original command, on the presumption that in some cases we won't want to call attention to them. Tested by spinning up single node, trying both 'describe' and 'desc' on a table I created. Ran through help, help 'ddl', help 'describe', and help 'desc'. Make sure desc didn't show up, save in the description of the describe command, but help 'desc' still worked. desc tablename shorthand for describe tablename, similar to how databases have -- Key: HBASE-8298 URL: https://issues.apache.org/jira/browse/HBASE-8298 Project: HBase Issue Type: Improvement Components: shell Affects Versions: 0.94.2 Reporter: Hari Sekhon Assignee: Sean Busbey Priority: Trivial Attachments: HBASE-8298-v3.patch, desc.patch, desc2.patch It would be nice if you could type desc tablename in hbase shell as shorthand for describe tablename similar to how you can in traditional databases. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11729) Document HFile v3
[ https://issues.apache.org/jira/browse/HBASE-11729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105656#comment-14105656 ] Andrew Purtell commented on HBASE-11729: I think we can ignore Jenkins in lieu of cut-and-paste of a successful conclusion of a local 'mvn site'. Document HFile v3 - Key: HBASE-11729 URL: https://issues.apache.org/jira/browse/HBASE-11729 Project: HBase Issue Type: Task Components: documentation, HFile Affects Versions: 0.98.0 Reporter: Sean Busbey Assignee: Sean Busbey Priority: Trivial Labels: beginner Attachments: HBASE-11729-v2.patch, HBASE-11729-v2.pdf, HBASE-11729.patch, HBASE-11729.pdf 0.98 added HFile v3. There are a couple of mentions of it in the book on the sections on cell tags, but there isn't an actual overview or design explanation like there is for [HFile v2|http://hbase.apache.org/book/hfilev2.html]. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11591) Scanner fails to retrieve KV from bulk loaded file with highest sequence id than the cell's mvcc in a non-bulk loaded file
[
https://issues.apache.org/jira/browse/HBASE-11591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105659#comment-14105659
]
Jeffrey Zhong commented on HBASE-11591:
---
[~ram_krish] {quote}
I think the change in HRegion.java to set the write Sequence number is a bug
fix? I still feel the patch would cause issue in the following scenario after
the above changes
- Assume a scan started and the read point is 20 at that time
- Bulk load is just getting completed and the scanner heap gets reset. The new
bulk loaded file with seqId 22 (for eg) gets added now to the scanner heap. But
remember that the read point is still 20.
- After this change we would just set the bulk load file's seqId to all its
KVs which is 22.
- Because there is no mvcc info in this bulk loaded file the scan would not be
able to skipTheKvsWithNewerReadPt() and hence the scan would still see the Kvs
with 22 as the seqId though the intention is to see only KVs with seqID 20.
I may be wrong. Am I missing something here? I may be wrong because for bulk
loaded files because there is no mvcc we are allowed to read anything in that
irrespective of the read pt?
{quote}
The situation above is valid. While existing behavior(like 0.98), we allow a
scan with lower readpt to read a bulk loaded file immediately as we can load a
hfile atomically. I think it's fine either keeping existing behavior or add
handling for such cases. Another option to handle such case you can set
hasMVCCInfo to true for a bulk loaded file because we will set its KVs' mvcc
using Hfile seqId.
[~jerryhe] {quote}It can be used backup HBase data and restore.{quote}
For such case, trunk code can handle it but you need to keep deleted cells
mvcc forever(using config hbase.hstore.compaction.keep.mvcc.period). When you
load a old hfile, its KVs will be sorted correctly based on their mvcc
values(LogSeqId).
Scanner fails to retrieve KV from bulk loaded file with highest sequence id
than the cell's mvcc in a non-bulk loaded file
---
Key: HBASE-11591
URL: https://issues.apache.org/jira/browse/HBASE-11591
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Critical
Fix For: 0.99.0
Attachments: HBASE-11591.patch, HBASE-11591_1.patch,
HBASE-11591_2.patch, HBASE-11591_3.patch, HBASE-11591_5.patch,
TestBulkload.java, hbase-11591-03-02.patch, hbase-11591-03-jeff.patch
See discussion in HBASE-11339.
When we have a case where there are same KVs in two files one produced by
flush/compaction and the other thro the bulk load.
Both the files have some same kvs which matches even in timestamp.
Steps:
Add some rows with a specific timestamp and flush the same.
Bulk load a file with the same data.. Enusre that assign seqnum property is
set.
The bulk load should use HFileOutputFormat2 (or ensure that we write the
bulk_time_output key).
This would ensure that the bulk loaded file has the highest seq num.
Assume the cell in the flushed/compacted store file is
row1,cf,cq,ts1, value1 and the cell in the bulk loaded file is
row1,cf,cq,ts1,value2
(There are no parallel scans).
Issue a scan on the table in 0.96. The retrieved value is
row1,cf1,cq,ts1,value2
But the same in 0.98 will retrieve row1,cf1,cq,ts2,value1.
This is a behaviour change. This is because of this code
{code}
public int compare(KeyValueScanner left, KeyValueScanner right) {
int comparison = compare(left.peek(), right.peek());
if (comparison != 0) {
return comparison;
} else {
// Since both the keys are exactly the same, we break the tie in favor
// of the key which came latest.
long leftSequenceID = left.getSequenceID();
long rightSequenceID = right.getSequenceID();
if (leftSequenceID rightSequenceID) {
return -1;
} else if (leftSequenceID rightSequenceID) {
return 1;
} else {
return 0;
}
}
}
{code}
Here in 0.96 case the mvcc of the cell in both the files will have 0 and so
the comparison will happen from the else condition . Where the seq id of the
bulk loaded file is greater and would sort out first ensuring that the scan
happens from that bulk loaded file.
In case of 0.98+ as we are retaining the mvcc+seqid we are not making the
mvcc as 0 (remains a non zero positive value). Hence the compare() sorts out
the cell in the flushed/compacted file. Which means though we know the
lateset file is the bulk loaded file we don't scan the data.
Seems to be a behaviour change. Will check on other corner cases
[jira] [Commented] (HBASE-11763) Move TTL handling into ScanQueryMatcher
[ https://issues.apache.org/jira/browse/HBASE-11763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105660#comment-14105660 ] Andrew Purtell commented on HBASE-11763: bq. I cannot remember the JIRA now, but there were some cases where users used filters with multi versions and the general doubt they had was whether the versions check happen first or the filter gets applied. That was us. :-) HBASE-10854 . The unit tests from that change still work after the change here. Move TTL handling into ScanQueryMatcher --- Key: HBASE-11763 URL: https://issues.apache.org/jira/browse/HBASE-11763 Project: HBase Issue Type: Sub-task Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.99.0, 2.0.0, 0.98.6 Attachments: HBASE-11763.patch, HBASE-11763.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11679) Replace HTable with HTableInterface where backwards-compatible
[
https://issues.apache.org/jira/browse/HBASE-11679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105696#comment-14105696
]
Enis Soztutar commented on HBASE-11679:
---
That would be awesome, I have my hands full on other stuff right now.
Replace HTable with HTableInterface where backwards-compatible
--
Key: HBASE-11679
URL: https://issues.apache.org/jira/browse/HBASE-11679
Project: HBase
Issue Type: Improvement
Reporter: Carter
Assignee: Carter
Attachments: HBASE_11679.patch, HBASE_11679.patch,
HBASE_11679_v3.patch
This is a refactor to move more of the code towards using interfaces for
proper encapsulation of logic.
The amount of code touched is large, but it should be fairly easy to review.
It changes variable declarations from HTable to HTableInterface where the
following holds:
# The declaration being updated won't break assignment
# The declaration change does not break the compile (eg trying to access
non-interface methods)
The two main situations are to change something like this:
{code}
HTable h = new HTable(c, tn);
{code}
to
{code}
HTableInterface h = new HTable(c, tn);
{code}
and this:
{code}
public void doSomething(HTable h) { ... }
{code}
to this:
{code}
public void doSomething(HTableInterface h) { ... }
{code}
This gets most of the obvious cases out of the way and prepares for more
complicated interface refactors in the future. In method signatures, I
changed parameters, but did _not_ change any public or protected method
return values, since that would violate criteria #1 above and break
compatibility.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HBASE-11796) Add client support for atomic checkAndMutate
Danny Purcell created HBASE-11796: - Summary: Add client support for atomic checkAndMutate Key: HBASE-11796 URL: https://issues.apache.org/jira/browse/HBASE-11796 Project: HBase Issue Type: Improvement Components: Client Reporter: Danny Purcell Currently HBase has support for atomic checkAndPut as well as checkAndDelete operations on individual rows. HBase also supports the atomic submission of multiple operations through mutateRow. It would be nice to support atomic checkAndMutate(RowMutations) as well. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11797) Create Table interface to replace HTableInterface
Carter created HBASE-11797: -- Summary: Create Table interface to replace HTableInterface Key: HBASE-11797 URL: https://issues.apache.org/jira/browse/HBASE-11797 Project: HBase Issue Type: Bug Reporter: Carter -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11679) Replace HTable with HTableInterface where backwards-compatible
[
https://issues.apache.org/jira/browse/HBASE-11679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105714#comment-14105714
]
Carter commented on HBASE-11679:
Created HBASE-11797 and starting discussion over there.
Replace HTable with HTableInterface where backwards-compatible
--
Key: HBASE-11679
URL: https://issues.apache.org/jira/browse/HBASE-11679
Project: HBase
Issue Type: Improvement
Reporter: Carter
Assignee: Carter
Attachments: HBASE_11679.patch, HBASE_11679.patch,
HBASE_11679_v3.patch
This is a refactor to move more of the code towards using interfaces for
proper encapsulation of logic.
The amount of code touched is large, but it should be fairly easy to review.
It changes variable declarations from HTable to HTableInterface where the
following holds:
# The declaration being updated won't break assignment
# The declaration change does not break the compile (eg trying to access
non-interface methods)
The two main situations are to change something like this:
{code}
HTable h = new HTable(c, tn);
{code}
to
{code}
HTableInterface h = new HTable(c, tn);
{code}
and this:
{code}
public void doSomething(HTable h) { ... }
{code}
to this:
{code}
public void doSomething(HTableInterface h) { ... }
{code}
This gets most of the obvious cases out of the way and prepares for more
complicated interface refactors in the future. In method signatures, I
changed parameters, but did _not_ change any public or protected method
return values, since that would violate criteria #1 above and break
compatibility.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11797) Create Table interface to replace HTableInterface
[ https://issues.apache.org/jira/browse/HBASE-11797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105723#comment-14105723 ] Jean-Marc Spaggiari commented on HBASE-11797: - Hi [~carterpage], Can you please share more background on this? Thanks. Create Table interface to replace HTableInterface - Key: HBASE-11797 URL: https://issues.apache.org/jira/browse/HBASE-11797 Project: HBase Issue Type: Bug Reporter: Carter -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HBASE-11797) Create Table interface to replace HTableInterface
[
https://issues.apache.org/jira/browse/HBASE-11797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carter reassigned HBASE-11797:
--
Assignee: Carter
Create Table interface to replace HTableInterface
-
Key: HBASE-11797
URL: https://issues.apache.org/jira/browse/HBASE-11797
Project: HBase
Issue Type: Bug
Reporter: Carter
Assignee: Carter
Basically doing this:
{code}
interface Table {
// get, put related stuff
}
@Deprecated
interface HTableInterface extends Table {
// users are encouraged to use the new Table interface
}
class HTable extends Table {
// all HTable constructors are deprecated
// Users are not encouraged to see this class
}
{code}
I'm proposing that in this JIRA I move everything from HTableInterface to
Table except the following:
* Anything deprecated
* Anything @InterfaceAudience.Private ({{coprocessorService(...)}} and
{{batchCoprocessorService(...)}})
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11797) Create Table interface to replace HTableInterface
[
https://issues.apache.org/jira/browse/HBASE-11797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Carter updated HBASE-11797:
---
Description:
Basically doing this:
{code}
interface Table {
// get, put related stuff
}
@Deprecated
interface HTableInterface extends Table {
// users are encouraged to use the new Table interface
}
class HTable extends Table {
// all HTable constructors are deprecated
// Users are not encouraged to see this class
}
{code}
I'm proposing that in this JIRA I move everything from HTableInterface to Table
except the following:
* Anything deprecated
* Anything @InterfaceAudience.Private ({{coprocessorService(...)}} and
{{batchCoprocessorService(...)}})
Create Table interface to replace HTableInterface
-
Key: HBASE-11797
URL: https://issues.apache.org/jira/browse/HBASE-11797
Project: HBase
Issue Type: Bug
Reporter: Carter
Basically doing this:
{code}
interface Table {
// get, put related stuff
}
@Deprecated
interface HTableInterface extends Table {
// users are encouraged to use the new Table interface
}
class HTable extends Table {
// all HTable constructors are deprecated
// Users are not encouraged to see this class
}
{code}
I'm proposing that in this JIRA I move everything from HTableInterface to
Table except the following:
* Anything deprecated
* Anything @InterfaceAudience.Private ({{coprocessorService(...)}} and
{{batchCoprocessorService(...)}})
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11786) Document web UI for tracking time spent in coprocessors
[ https://issues.apache.org/jira/browse/HBASE-11786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Srikanth Srungarapu updated HBASE-11786: Attachment: coprocessor_stats.png [~misty] Attaching the screenshot. Let me know if this works. Document web UI for tracking time spent in coprocessors --- Key: HBASE-11786 URL: https://issues.apache.org/jira/browse/HBASE-11786 Project: HBase Issue Type: Sub-task Components: Coprocessors, documentation Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Priority: Minor Attachments: HBASE-11786.patch, coprocessor_stats.png -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11797) Create Table interface to replace HTableInterface
[
https://issues.apache.org/jira/browse/HBASE-11797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105741#comment-14105741
]
Carter commented on HBASE-11797:
Description added that spells out my straw man.
Create Table interface to replace HTableInterface
-
Key: HBASE-11797
URL: https://issues.apache.org/jira/browse/HBASE-11797
Project: HBase
Issue Type: Bug
Reporter: Carter
Assignee: Carter
Basically doing this:
{code}
interface Table {
// get, put related stuff
}
@Deprecated
interface HTableInterface extends Table {
// users are encouraged to use the new Table interface
}
class HTable extends Table {
// all HTable constructors are deprecated
// Users are not encouraged to see this class
}
{code}
I'm proposing that in this JIRA I move everything from HTableInterface to
Table except the following:
* Anything deprecated
* Anything @InterfaceAudience.Private ({{coprocessorService(...)}} and
{{batchCoprocessorService(...)}})
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11789) LoadIncrementalHFiles is not picking up the -D option
[ https://issues.apache.org/jira/browse/HBASE-11789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105762#comment-14105762 ] Ishan Chhabra commented on HBASE-11789: --- Can you apply this to 0.96 and 0.94 head as well? LoadIncrementalHFiles is not picking up the -D option -- Key: HBASE-11789 URL: https://issues.apache.org/jira/browse/HBASE-11789 Project: HBase Issue Type: Bug Affects Versions: 0.99.0, 0.98.5, 2.0.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 0.99.0, 2.0.0, 0.98.6 Attachments: HBASE-11789-v0.patch LoadIncrementalHFiles is not using the Tool class correctly, preventing to use the -D options. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-7767) Get rid of ZKTable, and table enable/disable state in ZK
[
https://issues.apache.org/jira/browse/HBASE-7767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105758#comment-14105758
]
Hadoop QA commented on HBASE-7767:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12663365/HBASE-7767.patch
against trunk revision .
ATTACHMENT ID: 12663365
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 22 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
11 warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ \030\003 \001(\003\022\021\n\tis_shared\030\004
\001(\010\022\017\n\007purpose\030\005 \001 +
+ public MasterProtos.GetTableStateResponse getTableState(RpcController
controller, MasterProtos.GetTableStateRequest request) throws ServiceException {
+ public boolean setTableStateIfInStates(TableName tableName, TableState.State
newState, TableState.State... states) throws IOException {
+ public boolean setTableStateIfNotInStates(TableName tableName,
TableState.State newState, TableState.State... states) throws IOException {
+ !regionStates.isRegionInState(region,
org.apache.hadoop.hbase.master.RegionState.State.FAILED_CLOSE)) {
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/10518//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10518//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10518//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10518//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10518//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10518//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10518//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10518//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10518//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10518//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10518//console
This message is automatically generated.
Get rid of ZKTable, and table enable/disable state in ZK
-
Key: HBASE-7767
URL: https://issues.apache.org/jira/browse/HBASE-7767
Project: HBase
Issue Type: Sub-task
Components: Zookeeper
Affects Versions: 2.0.0
Reporter: Enis Soztutar
Assignee: Andrey Stepachev
Attachments: HBASE-7767.patch
As discussed table state in zookeeper for enable/disable state breaks our
zookeeper contract. It is also very intrusive, used from the client side,
master and region servers. We should get rid of it.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HBASE-11798) TestBucketWriterThread can zombie test
Alex Newman created HBASE-11798: --- Summary: TestBucketWriterThread can zombie test Key: HBASE-11798 URL: https://issues.apache.org/jira/browse/HBASE-11798 Project: HBase Issue Type: Bug Reporter: Alex Newman Assignee: Alex Newman -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11798) TestBucketWriterThread can zombie test
[
https://issues.apache.org/jira/browse/HBASE-11798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105794#comment-14105794
]
Alex Newman commented on HBASE-11798:
-
I noticed when trying to commit HBASE-4955 that TestBucketWriterThread can
zombie in its setup function.
I also noticed
main prio=10 tid=0x7fa9e000a800 nid=0x571 waiting on condition
[0x7fa9e6184000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.hbase.util.Threads.sleep(Threads.java:143)
at
org.apache.hadoop.hbase.io.hfile.bucket.TestBucketWriterThread.setUp(TestBucketWriterThread.java:78)
In the jstack at
https://builds.apache.org/job/PreCommit-HBASE-Build/10511/console
Looking at the code
this.plainCacheable = Mockito.mock(Cacheable.class);
bc.cacheBlock(this.plainKey, plainCacheable);
while(!bc.ramCache.isEmpty()) Threads.sleep(1); - where we hang
assertTrue(q.isEmpty());
// Now writer thread should be disabled.
At first I was confused but then I realized that isn't Thread.sleep it is
Threads.sleep
/**
* If interrupted, just prints out the interrupt on STDOUT, resets interrupt
and returns
* @param millis How long to sleep for in milliseconds.
*/
public static void sleep(long millis) {
try {
Thread.sleep(millis);
} catch (InterruptedException e) {
e.printStackTrace();
Thread.currentThread().interrupt();
}
}
I don't know if we need this. I am curious if we can fix it with a different
sleep command.
TestBucketWriterThread can zombie test
--
Key: HBASE-11798
URL: https://issues.apache.org/jira/browse/HBASE-11798
Project: HBase
Issue Type: Bug
Reporter: Alex Newman
Assignee: Alex Newman
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11789) LoadIncrementalHFiles is not picking up the -D option
[ https://issues.apache.org/jira/browse/HBASE-11789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105793#comment-14105793 ] Andrew Purtell commented on HBASE-11789: bq. Can you apply this to 0.96 and 0.94 head as well? 0.96 is EOL I believe. We could port back to 0.94. [~lhofhansl] ? LoadIncrementalHFiles is not picking up the -D option -- Key: HBASE-11789 URL: https://issues.apache.org/jira/browse/HBASE-11789 Project: HBase Issue Type: Bug Affects Versions: 0.99.0, 0.98.5, 2.0.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 0.99.0, 2.0.0, 0.98.6 Attachments: HBASE-11789-v0.patch LoadIncrementalHFiles is not using the Tool class correctly, preventing to use the -D options. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11798) TestBucketWriterThread can zombie test
[ https://issues.apache.org/jira/browse/HBASE-11798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Newman updated HBASE-11798: Status: Patch Available (was: Open) TestBucketWriterThread can zombie test -- Key: HBASE-11798 URL: https://issues.apache.org/jira/browse/HBASE-11798 Project: HBase Issue Type: Bug Reporter: Alex Newman Assignee: Alex Newman -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Work started] (HBASE-11798) TestBucketWriterThread can zombie test
[ https://issues.apache.org/jira/browse/HBASE-11798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-11798 started by Alex Newman. TestBucketWriterThread can zombie test -- Key: HBASE-11798 URL: https://issues.apache.org/jira/browse/HBASE-11798 Project: HBase Issue Type: Bug Reporter: Alex Newman Assignee: Alex Newman -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Work stopped] (HBASE-11798) TestBucketWriterThread can zombie test
[ https://issues.apache.org/jira/browse/HBASE-11798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-11798 stopped by Alex Newman. TestBucketWriterThread can zombie test -- Key: HBASE-11798 URL: https://issues.apache.org/jira/browse/HBASE-11798 Project: HBase Issue Type: Bug Reporter: Alex Newman Assignee: Alex Newman -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11798) TestBucketWriterThread can zombie test
[ https://issues.apache.org/jira/browse/HBASE-11798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Newman updated HBASE-11798: Attachment: HBASE-11798-v1.patch TestBucketWriterThread can zombie test -- Key: HBASE-11798 URL: https://issues.apache.org/jira/browse/HBASE-11798 Project: HBase Issue Type: Bug Reporter: Alex Newman Assignee: Alex Newman Attachments: HBASE-11798-v1.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-7782) HBaseTestingUtility.truncateTable() not acting like CLI
[ https://issues.apache.org/jira/browse/HBASE-7782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-7782: --- Attachment: HBASE-7782.patch patch: * preserves current behavior in deleteTableData method * updates existing tests to use deleteTableData * adds truncateTable methods that behave like the shell. HBaseTestingUtility.truncateTable() not acting like CLI --- Key: HBASE-7782 URL: https://issues.apache.org/jira/browse/HBASE-7782 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.3 Reporter: Adrien Mogenet Assignee: Sean Busbey Priority: Minor Labels: cli, hbasetest, test Attachments: HBASE-7782.patch I would like to discuss the behavior of the truncateTable() method of HBaseTestingUtility. It's currently only removing the data through a scan/delete pattern. However, the truncate command in CLI is doing additional things: it disables the tables, drop, creates (with similar column descriptors) and then enables the table. I think the truncateTable() method is misleading; for example I used it to force a coprocessor to be reloaded, but it did not. Of course I can disable and enable the table by myself within my unit test, but perhaps it deserves to be discussed? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-8298) desc tablename shorthand for describe tablename, similar to how databases have
[ https://issues.apache.org/jira/browse/HBASE-8298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105802#comment-14105802 ] Andrew Purtell commented on HBASE-8298: --- +1, clean, nice desc tablename shorthand for describe tablename, similar to how databases have -- Key: HBASE-8298 URL: https://issues.apache.org/jira/browse/HBASE-8298 Project: HBase Issue Type: Improvement Components: shell Affects Versions: 0.94.2 Reporter: Hari Sekhon Assignee: Sean Busbey Priority: Trivial Attachments: HBASE-8298-v3.patch, desc.patch, desc2.patch It would be nice if you could type desc tablename in hbase shell as shorthand for describe tablename similar to how you can in traditional databases. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-7782) HBaseTestingUtility.truncateTable() not acting like CLI
[ https://issues.apache.org/jira/browse/HBASE-7782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105803#comment-14105803 ] Matteo Bertozzi commented on HBASE-7782: +1 looks good to me HBaseTestingUtility.truncateTable() not acting like CLI --- Key: HBASE-7782 URL: https://issues.apache.org/jira/browse/HBASE-7782 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.3 Reporter: Adrien Mogenet Assignee: Sean Busbey Priority: Minor Labels: cli, hbasetest, test Attachments: HBASE-7782.patch I would like to discuss the behavior of the truncateTable() method of HBaseTestingUtility. It's currently only removing the data through a scan/delete pattern. However, the truncate command in CLI is doing additional things: it disables the tables, drop, creates (with similar column descriptors) and then enables the table. I think the truncateTable() method is misleading; for example I used it to force a coprocessor to be reloaded, but it did not. Of course I can disable and enable the table by myself within my unit test, but perhaps it deserves to be discussed? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-7782) HBaseTestingUtility.truncateTable() not acting like CLI
[ https://issues.apache.org/jira/browse/HBASE-7782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-7782: --- Status: Patch Available (was: Open) HBaseTestingUtility.truncateTable() not acting like CLI --- Key: HBASE-7782 URL: https://issues.apache.org/jira/browse/HBASE-7782 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.3 Reporter: Adrien Mogenet Assignee: Sean Busbey Priority: Minor Labels: cli, hbasetest, test Attachments: HBASE-7782.patch I would like to discuss the behavior of the truncateTable() method of HBaseTestingUtility. It's currently only removing the data through a scan/delete pattern. However, the truncate command in CLI is doing additional things: it disables the tables, drop, creates (with similar column descriptors) and then enables the table. I think the truncateTable() method is misleading; for example I used it to force a coprocessor to be reloaded, but it did not. Of course I can disable and enable the table by myself within my unit test, but perhaps it deserves to be discussed? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (HBASE-8298) desc tablename shorthand for describe tablename, similar to how databases have
[ https://issues.apache.org/jira/browse/HBASE-8298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105802#comment-14105802 ] Andrew Purtell edited comment on HBASE-8298 at 8/21/14 7:06 PM: +1, clean, nice Edit: Referring to v3 patch was (Author: apurtell): +1, clean, nice desc tablename shorthand for describe tablename, similar to how databases have -- Key: HBASE-8298 URL: https://issues.apache.org/jira/browse/HBASE-8298 Project: HBase Issue Type: Improvement Components: shell Affects Versions: 0.94.2 Reporter: Hari Sekhon Assignee: Sean Busbey Priority: Trivial Attachments: HBASE-8298-v3.patch, desc.patch, desc2.patch It would be nice if you could type desc tablename in hbase shell as shorthand for describe tablename similar to how you can in traditional databases. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-8298) desc tablename shorthand for describe tablename, similar to how databases have
[ https://issues.apache.org/jira/browse/HBASE-8298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105810#comment-14105810 ] Nick Dimiduk commented on HBASE-8298: - Hurray for aliasing! +1 desc tablename shorthand for describe tablename, similar to how databases have -- Key: HBASE-8298 URL: https://issues.apache.org/jira/browse/HBASE-8298 Project: HBase Issue Type: Improvement Components: shell Affects Versions: 0.94.2 Reporter: Hari Sekhon Assignee: Sean Busbey Priority: Trivial Attachments: HBASE-8298-v3.patch, desc.patch, desc2.patch It would be nice if you could type desc tablename in hbase shell as shorthand for describe tablename similar to how you can in traditional databases. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11797) Create Table interface to replace HTableInterface
[
https://issues.apache.org/jira/browse/HBASE-11797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105818#comment-14105818
]
Carter commented on HBASE-11797:
Also, review the issue links, especially HBASE-10602 for full context.
Create Table interface to replace HTableInterface
-
Key: HBASE-11797
URL: https://issues.apache.org/jira/browse/HBASE-11797
Project: HBase
Issue Type: Bug
Reporter: Carter
Assignee: Carter
Basically doing this:
{code}
interface Table {
// get, put related stuff
}
@Deprecated
interface HTableInterface extends Table {
// users are encouraged to use the new Table interface
}
class HTable extends Table {
// all HTable constructors are deprecated
// Users are not encouraged to see this class
}
{code}
I'm proposing that in this JIRA I move everything from HTableInterface to
Table except the following:
* Anything deprecated
* Anything @InterfaceAudience.Private ({{coprocessorService(...)}} and
{{batchCoprocessorService(...)}})
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-9746) RegionServer can't start when replication tries to replicate to an unknown host
[
https://issues.apache.org/jira/browse/HBASE-9746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-9746:
-
Fix Version/s: 0.94.24
0.98.7
2.0.0
0.99.0
RegionServer can't start when replication tries to replicate to an unknown
host
---
Key: HBASE-9746
URL: https://issues.apache.org/jira/browse/HBASE-9746
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.12
Reporter: Lars Hofhansl
Priority: Minor
Fix For: 0.99.0, 2.0.0, 0.98.7, 0.94.24
Just ran into this:
{code}
13/10/11 00:37:02 [regionserver60020] WARN zookeeper.ZKConfig(204):
java.net.UnknownHostException: old-host: Name or service not known
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:894)
at
java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1286)
at java.net.InetAddress.getAllByName0(InetAddress.java:1239)
at java.net.InetAddress.getAllByName(InetAddress.java:1155)
at java.net.InetAddress.getAllByName(InetAddress.java:1091)
at java.net.InetAddress.getByName(InetAddress.java:1041)
at
org.apache.hadoop.hbase.zookeeper.ZKConfig.getZKQuorumServersString(ZKConfig.java:201)
at
org.apache.hadoop.hbase.zookeeper.ZKConfig.getZKQuorumServersString(ZKConfig.java:245)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.init(ZooKeeperWatcher.java:147)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.init(ZooKeeperWatcher.java:127)
at
org.apache.hadoop.hbase.replication.ReplicationPeer.reloadZkWatcher(ReplicationPeer.java:170)
at
org.apache.hadoop.hbase.replication.ReplicationPeer.init(ReplicationPeer.java:69)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.getPeer(ReplicationZookeeper.java:343)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.connectToPeer(ReplicationZookeeper.java:308)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.connectExistingPeers(ReplicationZookeeper.java:189)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.init(ReplicationZookeeper.java:156)
at
org.apache.hadoop.hbase.replication.regionserver.Replication.initialize(Replication.java:89)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.newReplicationInstance(HRegionServer.java:3986)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.createNewReplicationInstance(HRegionServer.java:3955)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.setupWALAndReplication(HRegionServer.java:1412)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1096)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:749)
at java.lang.Thread.run(Thread.java:722)
13/10/11 00:37:02 [regionserver60020] ERROR zookeeper.ZKConfig(210): no valid
quorum servers found in zoo.cfg
13/10/11 00:37:02 [regionserver60020] WARN regionserver.HRegionServer(1108):
Exception in region server :
java.io.IOException: Unable to determine ZooKeeper ensemble
at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:116)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.init(ZooKeeperWatcher.java:153)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.init(ZooKeeperWatcher.java:127)
at
org.apache.hadoop.hbase.replication.ReplicationPeer.reloadZkWatcher(ReplicationPeer.java:170)
at
org.apache.hadoop.hbase.replication.ReplicationPeer.init(ReplicationPeer.java:69)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.getPeer(ReplicationZookeeper.java:343)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.connectToPeer(ReplicationZookeeper.java:308)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.connectExistingPeers(ReplicationZookeeper.java:189)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.init(ReplicationZookeeper.java:156)
at
org.apache.hadoop.hbase.replication.regionserver.Replication.initialize(Replication.java:89)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.newReplicationInstance(HRegionServer.java:3986)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.createNewReplicationInstance(HRegionServer.java:3955)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.setupWALAndReplication(HRegionServer.java:1412)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1096)
at
[jira] [Reopened] (HBASE-9746) RegionServer can't start when replication tries to replicate to an unknown host
[
https://issues.apache.org/jira/browse/HBASE-9746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl reopened HBASE-9746:
--
We've just seen this again. It's bad when HBase cannot come up due to missing
DNS entries for a slave cluster.
RegionServer can't start when replication tries to replicate to an unknown
host
---
Key: HBASE-9746
URL: https://issues.apache.org/jira/browse/HBASE-9746
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.12
Reporter: Lars Hofhansl
Priority: Minor
Fix For: 0.99.0, 2.0.0, 0.98.7, 0.94.24
Just ran into this:
{code}
13/10/11 00:37:02 [regionserver60020] WARN zookeeper.ZKConfig(204):
java.net.UnknownHostException: old-host: Name or service not known
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:894)
at
java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1286)
at java.net.InetAddress.getAllByName0(InetAddress.java:1239)
at java.net.InetAddress.getAllByName(InetAddress.java:1155)
at java.net.InetAddress.getAllByName(InetAddress.java:1091)
at java.net.InetAddress.getByName(InetAddress.java:1041)
at
org.apache.hadoop.hbase.zookeeper.ZKConfig.getZKQuorumServersString(ZKConfig.java:201)
at
org.apache.hadoop.hbase.zookeeper.ZKConfig.getZKQuorumServersString(ZKConfig.java:245)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.init(ZooKeeperWatcher.java:147)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.init(ZooKeeperWatcher.java:127)
at
org.apache.hadoop.hbase.replication.ReplicationPeer.reloadZkWatcher(ReplicationPeer.java:170)
at
org.apache.hadoop.hbase.replication.ReplicationPeer.init(ReplicationPeer.java:69)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.getPeer(ReplicationZookeeper.java:343)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.connectToPeer(ReplicationZookeeper.java:308)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.connectExistingPeers(ReplicationZookeeper.java:189)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.init(ReplicationZookeeper.java:156)
at
org.apache.hadoop.hbase.replication.regionserver.Replication.initialize(Replication.java:89)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.newReplicationInstance(HRegionServer.java:3986)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.createNewReplicationInstance(HRegionServer.java:3955)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.setupWALAndReplication(HRegionServer.java:1412)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1096)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:749)
at java.lang.Thread.run(Thread.java:722)
13/10/11 00:37:02 [regionserver60020] ERROR zookeeper.ZKConfig(210): no valid
quorum servers found in zoo.cfg
13/10/11 00:37:02 [regionserver60020] WARN regionserver.HRegionServer(1108):
Exception in region server :
java.io.IOException: Unable to determine ZooKeeper ensemble
at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:116)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.init(ZooKeeperWatcher.java:153)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.init(ZooKeeperWatcher.java:127)
at
org.apache.hadoop.hbase.replication.ReplicationPeer.reloadZkWatcher(ReplicationPeer.java:170)
at
org.apache.hadoop.hbase.replication.ReplicationPeer.init(ReplicationPeer.java:69)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.getPeer(ReplicationZookeeper.java:343)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.connectToPeer(ReplicationZookeeper.java:308)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.connectExistingPeers(ReplicationZookeeper.java:189)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.init(ReplicationZookeeper.java:156)
at
org.apache.hadoop.hbase.replication.regionserver.Replication.initialize(Replication.java:89)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.newReplicationInstance(HRegionServer.java:3986)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.createNewReplicationInstance(HRegionServer.java:3955)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.setupWALAndReplication(HRegionServer.java:1412)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1096)
at
[jira] [Commented] (HBASE-11788) hbase is not deleting the cell when a Put with a KeyValue, KeyValue.Type.Delete is submitted
[
https://issues.apache.org/jira/browse/HBASE-11788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105839#comment-14105839
]
Cristian Armaselu commented on HBASE-11788:
---
Consider 2 JVM executing the following on the same row key
JVM1
- (1) Put to update new columns,
- (2) Delete to remove column X value (since incoming data is null for column
X)
JVM2
- (1) Put to update new columns,
- (2) Delete to remove column Y value (since incoming data is null for column
X)
What happens if the hbase API requests are applied in this order (since 2 JVM,
2 threads of execution)
JVM1 (1) Put
JVM2 (2) Delete
JVM1 dies
JVM2 (2) Delete
Or you can think of any other combination with 4 operations instead of 2
The data is corrupted, none of the 2 new records were applied, we don't have
the previous record stored in hbase.
I cannot point in Hive where the issue is. I see the behavior in the queries
executed against it.
We were using Put with DeleteColumn for null incoming cells and the is
null/is not null operators in hive were running fine.
With the new Hbase 0.96.1.1 data is coming back from the query when it should
not come back since the incoming column is null, but hbase decided to put an
empty cell
hbase is not deleting the cell when a Put with a KeyValue,
KeyValue.Type.Delete is submitted
Key: HBASE-11788
URL: https://issues.apache.org/jira/browse/HBASE-11788
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0, 0.96.1.1, 0.98.5, 2.0.0
Environment: Cloudera CDH 5.1.x
Reporter: Cristian Armaselu
Attachments: TestPutWithDelete.java
Code executed:
{code}
@Test
public void testHbasePutDeleteCell() throws Exception {
TableName tableName = TableName.valueOf(my_test);
Configuration configuration = HBaseConfiguration.create();
HTableInterface table = new HTable(configuration, tableName);
final String rowKey = 12345;
final byte[] familly = Bytes.toBytes(default);
// put one row
Put put = new Put(Bytes.toBytes(rowKey));
put.add(familly, Bytes.toBytes(A), Bytes.toBytes(a));
put.add(familly, Bytes.toBytes(B), Bytes.toBytes(b));
put.add(familly, Bytes.toBytes(C), Bytes.toBytes(c));
table.put(put);
// get row back and assert the values
Get get = new Get(Bytes.toBytes(rowKey));
Result result = table.get(get);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(A))).equals(a), Column A value should be a);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(B))).equals(b), Column B value should be b);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(C))).equals(c), Column C value should be c);
// put the same row again with C column deleted
put = new Put(Bytes.toBytes(rowKey));
put.add(familly, Bytes.toBytes(A), Bytes.toBytes(a));
put.add(familly, Bytes.toBytes(B), Bytes.toBytes(b));
put.add(new KeyValue(Bytes.toBytes(rowKey), familly,
Bytes.toBytes(C), HConstants.LATEST_TIMESTAMP, KeyValue.Type.DeleteColumn));
table.put(put);
// get row back and assert the values
get = new Get(Bytes.toBytes(rowKey));
result = table.get(get);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(A))).equals(a), Column A value should be a);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(B))).equals(b), Column A value should be b);
Assert.isTrue(result.getValue(familly, Bytes.toBytes(C)) == null,
Column C should not exists);
}
{code}
This assertion fails, the cell is not deleted but rather the value is empty:
{code}
hbase(main):029:0 scan 'my_test'
ROW COLUMN+CELL
12345column=default:A,
timestamp=1408473082290, value=a
12345column=default:B,
timestamp=1408473082290, value=b
12345column=default:C,
timestamp=1408473082290, value=
{code}
This behavior is different than previous 4.8.x Cloudera version and is
currently corrupting all hive queries involving is null or is not null
operators on the columns mapped to
[jira] [Updated] (HBASE-11794) StripeStoreFlusher causes NullPointerException and Region down
[ https://issues.apache.org/jira/browse/HBASE-11794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-11794: Affects Version/s: 0.98.3 0.98.4 Good find [~eomiks]! Want to put together a patch to fix this? If not, I'll take care of it. StripeStoreFlusher causes NullPointerException and Region down -- Key: HBASE-11794 URL: https://issues.apache.org/jira/browse/HBASE-11794 Project: HBase Issue Type: Bug Components: Compaction Affects Versions: 0.98.3, 0.98.4, 0.98.5 Reporter: jeongmin kim Priority: Critical StoreFlusher.flushSnapshot() mustn't return null value. But StripeStoreFlusher.flushSnapshot() does. It cause NullPointerException at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:802) and this makes regions dead after exhaustive retries and no recovery available from it. the code (StripeStoreFlusher:64) has to be changed === from ListPath result = null to ListPath result = new ArrayListPath(); === to return a empty list not null value. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9746) RegionServer can't start when replication tries to replicate to an unknown host
[
https://issues.apache.org/jira/browse/HBASE-9746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105843#comment-14105843
]
Andrew Purtell commented on HBASE-9746:
---
I think we should allow the source RS to come up, with continued attempts to
start replication, and WARN level logging at each failed attempt. This means we
need to deal with UnknownHostException and related exceptions. We could make
the retry interval configurable. Using the WAL roll period as the default value
for that could be a reasonable thing to do.
RegionServer can't start when replication tries to replicate to an unknown
host
---
Key: HBASE-9746
URL: https://issues.apache.org/jira/browse/HBASE-9746
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.12
Reporter: Lars Hofhansl
Priority: Minor
Fix For: 0.99.0, 2.0.0, 0.98.7, 0.94.24
Just ran into this:
{code}
13/10/11 00:37:02 [regionserver60020] WARN zookeeper.ZKConfig(204):
java.net.UnknownHostException: old-host: Name or service not known
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:894)
at
java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1286)
at java.net.InetAddress.getAllByName0(InetAddress.java:1239)
at java.net.InetAddress.getAllByName(InetAddress.java:1155)
at java.net.InetAddress.getAllByName(InetAddress.java:1091)
at java.net.InetAddress.getByName(InetAddress.java:1041)
at
org.apache.hadoop.hbase.zookeeper.ZKConfig.getZKQuorumServersString(ZKConfig.java:201)
at
org.apache.hadoop.hbase.zookeeper.ZKConfig.getZKQuorumServersString(ZKConfig.java:245)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.init(ZooKeeperWatcher.java:147)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.init(ZooKeeperWatcher.java:127)
at
org.apache.hadoop.hbase.replication.ReplicationPeer.reloadZkWatcher(ReplicationPeer.java:170)
at
org.apache.hadoop.hbase.replication.ReplicationPeer.init(ReplicationPeer.java:69)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.getPeer(ReplicationZookeeper.java:343)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.connectToPeer(ReplicationZookeeper.java:308)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.connectExistingPeers(ReplicationZookeeper.java:189)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.init(ReplicationZookeeper.java:156)
at
org.apache.hadoop.hbase.replication.regionserver.Replication.initialize(Replication.java:89)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.newReplicationInstance(HRegionServer.java:3986)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.createNewReplicationInstance(HRegionServer.java:3955)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.setupWALAndReplication(HRegionServer.java:1412)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1096)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:749)
at java.lang.Thread.run(Thread.java:722)
13/10/11 00:37:02 [regionserver60020] ERROR zookeeper.ZKConfig(210): no valid
quorum servers found in zoo.cfg
13/10/11 00:37:02 [regionserver60020] WARN regionserver.HRegionServer(1108):
Exception in region server :
java.io.IOException: Unable to determine ZooKeeper ensemble
at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:116)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.init(ZooKeeperWatcher.java:153)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.init(ZooKeeperWatcher.java:127)
at
org.apache.hadoop.hbase.replication.ReplicationPeer.reloadZkWatcher(ReplicationPeer.java:170)
at
org.apache.hadoop.hbase.replication.ReplicationPeer.init(ReplicationPeer.java:69)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.getPeer(ReplicationZookeeper.java:343)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.connectToPeer(ReplicationZookeeper.java:308)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.connectExistingPeers(ReplicationZookeeper.java:189)
at
org.apache.hadoop.hbase.replication.ReplicationZookeeper.init(ReplicationZookeeper.java:156)
at
org.apache.hadoop.hbase.replication.regionserver.Replication.initialize(Replication.java:89)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.newReplicationInstance(HRegionServer.java:3986)
at
[jira] [Updated] (HBASE-8541) implement flush-into-stripes in stripe compactions
[ https://issues.apache.org/jira/browse/HBASE-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-8541: --- Fix Version/s: 0.98.3 implement flush-into-stripes in stripe compactions -- Key: HBASE-8541 URL: https://issues.apache.org/jira/browse/HBASE-8541 Project: HBase Issue Type: Sub-task Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.98.3 Attachments: HBASE-8541-latest-with-dependencies.patch, HBASE-8541-latest-with-dependencies.patch, HBASE-8541-latest-with-dependencies.patch, HBASE-8541-latest-with-dependencies.patch, HBASE-8541-v0.patch, HBASE-8541-v1.patch, HBASE-8541-v2.patch, HBASE-8541-v3.patch, HBASE-8541-v4.patch, HBASE-8541-v5.patch Flush will be able to flush into multiple files under this design, avoiding L0 I/O amplification. I have the patch which is missing just one feature - support for concurrent flushes and stripe changes. This can be done via extensive try-locking of stripe changes and flushes, or advisory flags without blocking flushes, dumping conflicting flushes into L0 in case of (very rare) collisions. For file loading for the latter, a set-cover-like problem needs to be solved to determine optimal stripes. That will also address Jimmy's concern of getting rid of metadata, btw. However currently I don't have time for that. I plan to attach the try-locking patch first, but this won't happen for a couple weeks probably and should not block main reviews. Hopefully this will be added on top of main reviews. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-8541) implement flush-into-stripes in stripe compactions
[ https://issues.apache.org/jira/browse/HBASE-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-8541: --- Fix Version/s: 2.0.0 1.0.0 implement flush-into-stripes in stripe compactions -- Key: HBASE-8541 URL: https://issues.apache.org/jira/browse/HBASE-8541 Project: HBase Issue Type: Sub-task Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 1.0.0, 0.98.3, 2.0.0 Attachments: HBASE-8541-latest-with-dependencies.patch, HBASE-8541-latest-with-dependencies.patch, HBASE-8541-latest-with-dependencies.patch, HBASE-8541-latest-with-dependencies.patch, HBASE-8541-v0.patch, HBASE-8541-v1.patch, HBASE-8541-v2.patch, HBASE-8541-v3.patch, HBASE-8541-v4.patch, HBASE-8541-v5.patch Flush will be able to flush into multiple files under this design, avoiding L0 I/O amplification. I have the patch which is missing just one feature - support for concurrent flushes and stripe changes. This can be done via extensive try-locking of stripe changes and flushes, or advisory flags without blocking flushes, dumping conflicting flushes into L0 in case of (very rare) collisions. For file loading for the latter, a set-cover-like problem needs to be solved to determine optimal stripes. That will also address Jimmy's concern of getting rid of metadata, btw. However currently I don't have time for that. I plan to attach the try-locking patch first, but this won't happen for a couple weeks probably and should not block main reviews. Hopefully this will be added on top of main reviews. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-2236) Upper bound of outstanding WALs can be overrun; take 2 (take 1 was hbase-2053)
[
https://issues.apache.org/jira/browse/HBASE-2236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-2236:
---
Component/s: wal
regionserver
Upper bound of outstanding WALs can be overrun; take 2 (take 1 was hbase-2053)
--
Key: HBASE-2236
URL: https://issues.apache.org/jira/browse/HBASE-2236
Project: HBase
Issue Type: Bug
Components: regionserver, wal
Reporter: stack
Priority: Critical
Labels: moved_from_0_20_5
So hbase-2053 is not aggressive enough. WALs can still overwhelm the upper
limit on log count. While the code added by HBASE-2053, when done, will
ensure we let go of the oldest WAL, to do it, we might have to flush many
regions. E.g:
{code}
2010-02-15 14:20:29,351 INFO org.apache.hadoop.hbase.regionserver.HLog: Too
many hlogs: logs=45, maxlogs=32; forcing flush of 5 regions(s):
test1,193717,1266095474624, test1,194375,1266108228663,
test1,195690,1266095539377, test1,196348,1266095539377,
test1,197939,1266069173999
{code}
This takes time. If we are taking on edits a furious rate, we might have
rolled the log again, meantime, maybe more than once.
Also log rolls happen inline with a put/delete as soon as it hits the 64MB
(default) boundary whereas the necessary flushing is done in background by a
single thread and the memstore can overrun the (default) 64MB size. Flushes
needed to release logs will be mixed in with natural flushes as memstores
fill. Flushes may take longer than the writing of an HLog because they can
be larger.
So, on an RS that is struggling the tendency would seem to be for a slight
rise in WALs. Only if the RS gets a breather will the flusher catch up.
If HBASE-2087 happens, then the count of WALs get a boost.
Ideas to fix this for good would be :
+ Priority queue for queuing up flushes with those that are queued to free up
WALs having priority
+ Improve the HBASE-2053 code so that it will free more than just the last
WAL, maybe even queuing flushes so we clear all WALs such that we are back
under the maximum WALS threshold again.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11789) LoadIncrementalHFiles is not picking up the -D option
[ https://issues.apache.org/jira/browse/HBASE-11789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] deepankar updated HBASE-11789: -- Attachment: HBASE-11789-0.94.patch Attaching the patch we used for our internal branch LoadIncrementalHFiles is not picking up the -D option -- Key: HBASE-11789 URL: https://issues.apache.org/jira/browse/HBASE-11789 Project: HBase Issue Type: Bug Affects Versions: 0.99.0, 0.98.5, 2.0.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 0.99.0, 2.0.0, 0.98.6 Attachments: HBASE-11789-0.94.patch, HBASE-11789-v0.patch LoadIncrementalHFiles is not using the Tool class correctly, preventing to use the -D options. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11331) [blockcache] lazy block decompression
[ https://issues.apache.org/jira/browse/HBASE-11331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105865#comment-14105865 ] Nick Dimiduk commented on HBASE-11331: -- [~vrodionov] I think PE has zipfian now too (or maybe that's just the value size?). I'll take a look at YCSB, thanks. [blockcache] lazy block decompression - Key: HBASE-11331 URL: https://issues.apache.org/jira/browse/HBASE-11331 Project: HBase Issue Type: Improvement Components: regionserver Reporter: Nick Dimiduk Assignee: Nick Dimiduk Attachments: HBASE-11331.00.patch, HBASE-11331.01.patch, HBASE-11331.02.patch, HBASE-11331.03.patch, HBASE-11331LazyBlockDecompressperfcompare.pdf, lazy-decompress.02.0.pdf, lazy-decompress.02.1.json, lazy-decompress.02.1.pdf, v03-20g-045g-false.pdf, v03-20g-045g-true.pdf Maintaining data in its compressed form in the block cache will greatly increase our effective blockcache size and should show a meaning improvement in cache hit rates in well designed applications. The idea here is to lazily decompress/decrypt blocks when they're consumed, rather than as soon as they're pulled off of disk. This is related to but less invasive than HBASE-8894. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HBASE-6189) Snappy build instructions are out of date
[ https://issues.apache.org/jira/browse/HBASE-6189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey reassigned HBASE-6189: -- Assignee: Sean Busbey Snappy build instructions are out of date - Key: HBASE-6189 URL: https://issues.apache.org/jira/browse/HBASE-6189 Project: HBase Issue Type: Bug Affects Versions: 0.92.1, 0.94.0, 0.95.2 Reporter: Dave Revell Assignee: Sean Busbey Priority: Critical In the ref guide (http://hbase.apache.org/book.html#build.snappy), it says to build snappy by passing -Dsnappy. Something's wrong here, because: 1. this causes the build to fail because the hadoop-snappy tar artifact can't be found by maven 2. the snappy classes are already included in hadoop 1.0, so using the snappy profile is unnecessary It would be great if someone who knows when/why to use the snappy profile could fix the instructions (and fix the POM if necessary). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11788) hbase is not deleting the cell when a Put with a KeyValue, KeyValue.Type.Delete is submitted
[
https://issues.apache.org/jira/browse/HBASE-11788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105878#comment-14105878
]
Andrew Purtell commented on HBASE-11788:
bq. I cannot point in Hive where the issue is. I see the behavior in the
queries executed against it.
Yes, that was my misunderstanding. I just saw in the related conversation on
dev@ an indication it's your code using the HBase API directly that is bundling
deletes with puts.
bq. (since 2 JVM, 2 threads of execution)
Assemble a Put and a Delete into a
[RowMutations|https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/RowMutations.html]
and each Put+Delete bundle (the RowMutations) from each JVM will operate
atomically with respect to the other.
hbase is not deleting the cell when a Put with a KeyValue,
KeyValue.Type.Delete is submitted
Key: HBASE-11788
URL: https://issues.apache.org/jira/browse/HBASE-11788
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0, 0.96.1.1, 0.98.5, 2.0.0
Environment: Cloudera CDH 5.1.x
Reporter: Cristian Armaselu
Attachments: TestPutWithDelete.java
Code executed:
{code}
@Test
public void testHbasePutDeleteCell() throws Exception {
TableName tableName = TableName.valueOf(my_test);
Configuration configuration = HBaseConfiguration.create();
HTableInterface table = new HTable(configuration, tableName);
final String rowKey = 12345;
final byte[] familly = Bytes.toBytes(default);
// put one row
Put put = new Put(Bytes.toBytes(rowKey));
put.add(familly, Bytes.toBytes(A), Bytes.toBytes(a));
put.add(familly, Bytes.toBytes(B), Bytes.toBytes(b));
put.add(familly, Bytes.toBytes(C), Bytes.toBytes(c));
table.put(put);
// get row back and assert the values
Get get = new Get(Bytes.toBytes(rowKey));
Result result = table.get(get);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(A))).equals(a), Column A value should be a);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(B))).equals(b), Column B value should be b);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(C))).equals(c), Column C value should be c);
// put the same row again with C column deleted
put = new Put(Bytes.toBytes(rowKey));
put.add(familly, Bytes.toBytes(A), Bytes.toBytes(a));
put.add(familly, Bytes.toBytes(B), Bytes.toBytes(b));
put.add(new KeyValue(Bytes.toBytes(rowKey), familly,
Bytes.toBytes(C), HConstants.LATEST_TIMESTAMP, KeyValue.Type.DeleteColumn));
table.put(put);
// get row back and assert the values
get = new Get(Bytes.toBytes(rowKey));
result = table.get(get);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(A))).equals(a), Column A value should be a);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(B))).equals(b), Column A value should be b);
Assert.isTrue(result.getValue(familly, Bytes.toBytes(C)) == null,
Column C should not exists);
}
{code}
This assertion fails, the cell is not deleted but rather the value is empty:
{code}
hbase(main):029:0 scan 'my_test'
ROW COLUMN+CELL
12345column=default:A,
timestamp=1408473082290, value=a
12345column=default:B,
timestamp=1408473082290, value=b
12345column=default:C,
timestamp=1408473082290, value=
{code}
This behavior is different than previous 4.8.x Cloudera version and is
currently corrupting all hive queries involving is null or is not null
operators on the columns mapped to hbase
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11798) TestBucketWriterThread can zombie test
[ https://issues.apache.org/jira/browse/HBASE-11798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105877#comment-14105877 ] Alex Newman commented on HBASE-11798: - Thinking about it, this test should sleep in a loop forever. Let's limit it to 1000 tries. If it can't succeed then, we will be stuck with a slow or flakey test. I think that causes more hard then good. TestBucketWriterThread can zombie test -- Key: HBASE-11798 URL: https://issues.apache.org/jira/browse/HBASE-11798 Project: HBase Issue Type: Bug Reporter: Alex Newman Assignee: Alex Newman Attachments: HBASE-11798-v1.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11331) [blockcache] lazy block decompression
[ https://issues.apache.org/jira/browse/HBASE-11331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-11331: - Attachment: v03-20g-045g-true-16h.pdf re: v03-20g-045g-true-16h.pdf I baked this patch overnight (16h + some warmup) with compress=true. Same --size=45. Things look stable. [blockcache] lazy block decompression - Key: HBASE-11331 URL: https://issues.apache.org/jira/browse/HBASE-11331 Project: HBase Issue Type: Improvement Components: regionserver Reporter: Nick Dimiduk Assignee: Nick Dimiduk Attachments: HBASE-11331.00.patch, HBASE-11331.01.patch, HBASE-11331.02.patch, HBASE-11331.03.patch, HBASE-11331LazyBlockDecompressperfcompare.pdf, lazy-decompress.02.0.pdf, lazy-decompress.02.1.json, lazy-decompress.02.1.pdf, v03-20g-045g-false.pdf, v03-20g-045g-true-16h.pdf, v03-20g-045g-true.pdf Maintaining data in its compressed form in the block cache will greatly increase our effective blockcache size and should show a meaning improvement in cache hit rates in well designed applications. The idea here is to lazily decompress/decrypt blocks when they're consumed, rather than as soon as they're pulled off of disk. This is related to but less invasive than HBASE-8894. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11798) TestBucketWriterThread can zombie test
[ https://issues.apache.org/jira/browse/HBASE-11798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Newman updated HBASE-11798: Attachment: HBASE-11798-v2.patch TestBucketWriterThread can zombie test -- Key: HBASE-11798 URL: https://issues.apache.org/jira/browse/HBASE-11798 Project: HBase Issue Type: Bug Reporter: Alex Newman Assignee: Alex Newman Attachments: HBASE-11798-v1.patch, HBASE-11798-v2.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11331) [blockcache] lazy block decompression
[ https://issues.apache.org/jira/browse/HBASE-11331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105888#comment-14105888 ] Nick Dimiduk commented on HBASE-11331: -- Interesting that proc.loadavg.1min decreases even though we know there's more decompress operations happening and the gc activity increases. Maybe that measurement includes iowait? [blockcache] lazy block decompression - Key: HBASE-11331 URL: https://issues.apache.org/jira/browse/HBASE-11331 Project: HBase Issue Type: Improvement Components: regionserver Reporter: Nick Dimiduk Assignee: Nick Dimiduk Attachments: HBASE-11331.00.patch, HBASE-11331.01.patch, HBASE-11331.02.patch, HBASE-11331.03.patch, HBASE-11331LazyBlockDecompressperfcompare.pdf, lazy-decompress.02.0.pdf, lazy-decompress.02.1.json, lazy-decompress.02.1.pdf, v03-20g-045g-false.pdf, v03-20g-045g-true-16h.pdf, v03-20g-045g-true.pdf Maintaining data in its compressed form in the block cache will greatly increase our effective blockcache size and should show a meaning improvement in cache hit rates in well designed applications. The idea here is to lazily decompress/decrypt blocks when they're consumed, rather than as soon as they're pulled off of disk. This is related to but less invasive than HBASE-8894. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9529) Audit of hbase-client @InterfaceAudience.Public apis
[
https://issues.apache.org/jira/browse/HBASE-9529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105889#comment-14105889
]
Gary Helmling commented on HBASE-9529:
--
Why were the {{HTableInterface.coprocessorService(...)}} methods marked as
private as part of this? They were explicitly written as part of the public
API.
Audit of hbase-client @InterfaceAudience.Public apis
Key: HBASE-9529
URL: https://issues.apache.org/jira/browse/HBASE-9529
Project: HBase
Issue Type: Sub-task
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Fix For: 0.98.0, 0.96.0
Attachments: hbase-9529.patch, hbase-9529.v2.patch
Similar to HBASE-9523, let's do an audit of the hbase-client public api.
This is easier to do now that the we can publish only the public api javadoc
http://hbase.apache.org/apidocs/ (notice it only has Public apis now!)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11072) Abstract WAL splitting from ZK
[ https://issues.apache.org/jira/browse/HBASE-11072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Soldatov updated HBASE-11072: Status: Open (was: Patch Available) Abstract WAL splitting from ZK -- Key: HBASE-11072 URL: https://issues.apache.org/jira/browse/HBASE-11072 Project: HBase Issue Type: Sub-task Components: Consensus, Zookeeper Affects Versions: 0.99.0 Reporter: Mikhail Antonov Assignee: Sergey Soldatov Attachments: HBASE-11072-1_v2.patch, HBASE-11072-1_v3.patch, HBASE-11072-1_v4.patch, HBASE-11072-2_v2.patch, HBASE-11072-v1.patch, HBASE-11072-v2.patch, HBASE-11072-v3.patch, HBASE-11072-v4.patch, HBASE-11072-v5.patch, HBASE-11072-v6.patch, HBASE-11072-v7.patch, HBASE-11072-v8.patch, HBASE-11072-v8.patch, HBASE-11072-v9.patch, HBASE_11072-1.patch HM side: - SplitLogManager RS side: - SplitLogWorker - HLogSplitter and a few handler classes. This jira may need to be split further apart into smaller ones. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11788) hbase is not deleting the cell when a Put with a KeyValue, KeyValue.Type.Delete is submitted
[
https://issues.apache.org/jira/browse/HBASE-11788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105891#comment-14105891
]
Cristian Armaselu commented on HBASE-11788:
---
Yes but that will not work with Htable.checkAndPut
hbase is not deleting the cell when a Put with a KeyValue,
KeyValue.Type.Delete is submitted
Key: HBASE-11788
URL: https://issues.apache.org/jira/browse/HBASE-11788
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0, 0.96.1.1, 0.98.5, 2.0.0
Environment: Cloudera CDH 5.1.x
Reporter: Cristian Armaselu
Attachments: TestPutWithDelete.java
Code executed:
{code}
@Test
public void testHbasePutDeleteCell() throws Exception {
TableName tableName = TableName.valueOf(my_test);
Configuration configuration = HBaseConfiguration.create();
HTableInterface table = new HTable(configuration, tableName);
final String rowKey = 12345;
final byte[] familly = Bytes.toBytes(default);
// put one row
Put put = new Put(Bytes.toBytes(rowKey));
put.add(familly, Bytes.toBytes(A), Bytes.toBytes(a));
put.add(familly, Bytes.toBytes(B), Bytes.toBytes(b));
put.add(familly, Bytes.toBytes(C), Bytes.toBytes(c));
table.put(put);
// get row back and assert the values
Get get = new Get(Bytes.toBytes(rowKey));
Result result = table.get(get);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(A))).equals(a), Column A value should be a);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(B))).equals(b), Column B value should be b);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(C))).equals(c), Column C value should be c);
// put the same row again with C column deleted
put = new Put(Bytes.toBytes(rowKey));
put.add(familly, Bytes.toBytes(A), Bytes.toBytes(a));
put.add(familly, Bytes.toBytes(B), Bytes.toBytes(b));
put.add(new KeyValue(Bytes.toBytes(rowKey), familly,
Bytes.toBytes(C), HConstants.LATEST_TIMESTAMP, KeyValue.Type.DeleteColumn));
table.put(put);
// get row back and assert the values
get = new Get(Bytes.toBytes(rowKey));
result = table.get(get);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(A))).equals(a), Column A value should be a);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(B))).equals(b), Column A value should be b);
Assert.isTrue(result.getValue(familly, Bytes.toBytes(C)) == null,
Column C should not exists);
}
{code}
This assertion fails, the cell is not deleted but rather the value is empty:
{code}
hbase(main):029:0 scan 'my_test'
ROW COLUMN+CELL
12345column=default:A,
timestamp=1408473082290, value=a
12345column=default:B,
timestamp=1408473082290, value=b
12345column=default:C,
timestamp=1408473082290, value=
{code}
This behavior is different than previous 4.8.x Cloudera version and is
currently corrupting all hive queries involving is null or is not null
operators on the columns mapped to hbase
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-7767) Get rid of ZKTable, and table enable/disable state in ZK
[ https://issues.apache.org/jira/browse/HBASE-7767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Stepachev updated HBASE-7767: Status: Open (was: Patch Available) Get rid of ZKTable, and table enable/disable state in ZK - Key: HBASE-7767 URL: https://issues.apache.org/jira/browse/HBASE-7767 Project: HBase Issue Type: Sub-task Components: Zookeeper Affects Versions: 2.0.0 Reporter: Enis Soztutar Assignee: Andrey Stepachev Attachments: HBASE-7767.patch As discussed table state in zookeeper for enable/disable state breaks our zookeeper contract. It is also very intrusive, used from the client side, master and region servers. We should get rid of it. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11072) Abstract WAL splitting from ZK
[ https://issues.apache.org/jira/browse/HBASE-11072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Soldatov updated HBASE-11072: Status: Patch Available (was: Open) Abstract WAL splitting from ZK -- Key: HBASE-11072 URL: https://issues.apache.org/jira/browse/HBASE-11072 Project: HBase Issue Type: Sub-task Components: Consensus, Zookeeper Affects Versions: 0.99.0 Reporter: Mikhail Antonov Assignee: Sergey Soldatov Attachments: HBASE-11072-1_v2.patch, HBASE-11072-1_v3.patch, HBASE-11072-1_v4.patch, HBASE-11072-2_v2.patch, HBASE-11072-v1.patch, HBASE-11072-v10.patch, HBASE-11072-v2.patch, HBASE-11072-v3.patch, HBASE-11072-v4.patch, HBASE-11072-v5.patch, HBASE-11072-v6.patch, HBASE-11072-v7.patch, HBASE-11072-v8.patch, HBASE-11072-v8.patch, HBASE-11072-v9.patch, HBASE_11072-1.patch HM side: - SplitLogManager RS side: - SplitLogWorker - HLogSplitter and a few handler classes. This jira may need to be split further apart into smaller ones. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-7767) Get rid of ZKTable, and table enable/disable state in ZK
[ https://issues.apache.org/jira/browse/HBASE-7767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Stepachev updated HBASE-7767: Attachment: HBASE-7767.patch fixing javadocs Get rid of ZKTable, and table enable/disable state in ZK - Key: HBASE-7767 URL: https://issues.apache.org/jira/browse/HBASE-7767 Project: HBase Issue Type: Sub-task Components: Zookeeper Affects Versions: 2.0.0 Reporter: Enis Soztutar Assignee: Andrey Stepachev Attachments: HBASE-7767.patch, HBASE-7767.patch As discussed table state in zookeeper for enable/disable state breaks our zookeeper contract. It is also very intrusive, used from the client side, master and region servers. We should get rid of it. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-7767) Get rid of ZKTable, and table enable/disable state in ZK
[ https://issues.apache.org/jira/browse/HBASE-7767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Stepachev updated HBASE-7767: Status: Patch Available (was: Open) Get rid of ZKTable, and table enable/disable state in ZK - Key: HBASE-7767 URL: https://issues.apache.org/jira/browse/HBASE-7767 Project: HBase Issue Type: Sub-task Components: Zookeeper Affects Versions: 2.0.0 Reporter: Enis Soztutar Assignee: Andrey Stepachev Attachments: HBASE-7767.patch, HBASE-7767.patch As discussed table state in zookeeper for enable/disable state breaks our zookeeper contract. It is also very intrusive, used from the client side, master and region servers. We should get rid of it. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11072) Abstract WAL splitting from ZK
[ https://issues.apache.org/jira/browse/HBASE-11072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Soldatov updated HBASE-11072: Attachment: HBASE-11072-v10.patch rebased + fixed comments Abstract WAL splitting from ZK -- Key: HBASE-11072 URL: https://issues.apache.org/jira/browse/HBASE-11072 Project: HBase Issue Type: Sub-task Components: Consensus, Zookeeper Affects Versions: 0.99.0 Reporter: Mikhail Antonov Assignee: Sergey Soldatov Attachments: HBASE-11072-1_v2.patch, HBASE-11072-1_v3.patch, HBASE-11072-1_v4.patch, HBASE-11072-2_v2.patch, HBASE-11072-v1.patch, HBASE-11072-v10.patch, HBASE-11072-v2.patch, HBASE-11072-v3.patch, HBASE-11072-v4.patch, HBASE-11072-v5.patch, HBASE-11072-v6.patch, HBASE-11072-v7.patch, HBASE-11072-v8.patch, HBASE-11072-v8.patch, HBASE-11072-v9.patch, HBASE_11072-1.patch HM side: - SplitLogManager RS side: - SplitLogWorker - HLogSplitter and a few handler classes. This jira may need to be split further apart into smaller ones. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11788) hbase is not deleting the cell when a Put with a KeyValue, KeyValue.Type.Delete is submitted
[
https://issues.apache.org/jira/browse/HBASE-11788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105902#comment-14105902
]
Srikanth Srungarapu commented on HBASE-11788:
-
Can you please let me know if someone is interested in picking this up? If not,
I can work on this.
hbase is not deleting the cell when a Put with a KeyValue,
KeyValue.Type.Delete is submitted
Key: HBASE-11788
URL: https://issues.apache.org/jira/browse/HBASE-11788
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0, 0.96.1.1, 0.98.5, 2.0.0
Environment: Cloudera CDH 5.1.x
Reporter: Cristian Armaselu
Attachments: TestPutWithDelete.java
Code executed:
{code}
@Test
public void testHbasePutDeleteCell() throws Exception {
TableName tableName = TableName.valueOf(my_test);
Configuration configuration = HBaseConfiguration.create();
HTableInterface table = new HTable(configuration, tableName);
final String rowKey = 12345;
final byte[] familly = Bytes.toBytes(default);
// put one row
Put put = new Put(Bytes.toBytes(rowKey));
put.add(familly, Bytes.toBytes(A), Bytes.toBytes(a));
put.add(familly, Bytes.toBytes(B), Bytes.toBytes(b));
put.add(familly, Bytes.toBytes(C), Bytes.toBytes(c));
table.put(put);
// get row back and assert the values
Get get = new Get(Bytes.toBytes(rowKey));
Result result = table.get(get);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(A))).equals(a), Column A value should be a);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(B))).equals(b), Column B value should be b);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(C))).equals(c), Column C value should be c);
// put the same row again with C column deleted
put = new Put(Bytes.toBytes(rowKey));
put.add(familly, Bytes.toBytes(A), Bytes.toBytes(a));
put.add(familly, Bytes.toBytes(B), Bytes.toBytes(b));
put.add(new KeyValue(Bytes.toBytes(rowKey), familly,
Bytes.toBytes(C), HConstants.LATEST_TIMESTAMP, KeyValue.Type.DeleteColumn));
table.put(put);
// get row back and assert the values
get = new Get(Bytes.toBytes(rowKey));
result = table.get(get);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(A))).equals(a), Column A value should be a);
Assert.isTrue(Bytes.toString(result.getValue(familly,
Bytes.toBytes(B))).equals(b), Column A value should be b);
Assert.isTrue(result.getValue(familly, Bytes.toBytes(C)) == null,
Column C should not exists);
}
{code}
This assertion fails, the cell is not deleted but rather the value is empty:
{code}
hbase(main):029:0 scan 'my_test'
ROW COLUMN+CELL
12345column=default:A,
timestamp=1408473082290, value=a
12345column=default:B,
timestamp=1408473082290, value=b
12345column=default:C,
timestamp=1408473082290, value=
{code}
This behavior is different than previous 4.8.x Cloudera version and is
currently corrupting all hive queries involving is null or is not null
operators on the columns mapped to hbase
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-6189) Snappy build instructions are out of date
[ https://issues.apache.org/jira/browse/HBASE-6189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105940#comment-14105940 ] Sean Busbey commented on HBASE-6189: The book currently says snappy isn't bundled because of a licensing problem. The google code repository says it's currently BSD licensed, which would be fine AFAIK. Anyone know what that was? I can clean it up as a part of this ticket. (also we could just use snappy-java, which is Apache v2 and available via maven deps) Snappy build instructions are out of date - Key: HBASE-6189 URL: https://issues.apache.org/jira/browse/HBASE-6189 Project: HBase Issue Type: Bug Affects Versions: 0.92.1, 0.94.0, 0.95.2 Reporter: Dave Revell Assignee: Sean Busbey Priority: Critical In the ref guide (http://hbase.apache.org/book.html#build.snappy), it says to build snappy by passing -Dsnappy. Something's wrong here, because: 1. this causes the build to fail because the hadoop-snappy tar artifact can't be found by maven 2. the snappy classes are already included in hadoop 1.0, so using the snappy profile is unnecessary It would be great if someone who knows when/why to use the snappy profile could fix the instructions (and fix the POM if necessary). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11799) AsyncProcess can get locked when running PE
[
https://issues.apache.org/jira/browse/HBASE-11799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elliott Clark updated HBASE-11799:
--
Attachment: jstack
AsyncProcess can get locked when running PE
---
Key: HBASE-11799
URL: https://issues.apache.org/jira/browse/HBASE-11799
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.98.7
Reporter: Elliott Clark
Attachments: jstack
On a clean checkout of 0.98.
{code}
mvn clean package -DskipTests
bin/start-hbase.sh
bin/hbase pe --nomapred randomWrite 10 1000{code}
Everything runs fine for a while then the client hangs.
The region server thinks that all requests have completed successfully. The
PE process is all stuck on the final flush.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
