[jira] [Commented] (HBASE-7847) Use zookeeper multi to clear znodes
[ https://issues.apache.org/jira/browse/HBASE-7847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934659#comment-13934659 ] Rakesh R commented on HBASE-7847: - Thank you again Stack, Rajeshbabu for the reviews. I've reworked and now I have used our own logic of traversal and preparing the nodes to be deleted. Use zookeeper multi to clear znodes --- Key: HBASE-7847 URL: https://issues.apache.org/jira/browse/HBASE-7847 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Attachments: 7847-v1.txt, HBASE-7847.patch, HBASE-7847.patch, HBASE-7847.patch, HBASE-7847_v4.patch In ZKProcedureUtil, clearChildZNodes() and clearZNodes(String procedureName) should utilize zookeeper multi so that they're atomic -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10514) Forward port HBASE-10466, possible data loss when failed flushes
[ https://issues.apache.org/jira/browse/HBASE-10514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934660#comment-13934660 ] Hudson commented on HBASE-10514: SUCCESS: Integrated in hbase-0.96 #346 (See [https://builds.apache.org/job/hbase-0.96/346/]) HBASE-10514 Forward port HBASE-10466, possible data loss when failed flushes (stack: rev 1577386) * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreConfigInformation.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionBusyWait.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java Forward port HBASE-10466, possible data loss when failed flushes Key: HBASE-10514 URL: https://issues.apache.org/jira/browse/HBASE-10514 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Priority: Critical Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.18 Attachments: 10154v4.096.txt, 10154v94.txt, 10514.txt, 10514v2.txt, 10514v3.txt, 10514v3.txt, 10514v4.txt, 10514v5.txt, 10514v6.098.txt Critical data loss issues that we need to ensure are not in branches beyond 0.89fb. Assigning myself. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10739) RS web UI NPE if master shuts down sooner
[ https://issues.apache.org/jira/browse/HBASE-10739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934661#comment-13934661 ] Hudson commented on HBASE-10739: SUCCESS: Integrated in hbase-0.96 #346 (See [https://builds.apache.org/job/hbase-0.96/346/]) HBASE-10739 RS web UI NPE if master shuts down sooner (jxiang: rev 1577411) * /hbase/branches/0.96/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/BackupMasterStatusTmpl.jamon * /hbase/branches/0.96/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSStatusServlet.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterStatusServlet.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressTracker.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRSStatusServlet.java RS web UI NPE if master shuts down sooner - Key: HBASE-10739 URL: https://issues.apache.org/jira/browse/HBASE-10739 Project: HBase Issue Type: Bug Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.96.2, 0.98.1, 0.99.0 Attachments: hbase-10739.patch, hbase-10739_v2.patch This is similar to HBASE-9294. However, the fix in HBASE-9294 can't fix the issue. hrs is unlikely null over there since we have an assert earlier. We also called hrs.isOnline eariler. It is very likely the master shuts down sooner so the master address is null instead. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10466) Bugs that make flushes skipped during HRegion close could cause data loss
[ https://issues.apache.org/jira/browse/HBASE-10466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934662#comment-13934662 ] Hudson commented on HBASE-10466: SUCCESS: Integrated in hbase-0.96 #346 (See [https://builds.apache.org/job/hbase-0.96/346/]) HBASE-10514 Forward port HBASE-10466, possible data loss when failed flushes (stack: rev 1577386) * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreConfigInformation.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionBusyWait.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java Bugs that make flushes skipped during HRegion close could cause data loss - Key: HBASE-10466 URL: https://issues.apache.org/jira/browse/HBASE-10466 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.89-fb Reporter: Yunfan Zhong Priority: Critical Fix For: 0.89-fb Attachments: Fix-bugs-that-causes-flushes-being-skipped-during-re.patch During region close, there are two flushes to ensure nothing is persisted in memory. When there is data in current memstore only, 1 flush is required. When there is data also in memstore's snapshot, 2 flushes are essential otherwise we have data loss. However, recently we found two bugs that lead to at least 1 flush skipped and caused data loss. Bug 1: Wrong calculation of HRegion.memstoreSize When a flush fails, data to be flushed is kept in each MemStore's snapshot and wait for next flush attempt to continue on it. But when the next flush succeeds, the counter of total memstore size in HRegion is always deduced by the sum of current memstore sizes instead of snapshots left from previous failed flush. This calculation is problematic that almost every time there is failed flush, HRegion.memstoreSize gets reduced by a wrong value. If region flush could not proceed for a couple cycles, the size in current memstore could be much larger than the snapshot. It's likely to drift memstoreSize much smaller than expected. In extreme case, if the error accumulates to even bigger than HRegion's memstore size limit, any further flush is skipped because flush does not do anything if memstoreSize is not larger than 0. When the region is closing, if the two flushes get skipped and leave data in current memstore and/or snapshot, we could lose data up to the memstore size limit of the region. The fix is deducing correct size of data that is going to be flushed from memstoreSize. Bug 2: Conditions for the first flush of region close (so-called pre-flush) If memstoreSize is smaller than a certain value, or when region close starts a flush is ongoing, the first flush is skipped and only the second flush takes place. However, two flushes are required in case previous flush fails and leaves some data in snapshot. The bug could cause loss of data in current memstore. The fix is removing all conditions except abort check so we ensure 2 flushes for region close. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10701) Cache invalidation improvements from client side
[
https://issues.apache.org/jira/browse/HBASE-10701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-10701:
--
Attachment: hbase-10701_v3.patch
Thanks Nicolas for the careful review.
I've changed the patch so that I dropped the approach or using HRL's with null
ServerNames. Instead we still set the HRL item as null inside RegionLocations.
RegionLocations now, can contain null elements at the tail of the array as
well. This enables the cache to know about how many replicas there are, but the
locations might still be unknown.
I've been testing this with
{code}
hbase
org.apache.hadoop.hbase.test.IntegrationTestTimeBoundedRequestsWithRegionReplicas
-Dhbase.IntegrationTestTimeBoundedRequestsWithRegionReplicas.runtime=60
-DIntegrationTestTimeBoundedRequestsWithRegionReplicas.num_write_threads=30
-DIntegrationTestTimeBoundedRequestsWithRegionReplicas.region_replication=3
-DIntegrationTestTimeBoundedRequestsWithRegionReplicas.num_read_threads=30
-Dhbase.ipc.client.allowsInterrupt=true
{code}
it seems the issues are fixed. However, I notice that the test most of the time
dies with OOM, cannot create native thread, because the number of threads grow
unbounded (north of 4K).
Tried setting -Dhbase.hconnection.threads.max=512 with no results so far.
One other issue (probably related) was that the RPC's would not start for a
long time and timeout the gets (10-20 secs) because the thread pool executor
does not schedule the tasks in the CompletionService from
RpcRetryingCallerWithReadReplicas. Do you have any opinion around this? Should
we create a secondary pool for the backup requests? If we address the thread
growing problem, probably this will be fixed as well.
The v3 patch also addresses your comments, except for the DoNotRetryEx. We'll
have to get this running consistently before addressing that I think.
Cache invalidation improvements from client side
Key: HBASE-10701
URL: https://issues.apache.org/jira/browse/HBASE-10701
Project: HBase
Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: hbase-10070
Attachments: hbase-10701_v1.patch, hbase-10701_v2.patch,
hbase-10701_v3.patch
Running the integration test in HBASE-10572, and HBASE-10355, it seems that
we need some changes for cache invalidation of meta entries from the client
side in backup RPCs.
Mainly the RPC's made for replicas should not invalidate the cache for all
the replicas (for example on RegionMovedException, connection error etc).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10642) Add M/R over snapshots to 0.94
[ https://issues.apache.org/jira/browse/HBASE-10642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934683#comment-13934683 ] Hudson commented on HBASE-10642: FAILURE: Integrated in HBase-0.94 #1323 (See [https://builds.apache.org/job/HBase-0.94/1323/]) HBASE-10642 Add M/R over snapshots to 0.94 (larsh: rev 1577420) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/HDFSBlocksDistribution.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormat.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableSnapshotInputFormat.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableSnapshotInputFormatScan.java Add M/R over snapshots to 0.94 -- Key: HBASE-10642 URL: https://issues.apache.org/jira/browse/HBASE-10642 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.18 Attachments: 10642-0.94-v2.txt, 10642-0.94-v3.txt, 10642-0.94-v4.txt, 10642-0.94-v5.txt, 10642-0.94.txt, SnapshotInputFormat.java I think we want drive towards all (or most) M/R over HBase to be against snapshots and HDFS directly. Adopting a simple input format (even if just as a sample) as part of HBase will allow us to direct users this way. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[
https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934691#comment-13934691
]
Hadoop QA commented on HBASE-10648:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12634649/HBASE-10648_V5.patch
against trunk revision .
ATTACHMENT ID: 12634649
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 26 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ KeyValue.COMPARATOR.compareRows(left.getRowArray(),
left.getRowOffset(), (int) left.getRowLength(), row1, 0, row1.length) == 0);
+System.out.println(iteration + took usec: + (System.nanoTime() -
nanos)/1000 + for: + cnt);
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestAdmin
org.apache.hadoop.hbase.client.TestHCM
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.regionserver.wal.TestLogRolling.testLogRollOnDatanodeDeath(TestLogRolling.java:368)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8986//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8986//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8986//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8986//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8986//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8986//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8986//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8986//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8986//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8986//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8986//console
This message is automatically generated.
Pluggable Memstore
--
Key: HBASE-10648
URL: https://issues.apache.org/jira/browse/HBASE-10648
Project: HBase
Issue Type: Sub-task
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Attachments: HBASE-10648.patch, HBASE-10648_V2.patch,
HBASE-10648_V3.patch, HBASE-10648_V4.patch, HBASE-10648_V5.patch
Make Memstore into an interface implementation. Also make it pluggable by
configuring the FQCN of the impl.
This will allow us to have different impl and optimizations in the Memstore
DataStructure and the upper layers untouched.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Reopened] (HBASE-10184) [Online Schema Change]: Add additional tests for online schema change
[ https://issues.apache.org/jira/browse/HBASE-10184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell reopened HBASE-10184: The changes passed a precommit build and locally, but have failed in more than one trunk and 0.98 build up on Jenkins and recently in a precommit build there as well. I will revert this from 0.98 now. [~aleksshulman], the failures may be tests working as intended, revealing issues with online schema change. If you are interested in debugging the failures on trunk, we can leave this committed there for a bit (unless objection) to facilitate that. [Online Schema Change]: Add additional tests for online schema change - Key: HBASE-10184 URL: https://issues.apache.org/jira/browse/HBASE-10184 Project: HBase Issue Type: Task Components: test Affects Versions: 0.96.1, 0.99.0 Reporter: Aleksandr Shulman Assignee: Aleksandr Shulman Labels: online_schema_change Fix For: 0.98.1, 0.99.0 Attachments: 10184-4.patch, 10184.addendum, HBASE-10184-trunk.diff There are some gaps in testing for Online Schema Change: Examples of some tests that should be added: 1. Splits with online schema change 2. Merge during online schema change 3. MR over HBase during online schema change 4. Bulk Load during online schema change 5. Online change table owner 6. Online Replication scope change 7. Online Bloom Filter change 8. Snapshots during online schema change (HBASE-10136) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10184) [Online Schema Change]: Add additional tests for online schema change
[ https://issues.apache.org/jira/browse/HBASE-10184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934698#comment-13934698 ] Andrew Purtell commented on HBASE-10184: Reverted from 0.98 branch in r1577439 [Online Schema Change]: Add additional tests for online schema change - Key: HBASE-10184 URL: https://issues.apache.org/jira/browse/HBASE-10184 Project: HBase Issue Type: Task Components: test Affects Versions: 0.96.1, 0.99.0 Reporter: Aleksandr Shulman Assignee: Aleksandr Shulman Labels: online_schema_change Fix For: 0.98.1, 0.99.0 Attachments: 10184-4.patch, 10184.addendum, HBASE-10184-trunk.diff There are some gaps in testing for Online Schema Change: Examples of some tests that should be added: 1. Splits with online schema change 2. Merge during online schema change 3. MR over HBase during online schema change 4. Bulk Load during online schema change 5. Online change table owner 6. Online Replication scope change 7. Online Bloom Filter change 8. Snapshots during online schema change (HBASE-10136) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10616) Integration test for multi-get calls
[ https://issues.apache.org/jira/browse/HBASE-10616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-10616: Attachment: 10616-add.txt I had to do the bugfix as in the attached patch to handle the case of multiget_batchsize == 0. Integration test for multi-get calls Key: HBASE-10616 URL: https://issues.apache.org/jira/browse/HBASE-10616 Project: HBase Issue Type: Sub-task Reporter: Devaraj Das Assignee: Devaraj Das Fix For: hbase-10070 Attachments: 10616-1.txt, 10616-2.2.txt, 10616-3.txt, 10616-4.1.txt, 10616-4.txt, 10616-add.txt HBASE-10572 adds a test that does 'get' from client. We should add a similar test for batch gets. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10184) [Online Schema Change]: Add additional tests for online schema change
[ https://issues.apache.org/jira/browse/HBASE-10184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934697#comment-13934697 ] Aleksandr Shulman commented on HBASE-10184: --- I will take a look. Should be interesting. [Online Schema Change]: Add additional tests for online schema change - Key: HBASE-10184 URL: https://issues.apache.org/jira/browse/HBASE-10184 Project: HBase Issue Type: Task Components: test Affects Versions: 0.96.1, 0.99.0 Reporter: Aleksandr Shulman Assignee: Aleksandr Shulman Labels: online_schema_change Fix For: 0.98.1, 0.99.0 Attachments: 10184-4.patch, 10184.addendum, HBASE-10184-trunk.diff There are some gaps in testing for Online Schema Change: Examples of some tests that should be added: 1. Splits with online schema change 2. Merge during online schema change 3. MR over HBase during online schema change 4. Bulk Load during online schema change 5. Online change table owner 6. Online Replication scope change 7. Online Bloom Filter change 8. Snapshots during online schema change (HBASE-10136) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10184) [Online Schema Change]: Add additional tests for online schema change
[ https://issues.apache.org/jira/browse/HBASE-10184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934699#comment-13934699 ] Andrew Purtell commented on HBASE-10184: bq. I will take a look. Should be interesting. Great, thanks. If we can get Jenkins consistently happy with the new tests in trunk then we can try putting them (and fixes) into 0.98 again. [Online Schema Change]: Add additional tests for online schema change - Key: HBASE-10184 URL: https://issues.apache.org/jira/browse/HBASE-10184 Project: HBase Issue Type: Task Components: test Affects Versions: 0.96.1, 0.99.0 Reporter: Aleksandr Shulman Assignee: Aleksandr Shulman Labels: online_schema_change Fix For: 0.98.1, 0.99.0 Attachments: 10184-4.patch, 10184.addendum, HBASE-10184-trunk.diff There are some gaps in testing for Online Schema Change: Examples of some tests that should be added: 1. Splits with online schema change 2. Merge during online schema change 3. MR over HBase during online schema change 4. Bulk Load during online schema change 5. Online change table owner 6. Online Replication scope change 7. Online Bloom Filter change 8. Snapshots during online schema change (HBASE-10136) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10184) [Online Schema Change]: Add additional tests for online schema change
[ https://issues.apache.org/jira/browse/HBASE-10184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-10184: --- Fix Version/s: (was: 0.98.1) 0.98.2 [Online Schema Change]: Add additional tests for online schema change - Key: HBASE-10184 URL: https://issues.apache.org/jira/browse/HBASE-10184 Project: HBase Issue Type: Task Components: test Affects Versions: 0.96.1, 0.99.0 Reporter: Aleksandr Shulman Assignee: Aleksandr Shulman Labels: online_schema_change Fix For: 0.99.0, 0.98.2 Attachments: 10184-4.patch, 10184.addendum, HBASE-10184-trunk.diff There are some gaps in testing for Online Schema Change: Examples of some tests that should be added: 1. Splits with online schema change 2. Merge during online schema change 3. MR over HBase during online schema change 4. Bulk Load during online schema change 5. Online change table owner 6. Online Replication scope change 7. Online Bloom Filter change 8. Snapshots during online schema change (HBASE-10136) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10184) [Online Schema Change]: Add additional tests for online schema change
[ https://issues.apache.org/jira/browse/HBASE-10184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934706#comment-13934706 ] Aleksandr Shulman commented on HBASE-10184: --- From the above runs (3), I only see one TF related to my tests: https://builds.apache.org/job/HBase-0.98/228/testReport/junit/org.apache.hadoop.hbase.regionserver/TestEndToEndSplitTransaction/testFromClientSideOnlineSchemaChangeWhileSplitting/ Can you point me to the additional failures? [Online Schema Change]: Add additional tests for online schema change - Key: HBASE-10184 URL: https://issues.apache.org/jira/browse/HBASE-10184 Project: HBase Issue Type: Task Components: test Affects Versions: 0.96.1, 0.99.0 Reporter: Aleksandr Shulman Assignee: Aleksandr Shulman Labels: online_schema_change Fix For: 0.99.0, 0.98.2 Attachments: 10184-4.patch, 10184.addendum, HBASE-10184-trunk.diff There are some gaps in testing for Online Schema Change: Examples of some tests that should be added: 1. Splits with online schema change 2. Merge during online schema change 3. MR over HBase during online schema change 4. Bulk Load during online schema change 5. Online change table owner 6. Online Replication scope change 7. Online Bloom Filter change 8. Snapshots during online schema change (HBASE-10136) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9964) start-hbase.sh returns success exit status and emits starting master message when master failed to start
[
https://issues.apache.org/jira/browse/HBASE-9964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934713#comment-13934713
]
Hadoop QA commented on HBASE-9964:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12634645/HBASE-9964.patch
against trunk revision .
ATTACHMENT ID: 12634645
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestHRegion
org.apache.hadoop.hbase.client.TestAdmin
org.apache.hadoop.hbase.regionserver.TestHRegionBusyWait
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.regionserver.wal.TestLogRolling.testLogRollOnDatanodeDeath(TestLogRolling.java:368)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8987//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8987//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8987//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8987//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8987//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8987//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8987//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8987//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8987//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8987//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8987//console
This message is automatically generated.
start-hbase.sh returns success exit status and emits starting master
message when master failed to start
--
Key: HBASE-9964
URL: https://issues.apache.org/jira/browse/HBASE-9964
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.13
Environment: Linux x86_64 (Centos 6.4)
Reporter: Jason Vas Dias
Attachments: HBASE-9964.patch
The bin/start-hbase.sh script returns an exit status indicating success, and
emits the message : 'starting master, logging to ...' , even though the
master failed to start:
$ ./bin/start-hbase.sh
starting master, logging to
/home/jason/3P/hbase-0.94.13/logs/hbase-jason-master-jvds.out
$ echo $?
0
# ^- this indicates successful exit status on Linux / UNIX-like systems
$ egrep 'start master|RuntimeException' logs/hbase-jason-master-jvds.log
2013-11-13 14:15:28,309 ERROR
org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master
java.lang.RuntimeException: Failed suppression of fs shutdown hook:
Thread[Thread-27,5,main]
The start-base.sh script should IMHO in the above scenario emit a message
like
'Failed to start the
[jira] [Commented] (HBASE-10616) Integration test for multi-get calls
[ https://issues.apache.org/jira/browse/HBASE-10616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934715#comment-13934715 ] Ted Yu commented on HBASE-10616: +1 on addendum. Integration test for multi-get calls Key: HBASE-10616 URL: https://issues.apache.org/jira/browse/HBASE-10616 Project: HBase Issue Type: Sub-task Reporter: Devaraj Das Assignee: Devaraj Das Fix For: hbase-10070 Attachments: 10616-1.txt, 10616-2.2.txt, 10616-3.txt, 10616-4.1.txt, 10616-4.txt, 10616-add.txt HBASE-10572 adds a test that does 'get' from client. We should add a similar test for batch gets. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10740) Upgrade zookeeper to 3.4.6 release
[ https://issues.apache.org/jira/browse/HBASE-10740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934717#comment-13934717 ] Hudson commented on HBASE-10740: FAILURE: Integrated in HBase-TRUNK #5009 (See [https://builds.apache.org/job/HBase-TRUNK/5009/]) HBASE-10740 Upgrade zookeeper to 3.4.6 release (tedyu: rev 1577412) * /hbase/trunk/pom.xml Upgrade zookeeper to 3.4.6 release -- Key: HBASE-10740 URL: https://issues.apache.org/jira/browse/HBASE-10740 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Priority: Minor Fix For: 0.99.0 Attachments: 10740-v1.txt Zookeeper 3.4.6 release has been released. This JIRA upgrades zookeeper dependency to 3.4.6 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10744) AM#CloseRegion no need to retry on FailedServerException
[ https://issues.apache.org/jira/browse/HBASE-10744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934716#comment-13934716 ] Hudson commented on HBASE-10744: FAILURE: Integrated in HBase-TRUNK #5009 (See [https://builds.apache.org/job/HBase-TRUNK/5009/]) HBASE-10744 AM#CloseRegion no need to retry on FailedServerException (jxiang: rev 1577416) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerOnCluster.java AM#CloseRegion no need to retry on FailedServerException Key: HBASE-10744 URL: https://issues.apache.org/jira/browse/HBASE-10744 Project: HBase Issue Type: Improvement Components: Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.96.2, 0.98.1, 0.99.0 Attachments: hbase-10744.patch When a regionserver restarts, AM#CloseRegion could get FailedServerException, or ServerNotRunningYetException. The call should not be retried since we know the original regionserver should be down. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10739) RS web UI NPE if master shuts down sooner
[ https://issues.apache.org/jira/browse/HBASE-10739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934718#comment-13934718 ] Hudson commented on HBASE-10739: FAILURE: Integrated in HBase-TRUNK #5009 (See [https://builds.apache.org/job/HBase-TRUNK/5009/]) HBASE-10739 RS web UI NPE if master shuts down sooner (jxiang: rev 1577415) * /hbase/trunk/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/BackupMasterStatusTmpl.jamon * /hbase/trunk/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSStatusServlet.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterStatusServlet.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressTracker.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRSStatusServlet.java RS web UI NPE if master shuts down sooner - Key: HBASE-10739 URL: https://issues.apache.org/jira/browse/HBASE-10739 Project: HBase Issue Type: Bug Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.96.2, 0.98.1, 0.99.0 Attachments: hbase-10739.patch, hbase-10739_v2.patch This is similar to HBASE-9294. However, the fix in HBASE-9294 can't fix the issue. hrs is unlikely null over there since we have an assert earlier. We also called hrs.isOnline eariler. It is very likely the master shuts down sooner so the master address is null instead. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10740) Upgrade zookeeper to 3.4.6 release
[ https://issues.apache.org/jira/browse/HBASE-10740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-10740: --- Resolution: Fixed Status: Resolved (was: Patch Available) Upgrade zookeeper to 3.4.6 release -- Key: HBASE-10740 URL: https://issues.apache.org/jira/browse/HBASE-10740 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Priority: Minor Fix For: 0.99.0 Attachments: 10740-v1.txt Zookeeper 3.4.6 release has been released. This JIRA upgrades zookeeper dependency to 3.4.6 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bharath v updated HBASE-8963: - Attachment: HBASE-8963.trunk.v4.patch Add configuration option to skip HFile archiving Key: HBASE-8963 URL: https://issues.apache.org/jira/browse/HBASE-8963 Project: HBase Issue Type: Improvement Reporter: Ted Yu Assignee: bharath v Fix For: 0.99.0 Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch, HBASE-8963.trunk.v3.patch, HBASE-8963.trunk.v4.patch Currently HFileArchiver is always called when a table is dropped. A configuration option (either global or per table) should be provided so that archiving can be skipped when table is deleted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[ https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934726#comment-13934726 ] bharath v commented on HBASE-8963: -- [~yuzhih...@gmail.com] Added v4 as per your comments [~mbertozzi] I did it for 2 reasons 1) All the archive cleaner plugins (except snapshot and hfilelink cleaner) are written assuming the hfile is already in archive . For example TimToLiveHFileCleaner which check if the ttl of the hfile inside the archive directory exceeds the configured timeout. That doesn't need to be run on the hfiles which are not yet archived and will always return true. Also any further custom implementation of users will expect the store files to be already in archive directory and would probably fail if we run the wholeCleanerChain 2) Given this runs only when skipArchive is enabled, users are ok with these hfiles getting deleted and won't expect these hfiles to be checked via their custom cleaners. Are these assumptions valid? Add configuration option to skip HFile archiving Key: HBASE-8963 URL: https://issues.apache.org/jira/browse/HBASE-8963 Project: HBase Issue Type: Improvement Reporter: Ted Yu Assignee: bharath v Fix For: 0.99.0 Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch, HBASE-8963.trunk.v3.patch, HBASE-8963.trunk.v4.patch Currently HFileArchiver is always called when a table is dropped. A configuration option (either global or per table) should be provided so that archiving can be skipped when table is deleted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10648) Pluggable Memstore
[ https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-10648: --- Attachment: HBASE-10648_V6.patch Patch solves line length problem. The failed tests passing locally. Don't think it is related to this patch. Pluggable Memstore -- Key: HBASE-10648 URL: https://issues.apache.org/jira/browse/HBASE-10648 Project: HBase Issue Type: Sub-task Reporter: Anoop Sam John Assignee: Anoop Sam John Attachments: HBASE-10648.patch, HBASE-10648_V2.patch, HBASE-10648_V3.patch, HBASE-10648_V4.patch, HBASE-10648_V5.patch, HBASE-10648_V6.patch Make Memstore into an interface implementation. Also make it pluggable by configuring the FQCN of the impl. This will allow us to have different impl and optimizations in the Memstore DataStructure and the upper layers untouched. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10744) AM#CloseRegion no need to retry on FailedServerException
[ https://issues.apache.org/jira/browse/HBASE-10744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934733#comment-13934733 ] Hudson commented on HBASE-10744: FAILURE: Integrated in HBase-0.98 #229 (See [https://builds.apache.org/job/HBase-0.98/229/]) HBASE-10744 AM#CloseRegion no need to retry on FailedServerException (jxiang: rev 1577417) * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerOnCluster.java AM#CloseRegion no need to retry on FailedServerException Key: HBASE-10744 URL: https://issues.apache.org/jira/browse/HBASE-10744 Project: HBase Issue Type: Improvement Components: Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.96.2, 0.98.1, 0.99.0 Attachments: hbase-10744.patch When a regionserver restarts, AM#CloseRegion could get FailedServerException, or ServerNotRunningYetException. The call should not be retried since we know the original regionserver should be down. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10739) RS web UI NPE if master shuts down sooner
[ https://issues.apache.org/jira/browse/HBASE-10739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934734#comment-13934734 ] Hudson commented on HBASE-10739: FAILURE: Integrated in HBase-0.98 #229 (See [https://builds.apache.org/job/HBase-0.98/229/]) HBASE-10739 RS web UI NPE if master shuts down sooner (jxiang: rev 1577414) * /hbase/branches/0.98/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/BackupMasterStatusTmpl.jamon * /hbase/branches/0.98/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSStatusServlet.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterStatusServlet.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressTracker.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRSStatusServlet.java RS web UI NPE if master shuts down sooner - Key: HBASE-10739 URL: https://issues.apache.org/jira/browse/HBASE-10739 Project: HBase Issue Type: Bug Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.96.2, 0.98.1, 0.99.0 Attachments: hbase-10739.patch, hbase-10739_v2.patch This is similar to HBASE-9294. However, the fix in HBASE-9294 can't fix the issue. hrs is unlikely null over there since we have an assert earlier. We also called hrs.isOnline eariler. It is very likely the master shuts down sooner so the master address is null instead. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10514) Forward port HBASE-10466, possible data loss when failed flushes
[
https://issues.apache.org/jira/browse/HBASE-10514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934736#comment-13934736
]
Yu Li commented on HBASE-10514:
---
Hi [~stack],
{quote}
Patch for 0.94. Does not include tests
{quote}
Will the v94 patch be integrated into 0.94 branch w/o tests? Or you're still
working on it?
Forward port HBASE-10466, possible data loss when failed flushes
Key: HBASE-10514
URL: https://issues.apache.org/jira/browse/HBASE-10514
Project: HBase
Issue Type: Bug
Reporter: stack
Assignee: stack
Priority: Critical
Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.18
Attachments: 10154v4.096.txt, 10154v94.txt, 10514.txt, 10514v2.txt,
10514v3.txt, 10514v3.txt, 10514v4.txt, 10514v5.txt, 10514v6.098.txt
Critical data loss issues that we need to ensure are not in branches beyond
0.89fb. Assigning myself.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10744) AM#CloseRegion no need to retry on FailedServerException
[ https://issues.apache.org/jira/browse/HBASE-10744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934756#comment-13934756 ] Hudson commented on HBASE-10744: FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #214 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/214/]) HBASE-10744 AM#CloseRegion no need to retry on FailedServerException (jxiang: rev 1577417) * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerOnCluster.java AM#CloseRegion no need to retry on FailedServerException Key: HBASE-10744 URL: https://issues.apache.org/jira/browse/HBASE-10744 Project: HBase Issue Type: Improvement Components: Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.96.2, 0.98.1, 0.99.0 Attachments: hbase-10744.patch When a regionserver restarts, AM#CloseRegion could get FailedServerException, or ServerNotRunningYetException. The call should not be retried since we know the original regionserver should be down. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10739) RS web UI NPE if master shuts down sooner
[ https://issues.apache.org/jira/browse/HBASE-10739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934757#comment-13934757 ] Hudson commented on HBASE-10739: FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #214 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/214/]) HBASE-10739 RS web UI NPE if master shuts down sooner (jxiang: rev 1577414) * /hbase/branches/0.98/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/BackupMasterStatusTmpl.jamon * /hbase/branches/0.98/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSStatusServlet.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterStatusServlet.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressTracker.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRSStatusServlet.java RS web UI NPE if master shuts down sooner - Key: HBASE-10739 URL: https://issues.apache.org/jira/browse/HBASE-10739 Project: HBase Issue Type: Bug Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.96.2, 0.98.1, 0.99.0 Attachments: hbase-10739.patch, hbase-10739_v2.patch This is similar to HBASE-9294. However, the fix in HBASE-9294 can't fix the issue. hrs is unlikely null over there since we have an assert earlier. We also called hrs.isOnline eariler. It is very likely the master shuts down sooner so the master address is null instead. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10514) Forward port HBASE-10466, possible data loss when failed flushes
[ https://issues.apache.org/jira/browse/HBASE-10514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934759#comment-13934759 ] Hudson commented on HBASE-10514: FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #117 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/117/]) HBASE-10514 Forward port HBASE-10466, possible data loss when failed flushes (stack: rev 1577353) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreConfigInformation.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionBusyWait.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java Forward port HBASE-10466, possible data loss when failed flushes Key: HBASE-10514 URL: https://issues.apache.org/jira/browse/HBASE-10514 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Priority: Critical Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.18 Attachments: 10154v4.096.txt, 10154v94.txt, 10514.txt, 10514v2.txt, 10514v3.txt, 10514v3.txt, 10514v4.txt, 10514v5.txt, 10514v6.098.txt Critical data loss issues that we need to ensure are not in branches beyond 0.89fb. Assigning myself. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10739) RS web UI NPE if master shuts down sooner
[ https://issues.apache.org/jira/browse/HBASE-10739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934762#comment-13934762 ] Hudson commented on HBASE-10739: FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #117 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/117/]) HBASE-10739 RS web UI NPE if master shuts down sooner (jxiang: rev 1577415) * /hbase/trunk/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/BackupMasterStatusTmpl.jamon * /hbase/trunk/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSStatusServlet.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterStatusServlet.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressTracker.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRSStatusServlet.java RS web UI NPE if master shuts down sooner - Key: HBASE-10739 URL: https://issues.apache.org/jira/browse/HBASE-10739 Project: HBase Issue Type: Bug Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.96.2, 0.98.1, 0.99.0 Attachments: hbase-10739.patch, hbase-10739_v2.patch This is similar to HBASE-9294. However, the fix in HBASE-9294 can't fix the issue. hrs is unlikely null over there since we have an assert earlier. We also called hrs.isOnline eariler. It is very likely the master shuts down sooner so the master address is null instead. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10184) [Online Schema Change]: Add additional tests for online schema change
[ https://issues.apache.org/jira/browse/HBASE-10184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934760#comment-13934760 ] Hudson commented on HBASE-10184: FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #117 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/117/]) HBASE-10184 Addendum fixes compilation against hadoop-1 (tedyu: rev 1577394) * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestRowCounter.java HBASE-10184 [Online Schema Change]: Add additional tests for online schema change (Aleksandr Shulman) (apurtell: rev 1577375) * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestRowCounter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java [Online Schema Change]: Add additional tests for online schema change - Key: HBASE-10184 URL: https://issues.apache.org/jira/browse/HBASE-10184 Project: HBase Issue Type: Task Components: test Affects Versions: 0.96.1, 0.99.0 Reporter: Aleksandr Shulman Assignee: Aleksandr Shulman Labels: online_schema_change Fix For: 0.99.0, 0.98.2 Attachments: 10184-4.patch, 10184.addendum, HBASE-10184-trunk.diff There are some gaps in testing for Online Schema Change: Examples of some tests that should be added: 1. Splits with online schema change 2. Merge during online schema change 3. MR over HBase during online schema change 4. Bulk Load during online schema change 5. Online change table owner 6. Online Replication scope change 7. Online Bloom Filter change 8. Snapshots during online schema change (HBASE-10136) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10738) AssignmentManager should shut down executors on stop
[ https://issues.apache.org/jira/browse/HBASE-10738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934764#comment-13934764 ] Hudson commented on HBASE-10738: FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #117 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/117/]) HBASE-10738 AssignmentManager should shut down executors on stop (jxiang: rev 1577357) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java AssignmentManager should shut down executors on stop Key: HBASE-10738 URL: https://issues.apache.org/jira/browse/HBASE-10738 Project: HBase Issue Type: Bug Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.18 Attachments: hbase-10738.patch This won't cause any issue in live cluster. However, in unit tests, it could leak threads and slow down tests. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10740) Upgrade zookeeper to 3.4.6 release
[ https://issues.apache.org/jira/browse/HBASE-10740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934761#comment-13934761 ] Hudson commented on HBASE-10740: FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #117 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/117/]) HBASE-10740 Upgrade zookeeper to 3.4.6 release (tedyu: rev 1577412) * /hbase/trunk/pom.xml Upgrade zookeeper to 3.4.6 release -- Key: HBASE-10740 URL: https://issues.apache.org/jira/browse/HBASE-10740 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Priority: Minor Fix For: 0.99.0 Attachments: 10740-v1.txt Zookeeper 3.4.6 release has been released. This JIRA upgrades zookeeper dependency to 3.4.6 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10744) AM#CloseRegion no need to retry on FailedServerException
[ https://issues.apache.org/jira/browse/HBASE-10744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934758#comment-13934758 ] Hudson commented on HBASE-10744: FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #117 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/117/]) HBASE-10744 AM#CloseRegion no need to retry on FailedServerException (jxiang: rev 1577416) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerOnCluster.java AM#CloseRegion no need to retry on FailedServerException Key: HBASE-10744 URL: https://issues.apache.org/jira/browse/HBASE-10744 Project: HBase Issue Type: Improvement Components: Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.96.2, 0.98.1, 0.99.0 Attachments: hbase-10744.patch When a regionserver restarts, AM#CloseRegion could get FailedServerException, or ServerNotRunningYetException. The call should not be retried since we know the original regionserver should be down. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10466) Bugs that make flushes skipped during HRegion close could cause data loss
[ https://issues.apache.org/jira/browse/HBASE-10466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934765#comment-13934765 ] Hudson commented on HBASE-10466: FAILURE: Integrated in HBase-TRUNK-on-Hadoop-1.1 #117 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-1.1/117/]) HBASE-10514 Forward port HBASE-10466, possible data loss when failed flushes (stack: rev 1577353) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreConfigInformation.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionBusyWait.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java Bugs that make flushes skipped during HRegion close could cause data loss - Key: HBASE-10466 URL: https://issues.apache.org/jira/browse/HBASE-10466 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.89-fb Reporter: Yunfan Zhong Priority: Critical Fix For: 0.89-fb Attachments: Fix-bugs-that-causes-flushes-being-skipped-during-re.patch During region close, there are two flushes to ensure nothing is persisted in memory. When there is data in current memstore only, 1 flush is required. When there is data also in memstore's snapshot, 2 flushes are essential otherwise we have data loss. However, recently we found two bugs that lead to at least 1 flush skipped and caused data loss. Bug 1: Wrong calculation of HRegion.memstoreSize When a flush fails, data to be flushed is kept in each MemStore's snapshot and wait for next flush attempt to continue on it. But when the next flush succeeds, the counter of total memstore size in HRegion is always deduced by the sum of current memstore sizes instead of snapshots left from previous failed flush. This calculation is problematic that almost every time there is failed flush, HRegion.memstoreSize gets reduced by a wrong value. If region flush could not proceed for a couple cycles, the size in current memstore could be much larger than the snapshot. It's likely to drift memstoreSize much smaller than expected. In extreme case, if the error accumulates to even bigger than HRegion's memstore size limit, any further flush is skipped because flush does not do anything if memstoreSize is not larger than 0. When the region is closing, if the two flushes get skipped and leave data in current memstore and/or snapshot, we could lose data up to the memstore size limit of the region. The fix is deducing correct size of data that is going to be flushed from memstoreSize. Bug 2: Conditions for the first flush of region close (so-called pre-flush) If memstoreSize is smaller than a certain value, or when region close starts a flush is ongoing, the first flush is skipped and only the second flush takes place. However, two flushes are required in case previous flush fails and leaves some data in snapshot. The bug could cause loss of data in current memstore. The fix is removing all conditions except abort check so we ensure 2 flushes for region close. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10184) [Online Schema Change]: Add additional tests for online schema change
[ https://issues.apache.org/jira/browse/HBASE-10184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934770#comment-13934770 ] Hudson commented on HBASE-10184: FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #215 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/215/]) Revert HBASE-10184 Add additional tests for online schema change; revert patch and addendum (apurtell: rev 1577439) * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestRowCounter.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java [Online Schema Change]: Add additional tests for online schema change - Key: HBASE-10184 URL: https://issues.apache.org/jira/browse/HBASE-10184 Project: HBase Issue Type: Task Components: test Affects Versions: 0.96.1, 0.99.0 Reporter: Aleksandr Shulman Assignee: Aleksandr Shulman Labels: online_schema_change Fix For: 0.99.0, 0.98.2 Attachments: 10184-4.patch, 10184.addendum, HBASE-10184-trunk.diff There are some gaps in testing for Online Schema Change: Examples of some tests that should be added: 1. Splits with online schema change 2. Merge during online schema change 3. MR over HBase during online schema change 4. Bulk Load during online schema change 5. Online change table owner 6. Online Replication scope change 7. Online Bloom Filter change 8. Snapshots during online schema change (HBASE-10136) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-7847) Use zookeeper multi to clear znodes
[
https://issues.apache.org/jira/browse/HBASE-7847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934780#comment-13934780
]
Hadoop QA commented on HBASE-7847:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12634657/HBASE-7847_v4.patch
against trunk revision .
ATTACHMENT ID: 12634657
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction
org.apache.hadoop.hbase.client.TestAdmin
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8988//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8988//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8988//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8988//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8988//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8988//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8988//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8988//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8988//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8988//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8988//console
This message is automatically generated.
Use zookeeper multi to clear znodes
---
Key: HBASE-7847
URL: https://issues.apache.org/jira/browse/HBASE-7847
Project: HBase
Issue Type: Sub-task
Reporter: Ted Yu
Attachments: 7847-v1.txt, HBASE-7847.patch, HBASE-7847.patch,
HBASE-7847.patch, HBASE-7847_v4.patch
In ZKProcedureUtil, clearChildZNodes() and clearZNodes(String procedureName)
should utilize zookeeper multi so that they're atomic
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[
https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934784#comment-13934784
]
Matteo Bertozzi commented on HBASE-8963:
{quote}All the archive cleaner plugins (except snapshot and hfilelink cleaner)
are written assuming the hfile is already in archive .{quote}
What about the custom ones that the user has written, or the ones that we will
write in the future?
I'll be the first to forget to update the archiver if I add a new cleaner like
the snapshot one.
{quote}Given this runs only when skipArchive is enabled, users are ok with
these hfiles getting deleted and won't expect these hfiles to be checked via
their custom cleaners.{quote}
I think users are ok to get unreferenced files deleted.
e.g. If I take a snapshot on the table and the delete the table, I want the
snapshot to be sane.
Also this settings is global and not per table which means.. that you may just
turn this on at some point in the future because your new table takes long time
archiving... but you don't want to lose snapshots (or other files that your
custom cleaner should keep around) for the other tables.
Add configuration option to skip HFile archiving
Key: HBASE-8963
URL: https://issues.apache.org/jira/browse/HBASE-8963
Project: HBase
Issue Type: Improvement
Reporter: Ted Yu
Assignee: bharath v
Fix For: 0.99.0
Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch,
HBASE-8963.trunk.v3.patch, HBASE-8963.trunk.v4.patch
Currently HFileArchiver is always called when a table is dropped.
A configuration option (either global or per table) should be provided so
that archiving can be skipped when table is deleted.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-10648) Pluggable Memstore
[ https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated HBASE-10648: -- Attachment: HBASE-10648-0.94_v1.patch Have done some work backporting this patch for 0.94. The attached patch is based on V4 patch, didn't follow V6 because HBASE-10514 patch for v94 still doesn't include the UT change there. Could update the patch later. [~anoop.hbase] please help review the patch, and hope I'm not doing any duplicated work here. And btw, you could see I'm ready to try out and help test patch for HBASE-10713 now :-) Pluggable Memstore -- Key: HBASE-10648 URL: https://issues.apache.org/jira/browse/HBASE-10648 Project: HBase Issue Type: Sub-task Reporter: Anoop Sam John Assignee: Anoop Sam John Attachments: HBASE-10648-0.94_v1.patch, HBASE-10648.patch, HBASE-10648_V2.patch, HBASE-10648_V3.patch, HBASE-10648_V4.patch, HBASE-10648_V5.patch, HBASE-10648_V6.patch Make Memstore into an interface implementation. Also make it pluggable by configuring the FQCN of the impl. This will allow us to have different impl and optimizations in the Memstore DataStructure and the upper layers untouched. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10744) AM#CloseRegion no need to retry on FailedServerException
[ https://issues.apache.org/jira/browse/HBASE-10744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934804#comment-13934804 ] Hudson commented on HBASE-10744: SUCCESS: Integrated in hbase-0.96 #347 (See [https://builds.apache.org/job/hbase-0.96/347/]) HBASE-10744 AM#CloseRegion no need to retry on FailedServerException (jxiang: rev 1577418) * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerOnCluster.java AM#CloseRegion no need to retry on FailedServerException Key: HBASE-10744 URL: https://issues.apache.org/jira/browse/HBASE-10744 Project: HBase Issue Type: Improvement Components: Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.96.2, 0.98.1, 0.99.0 Attachments: hbase-10744.patch When a regionserver restarts, AM#CloseRegion could get FailedServerException, or ServerNotRunningYetException. The call should not be retried since we know the original regionserver should be down. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[ https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934806#comment-13934806 ] Anoop Sam John commented on HBASE-10648: Sorry I will need some more time to post a patch for HBASE-10713. Before that I want to make the MsLAB also as pluggable. (On heap or off heap impls can come in) This change I am doing now. BTW I am not sure whether this kind of a change can go into an older version like 94. You want to internally maintain that (?) Will review the attached patch any way. Pluggable Memstore -- Key: HBASE-10648 URL: https://issues.apache.org/jira/browse/HBASE-10648 Project: HBase Issue Type: Sub-task Reporter: Anoop Sam John Assignee: Anoop Sam John Attachments: HBASE-10648-0.94_v1.patch, HBASE-10648.patch, HBASE-10648_V2.patch, HBASE-10648_V3.patch, HBASE-10648_V4.patch, HBASE-10648_V5.patch, HBASE-10648_V6.patch Make Memstore into an interface implementation. Also make it pluggable by configuring the FQCN of the impl. This will allow us to have different impl and optimizations in the Memstore DataStructure and the upper layers untouched. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-8963) Add configuration option to skip HFile archiving
[
https://issues.apache.org/jira/browse/HBASE-8963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934815#comment-13934815
]
Hadoop QA commented on HBASE-8963:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12634669/HBASE-8963.trunk.v4.patch
against trunk revision .
ATTACHMENT ID: 12634669
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestHRegion
org.apache.hadoop.hbase.client.TestAdmin
org.apache.hadoop.hbase.regionserver.TestHRegionBusyWait
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8989//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8989//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8989//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8989//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8989//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8989//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8989//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8989//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8989//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8989//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8989//console
This message is automatically generated.
Add configuration option to skip HFile archiving
Key: HBASE-8963
URL: https://issues.apache.org/jira/browse/HBASE-8963
Project: HBase
Issue Type: Improvement
Reporter: Ted Yu
Assignee: bharath v
Fix For: 0.99.0
Attachments: HBASE-8963.trunk.v1.patch, HBASE-8963.trunk.v2.patch,
HBASE-8963.trunk.v3.patch, HBASE-8963.trunk.v4.patch
Currently HFileArchiver is always called when a table is dropped.
A configuration option (either global or per table) should be provided so
that archiving can be skipped when table is deleted.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HBASE-10748) hbase-daemon.sh fails to execute with 'sh' command
Ashish Singhi created HBASE-10748: - Summary: hbase-daemon.sh fails to execute with 'sh' command Key: HBASE-10748 URL: https://issues.apache.org/jira/browse/HBASE-10748 Project: HBase Issue Type: Bug Affects Versions: 0.94.11 Reporter: Ashish Singhi hostname:HBASE_HOME/bin # sh hbase-daemon.sh restart master *hbase-daemon.sh: line 188: hbase-daemon.sh: command not found* *hbase-daemon.sh: line 196: hbase-daemon.sh: command not found* -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[
https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934819#comment-13934819
]
Hadoop QA commented on HBASE-10648:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12634676/HBASE-10648-0.94_v1.patch
against trunk revision .
ATTACHMENT ID: 12634676
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 18 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8991//console
This message is automatically generated.
Pluggable Memstore
--
Key: HBASE-10648
URL: https://issues.apache.org/jira/browse/HBASE-10648
Project: HBase
Issue Type: Sub-task
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Attachments: HBASE-10648-0.94_v1.patch, HBASE-10648.patch,
HBASE-10648_V2.patch, HBASE-10648_V3.patch, HBASE-10648_V4.patch,
HBASE-10648_V5.patch, HBASE-10648_V6.patch
Make Memstore into an interface implementation. Also make it pluggable by
configuring the FQCN of the impl.
This will allow us to have different impl and optimizations in the Memstore
DataStructure and the upper layers untouched.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-10748) hbase-daemon.sh fails to execute with 'sh' command
[ https://issues.apache.org/jira/browse/HBASE-10748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-10748: -- Attachment: HBASE-10748.patch Attaching a simple patch for this issue. hbase-daemon.sh fails to execute with 'sh' command -- Key: HBASE-10748 URL: https://issues.apache.org/jira/browse/HBASE-10748 Project: HBase Issue Type: Bug Affects Versions: 0.94.11 Reporter: Ashish Singhi Attachments: HBASE-10748.patch hostname:HBASE_HOME/bin # sh hbase-daemon.sh restart master *hbase-daemon.sh: line 188: hbase-daemon.sh: command not found* *hbase-daemon.sh: line 196: hbase-daemon.sh: command not found* -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10748) hbase-daemon.sh fails to execute with 'sh' command
[ https://issues.apache.org/jira/browse/HBASE-10748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-10748: -- Status: Patch Available (was: Open) Please review. hbase-daemon.sh fails to execute with 'sh' command -- Key: HBASE-10748 URL: https://issues.apache.org/jira/browse/HBASE-10748 Project: HBase Issue Type: Bug Affects Versions: 0.94.11 Reporter: Ashish Singhi Attachments: HBASE-10748.patch hostname:HBASE_HOME/bin # sh hbase-daemon.sh restart master *hbase-daemon.sh: line 188: hbase-daemon.sh: command not found* *hbase-daemon.sh: line 196: hbase-daemon.sh: command not found* -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[
https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934833#comment-13934833
]
Yu Li commented on HBASE-10648:
---
{quote}
Sorry I will need some more time to post a patch for HBASE-10713
{quote}
I see, it's ok, thanks for the update. In this case I might start writing some
code by myself (you know, the deadline thing :-)), and I'll appreciate a lot if
you can send your drafted patch to me through email, before it's ready for
public. Just want to avoid duplicated work as much as possible.
{quote}
BTW I am not sure whether this kind of a change can go into an older version
like 94
{quote}
Sorry but what's preventing the change to go into 94? The CSLM issue (when
there are too many KVs in it) also exists in v94, and we also want to get
relieved from that:-) Of course I'm not expecting the whole HBASE-10191
solution go into 94, but after making Memstore impl pluggable, at least
in-memory flush impl would fit well in 0.94, right?
{quote}
You want to internally maintain that?
{quote}
It's not an easy decision to upgrade to a new version in product env, and I'm
afraid we'll stay on 94 for some while, so yes we will maintain this
internally. But of course I'll try to contribute back to 94 branch after fully
test, so others using 94 can also benefit.
Pluggable Memstore
--
Key: HBASE-10648
URL: https://issues.apache.org/jira/browse/HBASE-10648
Project: HBase
Issue Type: Sub-task
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Attachments: HBASE-10648-0.94_v1.patch, HBASE-10648.patch,
HBASE-10648_V2.patch, HBASE-10648_V3.patch, HBASE-10648_V4.patch,
HBASE-10648_V5.patch, HBASE-10648_V6.patch
Make Memstore into an interface implementation. Also make it pluggable by
configuring the FQCN of the impl.
This will allow us to have different impl and optimizations in the Memstore
DataStructure and the upper layers untouched.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-10648) Pluggable Memstore
[ https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-10648: --- Attachment: (was: HBASE-10648_V6.patch) Pluggable Memstore -- Key: HBASE-10648 URL: https://issues.apache.org/jira/browse/HBASE-10648 Project: HBase Issue Type: Sub-task Reporter: Anoop Sam John Assignee: Anoop Sam John Attachments: HBASE-10648-0.94_v1.patch, HBASE-10648.patch, HBASE-10648_V2.patch, HBASE-10648_V3.patch, HBASE-10648_V4.patch, HBASE-10648_V5.patch Make Memstore into an interface implementation. Also make it pluggable by configuring the FQCN of the impl. This will allow us to have different impl and optimizations in the Memstore DataStructure and the upper layers untouched. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10648) Pluggable Memstore
[ https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-10648: --- Attachment: HBASE-10648_V6.patch QA has taken the latest attachment (94 patch). Reattaching trunk patch to get a QA run. Pluggable Memstore -- Key: HBASE-10648 URL: https://issues.apache.org/jira/browse/HBASE-10648 Project: HBase Issue Type: Sub-task Reporter: Anoop Sam John Assignee: Anoop Sam John Attachments: HBASE-10648-0.94_v1.patch, HBASE-10648.patch, HBASE-10648_V2.patch, HBASE-10648_V3.patch, HBASE-10648_V4.patch, HBASE-10648_V5.patch, HBASE-10648_V6.patch Make Memstore into an interface implementation. Also make it pluggable by configuring the FQCN of the impl. This will allow us to have different impl and optimizations in the Memstore DataStructure and the upper layers untouched. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[ https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934844#comment-13934844 ] Anoop Sam John commented on HBASE-10648: bq.Sorry but what's preventing the change to go into 94?The CSLM issue (when there are too many KVs in it) also exists in v94, and we also want to get relieved from that Of course I'm not expecting the whole HBASE-10191 solution go into 94, but after making Memstore impl pluggable, at least in-memory flush impl would fit well in 0.94, right? Ping [~larsh] abt the decision. Pluggable Memstore -- Key: HBASE-10648 URL: https://issues.apache.org/jira/browse/HBASE-10648 Project: HBase Issue Type: Sub-task Reporter: Anoop Sam John Assignee: Anoop Sam John Attachments: HBASE-10648-0.94_v1.patch, HBASE-10648.patch, HBASE-10648_V2.patch, HBASE-10648_V3.patch, HBASE-10648_V4.patch, HBASE-10648_V5.patch, HBASE-10648_V6.patch Make Memstore into an interface implementation. Also make it pluggable by configuring the FQCN of the impl. This will allow us to have different impl and optimizations in the Memstore DataStructure and the upper layers untouched. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[
https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934846#comment-13934846
]
Hadoop QA commented on HBASE-10648:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12634670/HBASE-10648_V6.patch
against trunk revision .
ATTACHMENT ID: 12634670
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 26 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestHRegionBusyWait
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.regionserver.wal.TestLogRolling.testLogRollOnDatanodeDeath(TestLogRolling.java:368)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8990//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8990//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8990//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8990//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8990//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8990//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8990//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8990//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8990//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8990//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8990//console
This message is automatically generated.
Pluggable Memstore
--
Key: HBASE-10648
URL: https://issues.apache.org/jira/browse/HBASE-10648
Project: HBase
Issue Type: Sub-task
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Attachments: HBASE-10648-0.94_v1.patch, HBASE-10648.patch,
HBASE-10648_V2.patch, HBASE-10648_V3.patch, HBASE-10648_V4.patch,
HBASE-10648_V5.patch, HBASE-10648_V6.patch
Make Memstore into an interface implementation. Also make it pluggable by
configuring the FQCN of the impl.
This will allow us to have different impl and optimizations in the Memstore
DataStructure and the upper layers untouched.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HBASE-10749) CellComparator.compareStatic() compares type wrongly
ramkrishna.s.vasudevan created HBASE-10749:
--
Summary: CellComparator.compareStatic() compares type wrongly
Key: HBASE-10749
URL: https://issues.apache.org/jira/browse/HBASE-10749
Project: HBase
Issue Type: Bug
Affects Versions: 0.96.1.1, 0.98.1, 0.99.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Critical
Fix For: 0.98.1, 0.99.0
CellComparator.compareStatic(Cell a, Cell b) compares type wrongly.
{code}
//type
c = (0xff a.getTypeByte()) - (0xff b.getTypeByte());
if (c != 0) return c;
{code}
In the normal case it should be the other way
{code}
// Compare types. Let the delete types sort ahead of puts; i.e. types
// of higher numbers sort before those of lesser numbers. Maximum (255)
// appears ahead of everything, and minimum (0) appears after
// everything.
return (0xff rtype) - (0xff ltype);
{code}
Found this issue while replacing KVComparator to use CellComparator.
[~mcorgan]
Could you confirm this once? Currently PrefixTree code path uses this
compareStatic method.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[
https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934850#comment-13934850
]
Anoop Sam John commented on HBASE-10648:
Gone through the 94 patch. Looks good.
{code}
-flushed += this.memstore.heapSizeChange(kv, true);
+flushed += DefaultMemStore.heapSizeChange(kv, true);
{code}
It won't be good to directly use some impl here. We use flushed variable for
logging purpose only right (?) In trunk it is. Can you use the size available
in MemStoreSnapshot (?)
DefaultMemStore#snapshot()
{code}
+ long snapshotSize = 0;
...
return new MemStoreSnapshot(this.snapshotId, snapshot.size(),
+ snapshotSize, this.snapshotTimeRangeTracker,
+ new CollectionBackedScanner(snapshot, this.comparator));
{code}
Make snapshotSize = keySize().
Any way once HBASE-10514 is in to 94, will need some change here and
snapshotSize will come as an instance member.
Pluggable Memstore
--
Key: HBASE-10648
URL: https://issues.apache.org/jira/browse/HBASE-10648
Project: HBase
Issue Type: Sub-task
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Attachments: HBASE-10648-0.94_v1.patch, HBASE-10648.patch,
HBASE-10648_V2.patch, HBASE-10648_V3.patch, HBASE-10648_V4.patch,
HBASE-10648_V5.patch, HBASE-10648_V6.patch
Make Memstore into an interface implementation. Also make it pluggable by
configuring the FQCN of the impl.
This will allow us to have different impl and optimizations in the Memstore
DataStructure and the upper layers untouched.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10748) hbase-daemon.sh fails to execute with 'sh' command
[
https://issues.apache.org/jira/browse/HBASE-10748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934852#comment-13934852
]
Kashif J S commented on HBASE-10748:
For consistency across shell instead of using name hbase-daemon.sh, you can
make use of 'basename' cmd like below
thiscmd=$bin/`basename ${BASH_SOURCE-$0}`
OR
thiscmd=$bin/$(basename ${BASH_SOURCE-$0})
hbase-daemon.sh fails to execute with 'sh' command
--
Key: HBASE-10748
URL: https://issues.apache.org/jira/browse/HBASE-10748
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.11
Reporter: Ashish Singhi
Attachments: HBASE-10748.patch
hostname:HBASE_HOME/bin # sh hbase-daemon.sh restart master
*hbase-daemon.sh: line 188: hbase-daemon.sh: command not found*
*hbase-daemon.sh: line 196: hbase-daemon.sh: command not found*
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10749) CellComparator.compareStatic() compares type wrongly
[
https://issues.apache.org/jira/browse/HBASE-10749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934856#comment-13934856
]
Anoop Sam John commented on HBASE-10749:
Correct Ram. Cells with higher type should come before.
Also there is some special handling for Minimum and Maximum types. We will
need that also here?
CellComparator.compareStatic() compares type wrongly
Key: HBASE-10749
URL: https://issues.apache.org/jira/browse/HBASE-10749
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.1, 0.99.0, 0.96.1.1
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Critical
Fix For: 0.98.1, 0.99.0
CellComparator.compareStatic(Cell a, Cell b) compares type wrongly.
{code}
//type
c = (0xff a.getTypeByte()) - (0xff b.getTypeByte());
if (c != 0) return c;
{code}
In the normal case it should be the other way
{code}
// Compare types. Let the delete types sort ahead of puts; i.e. types
// of higher numbers sort before those of lesser numbers. Maximum (255)
// appears ahead of everything, and minimum (0) appears after
// everything.
return (0xff rtype) - (0xff ltype);
{code}
Found this issue while replacing KVComparator to use CellComparator.
[~mcorgan]
Could you confirm this once? Currently PrefixTree code path uses this
compareStatic method.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HBASE-10750) Pluggable MemStoreLAB
Anoop Sam John created HBASE-10750: -- Summary: Pluggable MemStoreLAB Key: HBASE-10750 URL: https://issues.apache.org/jira/browse/HBASE-10750 Project: HBase Issue Type: Sub-task Components: regionserver Reporter: Anoop Sam John Assignee: Anoop Sam John This can allow one to have offheap MSLAB also. The allocate() returns a byte[] now. This has to be changed to ByteRange so that the impl can return on heap byte[] backed BR or offheap buffer backed BR. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10748) hbase-daemon.sh fails to execute with 'sh' command
[ https://issues.apache.org/jira/browse/HBASE-10748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-10748: -- Attachment: HBASE-10748.patch Thanks [~kashi...@huawei.com] for taking a look into the patch. Updated the patch as per comment. hbase-daemon.sh fails to execute with 'sh' command -- Key: HBASE-10748 URL: https://issues.apache.org/jira/browse/HBASE-10748 Project: HBase Issue Type: Bug Affects Versions: 0.94.11 Reporter: Ashish Singhi Attachments: HBASE-10748.patch, HBASE-10748.patch hostname:HBASE_HOME/bin # sh hbase-daemon.sh restart master *hbase-daemon.sh: line 188: hbase-daemon.sh: command not found* *hbase-daemon.sh: line 196: hbase-daemon.sh: command not found* -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10750) Pluggable MemStoreLAB
[ https://issues.apache.org/jira/browse/HBASE-10750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934866#comment-13934866 ] Anoop Sam John commented on HBASE-10750: Patch is ready. Will attach once HBASE-10648 goes in. Pluggable MemStoreLAB - Key: HBASE-10750 URL: https://issues.apache.org/jira/browse/HBASE-10750 Project: HBase Issue Type: Sub-task Components: regionserver Reporter: Anoop Sam John Assignee: Anoop Sam John This can allow one to have offheap MSLAB also. The allocate() returns a byte[] now. This has to be changed to ByteRange so that the impl can return on heap byte[] backed BR or offheap buffer backed BR. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10184) [Online Schema Change]: Add additional tests for online schema change
[ https://issues.apache.org/jira/browse/HBASE-10184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934882#comment-13934882 ] Hudson commented on HBASE-10184: FAILURE: Integrated in HBase-0.98 #230 (See [https://builds.apache.org/job/HBase-0.98/230/]) Revert HBASE-10184 Add additional tests for online schema change; revert patch and addendum (apurtell: rev 1577439) * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestRowCounter.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java [Online Schema Change]: Add additional tests for online schema change - Key: HBASE-10184 URL: https://issues.apache.org/jira/browse/HBASE-10184 Project: HBase Issue Type: Task Components: test Affects Versions: 0.96.1, 0.99.0 Reporter: Aleksandr Shulman Assignee: Aleksandr Shulman Labels: online_schema_change Fix For: 0.99.0, 0.98.2 Attachments: 10184-4.patch, 10184.addendum, HBASE-10184-trunk.diff There are some gaps in testing for Online Schema Change: Examples of some tests that should be added: 1. Splits with online schema change 2. Merge during online schema change 3. MR over HBase during online schema change 4. Bulk Load during online schema change 5. Online change table owner 6. Online Replication scope change 7. Online Bloom Filter change 8. Snapshots during online schema change (HBASE-10136) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10749) CellComparator.compareStatic() compares type wrongly
[
https://issues.apache.org/jira/browse/HBASE-10749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934896#comment-13934896
]
ramkrishna.s.vasudevan commented on HBASE-10749:
bq.Also there is some special handling for Minimum and Maximum types
I think that is already happening in the current code also.
CellComparator.compareStatic() compares type wrongly
Key: HBASE-10749
URL: https://issues.apache.org/jira/browse/HBASE-10749
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.1, 0.99.0, 0.96.1.1
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Critical
Fix For: 0.98.1, 0.99.0
CellComparator.compareStatic(Cell a, Cell b) compares type wrongly.
{code}
//type
c = (0xff a.getTypeByte()) - (0xff b.getTypeByte());
if (c != 0) return c;
{code}
In the normal case it should be the other way
{code}
// Compare types. Let the delete types sort ahead of puts; i.e. types
// of higher numbers sort before those of lesser numbers. Maximum (255)
// appears ahead of everything, and minimum (0) appears after
// everything.
return (0xff rtype) - (0xff ltype);
{code}
Found this issue while replacing KVComparator to use CellComparator.
[~mcorgan]
Could you confirm this once? Currently PrefixTree code path uses this
compareStatic method.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10749) CellComparator.compareStatic() compares type wrongly
[
https://issues.apache.org/jira/browse/HBASE-10749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934895#comment-13934895
]
ramkrishna.s.vasudevan commented on HBASE-10749:
Ya true. Just wanted to confirm once was it done on any purpose?
CellComparator.compareStatic() compares type wrongly
Key: HBASE-10749
URL: https://issues.apache.org/jira/browse/HBASE-10749
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.1, 0.99.0, 0.96.1.1
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Critical
Fix For: 0.98.1, 0.99.0
CellComparator.compareStatic(Cell a, Cell b) compares type wrongly.
{code}
//type
c = (0xff a.getTypeByte()) - (0xff b.getTypeByte());
if (c != 0) return c;
{code}
In the normal case it should be the other way
{code}
// Compare types. Let the delete types sort ahead of puts; i.e. types
// of higher numbers sort before those of lesser numbers. Maximum (255)
// appears ahead of everything, and minimum (0) appears after
// everything.
return (0xff rtype) - (0xff ltype);
{code}
Found this issue while replacing KVComparator to use CellComparator.
[~mcorgan]
Could you confirm this once? Currently PrefixTree code path uses this
compareStatic method.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10531) Revisit how the key byte[] is passed to HFileScanner.seekTo and reseekTo
[ https://issues.apache.org/jira/browse/HBASE-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934903#comment-13934903 ] ramkrishna.s.vasudevan commented on HBASE-10531: bq.Shouldn't this be unsupportedoperationexception in your new key only class? For the value part we can throw UnSupportedException but for the Tag part we cannot because in the latest comparisons we do we also compare the seqid added during WAL replay which is inside a tag. So to find an exact key we may end up comparing the tag with seqid also. Revisit how the key byte[] is passed to HFileScanner.seekTo and reseekTo Key: HBASE-10531 URL: https://issues.apache.org/jira/browse/HBASE-10531 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0 Attachments: HBASE-10531.patch, HBASE-10531_1.patch, HBASE-10531_2.patch Currently the byte[] key passed to HFileScanner.seekTo and HFileScanner.reseekTo, is a combination of row, cf, qual, type and ts. And the caller forms this by using kv.getBuffer, which is actually deprecated. So see how this can be achieved considering kv.getBuffer is removed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10748) hbase-daemon.sh fails to execute with 'sh' command
[
https://issues.apache.org/jira/browse/HBASE-10748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934906#comment-13934906
]
Hadoop QA commented on HBASE-10748:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12634684/HBASE-10748.patch
against trunk revision .
ATTACHMENT ID: 12634684
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestHRegion
org.apache.hadoop.hbase.regionserver.TestHRegionBusyWait
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8992//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8992//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8992//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8992//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8992//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8992//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8992//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8992//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8992//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8992//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8992//console
This message is automatically generated.
hbase-daemon.sh fails to execute with 'sh' command
--
Key: HBASE-10748
URL: https://issues.apache.org/jira/browse/HBASE-10748
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.11
Reporter: Ashish Singhi
Attachments: HBASE-10748.patch, HBASE-10748.patch
hostname:HBASE_HOME/bin # sh hbase-daemon.sh restart master
*hbase-daemon.sh: line 188: hbase-daemon.sh: command not found*
*hbase-daemon.sh: line 196: hbase-daemon.sh: command not found*
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10278) Provide better write predictability
[ https://issues.apache.org/jira/browse/HBASE-10278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934926#comment-13934926 ] Jean-Marc Spaggiari commented on HBASE-10278: - Is there any metrics to say we have switched that number of times, we have waiting an average of before switching, in total operation took x ms, and in total it will have taken x ms (based on the duration of the first thread)? Or we do not record that for now? Provide better write predictability --- Key: HBASE-10278 URL: https://issues.apache.org/jira/browse/HBASE-10278 Project: HBase Issue Type: New Feature Reporter: Himanshu Vashishtha Assignee: Himanshu Vashishtha Attachments: 10278-wip-1.1.patch, Multiwaldesigndoc.pdf, SwitchWriterFlow.pptx Currently, HBase has one WAL per region server. Whenever there is any latency in the write pipeline (due to whatever reasons such as n/w blip, a node in the pipeline having a bad disk, etc), the overall write latency suffers. Jonathan Hsieh and I analyzed various approaches to tackle this issue. We also looked at HBASE-5699, which talks about adding concurrent multi WALs. Along with performance numbers, we also focussed on design simplicity, minimum impact on MTTR Replication, and compatibility with 0.96 and 0.98. Considering all these parameters, we propose a new HLog implementation with WAL Switching functionality. Please find attached the design doc for the same. It introduces the WAL Switching feature, and experiments/results of a prototype implementation, showing the benefits of this feature. The second goal of this work is to serve as a building block for concurrent multiple WALs feature. Please review the doc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[
https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934931#comment-13934931
]
Hadoop QA commented on HBASE-10648:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12634686/HBASE-10648_V6.patch
against trunk revision .
ATTACHMENT ID: 12634686
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 26 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.regionserver.wal.TestLogRolling.testLogRollOnDatanodeDeath(TestLogRolling.java:368)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8993//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8993//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8993//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8993//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8993//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8993//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8993//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8993//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8993//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8993//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8993//console
This message is automatically generated.
Pluggable Memstore
--
Key: HBASE-10648
URL: https://issues.apache.org/jira/browse/HBASE-10648
Project: HBase
Issue Type: Sub-task
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Attachments: HBASE-10648-0.94_v1.patch, HBASE-10648.patch,
HBASE-10648_V2.patch, HBASE-10648_V3.patch, HBASE-10648_V4.patch,
HBASE-10648_V5.patch, HBASE-10648_V6.patch
Make Memstore into an interface implementation. Also make it pluggable by
configuring the FQCN of the impl.
This will allow us to have different impl and optimizations in the Memstore
DataStructure and the upper layers untouched.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-10531) Revisit how the key byte[] is passed to HFileScanner.seekTo and reseekTo
[ https://issues.apache.org/jira/browse/HBASE-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-10531: --- Attachment: HBASE-10531_3.patch Updated patch. Removes repetitive code. Addresses review comments. Stack, the changes from using KVComparator to CellComparator has to be done in another issue. For now using the CellComparator and calling it in KVComparator. Reviews and comments are welcome. Revisit how the key byte[] is passed to HFileScanner.seekTo and reseekTo Key: HBASE-10531 URL: https://issues.apache.org/jira/browse/HBASE-10531 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0 Attachments: HBASE-10531.patch, HBASE-10531_1.patch, HBASE-10531_2.patch, HBASE-10531_3.patch Currently the byte[] key passed to HFileScanner.seekTo and HFileScanner.reseekTo, is a combination of row, cf, qual, type and ts. And the caller forms this by using kv.getBuffer, which is actually deprecated. So see how this can be achieved considering kv.getBuffer is removed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10531) Revisit how the key byte[] is passed to HFileScanner.seekTo and reseekTo
[ https://issues.apache.org/jira/browse/HBASE-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934954#comment-13934954 ] ramkrishna.s.vasudevan commented on HBASE-10531: I think some problem with the RB. Getting this error 'The file hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparator.java (revision 18d54f8) was not found in the repository'. HFilePerformanceEvaluation and TestHFilePerformance are still using KeyValue.RawBytesComparator. Can we change that? Will prepare a micro benchmark report in the beginning of next week. Revisit how the key byte[] is passed to HFileScanner.seekTo and reseekTo Key: HBASE-10531 URL: https://issues.apache.org/jira/browse/HBASE-10531 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0 Attachments: HBASE-10531.patch, HBASE-10531_1.patch, HBASE-10531_2.patch, HBASE-10531_3.patch Currently the byte[] key passed to HFileScanner.seekTo and HFileScanner.reseekTo, is a combination of row, cf, qual, type and ts. And the caller forms this by using kv.getBuffer, which is actually deprecated. So see how this can be achieved considering kv.getBuffer is removed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10531) Revisit how the key byte[] is passed to HFileScanner.seekTo and reseekTo
[ https://issues.apache.org/jira/browse/HBASE-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-10531: --- Status: Patch Available (was: Open) Revisit how the key byte[] is passed to HFileScanner.seekTo and reseekTo Key: HBASE-10531 URL: https://issues.apache.org/jira/browse/HBASE-10531 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0 Attachments: HBASE-10531.patch, HBASE-10531_1.patch, HBASE-10531_2.patch, HBASE-10531_3.patch Currently the byte[] key passed to HFileScanner.seekTo and HFileScanner.reseekTo, is a combination of row, cf, qual, type and ts. And the caller forms this by using kv.getBuffer, which is actually deprecated. So see how this can be achieved considering kv.getBuffer is removed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10749) CellComparator.compareStatic() compares type wrongly
[
https://issues.apache.org/jira/browse/HBASE-10749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-10749:
---
Attachment: HBASE-10749.patch
Patch for trunk.
CellComparator.compareStatic() compares type wrongly
Key: HBASE-10749
URL: https://issues.apache.org/jira/browse/HBASE-10749
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.1, 0.99.0, 0.96.1.1
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Critical
Fix For: 0.98.1, 0.99.0
Attachments: HBASE-10749.patch
CellComparator.compareStatic(Cell a, Cell b) compares type wrongly.
{code}
//type
c = (0xff a.getTypeByte()) - (0xff b.getTypeByte());
if (c != 0) return c;
{code}
In the normal case it should be the other way
{code}
// Compare types. Let the delete types sort ahead of puts; i.e. types
// of higher numbers sort before those of lesser numbers. Maximum (255)
// appears ahead of everything, and minimum (0) appears after
// everything.
return (0xff rtype) - (0xff ltype);
{code}
Found this issue while replacing KVComparator to use CellComparator.
[~mcorgan]
Could you confirm this once? Currently PrefixTree code path uses this
compareStatic method.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-10749) CellComparator.compareStatic() compares type wrongly
[
https://issues.apache.org/jira/browse/HBASE-10749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-10749:
---
Status: Patch Available (was: Open)
CellComparator.compareStatic() compares type wrongly
Key: HBASE-10749
URL: https://issues.apache.org/jira/browse/HBASE-10749
Project: HBase
Issue Type: Bug
Affects Versions: 0.96.1.1, 0.98.1, 0.99.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Critical
Fix For: 0.98.1, 0.99.0
Attachments: HBASE-10749.patch
CellComparator.compareStatic(Cell a, Cell b) compares type wrongly.
{code}
//type
c = (0xff a.getTypeByte()) - (0xff b.getTypeByte());
if (c != 0) return c;
{code}
In the normal case it should be the other way
{code}
// Compare types. Let the delete types sort ahead of puts; i.e. types
// of higher numbers sort before those of lesser numbers. Maximum (255)
// appears ahead of everything, and minimum (0) appears after
// everything.
return (0xff rtype) - (0xff ltype);
{code}
Found this issue while replacing KVComparator to use CellComparator.
[~mcorgan]
Could you confirm this once? Currently PrefixTree code path uses this
compareStatic method.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10531) Revisit how the key byte[] is passed to HFileScanner.seekTo and reseekTo
[
https://issues.apache.org/jira/browse/HBASE-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13934979#comment-13934979
]
Hadoop QA commented on HBASE-10531:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12634704/HBASE-10531_3.patch
against trunk revision .
ATTACHMENT ID: 12634704
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 21 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8995//console
This message is automatically generated.
Revisit how the key byte[] is passed to HFileScanner.seekTo and reseekTo
Key: HBASE-10531
URL: https://issues.apache.org/jira/browse/HBASE-10531
Project: HBase
Issue Type: Sub-task
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.99.0
Attachments: HBASE-10531.patch, HBASE-10531_1.patch,
HBASE-10531_2.patch, HBASE-10531_3.patch
Currently the byte[] key passed to HFileScanner.seekTo and
HFileScanner.reseekTo, is a combination of row, cf, qual, type and ts. And
the caller forms this by using kv.getBuffer, which is actually deprecated.
So see how this can be achieved considering kv.getBuffer is removed.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-10549) When there is a hole, LoadIncrementalHFiles will hang in an infinite loop.
[ https://issues.apache.org/jira/browse/HBASE-10549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yuanxinen updated HBASE-10549: -- Attachment: HBASE-10549-0.94 The patch is for 0.94 When there is a hole, LoadIncrementalHFiles will hang in an infinite loop. -- Key: HBASE-10549 URL: https://issues.apache.org/jira/browse/HBASE-10549 Project: HBase Issue Type: Bug Components: HFile Affects Versions: 0.98.0, 0.94.11, 0.96.1.1 Reporter: yuanxinen Assignee: yuanxinen Fix For: 0.96.2, 0.99.0, 0.94.19, 0.98.2 Attachments: HBASE-10549-0.94, HBASE-10549-trunk-2014-03-13.patch, HBASE-10549-trunk.patch First,I explan my test steps. 1.importtsv 2.split the region 3.delete the region info from .META.(make a hole) 4.LoadIncrementalHFiles (this step will hung up in an infinite loop) I check the log,there are two issues 1.it create _tmp folder in an infinite loop. hdfs://hacluster/output3/i/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/test_table,136.bottom 2.when slpliting the hfile,it put the first line data(1211) into two files(top and bottom) Input File=hdfs://hacluster/output3/i/3ac6ec287c644a8fb72d96b13e31f576,outFile=hdfs://hacluster/output3/i/_tmp/test_table,2.top,KeyValue=1211/i:value/1390469306407/Put/vlen=1/ts=0 Input File=hdfs://hacluster/output3/i/3ac6ec287c644a8fb72d96b13e31f576,outFile=hdfs://hacluster/output3/i/_tmp/test_table,2.bottom,KeyValue=1211/i:value/1390469306407/Put/vlen=1/ts=0 and then I check the code. So I think before spliting the hfile,we should check the consistency of startkey and endkey,if something wrong,we should throw the exception,and stop LoadIncrementalHFiles. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10748) hbase-daemon.sh fails to execute with 'sh' command
[
https://issues.apache.org/jira/browse/HBASE-10748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935007#comment-13935007
]
Hadoop QA commented on HBASE-10748:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12634691/HBASE-10748.patch
against trunk revision .
ATTACHMENT ID: 12634691
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestAdmin
org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction
org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.mapreduce.TestImportExport.testImport94Table(TestImportExport.java:230)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8994//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8994//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8994//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8994//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8994//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8994//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8994//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8994//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8994//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8994//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8994//console
This message is automatically generated.
hbase-daemon.sh fails to execute with 'sh' command
--
Key: HBASE-10748
URL: https://issues.apache.org/jira/browse/HBASE-10748
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.11
Reporter: Ashish Singhi
Attachments: HBASE-10748.patch, HBASE-10748.patch
hostname:HBASE_HOME/bin # sh hbase-daemon.sh restart master
*hbase-daemon.sh: line 188: hbase-daemon.sh: command not found*
*hbase-daemon.sh: line 196: hbase-daemon.sh: command not found*
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10746) Bump the version of HTrace to 3.0
[ https://issues.apache.org/jira/browse/HBASE-10746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935009#comment-13935009 ] Masatake Iwasaki commented on HBASE-10746: -- Thanks for the udpate [~stack]! I'm waiting the release. Bump the version of HTrace to 3.0 - Key: HBASE-10746 URL: https://issues.apache.org/jira/browse/HBASE-10746 Project: HBase Issue Type: Improvement Components: util Affects Versions: 0.96.2, 0.98.1, 0.99.0 Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Priority: Minor Upgrading the version of HTrace from 2.04 to 3.0. The package name has been changed from org.cloudera.htrace to org.htrace. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[ https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935014#comment-13935014 ] Anoop Sam John commented on HBASE-10648: Test fail seems not related. Going to commit in some time. Pluggable Memstore -- Key: HBASE-10648 URL: https://issues.apache.org/jira/browse/HBASE-10648 Project: HBase Issue Type: Sub-task Reporter: Anoop Sam John Assignee: Anoop Sam John Attachments: HBASE-10648-0.94_v1.patch, HBASE-10648.patch, HBASE-10648_V2.patch, HBASE-10648_V3.patch, HBASE-10648_V4.patch, HBASE-10648_V5.patch, HBASE-10648_V6.patch Make Memstore into an interface implementation. Also make it pluggable by configuring the FQCN of the impl. This will allow us to have different impl and optimizations in the Memstore DataStructure and the upper layers untouched. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10319) HLog should roll periodically to allow DN decommission to eventually complete.
[ https://issues.apache.org/jira/browse/HBASE-10319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935016#comment-13935016 ] Himanshu Vashishtha commented on HBASE-10319: - bq. having hbase periodically force a log roll. This would enable the hdfs dn con complete. A roll creates a new file, writes and sync its header, and then closes the old writer. I wonder how it solve the original issue because you would still have one open file ? I guess decommissioning blacklists a DN, so when a new file is created, NN ensures no blocks are allocated on this DN. Just want to confirm. HLog should roll periodically to allow DN decommission to eventually complete. -- Key: HBASE-10319 URL: https://issues.apache.org/jira/browse/HBASE-10319 Project: HBase Issue Type: Bug Reporter: Jonathan Hsieh Assignee: Matteo Bertozzi Fix For: 0.98.0, 0.96.2, 0.94.17 Attachments: HBASE-10319-v0.patch, HBASE-10319-v1.patch We encountered a situation where we had an esseitially read only table and attempted to do a clean HDFS DN decommission. DN's cannot decomission if there are open blocks being written to currently on it. Because the hbase Hlog file was open, had some data (hlog header), the DN could not decommission itself. Since no new data is ever written, the existing periodic check is not activated. After discussing with [~atm], it seems that although an hdfs semantics change would be ideal (e.g. hbase doesn't have to be aware of hdfs decommission and the client would roll over) this would take much more effort than having hbase periodically force a log roll. This would enable the hdfs dn con complete. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10648) Pluggable Memstore
[ https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-10648: --- Resolution: Fixed Fix Version/s: 0.99.0 Release Note: A new implementation for the Memstore can be plugged into region server. For this implement org.apache.hadoop.hbase.regionserver.MemStore interface and configure the FQCN of the impl class against the key hbase.regionserver.memstore.class Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pluggable Memstore -- Key: HBASE-10648 URL: https://issues.apache.org/jira/browse/HBASE-10648 Project: HBase Issue Type: Sub-task Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10648-0.94_v1.patch, HBASE-10648.patch, HBASE-10648_V2.patch, HBASE-10648_V3.patch, HBASE-10648_V4.patch, HBASE-10648_V5.patch, HBASE-10648_V6.patch Make Memstore into an interface implementation. Also make it pluggable by configuring the FQCN of the impl. This will allow us to have different impl and optimizations in the Memstore DataStructure and the upper layers untouched. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[ https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935055#comment-13935055 ] Anoop Sam John commented on HBASE-10648: Committed to trunk. Thanks for the reviews. Pluggable Memstore -- Key: HBASE-10648 URL: https://issues.apache.org/jira/browse/HBASE-10648 Project: HBase Issue Type: Sub-task Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10648-0.94_v1.patch, HBASE-10648.patch, HBASE-10648_V2.patch, HBASE-10648_V3.patch, HBASE-10648_V4.patch, HBASE-10648_V5.patch, HBASE-10648_V6.patch Make Memstore into an interface implementation. Also make it pluggable by configuring the FQCN of the impl. This will allow us to have different impl and optimizations in the Memstore DataStructure and the upper layers untouched. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[
https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935079#comment-13935079
]
Yu Li commented on HBASE-10648:
---
Hi [~anoop.hbase],
{quote}
It won't be good to directly use some impl here. We use flushed variable for
logging purpose only right? In trunk it is. Can you use the size available in
MemStoreSnapshot?
{quote}
Yes, the flushedSize is for logging purpose, or say monitoring purpose.
Actually this is kind of a necessary metrics for our monitoring system and we
think it useful for analyzing online problem when there's any.
From the code segment of the heapSizeChange method, it's for computing the
heapsize change when adding/removing a previously-non-existing kv, so I don't
think any size available in MemStoreSnapshot can be used here. If we decided
to keep the flushedSize metrics, I think it's even better to put
heapSizeChange method into the MemStore interface, since different impl of
MemStore might have different heap size to store one kv, depending on the
structure(not only CSLM) we use to store it. What's your opinion?
{code}
static long heapSizeChange(final KeyValue kv, final boolean notpresent) {
return notpresent ?
ClassSize.align(ClassSize.CONCURRENT_SKIPLISTMAP_ENTRY + kv.heapSize()):
0;
}
{code}
{quote}
Make snapshotSize = keySize().
Any way once HBASE-10514 is in to 94, will need some change here and
snapshotSize will come as an instance member.
{quote}
Make sense. Let me make the change in one go after HBASE-10514 goes into 94.
Pluggable Memstore
--
Key: HBASE-10648
URL: https://issues.apache.org/jira/browse/HBASE-10648
Project: HBase
Issue Type: Sub-task
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 0.99.0
Attachments: HBASE-10648-0.94_v1.patch, HBASE-10648.patch,
HBASE-10648_V2.patch, HBASE-10648_V3.patch, HBASE-10648_V4.patch,
HBASE-10648_V5.patch, HBASE-10648_V6.patch
Make Memstore into an interface implementation. Also make it pluggable by
configuring the FQCN of the impl.
This will allow us to have different impl and optimizations in the Memstore
DataStructure and the upper layers untouched.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10749) CellComparator.compareStatic() compares type wrongly
[
https://issues.apache.org/jira/browse/HBASE-10749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935116#comment-13935116
]
Hadoop QA commented on HBASE-10749:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12634710/HBASE-10749.patch
against trunk revision .
ATTACHMENT ID: 12634710
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestAdmin
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8996//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8996//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8996//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8996//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8996//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8996//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8996//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8996//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8996//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8996//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8996//console
This message is automatically generated.
CellComparator.compareStatic() compares type wrongly
Key: HBASE-10749
URL: https://issues.apache.org/jira/browse/HBASE-10749
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.1, 0.99.0, 0.96.1.1
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Critical
Fix For: 0.98.1, 0.99.0
Attachments: HBASE-10749.patch
CellComparator.compareStatic(Cell a, Cell b) compares type wrongly.
{code}
//type
c = (0xff a.getTypeByte()) - (0xff b.getTypeByte());
if (c != 0) return c;
{code}
In the normal case it should be the other way
{code}
// Compare types. Let the delete types sort ahead of puts; i.e. types
// of higher numbers sort before those of lesser numbers. Maximum (255)
// appears ahead of everything, and minimum (0) appears after
// everything.
return (0xff rtype) - (0xff ltype);
{code}
Found this issue while replacing KVComparator to use CellComparator.
[~mcorgan]
Could you confirm this once? Currently PrefixTree code path uses this
compareStatic method.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10319) HLog should roll periodically to allow DN decommission to eventually complete.
[ https://issues.apache.org/jira/browse/HBASE-10319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935157#comment-13935157 ] Jonathan Hsieh commented on HBASE-10319: basically the dn enters decomissioning mode, prevents new blocks for write, and hangs until the open for write blocks closes. the hbase roll would close the file's block allowing the dn to shutdown and would be excluded as a potential place for new blocks from the new wal file. HLog should roll periodically to allow DN decommission to eventually complete. -- Key: HBASE-10319 URL: https://issues.apache.org/jira/browse/HBASE-10319 Project: HBase Issue Type: Bug Reporter: Jonathan Hsieh Assignee: Matteo Bertozzi Fix For: 0.98.0, 0.96.2, 0.94.17 Attachments: HBASE-10319-v0.patch, HBASE-10319-v1.patch We encountered a situation where we had an esseitially read only table and attempted to do a clean HDFS DN decommission. DN's cannot decomission if there are open blocks being written to currently on it. Because the hbase Hlog file was open, had some data (hlog header), the DN could not decommission itself. Since no new data is ever written, the existing periodic check is not activated. After discussing with [~atm], it seems that although an hdfs semantics change would be ideal (e.g. hbase doesn't have to be aware of hdfs decommission and the client would roll over) this would take much more effort than having hbase periodically force a log roll. This would enable the hdfs dn con complete. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-7847) Use zookeeper multi to clear znodes
[ https://issues.apache.org/jira/browse/HBASE-7847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935161#comment-13935161 ] Rakesh R commented on HBASE-7847: - These test cases are passing locally in my environment, I doubt whether these are due to my patch ? Use zookeeper multi to clear znodes --- Key: HBASE-7847 URL: https://issues.apache.org/jira/browse/HBASE-7847 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Attachments: 7847-v1.txt, HBASE-7847.patch, HBASE-7847.patch, HBASE-7847.patch, HBASE-7847_v4.patch In ZKProcedureUtil, clearChildZNodes() and clearZNodes(String procedureName) should utilize zookeeper multi so that they're atomic -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10745) Access ShutdownHook#fsShutdownHooks should be synchronized
[ https://issues.apache.org/jira/browse/HBASE-10745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-10745: Resolution: Fixed Fix Version/s: 0.94.18 0.99.0 0.98.1 0.96.2 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Access ShutdownHook#fsShutdownHooks should be synchronized -- Key: HBASE-10745 URL: https://issues.apache.org/jira/browse/HBASE-10745 Project: HBase Issue Type: Bug Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.18 Attachments: hbase-10745.patch Sometimes, I got weird RuntimeException. It turns out that ShutdownHook#fsShutdownHooks is a HashMap. We should synchronize its access. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10745) Access ShutdownHook#fsShutdownHooks should be synchronized
[ https://issues.apache.org/jira/browse/HBASE-10745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935183#comment-13935183 ] Jimmy Xiang commented on HBASE-10745: - Yes, all other accesses are synchronized. Integrated into 0.94, 0.96, 0.98, and trunk. Thanks. Access ShutdownHook#fsShutdownHooks should be synchronized -- Key: HBASE-10745 URL: https://issues.apache.org/jira/browse/HBASE-10745 Project: HBase Issue Type: Bug Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.18 Attachments: hbase-10745.patch Sometimes, I got weird RuntimeException. It turns out that ShutdownHook#fsShutdownHooks is a HashMap. We should synchronize its access. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10745) Access ShutdownHook#fsShutdownHooks should be synchronized
[ https://issues.apache.org/jira/browse/HBASE-10745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935212#comment-13935212 ] Hudson commented on HBASE-10745: FAILURE: Integrated in HBase-0.94-JDK7 #87 (See [https://builds.apache.org/job/HBase-0.94-JDK7/87/]) HBASE-10745 Access ShutdownHook#fsShutdownHooks should be synchronized (jxiang: rev 1577591) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/ShutdownHook.java Access ShutdownHook#fsShutdownHooks should be synchronized -- Key: HBASE-10745 URL: https://issues.apache.org/jira/browse/HBASE-10745 Project: HBase Issue Type: Bug Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.18 Attachments: hbase-10745.patch Sometimes, I got weird RuntimeException. It turns out that ShutdownHook#fsShutdownHooks is a HashMap. We should synchronize its access. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-5175) Add DoubleColumnInterpreter
[ https://issues.apache.org/jira/browse/HBASE-5175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julian Wissmann updated HBASE-5175: --- Attachment: DoubleColumnInterpreterV2.patch Re-Created patch. Issues of the previous patch should be fixed. Add DoubleColumnInterpreter --- Key: HBASE-5175 URL: https://issues.apache.org/jira/browse/HBASE-5175 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Assignee: Julian Wissmann Labels: aggregator, features Fix For: 0.99.0 Attachments: DoubleColumnInterpreter.java, DoubleColumnInterpreter.patch, DoubleColumnInterpreterV2.patch, HBase.proto, TestDoubleColumnInterpreter.java DoubleColumnInterpreter was requested by Royston Sellman. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-5175) Add DoubleColumnInterpreter
[ https://issues.apache.org/jira/browse/HBASE-5175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-5175: -- Status: Patch Available (was: Open) Add DoubleColumnInterpreter --- Key: HBASE-5175 URL: https://issues.apache.org/jira/browse/HBASE-5175 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Assignee: Julian Wissmann Labels: aggregator, features Fix For: 0.99.0 Attachments: DoubleColumnInterpreter.java, DoubleColumnInterpreter.patch, DoubleColumnInterpreterV2.patch, HBase.proto, TestDoubleColumnInterpreter.java DoubleColumnInterpreter was requested by Royston Sellman. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-5175) Add DoubleColumnInterpreter
[
https://issues.apache.org/jira/browse/HBASE-5175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935252#comment-13935252
]
Hadoop QA commented on HBASE-5175:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12634745/DoubleColumnInterpreterV2.patch
against trunk revision .
ATTACHMENT ID: 12634745
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8998//console
This message is automatically generated.
Add DoubleColumnInterpreter
---
Key: HBASE-5175
URL: https://issues.apache.org/jira/browse/HBASE-5175
Project: HBase
Issue Type: Sub-task
Reporter: Ted Yu
Assignee: Julian Wissmann
Labels: aggregator, features
Fix For: 0.99.0
Attachments: DoubleColumnInterpreter.java,
DoubleColumnInterpreter.patch, DoubleColumnInterpreterV2.patch, HBase.proto,
TestDoubleColumnInterpreter.java
DoubleColumnInterpreter was requested by Royston Sellman.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10745) Access ShutdownHook#fsShutdownHooks should be synchronized
[ https://issues.apache.org/jira/browse/HBASE-10745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935270#comment-13935270 ] Hudson commented on HBASE-10745: FAILURE: Integrated in HBase-0.94-security #443 (See [https://builds.apache.org/job/HBase-0.94-security/443/]) HBASE-10745 Access ShutdownHook#fsShutdownHooks should be synchronized (jxiang: rev 1577591) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/ShutdownHook.java Access ShutdownHook#fsShutdownHooks should be synchronized -- Key: HBASE-10745 URL: https://issues.apache.org/jira/browse/HBASE-10745 Project: HBase Issue Type: Bug Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.18 Attachments: hbase-10745.patch Sometimes, I got weird RuntimeException. It turns out that ShutdownHook#fsShutdownHooks is a HashMap. We should synchronize its access. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10749) CellComparator.compareStatic() compares type wrongly
[
https://issues.apache.org/jira/browse/HBASE-10749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935275#comment-13935275
]
Matt Corgan commented on HBASE-10749:
-
yikes - that was not on purpose
CellComparator.compareStatic() compares type wrongly
Key: HBASE-10749
URL: https://issues.apache.org/jira/browse/HBASE-10749
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.1, 0.99.0, 0.96.1.1
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Critical
Fix For: 0.98.1, 0.99.0
Attachments: HBASE-10749.patch
CellComparator.compareStatic(Cell a, Cell b) compares type wrongly.
{code}
//type
c = (0xff a.getTypeByte()) - (0xff b.getTypeByte());
if (c != 0) return c;
{code}
In the normal case it should be the other way
{code}
// Compare types. Let the delete types sort ahead of puts; i.e. types
// of higher numbers sort before those of lesser numbers. Maximum (255)
// appears ahead of everything, and minimum (0) appears after
// everything.
return (0xff rtype) - (0xff ltype);
{code}
Found this issue while replacing KVComparator to use CellComparator.
[~mcorgan]
Could you confirm this once? Currently PrefixTree code path uses this
compareStatic method.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10745) Access ShutdownHook#fsShutdownHooks should be synchronized
[ https://issues.apache.org/jira/browse/HBASE-10745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935278#comment-13935278 ] Hudson commented on HBASE-10745: FAILURE: Integrated in HBase-0.94-on-Hadoop-2 #53 (See [https://builds.apache.org/job/HBase-0.94-on-Hadoop-2/53/]) HBASE-10745 Access ShutdownHook#fsShutdownHooks should be synchronized (jxiang: rev 1577591) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/ShutdownHook.java Access ShutdownHook#fsShutdownHooks should be synchronized -- Key: HBASE-10745 URL: https://issues.apache.org/jira/browse/HBASE-10745 Project: HBase Issue Type: Bug Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.18 Attachments: hbase-10745.patch Sometimes, I got weird RuntimeException. It turns out that ShutdownHook#fsShutdownHooks is a HashMap. We should synchronize its access. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10745) Access ShutdownHook#fsShutdownHooks should be synchronized
[ https://issues.apache.org/jira/browse/HBASE-10745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935322#comment-13935322 ] Hudson commented on HBASE-10745: SUCCESS: Integrated in HBase-0.94 #1324 (See [https://builds.apache.org/job/HBase-0.94/1324/]) HBASE-10745 Access ShutdownHook#fsShutdownHooks should be synchronized (jxiang: rev 1577591) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/ShutdownHook.java Access ShutdownHook#fsShutdownHooks should be synchronized -- Key: HBASE-10745 URL: https://issues.apache.org/jira/browse/HBASE-10745 Project: HBase Issue Type: Bug Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.96.2, 0.98.1, 0.99.0, 0.94.18 Attachments: hbase-10745.patch Sometimes, I got weird RuntimeException. It turns out that ShutdownHook#fsShutdownHooks is a HashMap. We should synchronize its access. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10741) Deprecate HTablePool and HTableFactory
[
https://issues.apache.org/jira/browse/HBASE-10741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935325#comment-13935325
]
Nick Dimiduk commented on HBASE-10741:
--
I don't want to confuse with the @since annotation
(http://www.oracle.com/technetwork/java/javase/documentation/index-137868.html#@since).
Will instead use:
{noformat}
@deprecated as of 0.98.1. See HConnectionManager#createConnection
{noformat}
Deprecate HTablePool and HTableFactory
--
Key: HBASE-10741
URL: https://issues.apache.org/jira/browse/HBASE-10741
Project: HBase
Issue Type: Sub-task
Components: Client
Affects Versions: 0.98.0, 0.96.0, 0.99.0
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
Fix For: 0.96.2, 0.99.0, 0.98.2
Attachments: HBASE-10741.00.patch
Per parent ticket, deprecate these ways to retrieve an HTable instance.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10648) Pluggable Memstore
[ https://issues.apache.org/jira/browse/HBASE-10648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935337#comment-13935337 ] Hudson commented on HBASE-10648: FAILURE: Integrated in HBase-TRUNK #5010 (See [https://builds.apache.org/job/HBase-TRUNK/5010/]) HBASE-10648 Pluggable Memstore.(Anoop) (anoopsamjohn: rev 1577541) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultMemStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFlusher.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFlusher.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StripeStoreFlusher.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHeapSize.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultMemStore.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreChunkPool.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java Pluggable Memstore -- Key: HBASE-10648 URL: https://issues.apache.org/jira/browse/HBASE-10648 Project: HBase Issue Type: Sub-task Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10648-0.94_v1.patch, HBASE-10648.patch, HBASE-10648_V2.patch, HBASE-10648_V3.patch, HBASE-10648_V4.patch, HBASE-10648_V5.patch, HBASE-10648_V6.patch Make Memstore into an interface implementation. Also make it pluggable by configuring the FQCN of the impl. This will allow us to have different impl and optimizations in the Memstore DataStructure and the upper layers untouched. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10741) Deprecate HTablePool and HTableFactory
[ https://issues.apache.org/jira/browse/HBASE-10741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-10741: - Resolution: Fixed Fix Version/s: (was: 0.98.2) 0.98.1 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to three branches. Thanks for the review. Deprecate HTablePool and HTableFactory -- Key: HBASE-10741 URL: https://issues.apache.org/jira/browse/HBASE-10741 Project: HBase Issue Type: Sub-task Components: Client Affects Versions: 0.98.0, 0.96.0, 0.99.0 Reporter: Nick Dimiduk Assignee: Nick Dimiduk Fix For: 0.96.2, 0.98.1, 0.99.0 Attachments: HBASE-10741.00.patch Per parent ticket, deprecate these ways to retrieve an HTable instance. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10741) Deprecate HTablePool and HTableFactory
[ https://issues.apache.org/jira/browse/HBASE-10741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-10741: - Attachment: HBASE-10741.01.patch Attaching what was committed. Deprecate HTablePool and HTableFactory -- Key: HBASE-10741 URL: https://issues.apache.org/jira/browse/HBASE-10741 Project: HBase Issue Type: Sub-task Components: Client Affects Versions: 0.98.0, 0.96.0, 0.99.0 Reporter: Nick Dimiduk Assignee: Nick Dimiduk Fix For: 0.96.2, 0.98.1, 0.99.0 Attachments: HBASE-10741.00.patch, HBASE-10741.01.patch Per parent ticket, deprecate these ways to retrieve an HTable instance. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-10751) TestHRegion testWritesWhileScanning occasional fail since HBASE-10514 went in
stack created HBASE-10751:
-
Summary: TestHRegion testWritesWhileScanning occasional fail since
HBASE-10514 went in
Key: HBASE-10751
URL: https://issues.apache.org/jira/browse/HBASE-10751
Project: HBase
Issue Type: Bug
Reporter: stack
Assignee: stack
I saw this here
https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/213/testReport/junit/org.apache.hadoop.hbase.regionserver/TestHRegion/testWritesWhileScanning/
This patch looks to have exposed a problem in our HStore commit logic. We are
supposed to crash out if we fail to write but we keep going here. I am having
trouble figuring why. Let me write a little test:
{code}
2014-03-14 01:58:48,647 DEBUG [Thread-3] regionserver.HRegionFileSystem(339):
Committing store file
/home/jenkins/jenkins-slave/workspace/HBase-0.98-on-Hadoop-1.1/0.98-hadoop1.1/hbase-server/target/test-data/f7999012-e166-4619-ab3c-5014e0f65007/data/default/testWritesWhileScanning/306ea000673d780f06daf2469e7f9bab/.tmp/a0e6579af25f463ebb7eebe3c043b8a0
as
/home/jenkins/jenkins-slave/workspace/HBase-0.98-on-Hadoop-1.1/0.98-hadoop1.1/hbase-server/target/test-data/f7999012-e166-4619-ab3c-5014e0f65007/data/default/testWritesWhileScanning/306ea000673d780f06daf2469e7f9bab/family7/a0e6579af25f463ebb7eebe3c043b8a0
2014-03-14 01:58:48,647 INFO [Thread-2] regionserver.HRegion(5779): writing
data to region
testWritesWhileScanning,,1394762315120.306ea000673d780f06daf2469e7f9bab. with
WAL disabled. Data may be lost in the event of a crash.
2014-03-14 01:58:48,648 ERROR [Thread-3]
regionserver.HStore$StoreFlusherImpl(1964): Failed to commit store file
/home/jenkins/jenkins-slave/workspace/HBase-0.98-on-Hadoop-1.1/0.98-hadoop1.1/hbase-server/target/test-data/f7999012-e166-4619-ab3c-5014e0f65007/data/default/testWritesWhileScanning/306ea000673d780f06daf2469e7f9bab/.tmp/a0e6579af25f463ebb7eebe3c043b8a0
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading HFile
Trailer from file
file:/home/jenkins/jenkins-slave/workspace/HBase-0.98-on-Hadoop-1.1/0.98-hadoop1.1/hbase-server/target/test-data/f7999012-e166-4619-ab3c-5014e0f65007/data/default/testWritesWhileScanning/306ea000673d780f06daf2469e7f9bab/family7/a0e6579af25f463ebb7eebe3c043b8a0
at
org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:552)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:580)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1019)
at
org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:211)
at
org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:350)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:445)
at
org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:551)
at
org.apache.hadoop.hbase.regionserver.HStore.commitFile(HStore.java:842)
at
org.apache.hadoop.hbase.regionserver.HStore.access$200(HStore.java:118)
at
org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit(HStore.java:1961)
at
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1706)
at
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1583)
at
org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1498)
at
org.apache.hadoop.hbase.regionserver.TestHRegion$FlushThread.run(TestHRegion.java:3034)
Caused by: java.nio.channels.ClosedByInterruptException
at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:282)
at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.seek(RawLocalFileSystem.java:111)
at
org.apache.hadoop.fs.BufferedFSInputStream.seek(BufferedFSInputStream.java:78)
at
org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:37)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:206)
at
org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:237)
at org.apache.hadoop.fs.FSInputChecker.fill(FSInputChecker.java:176)
at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:193)
at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
at
org.apache.hadoop.fs.FSInputChecker.readFully(FSInputChecker.java:384)
at org.apache.hadoop.fs.FSInputChecker.seek(FSInputChecker.java:365)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.seek(ChecksumFileSystem.java:271)
at
org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:37)
at
[jira] [Commented] (HBASE-10719) Cell interface may need a ByteBuffer rather than a byte[]
[ https://issues.apache.org/jira/browse/HBASE-10719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935366#comment-13935366 ] Nick Dimiduk commented on HBASE-10719: -- Shouldn't we consider ByteBuffers for values so that we can work toward an entirely off-heap data pathway? This will enable larger value blobs (10's - 100's of MB) for users and open the way for (closer to) zero copy pipeline. Cell interface may need a ByteBuffer rather than a byte[] - Key: HBASE-10719 URL: https://issues.apache.org/jira/browse/HBASE-10719 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan See discussion in HBASE-10531. The byte[] backing a cell may be off heap also. In such a case having a byte[] may require a copy offheap to on heap. So better to have ByteBuffer backing a Cell. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10184) [Online Schema Change]: Add additional tests for online schema change
[ https://issues.apache.org/jira/browse/HBASE-10184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935403#comment-13935403 ] Andrew Purtell commented on HBASE-10184: bq. Can you point me to the additional failures? Sure [~aleksshulman], please see: https://builds.apache.org/job/HBase-0.98/228/ https://builds.apache.org/job/HBase-TRUNK/5009/ https://builds.apache.org/job/PreCommit-HBASE-Build/8984//testReport/ https://builds.apache.org/job/PreCommit-HBASE-Build/8988// https://builds.apache.org/job/PreCommit-HBASE-Build/8989// https://builds.apache.org/job/PreCommit-HBASE-Build/8994// https://builds.apache.org/job/PreCommit-HBASE-Build/8996// And possibly more, I stopped looking after here. [Online Schema Change]: Add additional tests for online schema change - Key: HBASE-10184 URL: https://issues.apache.org/jira/browse/HBASE-10184 Project: HBase Issue Type: Task Components: test Affects Versions: 0.96.1, 0.99.0 Reporter: Aleksandr Shulman Assignee: Aleksandr Shulman Labels: online_schema_change Fix For: 0.99.0, 0.98.2 Attachments: 10184-4.patch, 10184.addendum, HBASE-10184-trunk.diff There are some gaps in testing for Online Schema Change: Examples of some tests that should be added: 1. Splits with online schema change 2. Merge during online schema change 3. MR over HBase during online schema change 4. Bulk Load during online schema change 5. Online change table owner 6. Online Replication scope change 7. Online Bloom Filter change 8. Snapshots during online schema change (HBASE-10136) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10750) Pluggable MemStoreLAB
[ https://issues.apache.org/jira/browse/HBASE-10750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-10750: --- Status: Patch Available (was: Open) Pluggable MemStoreLAB - Key: HBASE-10750 URL: https://issues.apache.org/jira/browse/HBASE-10750 Project: HBase Issue Type: Sub-task Components: regionserver Reporter: Anoop Sam John Assignee: Anoop Sam John Attachments: HBASE-10750.patch This can allow one to have offheap MSLAB also. The allocate() returns a byte[] now. This has to be changed to ByteRange so that the impl can return on heap byte[] backed BR or offheap buffer backed BR. -- This message was sent by Atlassian JIRA (v6.2#6252)
