[jira] [Work started] (HBASE-13212) Procedure V2 - master Create/Modify/Delete namespace
[ https://issues.apache.org/jira/browse/HBASE-13212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-13212 started by Stephen Yuan Jiang. -- Procedure V2 - master Create/Modify/Delete namespace Key: HBASE-13212 URL: https://issues.apache.org/jira/browse/HBASE-13212 Project: HBase Issue Type: Sub-task Components: master Affects Versions: 2.0.0 Reporter: Stephen Yuan Jiang Assignee: Stephen Yuan Jiang Labels: reliability Original Estimate: 168h Remaining Estimate: 168h master side, part of HBASE-12439 starts up the procedure executor on the master and replaces the create/modify/delete namespace handlers with the procedure version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14104) Add vectorportal.com to NOTICES.txt as src of our logo
[ https://issues.apache.org/jira/browse/HBASE-14104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-14104. --- Resolution: Fixed Assignee: stack Hadoop Flags: Reviewed Fix Version/s: 1.0.3 1.1.2 1.2.0 2.0.0 Pushed to all branches 1.0+ Add vectorportal.com to NOTICES.txt as src of our logo -- Key: HBASE-14104 URL: https://issues.apache.org/jira/browse/HBASE-14104 Project: HBase Issue Type: Sub-task Reporter: stack Assignee: stack Fix For: 2.0.0, 1.2.0, 1.1.2, 1.0.3 Attachments: 14104.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14106) TestProcedureRecovery is flaky
[
https://issues.apache.org/jira/browse/HBASE-14106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630052#comment-14630052
]
Matteo Bertozzi commented on HBASE-14106:
-
yeah, the test if flaky because the TestMultiStepProcedure is killing the
executor when doing the execute step.
{code}
restart();
--- the executor may already be killed if the proc1 executed another step here.
Procedure proc2 = new TestMultiStepProcedure();
long procId2 = ProcedureTestingUtility.submitAndWait(procExecutor, proc2,
nonceGroup, nonce);
{code}
let me rewrite this test
TestProcedureRecovery is flaky
--
Key: HBASE-14106
URL: https://issues.apache.org/jira/browse/HBASE-14106
Project: HBase
Issue Type: Bug
Components: proc-v2, test
Reporter: Andrew Purtell
Encountered this when running master tests locally using 7u79:
{noformat}
Tests run: 8, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 12.28 sec
FAILURE! - in org.apache.hadoop.hbase.procedure2.TestProcedureRecovery
testRunningProcWithSameNonce(org.apache.hadoop.hbase.procedure2.TestProcedureRecovery)
Time elapsed: 0.318 sec ERROR!
java.lang.IllegalArgumentException: null
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:76)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:595)
at
org.apache.hadoop.hbase.procedure2.ProcedureTestingUtility.submitAndWait(ProcedureTestingUtility.java:137)
at
org.apache.hadoop.hbase.procedure2.TestProcedureRecovery.testRunningProcWithSameNonce(TestProcedureRecovery.java:321)
{noformat}
{noformat}
Flaked tests:
org.apache.hadoop.hbase.procedure2.TestProcedureRecovery.testRunningProcWithSameNonce(org.apache.hadoop.hbase.procedure2.TestProcedureRecovery)
Run 1: TestProcedureRecovery.testRunningProcWithSameNonce:321 »
IllegalArgument
Run 2: PASS
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-14106) TestProcedureRecovery is flaky
[
https://issues.apache.org/jira/browse/HBASE-14106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi reassigned HBASE-14106:
---
Assignee: Matteo Bertozzi
TestProcedureRecovery is flaky
--
Key: HBASE-14106
URL: https://issues.apache.org/jira/browse/HBASE-14106
Project: HBase
Issue Type: Bug
Components: proc-v2, test
Reporter: Andrew Purtell
Assignee: Matteo Bertozzi
Encountered this when running master tests locally using 7u79:
{noformat}
Tests run: 8, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 12.28 sec
FAILURE! - in org.apache.hadoop.hbase.procedure2.TestProcedureRecovery
testRunningProcWithSameNonce(org.apache.hadoop.hbase.procedure2.TestProcedureRecovery)
Time elapsed: 0.318 sec ERROR!
java.lang.IllegalArgumentException: null
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:76)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:595)
at
org.apache.hadoop.hbase.procedure2.ProcedureTestingUtility.submitAndWait(ProcedureTestingUtility.java:137)
at
org.apache.hadoop.hbase.procedure2.TestProcedureRecovery.testRunningProcWithSameNonce(TestProcedureRecovery.java:321)
{noformat}
{noformat}
Flaked tests:
org.apache.hadoop.hbase.procedure2.TestProcedureRecovery.testRunningProcWithSameNonce(org.apache.hadoop.hbase.procedure2.TestProcedureRecovery)
Run 1: TestProcedureRecovery.testRunningProcWithSameNonce:321 »
IllegalArgument
Run 2: PASS
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14000) Region server failed to report Master and stuck in reportForDuty retry loop
[
https://issues.apache.org/jira/browse/HBASE-14000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630073#comment-14630073
]
Jerry He commented on HBASE-14000:
--
The probably does not exist in 0.98.
From version 1+, the master shares code with HRegionServer and opens the RPC
server before the master finishes initialization.
The patch lgtm
Region server failed to report Master and stuck in reportForDuty retry loop
---
Key: HBASE-14000
URL: https://issues.apache.org/jira/browse/HBASE-14000
Project: HBase
Issue Type: Bug
Reporter: Pankaj Kumar
Assignee: Pankaj Kumar
Attachments: HBASE-14000.patch, HM_RS-Log_snippet.txt
In a HA cluster, region server got stuck in reportForDuty retry loop if the
active master is restarting and later on master switch happens before it
reports successfully.
Root cause is same as HBASE-13317, but the region server tried to connect
master when it was starting, so rssStub reset didnt happen as
{code}
if (ioe instanceof ServerNotRunningYetException) {
LOG.debug(Master is not running yet);
}
{code}
When master starts, master switch happened. So RS always tried to connect to
standby master.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12213) HFileBlock backed by Array of ByteBuffers
[ https://issues.apache.org/jira/browse/HBASE-12213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630092#comment-14630092 ] stack commented on HBASE-12213: --- Porting +1 from rb. Fix javadoc before committing. There are some followons on rb too. Nice work [~ramkrishna.s.vasude...@gmail.com] Add fat release note talking up new ByteBuff. HFileBlock backed by Array of ByteBuffers - Key: HBASE-12213 URL: https://issues.apache.org/jira/browse/HBASE-12213 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: ramkrishna.s.vasudevan Attachments: HBASE-12213_1.patch, HBASE-12213_10_withBBI.patch, HBASE-12213_11_withBBI.patch, HBASE-12213_12_withBBI.patch, HBASE-12213_12_withBBI.patch, HBASE-12213_13_withBBI.patch, HBASE-12213_14_withBBI.patch, HBASE-12213_2.patch, HBASE-12213_4.patch, HBASE-12213_8_withBBI.patch, HBASE-12213_9_withBBI.patch, HBASE-12213_jmh.zip In L2 cache (offheap) an HFile block might have been cached into multiple chunks of buffers. If HFileBlock need single BB, we will end up in recreation of bigger BB and copying. Instead we can make HFileBlock to serve data from an array of BBs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14106) TestProcedureRecovery is flaky
[
https://issues.apache.org/jira/browse/HBASE-14106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi updated HBASE-14106:
Fix Version/s: 1.1.2
1.2.0
2.0.0
Affects Version/s: 1.2.0
2.0.0
1.1.0.1
Status: Patch Available (was: Open)
TestProcedureRecovery is flaky
--
Key: HBASE-14106
URL: https://issues.apache.org/jira/browse/HBASE-14106
Project: HBase
Issue Type: Bug
Components: proc-v2, test
Affects Versions: 1.1.0.1, 2.0.0, 1.2.0
Reporter: Andrew Purtell
Assignee: Matteo Bertozzi
Fix For: 2.0.0, 1.2.0, 1.1.2
Attachments: HBASE-14106-v0.patch
Encountered this when running master tests locally using 7u79:
{noformat}
Tests run: 8, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 12.28 sec
FAILURE! - in org.apache.hadoop.hbase.procedure2.TestProcedureRecovery
testRunningProcWithSameNonce(org.apache.hadoop.hbase.procedure2.TestProcedureRecovery)
Time elapsed: 0.318 sec ERROR!
java.lang.IllegalArgumentException: null
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:76)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:595)
at
org.apache.hadoop.hbase.procedure2.ProcedureTestingUtility.submitAndWait(ProcedureTestingUtility.java:137)
at
org.apache.hadoop.hbase.procedure2.TestProcedureRecovery.testRunningProcWithSameNonce(TestProcedureRecovery.java:321)
{noformat}
{noformat}
Flaked tests:
org.apache.hadoop.hbase.procedure2.TestProcedureRecovery.testRunningProcWithSameNonce(org.apache.hadoop.hbase.procedure2.TestProcedureRecovery)
Run 1: TestProcedureRecovery.testRunningProcWithSameNonce:321 »
IllegalArgument
Run 2: PASS
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14106) TestProcedureRecovery is flaky
[
https://issues.apache.org/jira/browse/HBASE-14106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi updated HBASE-14106:
Attachment: HBASE-14106-v0.patch
TestProcedureRecovery is flaky
--
Key: HBASE-14106
URL: https://issues.apache.org/jira/browse/HBASE-14106
Project: HBase
Issue Type: Bug
Components: proc-v2, test
Reporter: Andrew Purtell
Assignee: Matteo Bertozzi
Attachments: HBASE-14106-v0.patch
Encountered this when running master tests locally using 7u79:
{noformat}
Tests run: 8, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 12.28 sec
FAILURE! - in org.apache.hadoop.hbase.procedure2.TestProcedureRecovery
testRunningProcWithSameNonce(org.apache.hadoop.hbase.procedure2.TestProcedureRecovery)
Time elapsed: 0.318 sec ERROR!
java.lang.IllegalArgumentException: null
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:76)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:595)
at
org.apache.hadoop.hbase.procedure2.ProcedureTestingUtility.submitAndWait(ProcedureTestingUtility.java:137)
at
org.apache.hadoop.hbase.procedure2.TestProcedureRecovery.testRunningProcWithSameNonce(TestProcedureRecovery.java:321)
{noformat}
{noformat}
Flaked tests:
org.apache.hadoop.hbase.procedure2.TestProcedureRecovery.testRunningProcWithSameNonce(org.apache.hadoop.hbase.procedure2.TestProcedureRecovery)
Run 1: TestProcedureRecovery.testRunningProcWithSameNonce:321 »
IllegalArgument
Run 2: PASS
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14111) Enable HBase ACL in REST operations
[ https://issues.apache.org/jira/browse/HBASE-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630572#comment-14630572 ] Andrew Purtell commented on HBASE-14111: That's not correct, if you have the AccessController running on the cluster and ACLs set up, the REST gateway cannot bypass them, it is just another client. I think you need to provide more detail on your setup. Enable HBase ACL in REST operations --- Key: HBASE-14111 URL: https://issues.apache.org/jira/browse/HBASE-14111 Project: HBase Issue Type: Improvement Components: REST, security Reporter: Roberto Arias-Yacupoma Priority: Minor Labels: patch, security Currently for any operations performed by users through REST service, the internal HBase ACL is bypassed and users can perform any operation without security restrictions (they can view and insert data to any location). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14109) NPE if we don't load fully before we are shutdown
stack created HBASE-14109:
-
Summary: NPE if we don't load fully before we are shutdown
Key: HBASE-14109
URL: https://issues.apache.org/jira/browse/HBASE-14109
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 1.2.0
Reporter: stack
Assignee: stack
Priority: Trivial
We failed to find ZK so:
{code}
2015-07-15 22:51:04,725 FATAL
[regionserver/c2022.halxg.cloudera.com/10.20.84.28:16020]
regionserver.HRegionServer: ABORTING region server
c2022.halxg.cloudera.com,16020,1437025808277: Initialization of RS failed.
Hence aborting RS.
java.io.IOException: Received the shutdown message while waiting.
at
org.apache.hadoop.hbase.regionserver.HRegionServer.blockAndCheckIfStopped(HRegionServer.java:807)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:756)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:732)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:876)
at java.lang.Thread.run(Thread.java:745)
{code}
... which got us this:
{code}
2015-07-15 22:51:04,734 FATAL
[regionserver/c2022.halxg.cloudera.com/10.20.84.28:16020]
regionserver.HRegionServer: ABORTING region server
c2022.halxg.cloudera.com,16020,1437025808277: Unhandled: null
java.lang.NullPointerException
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:911)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14098) Allow dropping caches behind compactions
[
https://issues.apache.org/jira/browse/HBASE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630189#comment-14630189
]
Hadoop QA commented on HBASE-14098:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12745644/HBASE-14098-v1.patch
against master branch at commit 97e0af001bdef74d9e0cc23e09cf7105bb8e0f64.
ATTACHMENT ID: 12745644
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 8 zombie test(s):
at
org.apache.hadoop.hbase.mapreduce.TestImportTSVWithVisibilityLabels.testMROnTableWithDeletes(TestImportTSVWithVisibilityLabels.java:180)
at
org.apache.hadoop.hbase.mapreduce.TestCopyTable.testStartStopRow(TestCopyTable.java:159)
at
org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testExportFileSystemState(TestExportSnapshot.java:288)
at
org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testExportFileSystemState(TestExportSnapshot.java:262)
at
org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testSnapshotWithRefsExportFileSystemState(TestExportSnapshot.java:256)
at
org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testSnapshotWithRefsExportFileSystemState(TestExportSnapshot.java:236)
at
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.testExcludeMinorCompaction(TestHFileOutputFormat.java:977)
at
org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster.testMoveRegion(TestAssignmentManagerOnCluster.java:389)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14803//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14803//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14803//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14803//console
This message is automatically generated.
Allow dropping caches behind compactions
Key: HBASE-14098
URL: https://issues.apache.org/jira/browse/HBASE-14098
Project: HBase
Issue Type: Bug
Components: Compaction, hadoop2, HFile
Affects Versions: 2.0.0, 1.3.0
Reporter: Elliott Clark
Assignee: Elliott Clark
Fix For: 2.0.0, 1.3.0
Attachments: HBASE-14098-v1.patch, HBASE-14098.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14110) Add CostFunction for balancing primary region replicas
Ted Yu created HBASE-14110: -- Summary: Add CostFunction for balancing primary region replicas Key: HBASE-14110 URL: https://issues.apache.org/jira/browse/HBASE-14110 Project: HBase Issue Type: Improvement Reporter: Ted Yu Assignee: Ted Yu Currently replicas for the same region are spread across region servers. However, one region server may serve disproportionately high number of primary region replicas. This has been observed in production cluster. This issue adds PrimaryRegionCountSkewCostFunction which facilitates balancing primary region replicas evenly across region servers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14110) Add CostFunction for balancing primary region replicas
[ https://issues.apache.org/jira/browse/HBASE-14110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14110: --- Attachment: 14110-v1.txt Balancer related tests pass for patch v1. Add CostFunction for balancing primary region replicas -- Key: HBASE-14110 URL: https://issues.apache.org/jira/browse/HBASE-14110 Project: HBase Issue Type: Improvement Reporter: Ted Yu Assignee: Ted Yu Attachments: 14110-v1.txt Currently replicas for the same region are spread across region servers. However, one region server may serve disproportionately high number of primary region replicas. This has been observed in production cluster. This issue adds PrimaryRegionCountSkewCostFunction which facilitates balancing primary region replicas evenly across region servers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14089) Remove unnecessary draw of system entropy from RecoverableZooKeeper
[ https://issues.apache.org/jira/browse/HBASE-14089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629266#comment-14629266 ] Hudson commented on HBASE-14089: SUCCESS: Integrated in HBase-0.98 #1059 (See [https://builds.apache.org/job/HBase-0.98/1059/]) HBASE-14089 Remove unnecessary draw of system entropy from RecoverableZooKeeper (apurtell: rev 9a7df6936e8f638116bb6e27bff3ab233aa4b47b) * hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java Remove unnecessary draw of system entropy from RecoverableZooKeeper --- Key: HBASE-14089 URL: https://issues.apache.org/jira/browse/HBASE-14089 Project: HBase Issue Type: Bug Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Fix For: 2.0.0, 0.98.14, 1.1.2, 1.3.0, 1.2.1, 1.0.3 Attachments: HBASE-14089.patch I had a look at instances where we use SecureRandom, which could block if insufficient entropy, in the 0.98 and master branch code. (Random in contrast is a PRNG seeded by System#nanoTime, it doesn't draw from system entropy.) Most uses are in encryption related code, our native encryption and SSL, but we do also use SecureRandom for salting znode metadata in RecoverableZooKeeper#appendMetadata, which is called whenever we do setData. Conceivably we could block unexpectedly when constructing data to write out to a znode if entropy gets too low until more is available. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-8642) [Snapshot] List and delete snapshot by table
[ https://issues.apache.org/jira/browse/HBASE-8642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629267#comment-14629267 ] Hudson commented on HBASE-8642: --- SUCCESS: Integrated in HBase-0.98 #1059 (See [https://builds.apache.org/job/HBase-0.98/1059/]) HBASE-8642 [Snapshot] List and delete snapshot by table (apurtell: rev f15ad3da641785ae6105f9100d486dc270aa8dcd) * hbase-shell/src/main/ruby/hbase/admin.rb * hbase-shell/src/main/ruby/shell/commands/delete_table_snapshots.rb * hbase-shell/src/main/ruby/shell.rb * hbase-shell/src/main/ruby/shell/commands/list_table_snapshots.rb * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotFromClient.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java [Snapshot] List and delete snapshot by table Key: HBASE-8642 URL: https://issues.apache.org/jira/browse/HBASE-8642 Project: HBase Issue Type: Improvement Components: snapshots Affects Versions: 0.98.0, 0.95.0, 0.95.1, 0.95.2 Reporter: Julian Zhou Assignee: Ashish Singhi Fix For: 2.0.0, 0.98.14, 1.3.0 Attachments: 8642-trunk-0.95-v0.patch, 8642-trunk-0.95-v1.patch, 8642-trunk-0.95-v2.patch, HBASE-8642-0.98.patch, HBASE-8642-v1.patch, HBASE-8642-v2.patch, HBASE-8642-v3.patch, HBASE-8642-v4.patch, HBASE-8642.patch Support list and delete snapshots by table names. User scenario: A user wants to delete all the snapshots which were taken in January month for a table 't' where snapshot names starts with 'Jan'. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13971) Flushes stuck since 6 hours on a regionserver.
[ https://issues.apache.org/jira/browse/HBASE-13971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629426#comment-14629426 ] Ted Yu commented on HBASE-13971: Is there more review comment ? If not, planning to commit later today. Flushes stuck since 6 hours on a regionserver. -- Key: HBASE-13971 URL: https://issues.apache.org/jira/browse/HBASE-13971 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 1.3.0 Environment: Caused while running IntegrationTestLoadAndVerify for 20 M rows on cluster with 32 region servers each with max heap size of 24GBs. Reporter: Abhilash Assignee: Ted Yu Priority: Critical Attachments: 13971-v1.txt, 13971-v1.txt, 13971-v1.txt, jstack.1, jstack.2, jstack.3, jstack.4, jstack.5, rsDebugDump.txt, screenshot-1.png One region server stuck while flushing(possible deadlock). Its trying to flush two regions since last 6 hours (see the screenshot). Caused while running IntegrationTestLoadAndVerify for 20 M rows with 600 mapper jobs and 100 back references. ~37 Million writes on each regionserver till now but no writes happening on any regionserver from past 6 hours and their memstore size is zero(I dont know if this is related). But this particular regionserver has memstore size of 9GBs from past 6 hours. Relevant snaps from debug dump: Tasks: === Task: Flushing IntegrationTestLoadAndVerify,R\x9B\x1B\xBF\xAE\x08\xD1\xA2,1435179555993.8e2d075f94ce7699f416ec4ced9873cd. Status: RUNNING:Preparing to flush by snapshotting stores in 8e2d075f94ce7699f416ec4ced9873cd Running for 22034s Task: Flushing IntegrationTestLoadAndVerify,\x93\xA385\x81Z\x11\xE6,1435179555993.9f8d0e01a40405b835bf6e5a22a86390. Status: RUNNING:Preparing to flush by snapshotting stores in 9f8d0e01a40405b835bf6e5a22a86390 Running for 22033s Executors: === ... Thread 139 (MemStoreFlusher.1): State: WAITING Blocked count: 139711 Waited count: 239212 Waiting on java.util.concurrent.CountDownLatch$Sync@b9c094a Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231) org.apache.hadoop.hbase.wal.WALKey.getSequenceId(WALKey.java:305) org.apache.hadoop.hbase.regionserver.HRegion.getNextSequenceId(HRegion.java:2422) org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2168) org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2047) org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2011) org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1902) org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1828) org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:510) org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:471) org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$900(MemStoreFlusher.java:75) org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:259) java.lang.Thread.run(Thread.java:745) Thread 137 (MemStoreFlusher.0): State: WAITING Blocked count: 138931 Waited count: 237448 Waiting on java.util.concurrent.CountDownLatch$Sync@53f41f76 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231) org.apache.hadoop.hbase.wal.WALKey.getSequenceId(WALKey.java:305) org.apache.hadoop.hbase.regionserver.HRegion.getNextSequenceId(HRegion.java:2422) org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2168)
[jira] [Created] (HBASE-14098) Allow dropping caches behind compactions
Elliott Clark created HBASE-14098: - Summary: Allow dropping caches behind compactions Key: HBASE-14098 URL: https://issues.apache.org/jira/browse/HBASE-14098 Project: HBase Issue Type: Bug Components: Compaction, hadoop2, HFile Reporter: Elliott Clark Assignee: Elliott Clark -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14075) HBaseClusterManager should use port(if given) to find pid
[
https://issues.apache.org/jira/browse/HBASE-14075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-14075:
--
Attachment: HBASE-14075-master_v6.patch
Update the patch to resolve ling length issue
[~dimaspivak] could you help review the updated patch on RB? Thanks
[~tedyu] could you also help review the patch? Thanks
HBaseClusterManager should use port(if given) to find pid
-
Key: HBASE-14075
URL: https://issues.apache.org/jira/browse/HBASE-14075
Project: HBase
Issue Type: Bug
Reporter: Yu Li
Assignee: Yu Li
Priority: Minor
Attachments: HBASE-14075-master_v2.patch,
HBASE-14075-master_v3.patch, HBASE-14075-master_v4.patch,
HBASE-14075-master_v5.patch, HBASE-14075-master_v6.patch, HBASE-14075.patch
This issue is found while we run ITBLL in distributed cluster. Our testing
env is kind of special that we run multiple regionserver instance on a single
physical machine, so {noformat}ps -ef | grep proc_regionserver{noformat} will
return more than one line, thus cause the tool might check/kill the wrong
process
Actually in HBaseClusterManager we already introduce port as a parameter for
methods like isRunning, kill, etc. So the only thing to do here is to get pid
through port if port is given
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14090) Redo FS layout; let go of tables/regions/stores directory hierarchy in DFS
[ https://issues.apache.org/jira/browse/HBASE-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629516#comment-14629516 ] Lars Hofhansl commented on HBASE-14090: --- To throw out a radical idea: We _could_ go as far as having HBase manage a block pool directly, without the NN being involved at all. We'd lose things like distCP, etc, but we'd be in control of all blocks directly (placement, etc), and no need to run a pesky NN (with QJMs, etc). Let me dream for a bit :) Redo FS layout; let go of tables/regions/stores directory hierarchy in DFS -- Key: HBASE-14090 URL: https://issues.apache.org/jira/browse/HBASE-14090 Project: HBase Issue Type: Sub-task Reporter: stack Our layout as is won't work if 1M regions; e.g. HDFS will fall over if directories of hundreds of thousands of files. HBASE-13991 (Humongous Tables) would address this specific directory problem only by adding subdirs under table dir but there are other issues with our current layout: * Our table/regions/column family 'facade' has to be maintained in two locations -- in master memory and in the hdfs directory layout -- and the farce needs to be kept synced or worse, the model management is split between master memory and DFS layout. 'Syncing' in HDFS has us dropping constructs such as 'Reference' and 'HalfHFiles' on split, 'HFileLinks' when archiving, and so on. This 'tie' makes it hard to make changes. * While HDFS has atomic rename, useful for fencing and for having files added atomically, if the model were solely owned by hbase, there are hbase primitives we could make use of -- changes in a row are atomic and coprocessors -- to simplify table transactions and provide more consistent views of our model to clients; file 'moves' could be a memory operation only rather than an HDFS call; sharing files between tables/snapshots and when it is safe to remove them would be simplified if one owner only; and so on. This is an umbrella blue-sky issue to discuss what a new layout would look like and how we might get there. I'll follow up with some sketches of what new layout could look like that come of some chats a few of us have been having. We are also under the 'delusion' that move to a new layout could be done as part of a rolling upgrade and that the amount of work involved is not gargantuan. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14084) Observe some out-of-date doc on Integration Tests part
[ https://issues.apache.org/jira/browse/HBASE-14084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629521#comment-14629521 ] Yu Li commented on HBASE-14084: --- I'd like to try but also cannot assure the time, will attach the patch if got it done, and will leave the jira unassigned in case more experienced people want to take it. :-) Observe some out-of-date doc on Integration Tests part Key: HBASE-14084 URL: https://issues.apache.org/jira/browse/HBASE-14084 Project: HBase Issue Type: Task Components: documentation Affects Versions: 1.1.0 Reporter: Yu Li As titled, have checked src/main/asciidoc/_chapters/developer.adoc and confirmed some out-of-date part, such as the doc still refers to org.apache.hadoop.hbase.util.ChaosMonkey which doesn't exist anymore On the other hand, I think run ITBLL against distributed cluster is a really good way to do real-world integration/system testing, but existing document about this is not explicit enough. Actually it costs me quite a while to setup the env and make the testing run smoothly (encountered issues like always launching minicluster if run with bin/hbase script, finally got it run using bin/hadoop script, etc.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14098) Allow dropping caches behind compactions
[ https://issues.apache.org/jira/browse/HBASE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-14098: -- Attachment: HBASE-14098.patch Pretty simple patch to try and see how this works. Allow dropping caches behind compactions Key: HBASE-14098 URL: https://issues.apache.org/jira/browse/HBASE-14098 Project: HBase Issue Type: Bug Components: Compaction, hadoop2, HFile Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HBASE-14098.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14098) Allow dropping caches behind compactions
[ https://issues.apache.org/jira/browse/HBASE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629479#comment-14629479 ] Elliott Clark commented on HBASE-14098: --- Yeah related but on the other side. Allow dropping caches behind compactions Key: HBASE-14098 URL: https://issues.apache.org/jira/browse/HBASE-14098 Project: HBase Issue Type: Bug Components: Compaction, hadoop2, HFile Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HBASE-14098.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14075) HBaseClusterManager should use port(if given) to find pid
[
https://issues.apache.org/jira/browse/HBASE-14075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629504#comment-14629504
]
Hadoop QA commented on HBASE-14075:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12745583/HBASE-14075-master_v6.patch
against master branch at commit ebdac4b52e67614db70b59be8cd8143efe701911.
ATTACHMENT ID: 12745583
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 56 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14797//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14797//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14797//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14797//console
This message is automatically generated.
HBaseClusterManager should use port(if given) to find pid
-
Key: HBASE-14075
URL: https://issues.apache.org/jira/browse/HBASE-14075
Project: HBase
Issue Type: Bug
Reporter: Yu Li
Assignee: Yu Li
Priority: Minor
Attachments: HBASE-14075-master_v2.patch,
HBASE-14075-master_v3.patch, HBASE-14075-master_v4.patch,
HBASE-14075-master_v5.patch, HBASE-14075-master_v6.patch, HBASE-14075.patch
This issue is found while we run ITBLL in distributed cluster. Our testing
env is kind of special that we run multiple regionserver instance on a single
physical machine, so {noformat}ps -ef | grep proc_regionserver{noformat} will
return more than one line, thus cause the tool might check/kill the wrong
process
Actually in HBaseClusterManager we already introduce port as a parameter for
methods like isRunning, kill, etc. So the only thing to do here is to get pid
through port if port is given
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14098) Allow dropping caches behind compactions
[ https://issues.apache.org/jira/browse/HBASE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-14098: -- Fix Version/s: 1.3.0 2.0.0 Affects Version/s: 1.3.0 2.0.0 Status: Patch Available (was: Open) Allow dropping caches behind compactions Key: HBASE-14098 URL: https://issues.apache.org/jira/browse/HBASE-14098 Project: HBase Issue Type: Bug Components: Compaction, hadoop2, HFile Affects Versions: 2.0.0, 1.3.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0, 1.3.0 Attachments: HBASE-14098.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14084) Observe some out-of-date doc on Integration Tests part
[ https://issues.apache.org/jira/browse/HBASE-14084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629519#comment-14629519 ] Yu Li commented on HBASE-14084: --- Thanks for mentioning HBASE-11276 Dima, actually I'm just planning to write a standalone ChaosMonkey runner since ITBLL cannot cover system testing for tools like Bulkload (i.e. testing bulkload during region move/split/flush or rs/master crash). Now I could upload a patch there instead of creating a new jira when I got the work done :-) Observe some out-of-date doc on Integration Tests part Key: HBASE-14084 URL: https://issues.apache.org/jira/browse/HBASE-14084 Project: HBase Issue Type: Task Components: documentation Affects Versions: 1.1.0 Reporter: Yu Li As titled, have checked src/main/asciidoc/_chapters/developer.adoc and confirmed some out-of-date part, such as the doc still refers to org.apache.hadoop.hbase.util.ChaosMonkey which doesn't exist anymore On the other hand, I think run ITBLL against distributed cluster is a really good way to do real-world integration/system testing, but existing document about this is not explicit enough. Actually it costs me quite a while to setup the env and make the testing run smoothly (encountered issues like always launching minicluster if run with bin/hbase script, finally got it run using bin/hadoop script, etc.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14076) ResultSerialization and MutationSerialization can throw InvalidProtocolBufferException when serializing a cell larger than 64MB
[
https://issues.apache.org/jira/browse/HBASE-14076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629522#comment-14629522
]
Hadoop QA commented on HBASE-14076:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12745587/HBASE-14076.hbase-11339.patch
against hbase-11339 branch at commit ebdac4b52e67614db70b59be8cd8143efe701911.
ATTACHMENT ID: 12745587
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1923 checkstyle errors (more than the master's current 1922 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14798//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14798//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14798//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14798//console
This message is automatically generated.
ResultSerialization and MutationSerialization can throw
InvalidProtocolBufferException when serializing a cell larger than 64MB
---
Key: HBASE-14076
URL: https://issues.apache.org/jira/browse/HBASE-14076
Project: HBase
Issue Type: Bug
Affects Versions: 2.0.0, hbase-11339, 1.2.0
Reporter: Esteban Gutierrez
Assignee: Esteban Gutierrez
Attachments: HBASE-14076.hbase-11339.patch,
HBASE-14076.hbase-11339.patch
This was reported in CRUNCH-534 but is a problem how we handle
deserialization of large Cells ( 64MB) in ResultSerialization and
MutationSerialization.
The fix is just re-using what it was done in HBASE-13230.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14098) Allow dropping caches behind compactions
[ https://issues.apache.org/jira/browse/HBASE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629498#comment-14629498 ] Lars Hofhansl commented on HBASE-14098: --- Oh... This is for the output files of a compaction... Aren't we likely going to read the compacted data again (especially since we just dropped it all from the block cache)? Allow dropping caches behind compactions Key: HBASE-14098 URL: https://issues.apache.org/jira/browse/HBASE-14098 Project: HBase Issue Type: Bug Components: Compaction, hadoop2, HFile Affects Versions: 2.0.0, 1.3.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0, 1.3.0 Attachments: HBASE-14098.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14098) Allow dropping caches behind compactions
[
https://issues.apache.org/jira/browse/HBASE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629537#comment-14629537
]
Hadoop QA commented on HBASE-14098:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12745598/HBASE-14098.patch
against master branch at commit ebdac4b52e67614db70b59be8cd8143efe701911.
ATTACHMENT ID: 12745598
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.io.hfile.TestLazyDataBlockDecompression
org.apache.hadoop.hbase.regionserver.TestBlocksScanned
org.apache.hadoop.hbase.io.hfile.TestReseekTo
org.apache.hadoop.hbase.regionserver.TestSplitTransaction
org.apache.hadoop.hbase.io.hfile.TestSeekTo
org.apache.hadoop.hbase.regionserver.TestColumnSeeking
org.apache.hadoop.hbase.coprocessor.TestCoprocessorInterface
org.apache.hadoop.hbase.io.hfile.TestHFileInlineToRootChunkConversion
org.apache.hadoop.hbase.io.hfile.TestPrefetch
org.apache.hadoop.hbase.regionserver.TestStoreFile
org.apache.hadoop.hbase.io.encoding.TestPrefixTree
org.apache.hadoop.hbase.io.hfile.TestHFileWriterV3
org.apache.hadoop.hbase.regionserver.TestBulkLoad
org.apache.hadoop.hbase.io.hfile.TestHFileEncryption
org.apache.hadoop.hbase.filter.TestColumnPrefixFilter
org.apache.hadoop.hbase.regionserver.TestScanner
org.apache.hadoop.hbase.filter.TestDependentColumnFilter
org.apache.hadoop.hbase.regionserver.TestResettingCounters
org.apache.hadoop.hbase.regionserver.TestRegionMergeTransaction
org.apache.hadoop.hbase.regionserver.TestKeepDeletes
org.apache.hadoop.hbase.regionserver.TestStoreFileRefresherChore
org.apache.hadoop.hbase.coprocessor.TestRegionObserverStacking
org.apache.hadoop.hbase.io.hfile.TestScannerSelectionUsingKeyRange
org.apache.hadoop.hbase.io.hfile.TestHFile
org.apache.hadoop.hbase.regionserver.TestScanWithBloomError
org.apache.hadoop.hbase.filter.TestFilter
org.apache.hadoop.hbase.filter.TestInvocationRecordFilter
org.apache.hadoop.hbase.filter.TestMultipleColumnPrefixFilter
org.apache.hadoop.hbase.regionserver.TestWideScanner
org.apache.hadoop.hbase.io.TestHalfStoreFileReader
org.apache.hadoop.hbase.regionserver.TestStoreFileScannerWithTagCompression
org.apache.hadoop.hbase.client.TestIntraRowPagination
org.apache.hadoop.hbase.regionserver.TestMinVersions
org.apache.hadoop.hbase.io.hfile.TestHFileWriterV2
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14799//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14799//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14799//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14799//console
This message is automatically generated.
Allow dropping caches behind compactions
Key: HBASE-14098
URL: https://issues.apache.org/jira/browse/HBASE-14098
[jira] [Commented] (HBASE-14089) Remove unnecessary draw of system entropy from RecoverableZooKeeper
[ https://issues.apache.org/jira/browse/HBASE-14089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629456#comment-14629456 ] Hudson commented on HBASE-14089: SUCCESS: Integrated in HBase-1.3-IT #44 (See [https://builds.apache.org/job/HBase-1.3-IT/44/]) HBASE-14089 Remove unnecessary draw of system entropy from RecoverableZooKeeper (apurtell: rev c42fc1dd56a651a52b00d03833b00fb800fa5cd9) * hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java Remove unnecessary draw of system entropy from RecoverableZooKeeper --- Key: HBASE-14089 URL: https://issues.apache.org/jira/browse/HBASE-14089 Project: HBase Issue Type: Bug Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Fix For: 2.0.0, 0.98.14, 1.1.2, 1.3.0, 1.2.1, 1.0.3 Attachments: HBASE-14089.patch I had a look at instances where we use SecureRandom, which could block if insufficient entropy, in the 0.98 and master branch code. (Random in contrast is a PRNG seeded by System#nanoTime, it doesn't draw from system entropy.) Most uses are in encryption related code, our native encryption and SSL, but we do also use SecureRandom for salting znode metadata in RecoverableZooKeeper#appendMetadata, which is called whenever we do setData. Conceivably we could block unexpectedly when constructing data to write out to a znode if entropy gets too low until more is available. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14097) Log link to client scan troubleshooting section when scanner exceptions happen.
[
https://issues.apache.org/jira/browse/HBASE-14097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629322#comment-14629322
]
Hadoop QA commented on HBASE-14097:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12745571/HBASE-14097.patch
against master branch at commit ebdac4b52e67614db70b59be8cd8143efe701911.
ATTACHMENT ID: 12745571
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 5 zombie test(s):
at
org.apache.hadoop.hbase.io.encoding.TestEncodedSeekers.testEncodedSeeker(TestEncodedSeekers.java:122)
at
org.apache.hadoop.hbase.io.encoding.TestDataBlockEncoders.testSeekingOnSample(TestDataBlockEncoders.java:209)
at
org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite.testNotCachingDataBlocksDuringCompactionInternals(TestCacheOnWrite.java:454)
at
org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite.testNotCachingDataBlocksDuringCompaction(TestCacheOnWrite.java:478)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14796//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14796//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14796//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14796//console
This message is automatically generated.
Log link to client scan troubleshooting section when scanner exceptions
happen.
---
Key: HBASE-14097
URL: https://issues.apache.org/jira/browse/HBASE-14097
Project: HBase
Issue Type: Improvement
Reporter: Srikanth Srungarapu
Assignee: Srikanth Srungarapu
Priority: Trivial
Attachments: HBASE-14097.patch
As per description.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-8642) [Snapshot] List and delete snapshot by table
[ https://issues.apache.org/jira/browse/HBASE-8642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629355#comment-14629355 ] Hudson commented on HBASE-8642: --- FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1012 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1012/]) HBASE-8642 [Snapshot] List and delete snapshot by table (apurtell: rev f15ad3da641785ae6105f9100d486dc270aa8dcd) * hbase-shell/src/main/ruby/hbase/admin.rb * hbase-shell/src/main/ruby/shell/commands/list_table_snapshots.rb * hbase-shell/src/main/ruby/shell.rb * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotFromClient.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java * hbase-shell/src/main/ruby/shell/commands/delete_table_snapshots.rb [Snapshot] List and delete snapshot by table Key: HBASE-8642 URL: https://issues.apache.org/jira/browse/HBASE-8642 Project: HBase Issue Type: Improvement Components: snapshots Affects Versions: 0.98.0, 0.95.0, 0.95.1, 0.95.2 Reporter: Julian Zhou Assignee: Ashish Singhi Fix For: 2.0.0, 0.98.14, 1.3.0 Attachments: 8642-trunk-0.95-v0.patch, 8642-trunk-0.95-v1.patch, 8642-trunk-0.95-v2.patch, HBASE-8642-0.98.patch, HBASE-8642-v1.patch, HBASE-8642-v2.patch, HBASE-8642-v3.patch, HBASE-8642-v4.patch, HBASE-8642.patch Support list and delete snapshots by table names. User scenario: A user wants to delete all the snapshots which were taken in January month for a table 't' where snapshot names starts with 'Jan'. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14089) Remove unnecessary draw of system entropy from RecoverableZooKeeper
[ https://issues.apache.org/jira/browse/HBASE-14089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629354#comment-14629354 ] Hudson commented on HBASE-14089: FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1012 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1012/]) HBASE-14089 Remove unnecessary draw of system entropy from RecoverableZooKeeper (apurtell: rev 9a7df6936e8f638116bb6e27bff3ab233aa4b47b) * hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java Remove unnecessary draw of system entropy from RecoverableZooKeeper --- Key: HBASE-14089 URL: https://issues.apache.org/jira/browse/HBASE-14089 Project: HBase Issue Type: Bug Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Fix For: 2.0.0, 0.98.14, 1.1.2, 1.3.0, 1.2.1, 1.0.3 Attachments: HBASE-14089.patch I had a look at instances where we use SecureRandom, which could block if insufficient entropy, in the 0.98 and master branch code. (Random in contrast is a PRNG seeded by System#nanoTime, it doesn't draw from system entropy.) Most uses are in encryption related code, our native encryption and SSL, but we do also use SecureRandom for salting znode metadata in RecoverableZooKeeper#appendMetadata, which is called whenever we do setData. Conceivably we could block unexpectedly when constructing data to write out to a znode if entropy gets too low until more is available. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14069) Add the ability for RegionSplitter to rolling split without using a SplitAlgorithm
[ https://issues.apache.org/jira/browse/HBASE-14069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629398#comment-14629398 ] Heng Chen commented on HBASE-14069: --- You will use HbaseAdmin.SplitRegion to split region directly , am I right ? Add the ability for RegionSplitter to rolling split without using a SplitAlgorithm -- Key: HBASE-14069 URL: https://issues.apache.org/jira/browse/HBASE-14069 Project: HBase Issue Type: New Feature Reporter: Elliott Clark Assignee: Abhilash RegionSplittler is the utility that can rolling split regions. It would be nice to be able to split regions and have the normal split points get computed for me so that I'm not reliant on knowing data distribution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12596) bulkload needs to follow locality
[ https://issues.apache.org/jira/browse/HBASE-12596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629448#comment-14629448 ] Lars Hofhansl commented on HBASE-12596: --- Hmm... We usually do not add improvements to earlier branches but not into later branches, So if it is in 0.98, we probably want this in 1.0, 1.1, and especially 1.2 as well. [~apurtell], thoughts? Thing similar to this came up for 0.94 every now and then, where folks complained either the earlier release forcing stuff into the later release, or alternatively that later releases could veto changes into the earlier releases. Not sure how we want to do this with 0.98, but the above is weird. Folks going from 0.98 to 1.0, 1.1, or 1.2 shouldn't suddenly lose an improvement. bulkload needs to follow locality - Key: HBASE-12596 URL: https://issues.apache.org/jira/browse/HBASE-12596 Project: HBase Issue Type: Improvement Components: HFile, regionserver Affects Versions: 0.98.8 Environment: hadoop-2.3.0, hbase-0.98.8, jdk1.7 Reporter: Victor Xu Assignee: Victor Xu Fix For: 2.0.0, 0.98.14, 1.3.0 Attachments: HBASE-12596-0.98-v1.patch, HBASE-12596-0.98-v2.patch, HBASE-12596-0.98-v3.patch, HBASE-12596-0.98-v4.patch, HBASE-12596-0.98-v5.patch, HBASE-12596-0.98-v6.patch, HBASE-12596-branch-1-v1.patch, HBASE-12596-branch-1-v2.patch, HBASE-12596-master-v1.patch, HBASE-12596-master-v2.patch, HBASE-12596-master-v3.patch, HBASE-12596-master-v4.patch, HBASE-12596-master-v5.patch, HBASE-12596-master-v6.patch, HBASE-12596.patch Normally, we have 2 steps to perform a bulkload: 1. use a job to write HFiles to be loaded; 2. Move these HFiles to the right hdfs directory. However, the locality could be loss during the first step. Why not just write the HFiles directly into the right place? We can do this easily because StoreFile.WriterBuilder has the withFavoredNodes method, and we just need to call it in HFileOutputFormat's getNewWriter(). This feature is enabled by default, and we could use 'hbase.bulkload.locality.sensitive.enabled=false' to disable it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14076) ResultSerialization and MutationSerialization can throw InvalidProtocolBufferException when serializing a cell larger than 64MB
[ https://issues.apache.org/jira/browse/HBASE-14076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esteban Gutierrez updated HBASE-14076: -- Attachment: HBASE-14076.hbase-11339.patch ResultSerialization and MutationSerialization can throw InvalidProtocolBufferException when serializing a cell larger than 64MB --- Key: HBASE-14076 URL: https://issues.apache.org/jira/browse/HBASE-14076 Project: HBase Issue Type: Bug Affects Versions: 2.0.0, hbase-11339, 1.2.0 Reporter: Esteban Gutierrez Assignee: Esteban Gutierrez Attachments: HBASE-14076.hbase-11339.patch, HBASE-14076.hbase-11339.patch This was reported in CRUNCH-534 but is a problem how we handle deserialization of large Cells ( 64MB) in ResultSerialization and MutationSerialization. The fix is just re-using what it was done in HBASE-13230. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14076) ResultSerialization and MutationSerialization can throw InvalidProtocolBufferException when serializing a cell larger than 64MB
[ https://issues.apache.org/jira/browse/HBASE-14076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629351#comment-14629351 ] Esteban Gutierrez commented on HBASE-14076: --- Both test failures don't seem to be related (those tests have failed in the past). Will kick another build just to be sure about the zombie tests too. ResultSerialization and MutationSerialization can throw InvalidProtocolBufferException when serializing a cell larger than 64MB --- Key: HBASE-14076 URL: https://issues.apache.org/jira/browse/HBASE-14076 Project: HBase Issue Type: Bug Affects Versions: 2.0.0, hbase-11339, 1.2.0 Reporter: Esteban Gutierrez Assignee: Esteban Gutierrez Attachments: HBASE-14076.hbase-11339.patch This was reported in CRUNCH-534 but is a problem how we handle deserialization of large Cells ( 64MB) in ResultSerialization and MutationSerialization. The fix is just re-using what it was done in HBASE-13230. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14098) Allow dropping caches behind compactions
[ https://issues.apache.org/jira/browse/HBASE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629459#comment-14629459 ] Lars Hofhansl commented on HBASE-14098: --- Related to HBASE-10052? Allow dropping caches behind compactions Key: HBASE-14098 URL: https://issues.apache.org/jira/browse/HBASE-14098 Project: HBase Issue Type: Bug Components: Compaction, hadoop2, HFile Reporter: Elliott Clark Assignee: Elliott Clark -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-8642) [Snapshot] List and delete snapshot by table
[ https://issues.apache.org/jira/browse/HBASE-8642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629457#comment-14629457 ] Hudson commented on HBASE-8642: --- SUCCESS: Integrated in HBase-1.3-IT #44 (See [https://builds.apache.org/job/HBase-1.3-IT/44/]) HBASE-8642 [Snapshot] List and delete snapshot by table (apurtell: rev 789d2a94b7e8c2d97c1b52f4f1f0d47922b711a2) * hbase-shell/src/main/ruby/shell.rb * hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java * hbase-shell/src/main/ruby/shell/commands/delete_table_snapshots.rb * hbase-shell/src/main/ruby/hbase/admin.rb * hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotFromClient.java * hbase-shell/src/main/ruby/shell/commands/list_table_snapshots.rb [Snapshot] List and delete snapshot by table Key: HBASE-8642 URL: https://issues.apache.org/jira/browse/HBASE-8642 Project: HBase Issue Type: Improvement Components: snapshots Affects Versions: 0.98.0, 0.95.0, 0.95.1, 0.95.2 Reporter: Julian Zhou Assignee: Ashish Singhi Fix For: 2.0.0, 0.98.14, 1.3.0 Attachments: 8642-trunk-0.95-v0.patch, 8642-trunk-0.95-v1.patch, 8642-trunk-0.95-v2.patch, HBASE-8642-0.98.patch, HBASE-8642-v1.patch, HBASE-8642-v2.patch, HBASE-8642-v3.patch, HBASE-8642-v4.patch, HBASE-8642.patch Support list and delete snapshots by table names. User scenario: A user wants to delete all the snapshots which were taken in January month for a table 't' where snapshot names starts with 'Jan'. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14099) StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row
[
https://issues.apache.org/jira/browse/HBASE-14099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-14099:
---
Fix Version/s: 2.0.0
StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start
and stop Row
---
Key: HBASE-14099
URL: https://issues.apache.org/jira/browse/HBASE-14099
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Minor
Fix For: 2.0.0
During profiling saw that the code here in passesKeyRangeFilter in Storefile
{code}
KeyValue smallestScanKeyValue = scan.isReversed() ? KeyValueUtil
.createFirstOnRow(scan.getStopRow()) :
KeyValueUtil.createFirstOnRow(scan
.getStartRow());
KeyValue largestScanKeyValue = scan.isReversed() ? KeyValueUtil
.createLastOnRow(scan.getStartRow()) :
KeyValueUtil.createLastOnRow(scan
.getStopRow());
{code}
This row need not be copied now considering that we have
CellComparator.compareRows(Cell, byte[]).
We have already refactored the firstKeyKv and lastKeyKV as part of other
JIRAs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14099) StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row
ramkrishna.s.vasudevan created HBASE-14099:
--
Summary: StoreFile.passesKeyRangeFilter need not create Cells from
the Scan's start and stop Row
Key: HBASE-14099
URL: https://issues.apache.org/jira/browse/HBASE-14099
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Minor
During profiling saw that the code here in passesKeyRangeFilter in Storefile
{code}
KeyValue smallestScanKeyValue = scan.isReversed() ? KeyValueUtil
.createFirstOnRow(scan.getStopRow()) :
KeyValueUtil.createFirstOnRow(scan
.getStartRow());
KeyValue largestScanKeyValue = scan.isReversed() ? KeyValueUtil
.createLastOnRow(scan.getStartRow()) :
KeyValueUtil.createLastOnRow(scan
.getStopRow());
{code}
This row need not be copied now considering that we have
CellComparator.compareRows(Cell, byte[]).
We have already refactored the firstKeyKv and lastKeyKV as part of other JIRAs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13954) Remove HTableInterface#getRowOrBefore related server side code
[
https://issues.apache.org/jira/browse/HBASE-13954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629628#comment-14629628
]
Ashish Singhi commented on HBASE-13954:
---
Attached patch for master branch which removes all the code related to
getClosestRowOrBefore.
Removed the respective non-deprecated API from Region and Store class also as
the interfaces are marked as LimitedPrivate(COPROC) which means they are not
public and need not go through deprecation cycle process.
Removed a non-deprecated api from Memstore interface as well which should not
be a problem as it marked for private audience.
I have deprecated setter and getter for the flag used for this API in Get class
as it is marked for public audience, where setter does not do any operation and
getter just returns false (default value).
I did not remove {{getClosestRowBefore}} operation from the acl matrix in the
book, as it is still valid for other branches and we do not have any separate
book for 2.0.0.
I modified the existing test cases to use reverse scan where ever the removed
apis where referenced.
Searched for {{GetClosestRow}} in *.rb scripts found no occurrence.
Uploaded the patch in RB as well, https://reviews.apache.org/r/36544/
Please review.
Remove HTableInterface#getRowOrBefore related server side code
--
Key: HBASE-13954
URL: https://issues.apache.org/jira/browse/HBASE-13954
Project: HBase
Issue Type: Sub-task
Components: API
Reporter: Ashish Singhi
Assignee: Ashish Singhi
Fix For: 2.0.0
Attachments: HBASE-13954.patch
As part of HBASE-13214 review, [~anoop.hbase] had a review comment on the
review board to remove all the server side related code for getRowOrBefore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14100) Fix high priority findbugs warnings
[ https://issues.apache.org/jira/browse/HBASE-14100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629735#comment-14629735 ] Duo Zhang commented on HBASE-14100: --- [~ram_krish] Fine. Is HBASE-12213 almost done? If not I think we could remove it first in this issue. Fix high priority findbugs warnings --- Key: HBASE-14100 URL: https://issues.apache.org/jira/browse/HBASE-14100 Project: HBase Issue Type: Bug Reporter: Duo Zhang Assignee: Duo Zhang See here: https://builds.apache.org/job/HBase-TRUNK/6654/findbugsResult/HIGH/ We have 6 high priority findbugs warnings. A high priority findbugs warning is usually a bug. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14090) Redo FS layout; let go of tables/regions/stores directory hierarchy in DFS
[
https://issues.apache.org/jira/browse/HBASE-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629662#comment-14629662
]
Matteo Bertozzi commented on HBASE-14090:
-
{quote}We could go as far as having HBase manage a block pool directly{quote}
[~lhofhansl] I'm with you on this, as I was mentioning on HBASE-7806. if we go
with block we can gain even more stuff like smarter compactions (instead of
rewriting everything, replace just the blocks that require modification).
deduplication when we do stuff like CopyTable or we may just have two tables
with some data in common, placements and so on.
but switching to block will probably be too much work, we were trying to think
how to split this stuff in intermediate steps and just changing the layout and
add files refs in meta seems to be an unsplittable giant step. but at least the
proposed stuff was designed with blocks in mind, so we can go there at some
point (unless there is a big push to do it now).
Redo FS layout; let go of tables/regions/stores directory hierarchy in DFS
--
Key: HBASE-14090
URL: https://issues.apache.org/jira/browse/HBASE-14090
Project: HBase
Issue Type: Sub-task
Reporter: stack
Our layout as is won't work if 1M regions; e.g. HDFS will fall over if
directories of hundreds of thousands of files. HBASE-13991 (Humongous Tables)
would address this specific directory problem only by adding subdirs under
table dir but there are other issues with our current layout:
* Our table/regions/column family 'facade' has to be maintained in two
locations -- in master memory and in the hdfs directory layout -- and the
farce needs to be kept synced or worse, the model management is split between
master memory and DFS layout. 'Syncing' in HDFS has us dropping constructs
such as 'Reference' and 'HalfHFiles' on split, 'HFileLinks' when archiving,
and so on. This 'tie' makes it hard to make changes.
* While HDFS has atomic rename, useful for fencing and for having files
added atomically, if the model were solely owned by hbase, there are hbase
primitives we could make use of -- changes in a row are atomic and
coprocessors -- to simplify table transactions and provide more consistent
views of our model to clients; file 'moves' could be a memory operation only
rather than an HDFS call; sharing files between tables/snapshots and when it
is safe to remove them would be simplified if one owner only; and so on.
This is an umbrella blue-sky issue to discuss what a new layout would look
like and how we might get there. I'll follow up with some sketches of what
new layout could look like that come of some chats a few of us have been
having. We are also under the 'delusion' that move to a new layout could be
done as part of a rolling upgrade and that the amount of work involved is not
gargantuan.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14100) Fix high priority findbugs warnings
[
https://issues.apache.org/jira/browse/HBASE-14100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629695#comment-14629695
]
Duo Zhang commented on HBASE-14100:
---
{code:title=WALSplitter.java}
1208 for (WriterThread t : writerThreads) {
1209 t.finish();
1210 }
1211 if (interrupt) {
1212 for (WriterThread t : writerThreads) {
1213 t.interrupt(); // interrupt the writer threads. We are stopping
now.
1214 }
1215 }
1216
1217 for (WriterThread t : writerThreads) {
1218 if (!progress_failed reporter != null !reporter.progress()) {
1219 progress_failed = true;
1220 }
1221 try {
1222 t.join();
1223 } catch (InterruptedException ie) {
1224 IOException iie = new InterruptedIOException();
1225 iie.initCause(ie);
1226 throw iie;
1227 }
1228 }
1229 controller.checkForErrors();
1230 LOG.info((this.writerThreads == null? 0: this.writerThreads.size())
+ // here we do a null check, but the above code use writeThreads directly
without a null check
1231 split writers finished; closing...);
{code}
And here is the defination of writerThreads
{code:title=WALSplitter.java}
1131 protected final ListWriterThread writerThreads =
Lists.newArrayList();
{code}
It is initialized at construction and marked as final, so I think we could just
remove the null check?
Fix high priority findbugs warnings
---
Key: HBASE-14100
URL: https://issues.apache.org/jira/browse/HBASE-14100
Project: HBase
Issue Type: Bug
Reporter: Duo Zhang
Assignee: Duo Zhang
See here:
https://builds.apache.org/job/HBase-TRUNK/6654/findbugsResult/HIGH/
We have 6 high priority findbugs warnings. A high priority findbugs warning
is usually a bug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12295) Prevent block eviction under us if reads are in progress from the BBs
[ https://issues.apache.org/jira/browse/HBASE-12295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-12295: --- Attachment: HBASE-12295_15.patch Updated patch addressing the comments in RB. Prevent block eviction under us if reads are in progress from the BBs - Key: HBASE-12295 URL: https://issues.apache.org/jira/browse/HBASE-12295 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 Attachments: HBASE-12295.pdf, HBASE-12295_1.patch, HBASE-12295_1.pdf, HBASE-12295_10.patch, HBASE-12295_12.patch, HBASE-12295_14.patch, HBASE-12295_15.patch, HBASE-12295_2.patch, HBASE-12295_4.patch, HBASE-12295_4.pdf, HBASE-12295_5.pdf, HBASE-12295_9.patch, HBASE-12295_trunk.patch While we try to serve the reads from the BBs directly from the block cache, we need to ensure that the blocks does not get evicted under us while reading. This JIRA is to discuss and implement a strategy for the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12295) Prevent block eviction under us if reads are in progress from the BBs
[ https://issues.apache.org/jira/browse/HBASE-12295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-12295: --- Status: Open (was: Patch Available) Prevent block eviction under us if reads are in progress from the BBs - Key: HBASE-12295 URL: https://issues.apache.org/jira/browse/HBASE-12295 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 Attachments: HBASE-12295.pdf, HBASE-12295_1.patch, HBASE-12295_1.pdf, HBASE-12295_10.patch, HBASE-12295_12.patch, HBASE-12295_14.patch, HBASE-12295_15.patch, HBASE-12295_2.patch, HBASE-12295_4.patch, HBASE-12295_4.pdf, HBASE-12295_5.pdf, HBASE-12295_9.patch, HBASE-12295_trunk.patch While we try to serve the reads from the BBs directly from the block cache, we need to ensure that the blocks does not get evicted under us while reading. This JIRA is to discuss and implement a strategy for the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13954) Remove HTableInterface#getRowOrBefore related server side code
[ https://issues.apache.org/jira/browse/HBASE-13954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-13954: -- Attachment: HBASE-13954.patch Remove HTableInterface#getRowOrBefore related server side code -- Key: HBASE-13954 URL: https://issues.apache.org/jira/browse/HBASE-13954 Project: HBase Issue Type: Sub-task Components: API Reporter: Ashish Singhi Assignee: Ashish Singhi Fix For: 2.0.0 Attachments: HBASE-13954.patch As part of HBASE-13214 review, [~anoop.hbase] had a review comment on the review board to remove all the server side related code for getRowOrBefore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14085) Correct LICENSE and NOTICE files in artifacts
[
https://issues.apache.org/jira/browse/HBASE-14085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629648#comment-14629648
]
Sean Busbey commented on HBASE-14085:
-
Current status: tracking down code copied from protobuf, building notice for
binary distribution artifact.
Correct LICENSE and NOTICE files in artifacts
-
Key: HBASE-14085
URL: https://issues.apache.org/jira/browse/HBASE-14085
Project: HBase
Issue Type: Task
Components: build
Affects Versions: 2.0.0, 0.94.28, 0.98.14, 1.0.2, 1.2.0, 1.1.2, 1.3.0
Reporter: Sean Busbey
Assignee: Sean Busbey
Priority: Blocker
+Problems:
* checked LICENSE/NOTICE on binary
** binary artifact LICENSE file has not been updated to include the
additional license terms for contained third party dependencies
** binary artifact NOTICE file does not include a copyright line
** binary artifact NOTICE file does not appear to propagate appropriate info
from the NOTICE files from bundled dependencies
* checked NOTICE on source
** source artifact NOTICE file does not include a copyright line
** source NOTICE file includes notices for third party dependencies not
included in the artifact
* checked NOTICE files shipped in maven jars
** copyright line only says 2015 when it's very likely the contents are under
copyright prior to this year
* nit: NOTICE file on jars in maven say HBase - ${module} rather than
Apache HBase - ${module} as required
refs:
http://www.apache.org/dev/licensing-howto.html#bundled-vs-non-bundled
http://www.apache.org/dev/licensing-howto.html#binary
http://www.apache.org/dev/licensing-howto.html#simple
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12295) Prevent block eviction under us if reads are in progress from the BBs
[ https://issues.apache.org/jira/browse/HBASE-12295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-12295: --- Status: Patch Available (was: Open) The 4 varieties of SizeCachedKeyValue and NoSizeCachedKeyValue has been removed and now we have only one SizecacheKeyvalue. The SharedMemory interface impl Cell is the other one that is added by this patch. Prevent block eviction under us if reads are in progress from the BBs - Key: HBASE-12295 URL: https://issues.apache.org/jira/browse/HBASE-12295 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 Attachments: HBASE-12295.pdf, HBASE-12295_1.patch, HBASE-12295_1.pdf, HBASE-12295_10.patch, HBASE-12295_12.patch, HBASE-12295_14.patch, HBASE-12295_15.patch, HBASE-12295_2.patch, HBASE-12295_4.patch, HBASE-12295_4.pdf, HBASE-12295_5.pdf, HBASE-12295_9.patch, HBASE-12295_trunk.patch While we try to serve the reads from the BBs directly from the block cache, we need to ensure that the blocks does not get evicted under us while reading. This JIRA is to discuss and implement a strategy for the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14100) Fix high priority findbugs warnings
[
https://issues.apache.org/jira/browse/HBASE-14100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629672#comment-14629672
]
Duo Zhang commented on HBASE-14100:
---
{{DMI_INVOKING_TOSTRING_ON_ARRAY}}
{code:title=SplitLogManager.java}
300 FileStatus[] files = fs.listStatus(logDir);
301 if (files != null files.length 0) {
302 LOG.warn(Returning success without actually splitting and
303 + deleting all the log files in path + logDir + : +
files, ioe); // files is an array, should we use Arrays.toString here? Is it
too noisy?
304 } else {
305 LOG.warn(Unable to delete log src dir. Ignoring. + logDir,
ioe);
306 }
{code}
{code:title=SimpleRegionNormalizer.java}
152 LOG.debug(Table + table + , largest region
153 + largestRegion.getFirst().getRegionName() + has size // Should
be getRegionNameAsString()?
154 + largestRegion.getSecond() + , more than 2 times than avg size,
splitting);
{code}
Fix high priority findbugs warnings
---
Key: HBASE-14100
URL: https://issues.apache.org/jira/browse/HBASE-14100
Project: HBase
Issue Type: Bug
Reporter: Duo Zhang
Assignee: Duo Zhang
See here:
https://builds.apache.org/job/HBase-TRUNK/6654/findbugsResult/HIGH/
We have 6 high priority findbugs warnings. A high priority findbugs warning
is usually a bug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14100) Fix high priority findbugs warnings
[ https://issues.apache.org/jira/browse/HBASE-14100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629707#comment-14629707 ] ramkrishna.s.vasudevan commented on HBASE-14100: The BloomFilterUtil methods that you have mentioned above will be removed as part of HBASE-12213. Is that fine with you [~Apache9]? Fix high priority findbugs warnings --- Key: HBASE-14100 URL: https://issues.apache.org/jira/browse/HBASE-14100 Project: HBase Issue Type: Bug Reporter: Duo Zhang Assignee: Duo Zhang See here: https://builds.apache.org/job/HBase-TRUNK/6654/findbugsResult/HIGH/ We have 6 high priority findbugs warnings. A high priority findbugs warning is usually a bug. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14100) Fix high priority findbugs warnings
[ https://issues.apache.org/jira/browse/HBASE-14100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-14100: -- Status: Patch Available (was: Open) Fix high priority findbugs warnings --- Key: HBASE-14100 URL: https://issues.apache.org/jira/browse/HBASE-14100 Project: HBase Issue Type: Bug Reporter: Duo Zhang Assignee: Duo Zhang Attachments: HBASE-14100.patch See here: https://builds.apache.org/job/HBase-TRUNK/6654/findbugsResult/HIGH/ We have 6 high priority findbugs warnings. A high priority findbugs warning is usually a bug. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14100) Fix high priority findbugs warnings
[ https://issues.apache.org/jira/browse/HBASE-14100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-14100: -- Attachment: HBASE-14100.patch Fix high priority findbugs warnings --- Key: HBASE-14100 URL: https://issues.apache.org/jira/browse/HBASE-14100 Project: HBase Issue Type: Bug Reporter: Duo Zhang Assignee: Duo Zhang Attachments: HBASE-14100.patch See here: https://builds.apache.org/job/HBase-TRUNK/6654/findbugsResult/HIGH/ We have 6 high priority findbugs warnings. A high priority findbugs warning is usually a bug. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13954) Remove HTableInterface#getRowOrBefore related server side code
[ https://issues.apache.org/jira/browse/HBASE-13954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-13954: -- Status: Patch Available (was: Open) Remove HTableInterface#getRowOrBefore related server side code -- Key: HBASE-13954 URL: https://issues.apache.org/jira/browse/HBASE-13954 Project: HBase Issue Type: Sub-task Components: API Reporter: Ashish Singhi Assignee: Ashish Singhi Fix For: 2.0.0 Attachments: HBASE-13954.patch As part of HBASE-13214 review, [~anoop.hbase] had a review comment on the review board to remove all the server side related code for getRowOrBefore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14090) Redo FS layout; let go of tables/regions/stores directory hierarchy in DFS
[ https://issues.apache.org/jira/browse/HBASE-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629806#comment-14629806 ] Dave Latham commented on HBASE-14090: - Going the block pool route sounds like we'd have to redo or find a way to share a lot of what HDFS does. Would be easier to manage locality though! Redo FS layout; let go of tables/regions/stores directory hierarchy in DFS -- Key: HBASE-14090 URL: https://issues.apache.org/jira/browse/HBASE-14090 Project: HBase Issue Type: Sub-task Reporter: stack Our layout as is won't work if 1M regions; e.g. HDFS will fall over if directories of hundreds of thousands of files. HBASE-13991 (Humongous Tables) would address this specific directory problem only by adding subdirs under table dir but there are other issues with our current layout: * Our table/regions/column family 'facade' has to be maintained in two locations -- in master memory and in the hdfs directory layout -- and the farce needs to be kept synced or worse, the model management is split between master memory and DFS layout. 'Syncing' in HDFS has us dropping constructs such as 'Reference' and 'HalfHFiles' on split, 'HFileLinks' when archiving, and so on. This 'tie' makes it hard to make changes. * While HDFS has atomic rename, useful for fencing and for having files added atomically, if the model were solely owned by hbase, there are hbase primitives we could make use of -- changes in a row are atomic and coprocessors -- to simplify table transactions and provide more consistent views of our model to clients; file 'moves' could be a memory operation only rather than an HDFS call; sharing files between tables/snapshots and when it is safe to remove them would be simplified if one owner only; and so on. This is an umbrella blue-sky issue to discuss what a new layout would look like and how we might get there. I'll follow up with some sketches of what new layout could look like that come of some chats a few of us have been having. We are also under the 'delusion' that move to a new layout could be done as part of a rolling upgrade and that the amount of work involved is not gargantuan. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12596) bulkload needs to follow locality
[ https://issues.apache.org/jira/browse/HBASE-12596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629880#comment-14629880 ] Andrew Purtell commented on HBASE-12596: bq. So if it is in 0.98, we probably want this in 1.0, 1.1, and especially 1.2 as well. You can ask Enis, Nick, and Sean. I've done this before and have found Enis does not accept changes for 1.0 that do not fit into the bug fix bucket. I'd presume that the case here but I'm not speaking for him. As 0.98 RM I can accept changes that add features on our minor releases (0.98.x) but not on our patch releases (0.98.x.y) as long as they do not break wire compatibility. that's the old policy set in discussions ahead of the 0.98.0 release. Consensus and user needs evolve though. Java binary compat became a pain point for downstreamers and Phoenix so now I also check that as part of the RC process. Consensus can evolve again, to a new policy that disallows anything but but fixes to 0.98. If so we should as a courtesy have a discussion with 0.98 users on the implications. Already the code divergence between 0.98 and branch-1 is significant enough to make all but trivial changes an effort to backport. If we change the 0.98 accept criteria to follow what Enis is doing with 1.0, for example, this shuts down all enhancements and that divergence will get a lot bigger. At that point I would advocate that 0.98 is EOLed. I'd retire as its RM. At some point those two actions should happen anyway. bulkload needs to follow locality - Key: HBASE-12596 URL: https://issues.apache.org/jira/browse/HBASE-12596 Project: HBase Issue Type: Improvement Components: HFile, regionserver Affects Versions: 0.98.8 Environment: hadoop-2.3.0, hbase-0.98.8, jdk1.7 Reporter: Victor Xu Assignee: Victor Xu Fix For: 2.0.0, 0.98.14, 1.3.0 Attachments: HBASE-12596-0.98-v1.patch, HBASE-12596-0.98-v2.patch, HBASE-12596-0.98-v3.patch, HBASE-12596-0.98-v4.patch, HBASE-12596-0.98-v5.patch, HBASE-12596-0.98-v6.patch, HBASE-12596-branch-1-v1.patch, HBASE-12596-branch-1-v2.patch, HBASE-12596-master-v1.patch, HBASE-12596-master-v2.patch, HBASE-12596-master-v3.patch, HBASE-12596-master-v4.patch, HBASE-12596-master-v5.patch, HBASE-12596-master-v6.patch, HBASE-12596.patch Normally, we have 2 steps to perform a bulkload: 1. use a job to write HFiles to be loaded; 2. Move these HFiles to the right hdfs directory. However, the locality could be loss during the first step. Why not just write the HFiles directly into the right place? We can do this easily because StoreFile.WriterBuilder has the withFavoredNodes method, and we just need to call it in HFileOutputFormat's getNewWriter(). This feature is enabled by default, and we could use 'hbase.bulkload.locality.sensitive.enabled=false' to disable it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (HBASE-12596) bulkload needs to follow locality
[ https://issues.apache.org/jira/browse/HBASE-12596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-12596: --- Comment: was deleted (was: bq. So if it is in 0.98, we probably want this in 1.0, 1.1, and especially 1.2 as well. You can ask Enis, Nick, and Sean. I've done this before and have found Enis does not accept changes for 1.0 that do not fit into the bug fix bucket. I'd presume that the case here but I'm not speaking for him. As 0.98 RM I can accept changes that add features on our minor releases (0.98.x) but not on our patch releases (0.98.x.y) as long as they do not break wire compatibility. that's the old policy set in discussions ahead of the 0.98.0 release. Consensus and user needs evolve though. Java binary compat became a pain point for downstreamers and Phoenix so now I also check that as part of the RC process. Consensus can evolve again, to a new policy that disallows anything but but fixes to 0.98. If so we should as a courtesy have a discussion with 0.98 users on the implications. Already the code divergence between 0.98 and branch-1 is significant enough to make all but trivial changes an effort to backport. If we change the 0.98 accept criteria to follow what Enis is doing with 1.0, for example, this shuts down all enhancements and that divergence will get a lot bigger. At that point I would advocate that 0.98 is EOLed. I'd retire as its RM. At some point those two actions should happen anyway. ) bulkload needs to follow locality - Key: HBASE-12596 URL: https://issues.apache.org/jira/browse/HBASE-12596 Project: HBase Issue Type: Improvement Components: HFile, regionserver Affects Versions: 0.98.8 Environment: hadoop-2.3.0, hbase-0.98.8, jdk1.7 Reporter: Victor Xu Assignee: Victor Xu Fix For: 2.0.0, 0.98.14, 1.3.0 Attachments: HBASE-12596-0.98-v1.patch, HBASE-12596-0.98-v2.patch, HBASE-12596-0.98-v3.patch, HBASE-12596-0.98-v4.patch, HBASE-12596-0.98-v5.patch, HBASE-12596-0.98-v6.patch, HBASE-12596-branch-1-v1.patch, HBASE-12596-branch-1-v2.patch, HBASE-12596-master-v1.patch, HBASE-12596-master-v2.patch, HBASE-12596-master-v3.patch, HBASE-12596-master-v4.patch, HBASE-12596-master-v5.patch, HBASE-12596-master-v6.patch, HBASE-12596.patch Normally, we have 2 steps to perform a bulkload: 1. use a job to write HFiles to be loaded; 2. Move these HFiles to the right hdfs directory. However, the locality could be loss during the first step. Why not just write the HFiles directly into the right place? We can do this easily because StoreFile.WriterBuilder has the withFavoredNodes method, and we just need to call it in HFileOutputFormat's getNewWriter(). This feature is enabled by default, and we could use 'hbase.bulkload.locality.sensitive.enabled=false' to disable it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-12596) bulkload needs to follow locality
[ https://issues.apache.org/jira/browse/HBASE-12596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629861#comment-14629861 ] Andrew Purtell edited comment on HBASE-12596 at 7/16/15 3:41 PM: - That's changed since the new rules went into effect for 1.0 and up. Here and elsewhere we are able to accept new features in 1.x minor revs (branch-1), 0.98 because of its continuation of old style numbering/policy, and master. There will be a rolling upgrade capable upgrade path from 0.98 to a branch-1 release with a contiguous feature set. This is the consequence of adopting semver for 1.0 and up but not earlier. As 0.98 RM I can accept changes that add features on our minor releases (0.98.x) but not on our patch releases (0.98.x.y) as long as they do not break wire compatibility. On branch-1 this policy is similar but now the minors are 1.x and the patch versions are 1.x.y. We have contributors and users wanting enhancements in 0.98 that are allowed there. I'm not going to retire 0.98 as RM until we have consensus it should be EOL. bq. So if it is in 0.98, we probably want this in 1.0, 1.1, and especially 1.2 as well. You can ask [~enis], [~ndimiduk], and [~busbey]. I've done this before and have found Enis does not accept changes for 1.0 that do not fit into the bug fix bucket. This makes sense as 1.0 operates under the new semver regime. I'd presume that the case here but I'm not speaking for him. The old policy for 0.98 (wire compat is the only dealbreaker) came about as a result of discussions leading up to the 0.98.0 release. Consensus and user needs evolve though. Java binary compat became a pain point for downstreamers and Phoenix so now I also check that as part of the RC process. Consensus can evolve again, to a new policy that disallows anything but bug fixes to 0.98. If so we should as a courtesy have a discussion with 0.98 users on the implications. Already the code divergence between 0.98 and branch-1 is significant enough to make all but trivial changes an effort to backport. If we change the 0.98 accept criteria to only allow what can be committed to all 1.x branches, this shuts down all enhancements to 0.98 because of change policies applied to the 1.x branches by their RMs, and that divergence will get a lot bigger. At that point I would advocate that 0.98 is EOLed and retire as its RM since it would no longer need dedicated attention. At some point those two actions should happen anyway. Edit: Consolidated two posts and at-mentions. was (Author: apurtell): That's changed since the new rules went into effect for 1.0 and up. Here and elsewhere we are able to accept new features in 1.x minor revs (branch-1), 0.98 because of its continuation of old style numbering/policy, and master. There will be a rolling upgrade capable upgrade path from 0.98 to a branch-1 release with a contiguous feature set. This is the consequence of adopting semver for 1.0 and up but not earlier. We have contributors and users wanting enhancements in 0.98 that are allowed there. I'm not going to retire 0.98 as RM until we have consensus it should be EOL. bulkload needs to follow locality - Key: HBASE-12596 URL: https://issues.apache.org/jira/browse/HBASE-12596 Project: HBase Issue Type: Improvement Components: HFile, regionserver Affects Versions: 0.98.8 Environment: hadoop-2.3.0, hbase-0.98.8, jdk1.7 Reporter: Victor Xu Assignee: Victor Xu Fix For: 2.0.0, 0.98.14, 1.3.0 Attachments: HBASE-12596-0.98-v1.patch, HBASE-12596-0.98-v2.patch, HBASE-12596-0.98-v3.patch, HBASE-12596-0.98-v4.patch, HBASE-12596-0.98-v5.patch, HBASE-12596-0.98-v6.patch, HBASE-12596-branch-1-v1.patch, HBASE-12596-branch-1-v2.patch, HBASE-12596-master-v1.patch, HBASE-12596-master-v2.patch, HBASE-12596-master-v3.patch, HBASE-12596-master-v4.patch, HBASE-12596-master-v5.patch, HBASE-12596-master-v6.patch, HBASE-12596.patch Normally, we have 2 steps to perform a bulkload: 1. use a job to write HFiles to be loaded; 2. Move these HFiles to the right hdfs directory. However, the locality could be loss during the first step. Why not just write the HFiles directly into the right place? We can do this easily because StoreFile.WriterBuilder has the withFavoredNodes method, and we just need to call it in HFileOutputFormat's getNewWriter(). This feature is enabled by default, and we could use 'hbase.bulkload.locality.sensitive.enabled=false' to disable it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-12596) bulkload needs to follow locality
[ https://issues.apache.org/jira/browse/HBASE-12596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629861#comment-14629861 ] Andrew Purtell edited comment on HBASE-12596 at 7/16/15 3:48 PM: - That's changed since the new rules went into effect for 1.0 and up. Here and elsewhere we are able to accept new features in 1.x minor revs (branch-1), 0.98 because of its continuation of old style numbering/policy, and master. There will be a rolling upgrade capable upgrade path from 0.98 to a branch-1 release with a contiguous feature set. This is the consequence of adopting semver for 1.0 and up but not earlier. As 0.98 RM I can accept changes that add features on our minor releases (0.98.x) but not on our patch releases (0.98.x.y) as long as they do not break wire compatibility. On branch-1 this policy is similar but now the minors are 1.x and the patch versions are 1.x.y. We have contributors and users wanting enhancements in 0.98 that are allowed there. I'm not going to retire 0.98 as RM until we have consensus it should be EOL. bq. So if it is in 0.98, we probably want this in 1.0, 1.1, and especially 1.2 as well. You can ask [~enis], [~ndimiduk], and [~busbey]. I've done this before and have found Enis does not accept changes for 1.0 that do not fit into the bug fix bucket. This makes sense as 1.0 operates under the new semver regime. I'd presume that the case here but I'm not speaking for him. (Not to single Enis out here, I also presume Nick and Sean will make similar choices.) The old policy for 0.98 (wire compat is the only dealbreaker) came about as a result of discussions leading up to the 0.98.0 release. Consensus and user needs evolve though. Java binary compat became a pain point for downstreamers and Phoenix so now I also check that as part of the RC process. Consensus can evolve again, to a new policy that disallows anything but bug fixes to 0.98. If so we should as a courtesy have a discussion with 0.98 users on the implications. Already the code divergence between 0.98 and branch-1 is significant enough to make all but trivial changes an effort to backport. If we change the 0.98 accept criteria to only allow what can be committed to all 1.x branches, this shuts down all enhancements to 0.98 because of change policies applied to the 1.x branches by their RMs, and that divergence will get a lot bigger. At that point I would advocate that 0.98 is EOLed and retire as its RM since it would no longer need dedicated attention. At some point those two actions should happen anyway. Edit: Consolidated two posts and at-mentions. was (Author: apurtell): That's changed since the new rules went into effect for 1.0 and up. Here and elsewhere we are able to accept new features in 1.x minor revs (branch-1), 0.98 because of its continuation of old style numbering/policy, and master. There will be a rolling upgrade capable upgrade path from 0.98 to a branch-1 release with a contiguous feature set. This is the consequence of adopting semver for 1.0 and up but not earlier. As 0.98 RM I can accept changes that add features on our minor releases (0.98.x) but not on our patch releases (0.98.x.y) as long as they do not break wire compatibility. On branch-1 this policy is similar but now the minors are 1.x and the patch versions are 1.x.y. We have contributors and users wanting enhancements in 0.98 that are allowed there. I'm not going to retire 0.98 as RM until we have consensus it should be EOL. bq. So if it is in 0.98, we probably want this in 1.0, 1.1, and especially 1.2 as well. You can ask [~enis], [~ndimiduk], and [~busbey]. I've done this before and have found Enis does not accept changes for 1.0 that do not fit into the bug fix bucket. This makes sense as 1.0 operates under the new semver regime. I'd presume that the case here but I'm not speaking for him. The old policy for 0.98 (wire compat is the only dealbreaker) came about as a result of discussions leading up to the 0.98.0 release. Consensus and user needs evolve though. Java binary compat became a pain point for downstreamers and Phoenix so now I also check that as part of the RC process. Consensus can evolve again, to a new policy that disallows anything but bug fixes to 0.98. If so we should as a courtesy have a discussion with 0.98 users on the implications. Already the code divergence between 0.98 and branch-1 is significant enough to make all but trivial changes an effort to backport. If we change the 0.98 accept criteria to only allow what can be committed to all 1.x branches, this shuts down all enhancements to 0.98 because of change policies applied to the 1.x branches by their RMs, and that divergence will get a lot bigger. At that point I would advocate that 0.98 is EOLed and retire as its RM since it would no longer need dedicated attention. At some point those
[jira] [Created] (HBASE-14102) Add thank you to our thanks page for vectorportal.com
stack created HBASE-14102: - Summary: Add thank you to our thanks page for vectorportal.com Key: HBASE-14102 URL: https://issues.apache.org/jira/browse/HBASE-14102 Project: HBase Issue Type: Sub-task Reporter: stack -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12596) bulkload needs to follow locality
[ https://issues.apache.org/jira/browse/HBASE-12596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629861#comment-14629861 ] Andrew Purtell commented on HBASE-12596: That's changed since the new rules went into effect for 1.0 and up. Here and elsewhere we are able to accept new features in 1.x minor revs (branch-1), 0.98 because of its continuation of old style numbering/policy, and master. There will be a rolling upgrade capable upgrade path from 0.98 to a branch-1 release with a contiguous feature set. This is the consequence of adopting semver for 1.0 and up but not earlier. We have contributors and users wanting enhancements in 0.98 that are allowed there. I'm not going to retire 0.98 as RM until we have consensus it should be EOL. bulkload needs to follow locality - Key: HBASE-12596 URL: https://issues.apache.org/jira/browse/HBASE-12596 Project: HBase Issue Type: Improvement Components: HFile, regionserver Affects Versions: 0.98.8 Environment: hadoop-2.3.0, hbase-0.98.8, jdk1.7 Reporter: Victor Xu Assignee: Victor Xu Fix For: 2.0.0, 0.98.14, 1.3.0 Attachments: HBASE-12596-0.98-v1.patch, HBASE-12596-0.98-v2.patch, HBASE-12596-0.98-v3.patch, HBASE-12596-0.98-v4.patch, HBASE-12596-0.98-v5.patch, HBASE-12596-0.98-v6.patch, HBASE-12596-branch-1-v1.patch, HBASE-12596-branch-1-v2.patch, HBASE-12596-master-v1.patch, HBASE-12596-master-v2.patch, HBASE-12596-master-v3.patch, HBASE-12596-master-v4.patch, HBASE-12596-master-v5.patch, HBASE-12596-master-v6.patch, HBASE-12596.patch Normally, we have 2 steps to perform a bulkload: 1. use a job to write HFiles to be loaded; 2. Move these HFiles to the right hdfs directory. However, the locality could be loss during the first step. Why not just write the HFiles directly into the right place? We can do this easily because StoreFile.WriterBuilder has the withFavoredNodes method, and we just need to call it in HFileOutputFormat's getNewWriter(). This feature is enabled by default, and we could use 'hbase.bulkload.locality.sensitive.enabled=false' to disable it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14103) Rebase our image atop the original source rather than a derivation
stack created HBASE-14103: - Summary: Rebase our image atop the original source rather than a derivation Key: HBASE-14103 URL: https://issues.apache.org/jira/browse/HBASE-14103 Project: HBase Issue Type: Sub-task Reporter: stack -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14104) Add vectorportal.com to NOTICES.txt as src of our logo
stack created HBASE-14104: - Summary: Add vectorportal.com to NOTICES.txt as src of our logo Key: HBASE-14104 URL: https://issues.apache.org/jira/browse/HBASE-14104 Project: HBase Issue Type: Sub-task Reporter: stack -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14104) Add vectorportal.com to NOTICES.txt as src of our logo
[ https://issues.apache.org/jira/browse/HBASE-14104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629893#comment-14629893 ] Sean Busbey commented on HBASE-14104: - +1 Add vectorportal.com to NOTICES.txt as src of our logo -- Key: HBASE-14104 URL: https://issues.apache.org/jira/browse/HBASE-14104 Project: HBase Issue Type: Sub-task Reporter: stack Attachments: 14104.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14101) Fix our logo
stack created HBASE-14101: - Summary: Fix our logo Key: HBASE-14101 URL: https://issues.apache.org/jira/browse/HBASE-14101 Project: HBase Issue Type: Bug Reporter: stack Our Sean found that our logo is a derivative work based on what is itself a derivation only the route to the original is a bit dodgy license-wise (this latter was his finding). See his note here: http://osdir.com/ml/general/2015-07/msg20779.html We wrote the owners of the original at vectorportal.com. They have been most accommodating and changed the license to be CC-BY 3.0. If you download from here, http://www.vectorportal.com/iFile/9136/KILLER-WHALE-FREE-VECTOR.eps/inc_downloading.asp, you will see the bundled license. Let me attach the zip file here. This issue is about rebasing our image so it is derivative of the original rather than of the derivative, http://www.vectorfree.com/jumping-orca , updating NOTICES.txt, and adding a thank you to the vectorportal.com to our thanks page. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14101) Fix our logo
[ https://issues.apache.org/jira/browse/HBASE-14101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14101: -- Attachment: orca-vector-image(1).zip If you click the DOWNLOAD button on this page [1], this is what you get. It is image and license. 1. http://www.vectorportal.com/subcategory/205/KILLER-WHALE-FREE-VECTOR.eps/ifile/9136/detailtest.asp Fix our logo Key: HBASE-14101 URL: https://issues.apache.org/jira/browse/HBASE-14101 Project: HBase Issue Type: Bug Reporter: stack Attachments: orca-vector-image(1).zip Our Sean found that our logo is a derivative work based on what is itself a derivation only the route to the original is a bit dodgy license-wise (this latter was his finding). See his note here: http://osdir.com/ml/general/2015-07/msg20779.html We wrote the owners of the original at vectorportal.com. They have been most accommodating and changed the license to be CC-BY 3.0. If you download from here, http://www.vectorportal.com/iFile/9136/KILLER-WHALE-FREE-VECTOR.eps/inc_downloading.asp, you will see the bundled license. Let me attach the zip file here. This issue is about rebasing our image so it is derivative of the original rather than of the derivative, http://www.vectorfree.com/jumping-orca , updating NOTICES.txt, and adding a thank you to the vectorportal.com to our thanks page. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14102) Add thank you to our thanks page for vectorportal.com
[ https://issues.apache.org/jira/browse/HBASE-14102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-14102. --- Resolution: Fixed Assignee: stack Fix Version/s: 2.0.0 Pushed one-liner that shows on site only to master branch. Add thank you to our thanks page for vectorportal.com - Key: HBASE-14102 URL: https://issues.apache.org/jira/browse/HBASE-14102 Project: HBase Issue Type: Sub-task Reporter: stack Assignee: stack Fix For: 2.0.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14104) Add vectorportal.com to NOTICES.txt as src of our logo
[
https://issues.apache.org/jira/browse/HBASE-14104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-14104:
--
Attachment: 14104.txt
Suggested temporary NOTICES txt until we redo the logo when we'll remove
pointer to dodgy derivative.
{code}diff --git a/NOTICE.txt b/NOTICE.txt
index 5e159f6..b7bc875 100644
--- a/NOTICE.txt
+++ b/NOTICE.txt
@@ -24,3 +24,9 @@ It is licensed Creative Commons Attribution 3.0.
See https://creativecommons.org/licenses/by/3.0/us/
We changed the logo by stripping the colored background, inverting
it and then rotating it some.
+
+Later we found that vectorfree.com image is not properly licensed.
+The original is owned by vectorportal.com. The original was
+relicensed so we could use it as Creative Commons Attribution 3.0.
+The license is bundled with the download available here:
+http://www.vectorportal.com/subcategory/205/KILLER-WHALE-FREE-VECTOR.eps/ifile/9136/detailtest.asp{code}
Add vectorportal.com to NOTICES.txt as src of our logo
--
Key: HBASE-14104
URL: https://issues.apache.org/jira/browse/HBASE-14104
Project: HBase
Issue Type: Sub-task
Reporter: stack
Attachments: 14104.txt
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-12596) bulkload needs to follow locality
[ https://issues.apache.org/jira/browse/HBASE-12596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629994#comment-14629994 ] Andrew Purtell edited comment on HBASE-12596 at 7/16/15 4:56 PM: - bq. It does show a couple differences in the processes though. By that definition, before 1.0 there were RMs for major versions (after 1.0 there are RMs for each minor version) This is a great observation. It seems to me that we evolved the branch RM informal role because each release branch represented a major version bump. The branch RM role is easier post 1.0. (Except: 0.94 and 0.98, grandfathered stuff and code divergence issues. Except: 2.0, we could use a shepherd for challenges with the major version bump.) was (Author: apurtell): bq. It does show a couple differences in the processes though. By that definition, before 1.0 there were RMs for major versions (after 1.0 there are RMs for each minor version) This is a great observation. It seems to me that we evolved the branch RM informal role because each release branch represented a major version bump. The branch RM role is easier post 1.0. (Except: 0.98, grandfathered stuff and code divergence issues. Except: 2.0, we could use a shepherd for challenges with the major version bump.) bulkload needs to follow locality - Key: HBASE-12596 URL: https://issues.apache.org/jira/browse/HBASE-12596 Project: HBase Issue Type: Improvement Components: HFile, regionserver Affects Versions: 0.98.8 Environment: hadoop-2.3.0, hbase-0.98.8, jdk1.7 Reporter: Victor Xu Assignee: Victor Xu Fix For: 2.0.0, 0.98.14, 1.3.0 Attachments: HBASE-12596-0.98-v1.patch, HBASE-12596-0.98-v2.patch, HBASE-12596-0.98-v3.patch, HBASE-12596-0.98-v4.patch, HBASE-12596-0.98-v5.patch, HBASE-12596-0.98-v6.patch, HBASE-12596-branch-1-v1.patch, HBASE-12596-branch-1-v2.patch, HBASE-12596-master-v1.patch, HBASE-12596-master-v2.patch, HBASE-12596-master-v3.patch, HBASE-12596-master-v4.patch, HBASE-12596-master-v5.patch, HBASE-12596-master-v6.patch, HBASE-12596.patch Normally, we have 2 steps to perform a bulkload: 1. use a job to write HFiles to be loaded; 2. Move these HFiles to the right hdfs directory. However, the locality could be loss during the first step. Why not just write the HFiles directly into the right place? We can do this easily because StoreFile.WriterBuilder has the withFavoredNodes method, and we just need to call it in HFileOutputFormat's getNewWriter(). This feature is enabled by default, and we could use 'hbase.bulkload.locality.sensitive.enabled=false' to disable it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14048) isPBMagicPrefix should always check for null data
[
https://issues.apache.org/jira/browse/HBASE-14048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-14048:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Pushed to 0.98
isPBMagicPrefix should always check for null data
-
Key: HBASE-14048
URL: https://issues.apache.org/jira/browse/HBASE-14048
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.13
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Fix For: 0.98.14
Attachments: HBASE-14048-0.98.patch
Example:
{noformat}
2015-07-09 04:20:30,649 ERROR [ver60020-EventThread] zookeeper.ClientCnxn -
Error while calling watcher
java.lang.NullPointerException
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.isPBMagicPrefix(ProtobufUtil.java:241)
at
org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.startNewSubprocedure(ZKProcedureMemberRpcs.java:203)
at
org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.waitForNewProcedures(ZKProcedureMemberRpcs.java:172)
at
org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.access$100(ZKProcedureMemberRpcs.java:55)
at
org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs$1.nodeChildrenChanged(ZKProcedureMemberRpcs.java:107)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:358)
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
{noformat}
This is observed with 0.98.
There may be a deeper cause but let's start by fixing the obvious problem.
Audit ProcedureV2 also on later branches.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14105) Add shell tests for Snapshot
Ashish Singhi created HBASE-14105: - Summary: Add shell tests for Snapshot Key: HBASE-14105 URL: https://issues.apache.org/jira/browse/HBASE-14105 Project: HBase Issue Type: Sub-task Reporter: Ashish Singhi Assignee: Ashish Singhi -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14050) NPE in org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess
[
https://issues.apache.org/jira/browse/HBASE-14050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-14050:
---
Assignee: Andrew Purtell
Fix Version/s: 1.0.3
1.1.2
1.2.0
0.98.14
2.0.0
Trivial patch
NPE in org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess
--
Key: HBASE-14050
URL: https://issues.apache.org/jira/browse/HBASE-14050
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.12.1
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Minor
Fix For: 2.0.0, 0.98.14, 1.2.0, 1.1.2, 1.0.3
Attachments: HBASE-14050.patch
{noformat}
2015-07-02 09:49:32,028 WARN [.reader=4,port=60020] ipc.RpcServer -
RpcServer.listener,port=60020: count of bytes read: 0
java.lang.NullPointerException
at
org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1479)
at
org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:854)
at
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:645)
at
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:620)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14050) NPE in org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess
[
https://issues.apache.org/jira/browse/HBASE-14050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-14050:
---
Status: Patch Available (was: Open)
NPE in org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess
--
Key: HBASE-14050
URL: https://issues.apache.org/jira/browse/HBASE-14050
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.12.1
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Minor
Fix For: 2.0.0, 0.98.14, 1.2.0, 1.1.2, 1.0.3
Attachments: HBASE-14050.patch
{noformat}
2015-07-02 09:49:32,028 WARN [.reader=4,port=60020] ipc.RpcServer -
RpcServer.listener,port=60020: count of bytes read: 0
java.lang.NullPointerException
at
org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1479)
at
org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:854)
at
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:645)
at
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:620)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14050) NPE in org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess
[
https://issues.apache.org/jira/browse/HBASE-14050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-14050:
---
Attachment: HBASE-14050.patch
NPE in org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess
--
Key: HBASE-14050
URL: https://issues.apache.org/jira/browse/HBASE-14050
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.12.1
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Minor
Fix For: 2.0.0, 0.98.14, 1.2.0, 1.1.2, 1.0.3
Attachments: HBASE-14050.patch
{noformat}
2015-07-02 09:49:32,028 WARN [.reader=4,port=60020] ipc.RpcServer -
RpcServer.listener,port=60020: count of bytes read: 0
java.lang.NullPointerException
at
org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1479)
at
org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:854)
at
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:645)
at
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:620)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14106) TestProcedureRecovery is flaky
[
https://issues.apache.org/jira/browse/HBASE-14106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-14106:
---
Summary: TestProcedureRecovery is flaky (was: TestProcedureRecovery is
flaky on master)
TestProcedureRecovery is flaky
--
Key: HBASE-14106
URL: https://issues.apache.org/jira/browse/HBASE-14106
Project: HBase
Issue Type: Bug
Components: proc-v2, test
Reporter: Andrew Purtell
Encountered this when running master tests locally using 7u79:
{noformat}
Tests run: 8, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 12.28 sec
FAILURE! - in org.apache.hadoop.hbase.procedure2.TestProcedureRecovery
testRunningProcWithSameNonce(org.apache.hadoop.hbase.procedure2.TestProcedureRecovery)
Time elapsed: 0.318 sec ERROR!
java.lang.IllegalArgumentException: null
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:76)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:595)
at
org.apache.hadoop.hbase.procedure2.ProcedureTestingUtility.submitAndWait(ProcedureTestingUtility.java:137)
at
org.apache.hadoop.hbase.procedure2.TestProcedureRecovery.testRunningProcWithSameNonce(TestProcedureRecovery.java:321)
{noformat}
{noformat}
Flaked tests:
org.apache.hadoop.hbase.procedure2.TestProcedureRecovery.testRunningProcWithSameNonce(org.apache.hadoop.hbase.procedure2.TestProcedureRecovery)
Run 1: TestProcedureRecovery.testRunningProcWithSameNonce:321 »
IllegalArgument
Run 2: PASS
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14106) TestProcedureRecovery is flaky on master
Andrew Purtell created HBASE-14106:
--
Summary: TestProcedureRecovery is flaky on master
Key: HBASE-14106
URL: https://issues.apache.org/jira/browse/HBASE-14106
Project: HBase
Issue Type: Bug
Components: proc-v2, test
Reporter: Andrew Purtell
Encountered this when running master tests locally using 7u79:
{noformat}
Tests run: 8, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 12.28 sec
FAILURE! - in org.apache.hadoop.hbase.procedure2.TestProcedureRecovery
testRunningProcWithSameNonce(org.apache.hadoop.hbase.procedure2.TestProcedureRecovery)
Time elapsed: 0.318 sec ERROR!
java.lang.IllegalArgumentException: null
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:76)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:595)
at
org.apache.hadoop.hbase.procedure2.ProcedureTestingUtility.submitAndWait(ProcedureTestingUtility.java:137)
at
org.apache.hadoop.hbase.procedure2.TestProcedureRecovery.testRunningProcWithSameNonce(TestProcedureRecovery.java:321)
{noformat}
{noformat}
Flaked tests:
org.apache.hadoop.hbase.procedure2.TestProcedureRecovery.testRunningProcWithSameNonce(org.apache.hadoop.hbase.procedure2.TestProcedureRecovery)
Run 1: TestProcedureRecovery.testRunningProcWithSameNonce:321 »
IllegalArgument
Run 2: PASS
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14107) Administrative Task: Provide an API to List all procedures
Stephen Yuan Jiang created HBASE-14107: -- Summary: Administrative Task: Provide an API to List all procedures Key: HBASE-14107 URL: https://issues.apache.org/jira/browse/HBASE-14107 Project: HBase Issue Type: Sub-task Components: proc-v2 Affects Versions: 2.0.0, 1.3.0 Reporter: Stephen Yuan Jiang Assignee: Stephen Yuan Jiang With Procedure V2 in production since HBASE 1.1 release, there is a need to list all procedures (running, queued, recently completed) from HBASE shell (or Web UI). This JIRA is to track the work to add a API to list all procedures. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14108) Administrative Task: provide an API to abort a procedure
Stephen Yuan Jiang created HBASE-14108: -- Summary: Administrative Task: provide an API to abort a procedure Key: HBASE-14108 URL: https://issues.apache.org/jira/browse/HBASE-14108 Project: HBase Issue Type: Sub-task Components: proc-v2 Affects Versions: 2.0.0, 1.3.0 Reporter: Stephen Yuan Jiang Assignee: Stephen Yuan Jiang With Procedure-V2 in production since HBASE 1.1 release, there is a need to abort a procedure (eg. for a long-running procedure that stucks somewhere and blocks others). The command could either from shell or Web UI. This task tracks the work to provide an API to abort a procedure (either rollback or simply quit). This API could be used either from shell or Web UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14100) Fix high priority findbugs warnings
Duo Zhang created HBASE-14100: - Summary: Fix high priority findbugs warnings Key: HBASE-14100 URL: https://issues.apache.org/jira/browse/HBASE-14100 Project: HBase Issue Type: Bug Reporter: Duo Zhang Assignee: Duo Zhang See here: https://builds.apache.org/job/HBase-TRUNK/6654/findbugsResult/HIGH/ We have 6 high priority findbugs warnings. A high priority findbugs warning is usually a bug. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14100) Fix high priority findbugs warnings
[
https://issues.apache.org/jira/browse/HBASE-14100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629690#comment-14629690
]
Duo Zhang commented on HBASE-14100:
---
{{IL_INFINITE_RECURSIVE_LOOP}}
{code:title=BloomFilterUtil.java}
195 /** Should only be used in tests */
196 public static boolean contains(byte[] buf, int offset, int length,
ByteBuffer bloom) { // no reference to this method, so just remove it?
197 return contains(buf, offset, length, bloom);
198 }
{code}
{code:title=RateLimiter.java}
201 // This method is for strictly testing purpose only
202 @VisibleForTesting
203 public void setNextRefillTime(long nextRefillTime) {
204 this.setNextRefillTime(nextRefillTime);
205 }
206
207 public long getNextRefillTime() {
208 return this.getNextRefillTime();
209 }
{code}
These two methods are all overridden by sub classes so no actual problem now.
But I do not think it is a good idea to leave a confusing code here? Just mark
the method as abstract? Or leave the implementation empty?
Fix high priority findbugs warnings
---
Key: HBASE-14100
URL: https://issues.apache.org/jira/browse/HBASE-14100
Project: HBase
Issue Type: Bug
Reporter: Duo Zhang
Assignee: Duo Zhang
See here:
https://builds.apache.org/job/HBase-TRUNK/6654/findbugsResult/HIGH/
We have 6 high priority findbugs warnings. A high priority findbugs warning
is usually a bug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12596) bulkload needs to follow locality
[ https://issues.apache.org/jira/browse/HBASE-12596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629994#comment-14629994 ] Andrew Purtell commented on HBASE-12596: bq. It does show a couple differences in the processes though. By that definition, before 1.0 there were RMs for major versions (after 1.0 there are RMs for each minor version) This is a great observation. It seems to me that we evolved the branch RM informal role because each release branch represented a major version bump. The branch RM role is easier post 1.0. (Except: 0.98, grandfathered stuff and code divergence issues. Except: 2.0, we could use a shepherd for challenges with the major version bump.) bulkload needs to follow locality - Key: HBASE-12596 URL: https://issues.apache.org/jira/browse/HBASE-12596 Project: HBase Issue Type: Improvement Components: HFile, regionserver Affects Versions: 0.98.8 Environment: hadoop-2.3.0, hbase-0.98.8, jdk1.7 Reporter: Victor Xu Assignee: Victor Xu Fix For: 2.0.0, 0.98.14, 1.3.0 Attachments: HBASE-12596-0.98-v1.patch, HBASE-12596-0.98-v2.patch, HBASE-12596-0.98-v3.patch, HBASE-12596-0.98-v4.patch, HBASE-12596-0.98-v5.patch, HBASE-12596-0.98-v6.patch, HBASE-12596-branch-1-v1.patch, HBASE-12596-branch-1-v2.patch, HBASE-12596-master-v1.patch, HBASE-12596-master-v2.patch, HBASE-12596-master-v3.patch, HBASE-12596-master-v4.patch, HBASE-12596-master-v5.patch, HBASE-12596-master-v6.patch, HBASE-12596.patch Normally, we have 2 steps to perform a bulkload: 1. use a job to write HFiles to be loaded; 2. Move these HFiles to the right hdfs directory. However, the locality could be loss during the first step. Why not just write the HFiles directly into the right place? We can do this easily because StoreFile.WriterBuilder has the withFavoredNodes method, and we just need to call it in HFileOutputFormat's getNewWriter(). This feature is enabled by default, and we could use 'hbase.bulkload.locality.sensitive.enabled=false' to disable it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14100) Fix high priority findbugs warnings
[ https://issues.apache.org/jira/browse/HBASE-14100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629970#comment-14629970 ] ramkrishna.s.vasudevan commented on HBASE-14100: +1. Better to fix here along with other warnings. Fix high priority findbugs warnings --- Key: HBASE-14100 URL: https://issues.apache.org/jira/browse/HBASE-14100 Project: HBase Issue Type: Bug Reporter: Duo Zhang Assignee: Duo Zhang Attachments: HBASE-14100.patch See here: https://builds.apache.org/job/HBase-TRUNK/6654/findbugsResult/HIGH/ We have 6 high priority findbugs warnings. A high priority findbugs warning is usually a bug. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13954) Remove HTableInterface#getRowOrBefore related server side code
[
https://issues.apache.org/jira/browse/HBASE-13954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629971#comment-14629971
]
Ashish Singhi commented on HBASE-13954:
---
Unfortunately jenkins bot got aborted
{noformat}
Build was aborted
Aborted by anonymous
Archiving artifacts
Recording test results
ERROR: H0 is offline; cannot locate jdk-1.7u51
No workspace found for PreCommit-HBASE-Build #14801
ERROR: H0 is offline; cannot locate jdk-1.7u51
Description set: HBASE-13954
Finished: ABORTED
{noformat}
Remove HTableInterface#getRowOrBefore related server side code
--
Key: HBASE-13954
URL: https://issues.apache.org/jira/browse/HBASE-13954
Project: HBase
Issue Type: Sub-task
Components: API
Reporter: Ashish Singhi
Assignee: Ashish Singhi
Fix For: 2.0.0
Attachments: HBASE-13954.patch
As part of HBASE-13214 review, [~anoop.hbase] had a review comment on the
review board to remove all the server side related code for getRowOrBefore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13954) Remove HTableInterface#getRowOrBefore related server side code
[ https://issues.apache.org/jira/browse/HBASE-13954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-13954: -- Attachment: HBASE-13954(1).patch Reattach the same patch Remove HTableInterface#getRowOrBefore related server side code -- Key: HBASE-13954 URL: https://issues.apache.org/jira/browse/HBASE-13954 Project: HBase Issue Type: Sub-task Components: API Reporter: Ashish Singhi Assignee: Ashish Singhi Fix For: 2.0.0 Attachments: HBASE-13954(1).patch, HBASE-13954.patch As part of HBASE-13214 review, [~anoop.hbase] had a review comment on the review board to remove all the server side related code for getRowOrBefore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14069) Add the ability for RegionSplitter to rolling split without using a SplitAlgorithm
[ https://issues.apache.org/jira/browse/HBASE-14069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629976#comment-14629976 ] Abhilash commented on HBASE-14069: -- I am using org.apache.hadoop.hbase.client.Admin.SplitRegion(). Any specific concerns ? Add the ability for RegionSplitter to rolling split without using a SplitAlgorithm -- Key: HBASE-14069 URL: https://issues.apache.org/jira/browse/HBASE-14069 Project: HBase Issue Type: New Feature Reporter: Elliott Clark Assignee: Abhilash RegionSplittler is the utility that can rolling split regions. It would be nice to be able to split regions and have the normal split points get computed for me so that I'm not reliant on knowing data distribution. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12596) bulkload needs to follow locality
[ https://issues.apache.org/jira/browse/HBASE-12596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629920#comment-14629920 ] Andrew Purtell commented on HBASE-12596: Let me call this discussion out on dev@ with a pointer to the tail of the discussion on this issue. bulkload needs to follow locality - Key: HBASE-12596 URL: https://issues.apache.org/jira/browse/HBASE-12596 Project: HBase Issue Type: Improvement Components: HFile, regionserver Affects Versions: 0.98.8 Environment: hadoop-2.3.0, hbase-0.98.8, jdk1.7 Reporter: Victor Xu Assignee: Victor Xu Fix For: 2.0.0, 0.98.14, 1.3.0 Attachments: HBASE-12596-0.98-v1.patch, HBASE-12596-0.98-v2.patch, HBASE-12596-0.98-v3.patch, HBASE-12596-0.98-v4.patch, HBASE-12596-0.98-v5.patch, HBASE-12596-0.98-v6.patch, HBASE-12596-branch-1-v1.patch, HBASE-12596-branch-1-v2.patch, HBASE-12596-master-v1.patch, HBASE-12596-master-v2.patch, HBASE-12596-master-v3.patch, HBASE-12596-master-v4.patch, HBASE-12596-master-v5.patch, HBASE-12596-master-v6.patch, HBASE-12596.patch Normally, we have 2 steps to perform a bulkload: 1. use a job to write HFiles to be loaded; 2. Move these HFiles to the right hdfs directory. However, the locality could be loss during the first step. Why not just write the HFiles directly into the right place? We can do this easily because StoreFile.WriterBuilder has the withFavoredNodes method, and we just need to call it in HFileOutputFormat's getNewWriter(). This feature is enabled by default, and we could use 'hbase.bulkload.locality.sensitive.enabled=false' to disable it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-12596) bulkload needs to follow locality
[ https://issues.apache.org/jira/browse/HBASE-12596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629927#comment-14629927 ] Andrew Purtell edited comment on HBASE-12596 at 7/16/15 3:59 PM: - As Sean mentioned we have issues pending for artifact licensing and NOTICE file improvements. FWIW, those block any 0.98 release too so the change on this issue is presently committed to branch but not released. was (Author: apurtell): As Sean mentioned we have issues pending for artifact licensing and NOTICE file improvements pending. FWIW, those block any 0.98 release too so the change on this issue is presently committed to branch but not released. bulkload needs to follow locality - Key: HBASE-12596 URL: https://issues.apache.org/jira/browse/HBASE-12596 Project: HBase Issue Type: Improvement Components: HFile, regionserver Affects Versions: 0.98.8 Environment: hadoop-2.3.0, hbase-0.98.8, jdk1.7 Reporter: Victor Xu Assignee: Victor Xu Fix For: 2.0.0, 0.98.14, 1.3.0 Attachments: HBASE-12596-0.98-v1.patch, HBASE-12596-0.98-v2.patch, HBASE-12596-0.98-v3.patch, HBASE-12596-0.98-v4.patch, HBASE-12596-0.98-v5.patch, HBASE-12596-0.98-v6.patch, HBASE-12596-branch-1-v1.patch, HBASE-12596-branch-1-v2.patch, HBASE-12596-master-v1.patch, HBASE-12596-master-v2.patch, HBASE-12596-master-v3.patch, HBASE-12596-master-v4.patch, HBASE-12596-master-v5.patch, HBASE-12596-master-v6.patch, HBASE-12596.patch Normally, we have 2 steps to perform a bulkload: 1. use a job to write HFiles to be loaded; 2. Move these HFiles to the right hdfs directory. However, the locality could be loss during the first step. Why not just write the HFiles directly into the right place? We can do this easily because StoreFile.WriterBuilder has the withFavoredNodes method, and we just need to call it in HFileOutputFormat's getNewWriter(). This feature is enabled by default, and we could use 'hbase.bulkload.locality.sensitive.enabled=false' to disable it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14084) Observe some out-of-date doc on Integration Tests part
[ https://issues.apache.org/jira/browse/HBASE-14084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629948#comment-14629948 ] Dima Spivak commented on HBASE-14084: - [~carp84], why can't the existing ChaosMonkey be used during bulkload? IntegrationTestBulkload does exactly that, no? Either way, might not be worth the effort to reinvent the wheel when all we need with the ChaosMonkey tool is to reintroduce the main() that was removed during a previous refactoring effort. Observe some out-of-date doc on Integration Tests part Key: HBASE-14084 URL: https://issues.apache.org/jira/browse/HBASE-14084 Project: HBase Issue Type: Task Components: documentation Affects Versions: 1.1.0 Reporter: Yu Li As titled, have checked src/main/asciidoc/_chapters/developer.adoc and confirmed some out-of-date part, such as the doc still refers to org.apache.hadoop.hbase.util.ChaosMonkey which doesn't exist anymore On the other hand, I think run ITBLL against distributed cluster is a really good way to do real-world integration/system testing, but existing document about this is not explicit enough. Actually it costs me quite a while to setup the env and make the testing run smoothly (encountered issues like always launching minicluster if run with bin/hbase script, finally got it run using bin/hadoop script, etc.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14098) Allow dropping caches behind compactions
[ https://issues.apache.org/jira/browse/HBASE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629949#comment-14629949 ] Elliott Clark commented on HBASE-14098: --- Yes it's possible that we will read the file again. It's also likely that on large compactions we will just be blowing the fs cache out of the water. Compacting 32gb on machine with 32gb memory free means that nothing else can be in the fs cache. No /bin/bash, no inodes, nothing. My plan is likely to set this only for large compactions; the thought being that large compactions are much more likely to have stale data in them. I'm going to test the current patch out on a cluster that's doing really large compactions right now. If I see any positive changes then we can make this smarter. Allow dropping caches behind compactions Key: HBASE-14098 URL: https://issues.apache.org/jira/browse/HBASE-14098 Project: HBase Issue Type: Bug Components: Compaction, hadoop2, HFile Affects Versions: 2.0.0, 1.3.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0, 1.3.0 Attachments: HBASE-14098.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12596) bulkload needs to follow locality
[ https://issues.apache.org/jira/browse/HBASE-12596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629951#comment-14629951 ] Dave Latham commented on HBASE-12596: - Andrew's note that 0.98 is like major, 0.98.x is like minor, 0.98.x.y is like patch (not 0.98 minor, 0.98.x patch) is illuminating. It does show a couple differences in the processes though. By that definition, before 1.0 there were RMs for major versions (after 1.0 there are RMs for each minor version); before 1.0 patch releases were very rare and only happened for the most recent minor release (after 1.0 it looks like the intent would be to have patch releases for multiple minor releases living on). I think it reflects an intent for making more stable releases available by updating minor releases with patches rather than just focusing on the latest minor release and leaving other ones behind. bulkload needs to follow locality - Key: HBASE-12596 URL: https://issues.apache.org/jira/browse/HBASE-12596 Project: HBase Issue Type: Improvement Components: HFile, regionserver Affects Versions: 0.98.8 Environment: hadoop-2.3.0, hbase-0.98.8, jdk1.7 Reporter: Victor Xu Assignee: Victor Xu Fix For: 2.0.0, 0.98.14, 1.3.0 Attachments: HBASE-12596-0.98-v1.patch, HBASE-12596-0.98-v2.patch, HBASE-12596-0.98-v3.patch, HBASE-12596-0.98-v4.patch, HBASE-12596-0.98-v5.patch, HBASE-12596-0.98-v6.patch, HBASE-12596-branch-1-v1.patch, HBASE-12596-branch-1-v2.patch, HBASE-12596-master-v1.patch, HBASE-12596-master-v2.patch, HBASE-12596-master-v3.patch, HBASE-12596-master-v4.patch, HBASE-12596-master-v5.patch, HBASE-12596-master-v6.patch, HBASE-12596.patch Normally, we have 2 steps to perform a bulkload: 1. use a job to write HFiles to be loaded; 2. Move these HFiles to the right hdfs directory. However, the locality could be loss during the first step. Why not just write the HFiles directly into the right place? We can do this easily because StoreFile.WriterBuilder has the withFavoredNodes method, and we just need to call it in HFileOutputFormat's getNewWriter(). This feature is enabled by default, and we could use 'hbase.bulkload.locality.sensitive.enabled=false' to disable it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14098) Allow dropping caches behind compactions
[ https://issues.apache.org/jira/browse/HBASE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-14098: -- Attachment: HBASE-14098-v1.patch Allow dropping caches behind compactions Key: HBASE-14098 URL: https://issues.apache.org/jira/browse/HBASE-14098 Project: HBase Issue Type: Bug Components: Compaction, hadoop2, HFile Affects Versions: 2.0.0, 1.3.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0, 1.3.0 Attachments: HBASE-14098-v1.patch, HBASE-14098.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14100) Fix high priority findbugs warnings
[ https://issues.apache.org/jira/browse/HBASE-14100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629952#comment-14629952 ] Andrew Purtell commented on HBASE-14100: +1 The one concern I had are the changes to RateLimiter but I checked and that class is tagged as Private/Evolving. bq. I think we could remove it first in this issue Sounds good to me. Only presents a small fixup for HBASE-12213. Fix high priority findbugs warnings --- Key: HBASE-14100 URL: https://issues.apache.org/jira/browse/HBASE-14100 Project: HBase Issue Type: Bug Reporter: Duo Zhang Assignee: Duo Zhang Attachments: HBASE-14100.patch See here: https://builds.apache.org/job/HBase-TRUNK/6654/findbugsResult/HIGH/ We have 6 high priority findbugs warnings. A high priority findbugs warning is usually a bug. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12596) bulkload needs to follow locality
[ https://issues.apache.org/jira/browse/HBASE-12596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629915#comment-14629915 ] Sean Busbey commented on HBASE-12596: - I don't plan to accept feature additions to 1.2.z after 1.2.0. However, I still haven't gotten 1.2.0 RC0 out (:sadpanda) so I don't consider it feature frozen. I'd be fine with this going in there so long as it's before RC0. I expect the licensing issues on HBASE-14085 to take long enough that an RC won't happen before tomorrow. As an aside [~lhofhansl], if this issue goes into branch-1.2 leaving the 1.3.0 version number in place is perfectly fine. As the RM I'll remove the 1.3.0 once I've made sure it's present on 1.2.0 and in the changelog (and fix the versions here if, for example, it had to be reverted before an RC got promoted to a release). bulkload needs to follow locality - Key: HBASE-12596 URL: https://issues.apache.org/jira/browse/HBASE-12596 Project: HBase Issue Type: Improvement Components: HFile, regionserver Affects Versions: 0.98.8 Environment: hadoop-2.3.0, hbase-0.98.8, jdk1.7 Reporter: Victor Xu Assignee: Victor Xu Fix For: 2.0.0, 0.98.14, 1.3.0 Attachments: HBASE-12596-0.98-v1.patch, HBASE-12596-0.98-v2.patch, HBASE-12596-0.98-v3.patch, HBASE-12596-0.98-v4.patch, HBASE-12596-0.98-v5.patch, HBASE-12596-0.98-v6.patch, HBASE-12596-branch-1-v1.patch, HBASE-12596-branch-1-v2.patch, HBASE-12596-master-v1.patch, HBASE-12596-master-v2.patch, HBASE-12596-master-v3.patch, HBASE-12596-master-v4.patch, HBASE-12596-master-v5.patch, HBASE-12596-master-v6.patch, HBASE-12596.patch Normally, we have 2 steps to perform a bulkload: 1. use a job to write HFiles to be loaded; 2. Move these HFiles to the right hdfs directory. However, the locality could be loss during the first step. Why not just write the HFiles directly into the right place? We can do this easily because StoreFile.WriterBuilder has the withFavoredNodes method, and we just need to call it in HFileOutputFormat's getNewWriter(). This feature is enabled by default, and we could use 'hbase.bulkload.locality.sensitive.enabled=false' to disable it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-12596) bulkload needs to follow locality
[ https://issues.apache.org/jira/browse/HBASE-12596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629861#comment-14629861 ] Andrew Purtell edited comment on HBASE-12596 at 7/16/15 3:49 PM: - That's changed since the new rules went into effect for 1.0 and up. Here and elsewhere we are able to accept new features in 1.x minor revs (branch-1), 0.98 because of its continuation of old style numbering/policy, and master. There will be a rolling upgrade capable upgrade path from 0.98 to a branch-1 release with a contiguous feature set. This is the consequence of adopting semver for 1.0 and up but not earlier. As 0.98 RM I can accept changes that add features on our minor releases (0.98.x) but not on our patch releases (0.98.x.y) as long as they do not break wire compatibility. On branch-1 this policy is similar but now the minors are 1.x and the patch versions are 1.x.y. We have contributors and users wanting enhancements in 0.98 that are allowed there. I'm not going to retire 0.98 as RM until we have consensus it should be EOL. bq. So if it is in 0.98, we probably want this in 1.0, 1.1, and especially 1.2 as well. You can ask [~enis], [~ndimiduk], and [~busbey]. I've done this before and have found Enis does not accept changes for 1.0 that do not fit into the bug fix bucket. This makes sense as 1.0 operates under the new semver regime. I'd presume that the case here but I'm not speaking for him. (Not to single Enis out here, I also presume Nick and Sean will make similar choices, but there's a history of decisions for branch-1.0 that we can talk about.) The old policy for 0.98 (wire compat is the only dealbreaker) came about as a result of discussions leading up to the 0.98.0 release. Consensus and user needs evolve though. Java binary compat became a pain point for downstreamers and Phoenix so now I also check that as part of the RC process. Consensus can evolve again, to a new policy that disallows anything but bug fixes to 0.98. If so we should as a courtesy have a discussion with 0.98 users on the implications. Already the code divergence between 0.98 and branch-1 is significant enough to make all but trivial changes an effort to backport. If we change the 0.98 accept criteria to only allow what can be committed to all 1.x branches, this shuts down all enhancements to 0.98 because of change policies applied to the 1.x branches by their RMs, and that divergence will get a lot bigger. At that point I would advocate that 0.98 is EOLed and retire as its RM since it would no longer need dedicated attention. At some point those two actions should happen anyway. Edit: Consolidated two posts and at-mentions. was (Author: apurtell): That's changed since the new rules went into effect for 1.0 and up. Here and elsewhere we are able to accept new features in 1.x minor revs (branch-1), 0.98 because of its continuation of old style numbering/policy, and master. There will be a rolling upgrade capable upgrade path from 0.98 to a branch-1 release with a contiguous feature set. This is the consequence of adopting semver for 1.0 and up but not earlier. As 0.98 RM I can accept changes that add features on our minor releases (0.98.x) but not on our patch releases (0.98.x.y) as long as they do not break wire compatibility. On branch-1 this policy is similar but now the minors are 1.x and the patch versions are 1.x.y. We have contributors and users wanting enhancements in 0.98 that are allowed there. I'm not going to retire 0.98 as RM until we have consensus it should be EOL. bq. So if it is in 0.98, we probably want this in 1.0, 1.1, and especially 1.2 as well. You can ask [~enis], [~ndimiduk], and [~busbey]. I've done this before and have found Enis does not accept changes for 1.0 that do not fit into the bug fix bucket. This makes sense as 1.0 operates under the new semver regime. I'd presume that the case here but I'm not speaking for him. (Not to single Enis out here, I also presume Nick and Sean will make similar choices.) The old policy for 0.98 (wire compat is the only dealbreaker) came about as a result of discussions leading up to the 0.98.0 release. Consensus and user needs evolve though. Java binary compat became a pain point for downstreamers and Phoenix so now I also check that as part of the RC process. Consensus can evolve again, to a new policy that disallows anything but bug fixes to 0.98. If so we should as a courtesy have a discussion with 0.98 users on the implications. Already the code divergence between 0.98 and branch-1 is significant enough to make all but trivial changes an effort to backport. If we change the 0.98 accept criteria to only allow what can be committed to all 1.x branches, this shuts down all enhancements to 0.98 because of change policies applied to the 1.x branches by their RMs, and that divergence
[jira] [Commented] (HBASE-12596) bulkload needs to follow locality
[ https://issues.apache.org/jira/browse/HBASE-12596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629927#comment-14629927 ] Andrew Purtell commented on HBASE-12596: As Sean mentioned we have issues pending for artifact licensing and NOTICE file improvements pending. FWIW, those block any 0.98 release too so the change on this issue is presently committed to branch but not released. bulkload needs to follow locality - Key: HBASE-12596 URL: https://issues.apache.org/jira/browse/HBASE-12596 Project: HBase Issue Type: Improvement Components: HFile, regionserver Affects Versions: 0.98.8 Environment: hadoop-2.3.0, hbase-0.98.8, jdk1.7 Reporter: Victor Xu Assignee: Victor Xu Fix For: 2.0.0, 0.98.14, 1.3.0 Attachments: HBASE-12596-0.98-v1.patch, HBASE-12596-0.98-v2.patch, HBASE-12596-0.98-v3.patch, HBASE-12596-0.98-v4.patch, HBASE-12596-0.98-v5.patch, HBASE-12596-0.98-v6.patch, HBASE-12596-branch-1-v1.patch, HBASE-12596-branch-1-v2.patch, HBASE-12596-master-v1.patch, HBASE-12596-master-v2.patch, HBASE-12596-master-v3.patch, HBASE-12596-master-v4.patch, HBASE-12596-master-v5.patch, HBASE-12596-master-v6.patch, HBASE-12596.patch Normally, we have 2 steps to perform a bulkload: 1. use a job to write HFiles to be loaded; 2. Move these HFiles to the right hdfs directory. However, the locality could be loss during the first step. Why not just write the HFiles directly into the right place? We can do this easily because StoreFile.WriterBuilder has the withFavoredNodes method, and we just need to call it in HFileOutputFormat's getNewWriter(). This feature is enabled by default, and we could use 'hbase.bulkload.locality.sensitive.enabled=false' to disable it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14100) Fix high priority findbugs warnings
[
https://issues.apache.org/jira/browse/HBASE-14100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629957#comment-14629957
]
Hadoop QA commented on HBASE-14100:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12745627/HBASE-14100.patch
against master branch at commit ebdac4b52e67614db70b59be8cd8143efe701911.
ATTACHMENT ID: 12745627
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14802//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14802//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14802//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14802//console
This message is automatically generated.
Fix high priority findbugs warnings
---
Key: HBASE-14100
URL: https://issues.apache.org/jira/browse/HBASE-14100
Project: HBase
Issue Type: Bug
Reporter: Duo Zhang
Assignee: Duo Zhang
Attachments: HBASE-14100.patch
See here:
https://builds.apache.org/job/HBase-TRUNK/6654/findbugsResult/HIGH/
We have 6 high priority findbugs warnings. A high priority findbugs warning
is usually a bug.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-11276) Add back support for running ChaosMonkey as standalone tool
[ https://issues.apache.org/jira/browse/HBASE-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dima Spivak reassigned HBASE-11276: --- Assignee: Yu Li (was: Dima Spivak) Reassigning to Li, who's way ahead of me on this. :) Add back support for running ChaosMonkey as standalone tool --- Key: HBASE-11276 URL: https://issues.apache.org/jira/browse/HBASE-11276 Project: HBase Issue Type: Task Affects Versions: 0.98.0, 0.96.0, 0.99.0 Reporter: Dima Spivak Assignee: Yu Li Priority: Minor Attachments: HBASE-11276.patch [According to the ref guide|http://hbase.apache.org/book/hbase.tests.html#integration.tests], it was once possible to run ChaosMonkey as a standalone tool against a deployed cluster. After 0.94, this is no longer possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13992) Integrate SparkOnHBase into HBase
[ https://issues.apache.org/jira/browse/HBASE-13992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630135#comment-14630135 ] Ted Malaska commented on HBASE-13992: - [~lhofhansl] I would like to add 1.4 or 1.5 into another patch. This patch is so large already. My hope is to close this one out then start a couple more right off of this like: 1. Newer versions of Spark 2. Adding DataFrame Support 3. Documentation 4. UpdateStateBy Key that will work through HBase 5. Bulk load to HBase 6. Much More Thanks Integrate SparkOnHBase into HBase - Key: HBASE-13992 URL: https://issues.apache.org/jira/browse/HBASE-13992 Project: HBase Issue Type: New Feature Components: spark Reporter: Ted Malaska Assignee: Ted Malaska Fix For: 2.0.0 Attachments: HBASE-13992.patch, HBASE-13992.patch.3, HBASE-13992.patch.4 This Jira is to ask if SparkOnHBase can find a home in side HBase core. Here is the github: https://github.com/cloudera-labs/SparkOnHBase I am the core author of this project and the license is Apache 2.0 A blog explaining this project is here http://blog.cloudera.com/blog/2014/12/new-in-cloudera-labs-sparkonhbase/ A spark Streaming example is here http://blog.cloudera.com/blog/2014/11/how-to-do-near-real-time-sessionization-with-spark-streaming-and-apache-hadoop/ A real customer using this in produce is blogged here http://blog.cloudera.com/blog/2015/03/how-edmunds-com-used-spark-streaming-to-build-a-near-real-time-dashboard/ Please debate and let me know what I can do to make this happen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14109) NPE if we don't load fully before we are shutdown
[
https://issues.apache.org/jira/browse/HBASE-14109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-14109:
--
Attachment: 14109.txt
NPE if we don't load fully before we are shutdown
-
Key: HBASE-14109
URL: https://issues.apache.org/jira/browse/HBASE-14109
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 1.2.0
Reporter: stack
Assignee: stack
Priority: Trivial
Attachments: 14109.txt
We failed to find ZK so:
{code}
2015-07-15 22:51:04,725 FATAL
[regionserver/c2022.halxg.cloudera.com/10.20.84.28:16020]
regionserver.HRegionServer: ABORTING region server
c2022.halxg.cloudera.com,16020,1437025808277: Initialization of RS failed.
Hence aborting RS.
java.io.IOException: Received the shutdown message while waiting.
at
org.apache.hadoop.hbase.regionserver.HRegionServer.blockAndCheckIfStopped(HRegionServer.java:807)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:756)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:732)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:876)
at java.lang.Thread.run(Thread.java:745)
{code}
... which got us this:
{code}
2015-07-15 22:51:04,734 FATAL
[regionserver/c2022.halxg.cloudera.com/10.20.84.28:16020]
regionserver.HRegionServer: ABORTING region server
c2022.halxg.cloudera.com,16020,1437025808277: Unhandled: null
java.lang.NullPointerException
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:911)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14102) Add thank you to our thanks page for vectorportal.com
[ https://issues.apache.org/jira/browse/HBASE-14102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630147#comment-14630147 ] Hudson commented on HBASE-14102: SUCCESS: Integrated in HBase-TRUNK #6655 (See [https://builds.apache.org/job/HBase-TRUNK/6655/]) HBASE-14102 Add thank you to our thanks page for vectorportal.com (stack: rev 97e0af001bdef74d9e0cc23e09cf7105bb8e0f64) * src/main/site/asciidoc/sponsors.adoc Add thank you to our thanks page for vectorportal.com - Key: HBASE-14102 URL: https://issues.apache.org/jira/browse/HBASE-14102 Project: HBase Issue Type: Sub-task Reporter: stack Assignee: stack Fix For: 2.0.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
