[jira] [Created] (HBASE-12252) IntegrationTestBulkLoad fails with illegal partition error
Dima Spivak created HBASE-12252:
---
Summary: IntegrationTestBulkLoad fails with illegal partition error
Key: HBASE-12252
URL: https://issues.apache.org/jira/browse/HBASE-12252
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.98.6
Reporter: Dima Spivak
Assignee: Dima Spivak
While running {{IntegrationTestBulkLoad}} for a while, I encountered the
following error:
{code}
14/10/14 05:57:12 INFO mapreduce.IntegrationTestBulkLoad: Running check
14/10/14 05:57:12 INFO client.RMProxy: Connecting to ResourceManager at
node-1.internal/10.0.0.3:8032
14/10/14 05:57:16 INFO util.RegionSizeCalculator: Calculating region sizes for
table IntegrationTestBulkLoad.
14/10/14 05:57:16 INFO mapreduce.JobSubmitter: number of splits:16
14/10/14 05:57:17 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1413261370136_0012
14/10/14 05:57:17 INFO impl.YarnClientImpl: Submitted application
application_1413261370136_0012
14/10/14 05:57:17 INFO mapreduce.Job: The url to track the job:
http://node-1.internal:8088/proxy/application_1413261370136_0012/
14/10/14 05:57:17 INFO mapreduce.Job: Running job: job_1413261370136_0012
14/10/14 05:57:32 INFO mapreduce.Job: Job job_1413261370136_0012 running in
uber mode : false
14/10/14 05:57:32 INFO mapreduce.Job: map 0% reduce 0%
14/10/14 05:57:52 INFO mapreduce.Job: map 6% reduce 0%
14/10/14 05:57:53 INFO mapreduce.Job: Task Id :
attempt_1413261370136_0012_m_13_0, Status : FAILED
Error: java.io.IOException: Illegal partition for
org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad$LinkKey@133c4539 (-3)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1079)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:712)
at
org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
at
org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
at
org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad$LinkedListCheckingMapper.map(IntegrationTestBulkLoad.java:502)
at
org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad$LinkedListCheckingMapper.map(IntegrationTestBulkLoad.java:491)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
{code}
I'm pretty sure I've tracked this down to a partition generator that can return
a negative number; I'll post a fix once I've confirmed locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HBASE-12252) IntegrationTestBulkLoad fails with illegal partition error
[
https://issues.apache.org/jira/browse/HBASE-12252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-12252 started by Dima Spivak.
---
IntegrationTestBulkLoad fails with illegal partition error
--
Key: HBASE-12252
URL: https://issues.apache.org/jira/browse/HBASE-12252
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.98.6
Reporter: Dima Spivak
Assignee: Dima Spivak
While running {{IntegrationTestBulkLoad}} for a while, I encountered the
following error:
{code}
14/10/14 05:57:12 INFO mapreduce.IntegrationTestBulkLoad: Running check
14/10/14 05:57:12 INFO client.RMProxy: Connecting to ResourceManager at
node-1.internal/10.0.0.3:8032
14/10/14 05:57:16 INFO util.RegionSizeCalculator: Calculating region sizes
for table IntegrationTestBulkLoad.
14/10/14 05:57:16 INFO mapreduce.JobSubmitter: number of splits:16
14/10/14 05:57:17 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1413261370136_0012
14/10/14 05:57:17 INFO impl.YarnClientImpl: Submitted application
application_1413261370136_0012
14/10/14 05:57:17 INFO mapreduce.Job: The url to track the job:
http://node-1.internal:8088/proxy/application_1413261370136_0012/
14/10/14 05:57:17 INFO mapreduce.Job: Running job: job_1413261370136_0012
14/10/14 05:57:32 INFO mapreduce.Job: Job job_1413261370136_0012 running in
uber mode : false
14/10/14 05:57:32 INFO mapreduce.Job: map 0% reduce 0%
14/10/14 05:57:52 INFO mapreduce.Job: map 6% reduce 0%
14/10/14 05:57:53 INFO mapreduce.Job: Task Id :
attempt_1413261370136_0012_m_13_0, Status : FAILED
Error: java.io.IOException: Illegal partition for
org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad$LinkKey@133c4539
(-3)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1079)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:712)
at
org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
at
org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
at
org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad$LinkedListCheckingMapper.map(IntegrationTestBulkLoad.java:502)
at
org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad$LinkedListCheckingMapper.map(IntegrationTestBulkLoad.java:491)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
{code}
I'm pretty sure I've tracked this down to a partition generator that can
return a negative number; I'll post a fix once I've confirmed locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12251) [book] Hadoop compat matrix 0.94 section needs cleaned up
[
https://issues.apache.org/jira/browse/HBASE-12251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14170569#comment-14170569
]
Hadoop QA commented on HBASE-12251:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12674693/0001-HBASE-12251-book-Hadoop-compat-matrix-0.94-section-n.patch
against trunk revision .
ATTACHMENT ID: 12674693
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+programlisting language=bourne![CDATA[$
protoc -Isrc/main/protobuf --java_out=src/main/java
src/main/protobuf/hbase.proto]]/programlisting
+programlisting language=bourne![CDATA[$ protoc
-Isrc/main/protobuf --java_out=src/main/java
src/main/protobuf/ErrorHandling.proto]]/programlisting
+ screen language=bourne$ mvn clean install
assembly:single -Dhadoop.profile=2.0 -DskipTests/screen
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/11338//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11338//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11338//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11338//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11338//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11338//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11338//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11338//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11338//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11338//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11338//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11338//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11338//console
This message is automatically generated.
[book] Hadoop compat matrix 0.94 section needs cleaned up
-
Key: HBASE-12251
URL: https://issues.apache.org/jira/browse/HBASE-12251
Project: HBase
Issue Type: Improvement
Components: documentation
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
Fix For: 2.0.0
Attachments:
0001-HBASE-12251-book-Hadoop-compat-matrix-0.94-section-n.patch
The compatibility matrix has instructions for compiling 0.94 vs Hadoop 2.2
inline in the table. These should sit outside the compatibility matrix table
as a referenced section.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12147) Porting Online Config Change from 89-fb
[ https://issues.apache.org/jira/browse/HBASE-12147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14170599#comment-14170599 ] Hudson commented on HBASE-12147: SUCCESS: Integrated in HBase-TRUNK #5661 (See [https://builds.apache.org/job/HBase-TRUNK/5661/]) HBASE-12147 Porting Online Config Change from 89-fb -- JAVADOC FIX ADDENDUM (stack: rev 904654aa87d18cab4061a1b70553d691f5ba7f64) * hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java Porting Online Config Change from 89-fb --- Key: HBASE-12147 URL: https://issues.apache.org/jira/browse/HBASE-12147 Project: HBase Issue Type: Sub-task Affects Versions: 1.0.0 Reporter: Manukranth Kolloju Assignee: Manukranth Kolloju Fix For: 0.99.2 Attachments: 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch Original Estimate: 168h Remaining Estimate: 168h This jira is to track the forward port of HBASE-8805 and HBASE-8544 implemented by [~gaurav.menghani] in 89-fb. This improves operational efficiency in managing clusters that are serving production traffic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-10201) Port 'Make flush decisions per column family' to trunk
[ https://issues.apache.org/jira/browse/HBASE-10201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangduo updated HBASE-10201: - Attachment: HBASE-10201-0.98_1.patch According to Ted Yu's suggestion, add a testcase called testCompareStoreFileCount in TestPerColumnFamilyFlush to confirm we really reduce the number of store files with this patch. Port 'Make flush decisions per column family' to trunk -- Key: HBASE-10201 URL: https://issues.apache.org/jira/browse/HBASE-10201 Project: HBase Issue Type: Improvement Reporter: Ted Yu Attachments: 3149-trunk-v1.txt, HBASE-10201-0.98.patch, HBASE-10201-0.98_1.patch Currently the flush decision is made using the aggregate size of all column families. When large and small column families co-exist, this causes many small flushes of the smaller CF. We need to make per-CF flush decisions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-12132) mvn install -Dtest=testclass also runs the hbase-it tests.
[
https://issues.apache.org/jira/browse/HBASE-12132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dima Spivak resolved HBASE-12132.
-
Resolution: Not a Problem
Assignee: Dima Spivak
Hi Manukranth,
This is actually the designed behavior of Maven. If you run {{mvn install}}, it
will run all parts of the Maven lifecycle up to and including the phase
specified, which in this case means it will also go through the
integration-test phase. If you'd like to prevent IT tests from being run, add
{{-DskipITs}} to the end of your existing command.
mvn install -Dtest=testclass also runs the hbase-it tests.
-
Key: HBASE-12132
URL: https://issues.apache.org/jira/browse/HBASE-12132
Project: HBase
Issue Type: Bug
Reporter: Manukranth Kolloju
Assignee: Dima Spivak
Priority: Minor
Attachments: mvninstall.txt
mvn install -Dtest=testclass runs tests in it and they time out. This Jira
is intended to fix that behavior.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11368) Multi-column family BulkLoad fails if compactions go on too long
[
https://issues.apache.org/jira/browse/HBASE-11368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Qiang Tian updated HBASE-11368:
---
Attachment: hbase-11368-0.98.5.patch
I forgot StoreScanner is per CF..earlier analysis is wrong:
{quote}
After DefaultStoreFileManager#storefiles is updated in HStore#bulkLoadHFile,
notifyChangedReadersObservers is called to reset the StoreScanner#heap, so
checkReseek-resetScannerStack will be triggered in next scan/read to recreate
store scanners based on new storefiles.
so we could introduce a new region level rwlock multiCFLock,
HRegion#bulkLoadHFiles acquires the writelock before multi-CF
HStore.bulkLoadHFile call. and StoreScanner#resetScannerStack acquires the
readlock. this way the scanners are recreated after all CFs' store files are
populated.
{quote}
instead, the new lock should put at regionScanner layer. see the patch
attached.
the mvn test and TestHRegionServerBulkLoad(large test for atomic bulkload
test) passed, still need to run large tests and performance test(any
suggestions for it? YCSB?).
the lock can be further limited to a smaller scope by split
HStore#bulkLoadHFile into 2 parts:1) rename the bulkload files and put new
files into store files list 2) notifyChangedReadersObservers. only #2 needs the
lock.
if HDFS file rename is fast, the split may not be needed.
Multi-column family BulkLoad fails if compactions go on too long
Key: HBASE-11368
URL: https://issues.apache.org/jira/browse/HBASE-11368
Project: HBase
Issue Type: Bug
Reporter: stack
Assignee: Qiang Tian
Attachments: hbase-11368-0.98.5.patch
Compactions take a read lock. If a multi-column family region, before bulk
loading, we want to take a write lock on the region. If the compaction takes
too long, the bulk load fails.
Various recipes include:
+ Making smaller regions (lame)
+ [~victorunique] suggests major compacting just before bulk loading over in
HBASE-10882 as a work around.
Does the compaction need a read lock for that long? Does the bulk load need
a full write lock when multiple column families? Can we fail more gracefully
at least?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11368) Multi-column family BulkLoad fails if compactions go on too long
[ https://issues.apache.org/jira/browse/HBASE-11368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14170649#comment-14170649 ] Qiang Tian commented on HBASE-11368: it looks to me the patch could show the value only when there is long compaction + gets/scans, not sure if [~victorunique] wants to try it in some test env? thanks. Multi-column family BulkLoad fails if compactions go on too long Key: HBASE-11368 URL: https://issues.apache.org/jira/browse/HBASE-11368 Project: HBase Issue Type: Bug Reporter: stack Assignee: Qiang Tian Attachments: hbase-11368-0.98.5.patch Compactions take a read lock. If a multi-column family region, before bulk loading, we want to take a write lock on the region. If the compaction takes too long, the bulk load fails. Various recipes include: + Making smaller regions (lame) + [~victorunique] suggests major compacting just before bulk loading over in HBASE-10882 as a work around. Does the compaction need a read lock for that long? Does the bulk load need a full write lock when multiple column families? Can we fail more gracefully at least? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12251) [book] Hadoop compat matrix 0.94 section needs cleaned up
[ https://issues.apache.org/jira/browse/HBASE-12251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14170650#comment-14170650 ] Hudson commented on HBASE-12251: SUCCESS: Integrated in HBase-TRUNK #5662 (See [https://builds.apache.org/job/HBase-TRUNK/5662/]) HBASE-12251 [book] Hadoop compat matrix 0.94 section needs cleaned up (ndimiduk: rev 43e596becb441ecf02c823f112c3eefa65f5df0b) * src/main/docbkx/configuration.xml [book] Hadoop compat matrix 0.94 section needs cleaned up - Key: HBASE-12251 URL: https://issues.apache.org/jira/browse/HBASE-12251 Project: HBase Issue Type: Improvement Components: documentation Reporter: Nick Dimiduk Assignee: Nick Dimiduk Fix For: 2.0.0 Attachments: 0001-HBASE-12251-book-Hadoop-compat-matrix-0.94-section-n.patch The compatibility matrix has instructions for compiling 0.94 vs Hadoop 2.2 inline in the table. These should sit outside the compatibility matrix table as a referenced section. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12073) Shell command user_permission fails on the table created by user if he is not global admin.
[ https://issues.apache.org/jira/browse/HBASE-12073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14170688#comment-14170688 ] Matteo Bertozzi commented on HBASE-12073: - what is the rationale behind the decision of adding another method listMatchingTables() instead of using the one that we already have listTables()? Shell command user_permission fails on the table created by user if he is not global admin. -- Key: HBASE-12073 URL: https://issues.apache.org/jira/browse/HBASE-12073 Project: HBase Issue Type: Bug Reporter: Srikanth Srungarapu Assignee: Srikanth Srungarapu Priority: Minor Attachments: HBASE-12073.patch The command fails as the changes introduced by HBASE-10892 requires user (because of newly introduced call to getTableDescriptors) to have global admin permission. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12073) Shell command user_permission fails on the table created by user if he is not global admin.
[ https://issues.apache.org/jira/browse/HBASE-12073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-12073: Affects Version/s: 0.99.1 0.98.6.1 Fix Version/s: 2.0.0 Shell command user_permission fails on the table created by user if he is not global admin. -- Key: HBASE-12073 URL: https://issues.apache.org/jira/browse/HBASE-12073 Project: HBase Issue Type: Bug Affects Versions: 0.99.1, 0.98.6.1 Reporter: Srikanth Srungarapu Assignee: Srikanth Srungarapu Priority: Minor Fix For: 2.0.0 Attachments: HBASE-12073.patch The command fails as the changes introduced by HBASE-10892 requires user (because of newly introduced call to getTableDescriptors) to have global admin permission. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-11992: --- Attachment: HBASE-11992_0.98_1.patch Patch for 0.98. All tests passes. All the replication related tests also passes. [~apurtell] Address your review comments also. Coming to the cross version compatibility issues - Documentation would definitely be needed. But is it ok to take this up in a 0.98 minor release even? Backport HBASE-11367 (Pluggable replication endpoint) to 0.98 - Key: HBASE-11992 URL: https://issues.apache.org/jira/browse/HBASE-11992 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Attachments: HBASE-11992_0.98_1.patch, hbase-11367_0.98.patch ReplicationSource tails the logs for each peer. HBASE-11367 introduces ReplicationEndpoint which is customizable per peer. ReplicationEndpoint is run in the same RS process and instantiated per replication peer per region server. Implementations of this interface handle the actual shipping of WAL edits to the remote cluster. This issue is for backporting HBASE-11367 to 0.98. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12253) Backport HBASE-11920 (CP hooks for replication endpoint)
ramkrishna.s.vasudevan created HBASE-12253: -- Summary: Backport HBASE-11920 (CP hooks for replication endpoint) Key: HBASE-12253 URL: https://issues.apache.org/jira/browse/HBASE-12253 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.7 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.98.8 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12161) Add support for grant/revoke on namespaces in AccessControlClient
[
https://issues.apache.org/jira/browse/HBASE-12161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14170706#comment-14170706
]
Matteo Bertozzi commented on HBASE-12161:
-
something like that would be nice also for the existing grant/revoke, since
they are not using the ProtobufUtil.grant/revoke but reimplementing the logic.
but this thing doesn't work.
{code}
BlockingInterface getAccessControlServiceStub (HTable ht) {
...
ht = new HTable(...)
ht...
...
}
...
HTable ht = null;
stub = getAccessControlServiceStub(ht);
if (ht != null)
...
...
{code}
Add support for grant/revoke on namespaces in AccessControlClient
-
Key: HBASE-12161
URL: https://issues.apache.org/jira/browse/HBASE-12161
Project: HBase
Issue Type: Improvement
Reporter: Srikanth Srungarapu
Assignee: Srikanth Srungarapu
Priority: Minor
Attachments: HBASE-12161_0.98.patch, HBASE-12161_master.patch
As per the description.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12187) Review in source the paper Simple Testing Can Prevent Most Critical Failures
[ https://issues.apache.org/jira/browse/HBASE-12187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14170714#comment-14170714 ] Ding Yuan commented on HBASE-12187: --- Great! I will work on it and get back to you in a few days. Review in source the paper Simple Testing Can Prevent Most Critical Failures -- Key: HBASE-12187 URL: https://issues.apache.org/jira/browse/HBASE-12187 Project: HBase Issue Type: Bug Reporter: stack Priority: Critical Review the helpful paper https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-yuan.pdf It describes 'catastrophic failures', especially issues where exceptions are thrown but not properly handled. Their static analysis tool Aspirator turns up a bunch of the obvious offenders (Lets add to test-patch.sh alongside findbugs?). This issue is about going through code base making sub-issues to root out these and others (Don't we have the test described in figure #6 already? I thought we did? If we don't, need to add). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12241) The crash of regionServer when taking deadserver's replication queue break replication
[
https://issues.apache.org/jira/browse/HBASE-12241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui updated HBASE-12241:
Attachment: HBASE-12241-trunk-v1.diff
Set useMulti to be true in hbase-default.xml
The crash of regionServer when taking deadserver's replication queue break
replication
--
Key: HBASE-12241
URL: https://issues.apache.org/jira/browse/HBASE-12241
Project: HBase
Issue Type: Bug
Components: Replication
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Critical
Fix For: 2.0.0, 0.99.2
Attachments: HBASE-12241-trunk-v1.diff
When a regionserver crash, another regionserver will try to take over the
replication hlogs queue and help the the the dead regionserver to finish the
replcation.See NodeFailoverWorker in ReplicationSourceManager
Currently hbase.zookeeper.useMulti is false in default configuration. The
operation of taking over replication queue is not atomic. The
ReplicationSourceManager firstly lock the replication node of dead
regionserver and then copy the replication queue, and delete replication node
of dead regionserver at last. The operation of the lockOtherRS just creates a
persistent zk node named lock which prevent other regionserver taking over
the replication queue.
See:
{code}
public boolean lockOtherRS(String znode) {
try {
String parent = ZKUtil.joinZNode(this.rsZNode, znode);
if (parent.equals(rsServerNameZnode)) {
LOG.warn(Won't lock because this is us, we're dead!);
return false;
}
String p = ZKUtil.joinZNode(parent, RS_LOCK_ZNODE);
ZKUtil.createAndWatch(this.zookeeper, p,
Bytes.toBytes(rsServerNameZnode));
} catch (KeeperException e) {
...
return false;
}
return true;
}
{code}
But if a regionserver crashed after creating this lock zk node and before
coping the replication queue to its replication queue, the lock zk node
will be left forever and
no other regionserver can take over the replication queue.
In out production cluster, we encounter this problem. We found the
replication queue was there and no regionserver took over it and a lock zk
node left there.
{quote}
hbase.32561.log:2014-09-24,14:09:28,790 INFO
org.apache.hadoop.hbase.replication.ReplicationZookeeper: Won't transfer the
queue, another RS took care of it because of: KeeperErrorCode = NoNode for
/hbase/hhsrv-micloud/replication/rs/hh-hadoop-srv-st09.bj,12610,1410937824255/lock
hbase.32561.log:2014-09-24,14:14:45,148 INFO
org.apache.hadoop.hbase.replication.ReplicationZookeeper: Won't transfer the
queue, another RS took care of it because of: KeeperErrorCode = NoNode for
/hbase/hhsrv-micloud/replication/rs/hh-hadoop-srv-st10.bj,12600,1410937795685/lock
{quote}
A quick solution is that the lock operation just create an ephemeral lock
zookeeper node and when the lock node is deleted, other regionserver will be
notified to check if there are replication queue left.
Suggestions are welcomed! Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14170835#comment-14170835 ] ramkrishna.s.vasudevan commented on HBASE-11992: Test restart scenarios before and after the patch. 1) Without patch started two servers and inserted some data. Ensure replication was on between these two servers. Read back the data from the peer cluster. Data was found. 2) Stopped the parent cluster region server and master. 3) Applied the patch on the parent cluster. 4) Read back the data from both the clusters. 5) Inserted new data in the parent cluster. 6) Read back the data from the peer cluster. New data was found. 7) Stopped the peer cluster -both master and region server. 8) Applied the patch on the peer cluster. 9) Read back the data from peer cluster. 10) Inserted new data from the parent cluster. 11) Read back the data from the peer cluster. New data was found. [~apurtell],[~enis] Anything else to be verified? Let me know. Thanks!! Backport HBASE-11367 (Pluggable replication endpoint) to 0.98 - Key: HBASE-11992 URL: https://issues.apache.org/jira/browse/HBASE-11992 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Attachments: HBASE-11992_0.98_1.patch, hbase-11367_0.98.patch ReplicationSource tails the logs for each peer. HBASE-11367 introduces ReplicationEndpoint which is customizable per peer. ReplicationEndpoint is run in the same RS process and instantiated per replication peer per region server. Implementations of this interface handle the actual shipping of WAL edits to the remote cluster. This issue is for backporting HBASE-11367 to 0.98. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14170835#comment-14170835 ] ramkrishna.s.vasudevan edited comment on HBASE-11992 at 10/14/14 12:02 PM: --- Tested upgrade scenarios before and after the patch. 1) Without patch started two servers and inserted some data. Ensure replication was on between these two servers. Read back the data from the peer cluster. Data was found. 2) Stopped the parent cluster region server and master. 3) Applied the patch on the parent cluster. 4) Read back the data from both the clusters. 5) Inserted new data in the parent cluster. 6) Read back the data from the peer cluster. New data was found. 7) Stopped the peer cluster -both master and region server. 8) Applied the patch on the peer cluster. 9) Read back the data from peer cluster. 10) Inserted new data from the parent cluster. 11) Read back the data from the peer cluster. New data was found. [~apurtell],[~enis] Anything else to be verified? Let me know. Thanks!! was (Author: ram_krish): Test restart scenarios before and after the patch. 1) Without patch started two servers and inserted some data. Ensure replication was on between these two servers. Read back the data from the peer cluster. Data was found. 2) Stopped the parent cluster region server and master. 3) Applied the patch on the parent cluster. 4) Read back the data from both the clusters. 5) Inserted new data in the parent cluster. 6) Read back the data from the peer cluster. New data was found. 7) Stopped the peer cluster -both master and region server. 8) Applied the patch on the peer cluster. 9) Read back the data from peer cluster. 10) Inserted new data from the parent cluster. 11) Read back the data from the peer cluster. New data was found. [~apurtell],[~enis] Anything else to be verified? Let me know. Thanks!! Backport HBASE-11367 (Pluggable replication endpoint) to 0.98 - Key: HBASE-11992 URL: https://issues.apache.org/jira/browse/HBASE-11992 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Attachments: HBASE-11992_0.98_1.patch, hbase-11367_0.98.patch ReplicationSource tails the logs for each peer. HBASE-11367 introduces ReplicationEndpoint which is customizable per peer. ReplicationEndpoint is run in the same RS process and instantiated per replication peer per region server. Implementations of this interface handle the actual shipping of WAL edits to the remote cluster. This issue is for backporting HBASE-11367 to 0.98. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12254) Document limitations related to pluggable replication endpoint feature usage in 0.98
ramkrishna.s.vasudevan created HBASE-12254: -- Summary: Document limitations related to pluggable replication endpoint feature usage in 0.98 Key: HBASE-12254 URL: https://issues.apache.org/jira/browse/HBASE-12254 Project: HBase Issue Type: Sub-task Components: documentation Affects Versions: 0.98.7 Reporter: ramkrishna.s.vasudevan Fix For: 0.98.8 The pluggable Replication endpoint in 0.98 will need to be documented as how exactly it can be used because of limitations that we may have due to mixed version compatability where the peers may be in an older version of 0.98 where pluggable replication endpoint is not there. Also this feature adds some more data to the znodes like the name of the Endpoint impl, data and the Replication config. A peer cluster with the older version will not be able to read this data particularly when there is a custom replication configured. This JIRA aims at documenting such cases for the ease of user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12247) Use a user managed connection in TableInputFormat
[ https://issues.apache.org/jira/browse/HBASE-12247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14170899#comment-14170899 ] Solomon Duskis commented on HBASE-12247: [~stack]: I can't seem to find a clear place to shoehorn a table.close(). I couldn't find a place where HTable.close() is currently called. Use a user managed connection in TableInputFormat - Key: HBASE-12247 URL: https://issues.apache.org/jira/browse/HBASE-12247 Project: HBase Issue Type: Bug Affects Versions: 2.0.0, 0.99.2 Reporter: Solomon Duskis Assignee: Solomon Duskis TableInputFormat uses a new HTable() that manages its own connection. Switch that use to a manually created ConnectionFactory and then close it at the end of the operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12247) Use a user managed connection in TableInputFormat
[ https://issues.apache.org/jira/browse/HBASE-12247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14170906#comment-14170906 ] Solomon Duskis commented on HBASE-12247: I tried using some of the unit tests and a debugger to see where close() is called. I couldn't seem to get a hit. Use a user managed connection in TableInputFormat - Key: HBASE-12247 URL: https://issues.apache.org/jira/browse/HBASE-12247 Project: HBase Issue Type: Bug Affects Versions: 2.0.0, 0.99.2 Reporter: Solomon Duskis Assignee: Solomon Duskis TableInputFormat uses a new HTable() that manages its own connection. Switch that use to a manually created ConnectionFactory and then close it at the end of the operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Moved] (HBASE-12255) hbase table is unavailable after restart hdfs without stop hbase
[ https://issues.apache.org/jira/browse/HBASE-12255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell moved PHOENIX-1348 to HBASE-12255: - Affects Version/s: (was: 4.1) Workflow: no-reopen-closed, patch-avail (was: patch-available, re-open possible) Key: HBASE-12255 (was: PHOENIX-1348) Project: HBase (was: Phoenix) hbase table is unavailable after restart hdfs without stop hbase Key: HBASE-12255 URL: https://issues.apache.org/jira/browse/HBASE-12255 Project: HBase Issue Type: Bug Environment: hadoop-2.5.1 hbase-0.98 phonenix-4.1.0 Reporter: Junyong Li I have a phoenix table 'EVENT', and the table have a index 'IDX_DATE_HOUR_X'. I restarted hdfs without stop hbase, after then, the hbase table cann't be scaned. I try to restart hbase, the all hbase table still cann't be scaned. the regionserver log have many exception like this: 2014-10-13 19:33:05,287 INFO [A01101303447-V1,60020,1413199890407-recovery-writer--pool4-t3] client.AsyncProcess: #4, waiting for some tasks to finish. Expected max=0, tasksSent=9, tasksDone=8, currentTasksDone=8, retries=8 hasError=fal se, tableName=IDX_DATE_HOUR_X 2014-10-13 19:33:05,298 INFO [A01101303447-V1,60020,1413199890407-recovery-writer--pool4-t2] client.AsyncProcess: #5, waiting for some tasks to finish. Expected max=0, tasksSent=9, tasksDone=8, currentTasksDone=8, retries=8 hasError=fal se, tableName=IDX_DATE_HOUR_X 2014-10-13 19:33:05,311 INFO [A01101303447-V1,60020,1413199890407-recovery-writer--pool4-t1] client.AsyncProcess: #6, waiting for some tasks to finish. Expected max=0, tasksSent=9, tasksDone=8, currentTasksDone=8, retries=8 hasError=fal se, tableName=IDX_DATE_HOUR_X 2014-10-13 19:33:06,452 INFO [ReplicationExecutor-0] replication.ReplicationQueuesZKImpl: Moving A01101303447-V1,60020,1413199414409's hlogs to my queue 2014-10-13 19:33:15,325 INFO [A01101303447-V1,60020,1413199890407-recovery-writer--pool4-t1] client.AsyncProcess: #6, waiting for some tasks to finish. Expected max=0, tasksSent=10, tasksDone=9, currentTasksDone=9, retries=9 hasError=fa lse, tableName=IDX_DATE_HOUR_X 2014-10-13 19:33:15,333 INFO [htable-pool6-t2] client.AsyncProcess: #6, table=IDX_DATE_HOUR_X, attempt=10/350 failed 12 ops, last exception: org.apache.hadoop.hbase.exceptions.RegionOpeningException: org.apache.hadoop.hbase.exceptions.R egionOpeningException: Region IDX_DATE_HOUR_X,t\x00\x00\x00\x00\x00,1413186874829.9a92abb84768b129df3faedb877f7bea. is opening on A01101303447-V1,60020,1413199890407 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2759) at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:4213) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3437) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29593) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:744) --- After two days's try, i found that: If idisable 'EVENT', other tables can be scanned, then i enable 'EVENT' manually, the region log show that NullPointExceptin has occur then replaying WAL, the following is log: 2014-10-13 19:25:21,043 INFO [RS_OPEN_REGION-A01101303447-V1:60020-1] regionserver.HRegion: Replaying edits from hdfs://localhost/hbase-0.98/data/default/EVENT/def4a581d4ad963cbb8cad32cbfbab2e/recovered.edits/002 2014-10-13 19:25:21,048 INFO [A01101303447-V1,60020,1413199414409-recovery-writer--pool17-t2-SendThread(localhost:2182)] zookeeper.ClientCnxn: Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2182. Will not attempt to authe nticate using SASL (unknown error) 2014-10-13 19:25:21,049 INFO [A01101303447-V1,60020,1413199414409-recovery-writer--pool17-t2-SendThread(localhost:2182)] zookeeper.ClientCnxn: Socket connection established to localhost/0:0:0:0:0:0:0:1:2182, initiating session 2014-10-13 19:25:21,051 INFO [A01101303447-V1,60020,1413199414409-recovery-writer--pool17-t3] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x199c8484 connecting to ZooKeeper ensemble=localhost:2182 2014-10-13 19:25:21,051 INFO [A01101303447-V1,60020,1413199414409-recovery-writer--pool17-t3] zookeeper.ZooKeeper: Initiating client
[jira] [Commented] (HBASE-12255) hbase table is unavailable after restart hdfs without stop hbase
[ https://issues.apache.org/jira/browse/HBASE-12255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171047#comment-14171047 ] Andrew Purtell commented on HBASE-12255: This looks like an HBase bug, so I moved it to the HBase project. By any chance is a complete stacktrace for those NullPointerExceptions logged? Can you provide it here? hbase table is unavailable after restart hdfs without stop hbase Key: HBASE-12255 URL: https://issues.apache.org/jira/browse/HBASE-12255 Project: HBase Issue Type: Bug Environment: hadoop-2.5.1 hbase-0.98 phonenix-4.1.0 Reporter: Junyong Li I have a phoenix table 'EVENT', and the table have a index 'IDX_DATE_HOUR_X'. I restarted hdfs without stop hbase, after then, the hbase table cann't be scaned. I try to restart hbase, the all hbase table still cann't be scaned. the regionserver log have many exception like this: 2014-10-13 19:33:05,287 INFO [A01101303447-V1,60020,1413199890407-recovery-writer--pool4-t3] client.AsyncProcess: #4, waiting for some tasks to finish. Expected max=0, tasksSent=9, tasksDone=8, currentTasksDone=8, retries=8 hasError=fal se, tableName=IDX_DATE_HOUR_X 2014-10-13 19:33:05,298 INFO [A01101303447-V1,60020,1413199890407-recovery-writer--pool4-t2] client.AsyncProcess: #5, waiting for some tasks to finish. Expected max=0, tasksSent=9, tasksDone=8, currentTasksDone=8, retries=8 hasError=fal se, tableName=IDX_DATE_HOUR_X 2014-10-13 19:33:05,311 INFO [A01101303447-V1,60020,1413199890407-recovery-writer--pool4-t1] client.AsyncProcess: #6, waiting for some tasks to finish. Expected max=0, tasksSent=9, tasksDone=8, currentTasksDone=8, retries=8 hasError=fal se, tableName=IDX_DATE_HOUR_X 2014-10-13 19:33:06,452 INFO [ReplicationExecutor-0] replication.ReplicationQueuesZKImpl: Moving A01101303447-V1,60020,1413199414409's hlogs to my queue 2014-10-13 19:33:15,325 INFO [A01101303447-V1,60020,1413199890407-recovery-writer--pool4-t1] client.AsyncProcess: #6, waiting for some tasks to finish. Expected max=0, tasksSent=10, tasksDone=9, currentTasksDone=9, retries=9 hasError=fa lse, tableName=IDX_DATE_HOUR_X 2014-10-13 19:33:15,333 INFO [htable-pool6-t2] client.AsyncProcess: #6, table=IDX_DATE_HOUR_X, attempt=10/350 failed 12 ops, last exception: org.apache.hadoop.hbase.exceptions.RegionOpeningException: org.apache.hadoop.hbase.exceptions.R egionOpeningException: Region IDX_DATE_HOUR_X,t\x00\x00\x00\x00\x00,1413186874829.9a92abb84768b129df3faedb877f7bea. is opening on A01101303447-V1,60020,1413199890407 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2759) at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:4213) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3437) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29593) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:744) --- After two days's try, i found that: If idisable 'EVENT', other tables can be scanned, then i enable 'EVENT' manually, the region log show that NullPointExceptin has occur then replaying WAL, the following is log: 2014-10-13 19:25:21,043 INFO [RS_OPEN_REGION-A01101303447-V1:60020-1] regionserver.HRegion: Replaying edits from hdfs://localhost/hbase-0.98/data/default/EVENT/def4a581d4ad963cbb8cad32cbfbab2e/recovered.edits/002 2014-10-13 19:25:21,048 INFO [A01101303447-V1,60020,1413199414409-recovery-writer--pool17-t2-SendThread(localhost:2182)] zookeeper.ClientCnxn: Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2182. Will not attempt to authe nticate using SASL (unknown error) 2014-10-13 19:25:21,049 INFO [A01101303447-V1,60020,1413199414409-recovery-writer--pool17-t2-SendThread(localhost:2182)] zookeeper.ClientCnxn: Socket connection established to localhost/0:0:0:0:0:0:0:1:2182, initiating session 2014-10-13 19:25:21,051 INFO [A01101303447-V1,60020,1413199414409-recovery-writer--pool17-t3] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x199c8484 connecting to ZooKeeper ensemble=localhost:2182 2014-10-13 19:25:21,051 INFO [A01101303447-V1,60020,1413199414409-recovery-writer--pool17-t3] zookeeper.ZooKeeper: Initiating client connection,
[jira] [Updated] (HBASE-12255) NPE in OpenRegionHandler after restart hdfs without stop hbase
[ https://issues.apache.org/jira/browse/HBASE-12255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-12255: --- Summary: NPE in OpenRegionHandler after restart hdfs without stop hbase (was: hbase table is unavailable after restart hdfs without stop hbase) NPE in OpenRegionHandler after restart hdfs without stop hbase -- Key: HBASE-12255 URL: https://issues.apache.org/jira/browse/HBASE-12255 Project: HBase Issue Type: Bug Environment: hadoop-2.5.1 hbase-0.98 phonenix-4.1.0 Reporter: Junyong Li I have a phoenix table 'EVENT', and the table have a index 'IDX_DATE_HOUR_X'. I restarted hdfs without stop hbase, after then, the hbase table cann't be scaned. I try to restart hbase, the all hbase table still cann't be scaned. the regionserver log have many exception like this: 2014-10-13 19:33:05,287 INFO [A01101303447-V1,60020,1413199890407-recovery-writer--pool4-t3] client.AsyncProcess: #4, waiting for some tasks to finish. Expected max=0, tasksSent=9, tasksDone=8, currentTasksDone=8, retries=8 hasError=fal se, tableName=IDX_DATE_HOUR_X 2014-10-13 19:33:05,298 INFO [A01101303447-V1,60020,1413199890407-recovery-writer--pool4-t2] client.AsyncProcess: #5, waiting for some tasks to finish. Expected max=0, tasksSent=9, tasksDone=8, currentTasksDone=8, retries=8 hasError=fal se, tableName=IDX_DATE_HOUR_X 2014-10-13 19:33:05,311 INFO [A01101303447-V1,60020,1413199890407-recovery-writer--pool4-t1] client.AsyncProcess: #6, waiting for some tasks to finish. Expected max=0, tasksSent=9, tasksDone=8, currentTasksDone=8, retries=8 hasError=fal se, tableName=IDX_DATE_HOUR_X 2014-10-13 19:33:06,452 INFO [ReplicationExecutor-0] replication.ReplicationQueuesZKImpl: Moving A01101303447-V1,60020,1413199414409's hlogs to my queue 2014-10-13 19:33:15,325 INFO [A01101303447-V1,60020,1413199890407-recovery-writer--pool4-t1] client.AsyncProcess: #6, waiting for some tasks to finish. Expected max=0, tasksSent=10, tasksDone=9, currentTasksDone=9, retries=9 hasError=fa lse, tableName=IDX_DATE_HOUR_X 2014-10-13 19:33:15,333 INFO [htable-pool6-t2] client.AsyncProcess: #6, table=IDX_DATE_HOUR_X, attempt=10/350 failed 12 ops, last exception: org.apache.hadoop.hbase.exceptions.RegionOpeningException: org.apache.hadoop.hbase.exceptions.R egionOpeningException: Region IDX_DATE_HOUR_X,t\x00\x00\x00\x00\x00,1413186874829.9a92abb84768b129df3faedb877f7bea. is opening on A01101303447-V1,60020,1413199890407 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2759) at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:4213) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3437) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29593) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:744) --- After two days's try, i found that: If idisable 'EVENT', other tables can be scanned, then i enable 'EVENT' manually, the region log show that NullPointExceptin has occur then replaying WAL, the following is log: 2014-10-13 19:25:21,043 INFO [RS_OPEN_REGION-A01101303447-V1:60020-1] regionserver.HRegion: Replaying edits from hdfs://localhost/hbase-0.98/data/default/EVENT/def4a581d4ad963cbb8cad32cbfbab2e/recovered.edits/002 2014-10-13 19:25:21,048 INFO [A01101303447-V1,60020,1413199414409-recovery-writer--pool17-t2-SendThread(localhost:2182)] zookeeper.ClientCnxn: Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2182. Will not attempt to authe nticate using SASL (unknown error) 2014-10-13 19:25:21,049 INFO [A01101303447-V1,60020,1413199414409-recovery-writer--pool17-t2-SendThread(localhost:2182)] zookeeper.ClientCnxn: Socket connection established to localhost/0:0:0:0:0:0:0:1:2182, initiating session 2014-10-13 19:25:21,051 INFO [A01101303447-V1,60020,1413199414409-recovery-writer--pool17-t3] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x199c8484 connecting to ZooKeeper ensemble=localhost:2182 2014-10-13 19:25:21,051 INFO [A01101303447-V1,60020,1413199414409-recovery-writer--pool17-t3] zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2182 sessionTimeout=9 watcher=hconnection-0x199c8484, quorum=lo
[jira] [Commented] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171068#comment-14171068 ] Andrew Purtell commented on HBASE-11992: bq. Tested upgrade scenarios before and after the patch. So you started two clusters, one master and one RS each? That won't test exactly the scenario of concern. Would it be possible for you to test again with two clusters where at least two RSes are replicating in each? Only upgrade the RSes, one at a time. Backport HBASE-11367 (Pluggable replication endpoint) to 0.98 - Key: HBASE-11992 URL: https://issues.apache.org/jira/browse/HBASE-11992 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Attachments: HBASE-11992_0.98_1.patch, hbase-11367_0.98.patch ReplicationSource tails the logs for each peer. HBASE-11367 introduces ReplicationEndpoint which is customizable per peer. ReplicationEndpoint is run in the same RS process and instantiated per replication peer per region server. Implementations of this interface handle the actual shipping of WAL edits to the remote cluster. This issue is for backporting HBASE-11367 to 0.98. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12254) Document limitations related to pluggable replication endpoint feature usage in 0.98
[ https://issues.apache.org/jira/browse/HBASE-12254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171070#comment-14171070 ] Andrew Purtell commented on HBASE-12254: Sounds good. This would need to be committed before we could release a version with HBASE-11992 in it. Document limitations related to pluggable replication endpoint feature usage in 0.98 Key: HBASE-12254 URL: https://issues.apache.org/jira/browse/HBASE-12254 Project: HBase Issue Type: Sub-task Components: documentation Affects Versions: 0.98.7 Reporter: ramkrishna.s.vasudevan Fix For: 0.98.8 The pluggable Replication endpoint in 0.98 will need to be documented as how exactly it can be used because of limitations that we may have due to mixed version compatability where the peers may be in an older version of 0.98 where pluggable replication endpoint is not there. Also this feature adds some more data to the znodes like the name of the Endpoint impl, data and the Replication config. A peer cluster with the older version will not be able to read this data particularly when there is a custom replication configured. This JIRA aims at documenting such cases for the ease of user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12147) Porting Online Config Change from 89-fb
[ https://issues.apache.org/jira/browse/HBASE-12147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171102#comment-14171102 ] stack commented on HBASE-12147: --- Reverted from branch-1. I applied it wrong and broke branch-1 build. Let me try and figure what I did amiss. Porting Online Config Change from 89-fb --- Key: HBASE-12147 URL: https://issues.apache.org/jira/browse/HBASE-12147 Project: HBase Issue Type: Sub-task Affects Versions: 1.0.0 Reporter: Manukranth Kolloju Assignee: Manukranth Kolloju Fix For: 0.99.2 Attachments: 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch Original Estimate: 168h Remaining Estimate: 168h This jira is to track the forward port of HBASE-8805 and HBASE-8544 implemented by [~gaurav.menghani] in 89-fb. This improves operational efficiency in managing clusters that are serving production traffic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12256) Update patch submission guidelines to call out binary file support
Sean Busbey created HBASE-12256:
---
Summary: Update patch submission guidelines to call out binary
file support
Key: HBASE-12256
URL: https://issues.apache.org/jira/browse/HBASE-12256
Project: HBase
Issue Type: Improvement
Components: documentation
Reporter: Sean Busbey
Priority: Minor
Our guidelines should call out where users need to take special care if they
have binary files in their diff. (binary files here means anything {{file}}
detects as data)
By default {{git diff}} won't include binary files, instead it'll just say
something like binary files differ. That'll render the patch useless.
* Update reasons to favor format-patch to include can handle binary files
changing by default.
* Add a note on {{git diff}} (or change the default args) to use {{--binary}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12147) Porting Online Config Change from 89-fb
[ https://issues.apache.org/jira/browse/HBASE-12147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171107#comment-14171107 ] stack commented on HBASE-12147: --- Yeah, I had to revert. CompoundConfguration is missing addBytesMap When I try to fix it up, it has dependency on Bytes class -- there is stuff in Bytes that does not seem to have been backported. Any chance of looking at this [~manukranthk] else I'll dig in later t his morning Thanks. Porting Online Config Change from 89-fb --- Key: HBASE-12147 URL: https://issues.apache.org/jira/browse/HBASE-12147 Project: HBase Issue Type: Sub-task Affects Versions: 1.0.0 Reporter: Manukranth Kolloju Assignee: Manukranth Kolloju Fix For: 0.99.2 Attachments: 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch Original Estimate: 168h Remaining Estimate: 168h This jira is to track the forward port of HBASE-8805 and HBASE-8544 implemented by [~gaurav.menghani] in 89-fb. This improves operational efficiency in managing clusters that are serving production traffic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171162#comment-14171162 ] ramkrishna.s.vasudevan commented on HBASE-11992: bq.Would it be possible for you to test again with two clusters where at least two RSes are replicating in each? For now I don't have such a set up with me [~apurtell]. I will see what can I do. Backport HBASE-11367 (Pluggable replication endpoint) to 0.98 - Key: HBASE-11992 URL: https://issues.apache.org/jira/browse/HBASE-11992 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Attachments: HBASE-11992_0.98_1.patch, hbase-11367_0.98.patch ReplicationSource tails the logs for each peer. HBASE-11367 introduces ReplicationEndpoint which is customizable per peer. ReplicationEndpoint is run in the same RS process and instantiated per replication peer per region server. Implementations of this interface handle the actual shipping of WAL edits to the remote cluster. This issue is for backporting HBASE-11367 to 0.98. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12147) Porting Online Config Change from 89-fb
[ https://issues.apache.org/jira/browse/HBASE-12147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171185#comment-14171185 ] Manukranth Kolloju commented on HBASE-12147: Let me take a look in a half hour. Porting Online Config Change from 89-fb --- Key: HBASE-12147 URL: https://issues.apache.org/jira/browse/HBASE-12147 Project: HBase Issue Type: Sub-task Affects Versions: 1.0.0 Reporter: Manukranth Kolloju Assignee: Manukranth Kolloju Fix For: 0.99.2 Attachments: 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch Original Estimate: 168h Remaining Estimate: 168h This jira is to track the forward port of HBASE-8805 and HBASE-8544 implemented by [~gaurav.menghani] in 89-fb. This improves operational efficiency in managing clusters that are serving production traffic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12250) Adding an endpoint for updating the regionserver config
[ https://issues.apache.org/jira/browse/HBASE-12250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manukranth Kolloju updated HBASE-12250: --- Attachment: 0001-Add-admin-endpoint-for-updating-the-configuration-on.patch Forgot to rebase. Adding an endpoint for updating the regionserver config --- Key: HBASE-12250 URL: https://issues.apache.org/jira/browse/HBASE-12250 Project: HBase Issue Type: Task Components: regionserver Affects Versions: 2.0.0 Reporter: Manukranth Kolloju Assignee: Manukranth Kolloju Priority: Minor Fix For: 2.0.0 Attachments: 0001-Add-admin-endpoint-for-updating-the-configuration-on.patch, 0001-Add-admin-endpoint-for-updating-the-configuration-on.patch Original Estimate: 48h Remaining Estimate: 48h This is a follow up Jira that adds the end point for updating the configuration on the regionserver. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12250) Adding an endpoint for updating the regionserver config
[
https://issues.apache.org/jira/browse/HBASE-12250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171194#comment-14171194
]
Hadoop QA commented on HBASE-12250:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12674793/0001-Add-admin-endpoint-for-updating-the-configuration-on.patch
against trunk revision .
ATTACHMENT ID: 12674793
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11339//console
This message is automatically generated.
Adding an endpoint for updating the regionserver config
---
Key: HBASE-12250
URL: https://issues.apache.org/jira/browse/HBASE-12250
Project: HBase
Issue Type: Task
Components: regionserver
Affects Versions: 2.0.0
Reporter: Manukranth Kolloju
Assignee: Manukranth Kolloju
Priority: Minor
Fix For: 2.0.0
Attachments:
0001-Add-admin-endpoint-for-updating-the-configuration-on.patch,
0001-Add-admin-endpoint-for-updating-the-configuration-on.patch
Original Estimate: 48h
Remaining Estimate: 48h
This is a follow up Jira that adds the end point for updating the
configuration on the regionserver.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-10201) Port 'Make flush decisions per column family' to trunk
[ https://issues.apache.org/jira/browse/HBASE-10201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171204#comment-14171204 ] Ted Yu commented on HBASE-10201: There're calls to System.out.println() in the test, can you switch to calling LOG ? Port 'Make flush decisions per column family' to trunk -- Key: HBASE-10201 URL: https://issues.apache.org/jira/browse/HBASE-10201 Project: HBase Issue Type: Improvement Reporter: Ted Yu Attachments: 3149-trunk-v1.txt, HBASE-10201-0.98.patch, HBASE-10201-0.98_1.patch Currently the flush decision is made using the aggregate size of all column families. When large and small column families co-exist, this causes many small flushes of the smaller CF. We need to make per-CF flush decisions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12250) Adding an endpoint for updating the regionserver config
[ https://issues.apache.org/jira/browse/HBASE-12250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171263#comment-14171263 ] Ted Yu commented on HBASE-12250: Any chance of putting patch on Phabricator ? Adding an endpoint for updating the regionserver config --- Key: HBASE-12250 URL: https://issues.apache.org/jira/browse/HBASE-12250 Project: HBase Issue Type: Task Components: regionserver Affects Versions: 2.0.0 Reporter: Manukranth Kolloju Assignee: Manukranth Kolloju Priority: Minor Fix For: 2.0.0 Attachments: 0001-Add-admin-endpoint-for-updating-the-configuration-on.patch, 0001-Add-admin-endpoint-for-updating-the-configuration-on.patch Original Estimate: 48h Remaining Estimate: 48h This is a follow up Jira that adds the end point for updating the configuration on the regionserver. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12250) Adding an endpoint for updating the regionserver config
[ https://issues.apache.org/jira/browse/HBASE-12250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171271#comment-14171271 ] Manukranth Kolloju commented on HBASE-12250: Our internal mirror was out of date. Will put up the up to date patch and yeah, will put up the diff too. Adding an endpoint for updating the regionserver config --- Key: HBASE-12250 URL: https://issues.apache.org/jira/browse/HBASE-12250 Project: HBase Issue Type: Task Components: regionserver Affects Versions: 2.0.0 Reporter: Manukranth Kolloju Assignee: Manukranth Kolloju Priority: Minor Fix For: 2.0.0 Attachments: 0001-Add-admin-endpoint-for-updating-the-configuration-on.patch, 0001-Add-admin-endpoint-for-updating-the-configuration-on.patch Original Estimate: 48h Remaining Estimate: 48h This is a follow up Jira that adds the end point for updating the configuration on the regionserver. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12250) Adding an endpoint for updating the regionserver config
[
https://issues.apache.org/jira/browse/HBASE-12250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171270#comment-14171270
]
Ted Yu commented on HBASE-12250:
{code}
+admin.updateConfiguration(server);
{code}
Should there be some assertion around the above invocation ?
{code}
+ public UpdateConfigurationResponse updateConfiguration(
+ RpcController controller, UpdateConfigurationRequest request)
+ throws ServiceException {
+// TODO Auto-generated method stub
{code}
Remove TODO above.
{code}
+conf.reloadConfiguration();
+// TODO Notify all the observers that the configuration has changed.
{code}
Would the TODO be done in the next patch ?
Adding an endpoint for updating the regionserver config
---
Key: HBASE-12250
URL: https://issues.apache.org/jira/browse/HBASE-12250
Project: HBase
Issue Type: Task
Components: regionserver
Affects Versions: 2.0.0
Reporter: Manukranth Kolloju
Assignee: Manukranth Kolloju
Priority: Minor
Fix For: 2.0.0
Attachments:
0001-Add-admin-endpoint-for-updating-the-configuration-on.patch,
0001-Add-admin-endpoint-for-updating-the-configuration-on.patch
Original Estimate: 48h
Remaining Estimate: 48h
This is a follow up Jira that adds the end point for updating the
configuration on the regionserver.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12250) Adding an endpoint for updating the regionserver config
[ https://issues.apache.org/jira/browse/HBASE-12250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171285#comment-14171285 ] Manukranth Kolloju commented on HBASE-12250: Here is the diff : https://reviews.facebook.net/D24861 Since admin.updateConfiguration(server); was a void endpoint, as long as we don't have an exception I thought that it was good enough. Adding an endpoint for updating the regionserver config --- Key: HBASE-12250 URL: https://issues.apache.org/jira/browse/HBASE-12250 Project: HBase Issue Type: Task Components: regionserver Affects Versions: 2.0.0 Reporter: Manukranth Kolloju Assignee: Manukranth Kolloju Priority: Minor Fix For: 2.0.0 Attachments: 0001-Add-admin-endpoint-for-updating-the-configuration-on.patch, 0001-Add-admin-endpoint-for-updating-the-configuration-on.patch Original Estimate: 48h Remaining Estimate: 48h This is a follow up Jira that adds the end point for updating the configuration on the regionserver. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12247) Use a user managed connection in TableInputFormat
[ https://issues.apache.org/jira/browse/HBASE-12247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171287#comment-14171287 ] stack commented on HBASE-12247: --- bq. I tried using some of the unit tests and a debugger to see where close() is called. I couldn't seem to get a hit. In 0.94 code line? Or in trunk? Yeah, for a managed connection, no opportunity to 'close'. Will just 'close' when the jvm exits. Make a note this is what is going on. Anyone else have better idea? Use a user managed connection in TableInputFormat - Key: HBASE-12247 URL: https://issues.apache.org/jira/browse/HBASE-12247 Project: HBase Issue Type: Bug Affects Versions: 2.0.0, 0.99.2 Reporter: Solomon Duskis Assignee: Solomon Duskis TableInputFormat uses a new HTable() that manages its own connection. Switch that use to a manually created ConnectionFactory and then close it at the end of the operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12243) HBaseFsck should auto set ignorePreCheckPermission to true if no fix option is set.
[ https://issues.apache.org/jira/browse/HBASE-12243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171296#comment-14171296 ] Yi Deng commented on HBASE-12243: - Didn't find a Javadoc warning related to my diff. HBaseFsck should auto set ignorePreCheckPermission to true if no fix option is set. --- Key: HBASE-12243 URL: https://issues.apache.org/jira/browse/HBASE-12243 Project: HBase Issue Type: Improvement Components: hbck Affects Versions: 1.0.0, 2.0.0, 0.98.6.1 Reporter: Yi Deng Assignee: Yi Deng Priority: Minor Labels: hbck Fix For: 2.0.0 Attachments: 0001-HBASE-12243-HBaseFsck-should-auto-set-ignorePreCheck.patch It does not make sense for HBaseFsck to require WRITE access if no fix option is specifiy. This will be much convinient for operation script to write. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12247) Use a user managed connection in TableInputFormat
[ https://issues.apache.org/jira/browse/HBASE-12247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171301#comment-14171301 ] Solomon Duskis commented on HBASE-12247: I tried the debugger in trunk. I'll try it in 0.94. Use a user managed connection in TableInputFormat - Key: HBASE-12247 URL: https://issues.apache.org/jira/browse/HBASE-12247 Project: HBase Issue Type: Bug Affects Versions: 2.0.0, 0.99.2 Reporter: Solomon Duskis Assignee: Solomon Duskis TableInputFormat uses a new HTable() that manages its own connection. Switch that use to a manually created ConnectionFactory and then close it at the end of the operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-4955) Use the official versions of surefire junit
[ https://issues.apache.org/jira/browse/HBASE-4955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-4955: - Fix Version/s: 0.99.2 Use the official versions of surefire junit - Key: HBASE-4955 URL: https://issues.apache.org/jira/browse/HBASE-4955 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0, 0.98.0, 0.96.0, 0.99.0 Environment: all Reporter: Nicolas Liochon Assignee: Alex Newman Priority: Critical Fix For: 2.0.0, 0.99.2 Attachments: 4955.v1.patch, 4955.v2.patch, 4955.v2.patch, 4955.v2.patch, 4955.v2.patch, 4955.v3.patch, 4955.v3.patch, 4955.v3.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v5.patch, 4955.v6.patch, 4955.v7.patch, 4955.v7.patch, 4955.v8.patch, 4955.v9.patch, 8204.v4.patch, HBASE-4955-v10.patch, HBASE-4955-v10.patch, HBASE-4955-v10.patch We currently use private versions for Surefire JUnit since HBASE-4763. This JIRA traks what we need to move to official versions. Surefire 2.11 is just out, but, after some tests, it does not contain all what we need. JUnit. Could be for JUnit 4.11. Issue to monitor: https://github.com/KentBeck/junit/issues/359: fixed in our version, no feedback for an integration on trunk Surefire: Could be for Surefire 2.12. Issues to monitor are: 329 (category support): fixed, we use the official implementation from the trunk 786 (@Category with forkMode=always): fixed, we use the official implementation from the trunk 791 (incorrect elapsed time on test failure): fixed, we use the official implementation from the trunk 793 (incorrect time in the XML report): Not fixed (reopen) on trunk, fixed on our version. 760 (does not take into account the test method): fixed in trunk, not fixed in our version 798 (print immediately the test class name): not fixed in trunk, not fixed in our version 799 (Allow test parallelization when forkMode=always): not fixed in trunk, not fixed in our version 800 (redirectTestOutputToFile not taken into account): not yet fix on trunk, fixed on our version 800 793 are the more important to monitor, it's the only ones that are fixed in our version but not on trunk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-4955) Use the official versions of surefire junit
[ https://issues.apache.org/jira/browse/HBASE-4955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171327#comment-14171327 ] Elliott Clark commented on HBASE-4955: -- ok pulling this into branch-1. We've tested this internally enough that I feel confident pulling this to branch-1. Use the official versions of surefire junit - Key: HBASE-4955 URL: https://issues.apache.org/jira/browse/HBASE-4955 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0, 0.98.0, 0.96.0, 0.99.0 Environment: all Reporter: Nicolas Liochon Assignee: Alex Newman Priority: Critical Fix For: 2.0.0, 0.99.2 Attachments: 4955.v1.patch, 4955.v2.patch, 4955.v2.patch, 4955.v2.patch, 4955.v2.patch, 4955.v3.patch, 4955.v3.patch, 4955.v3.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v5.patch, 4955.v6.patch, 4955.v7.patch, 4955.v7.patch, 4955.v8.patch, 4955.v9.patch, 8204.v4.patch, HBASE-4955-v10.patch, HBASE-4955-v10.patch, HBASE-4955-v10.patch We currently use private versions for Surefire JUnit since HBASE-4763. This JIRA traks what we need to move to official versions. Surefire 2.11 is just out, but, after some tests, it does not contain all what we need. JUnit. Could be for JUnit 4.11. Issue to monitor: https://github.com/KentBeck/junit/issues/359: fixed in our version, no feedback for an integration on trunk Surefire: Could be for Surefire 2.12. Issues to monitor are: 329 (category support): fixed, we use the official implementation from the trunk 786 (@Category with forkMode=always): fixed, we use the official implementation from the trunk 791 (incorrect elapsed time on test failure): fixed, we use the official implementation from the trunk 793 (incorrect time in the XML report): Not fixed (reopen) on trunk, fixed on our version. 760 (does not take into account the test method): fixed in trunk, not fixed in our version 798 (print immediately the test class name): not fixed in trunk, not fixed in our version 799 (Allow test parallelization when forkMode=always): not fixed in trunk, not fixed in our version 800 (redirectTestOutputToFile not taken into account): not yet fix on trunk, fixed on our version 800 793 are the more important to monitor, it's the only ones that are fixed in our version but not on trunk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-4955) Use the official versions of surefire junit
[ https://issues.apache.org/jira/browse/HBASE-4955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171345#comment-14171345 ] stack commented on HBASE-4955: -- Go for it [~eclark] Use the official versions of surefire junit - Key: HBASE-4955 URL: https://issues.apache.org/jira/browse/HBASE-4955 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0, 0.98.0, 0.96.0, 0.99.0 Environment: all Reporter: Nicolas Liochon Assignee: Alex Newman Priority: Critical Fix For: 2.0.0, 0.99.2 Attachments: 4955.v1.patch, 4955.v2.patch, 4955.v2.patch, 4955.v2.patch, 4955.v2.patch, 4955.v3.patch, 4955.v3.patch, 4955.v3.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v5.patch, 4955.v6.patch, 4955.v7.patch, 4955.v7.patch, 4955.v8.patch, 4955.v9.patch, 8204.v4.patch, HBASE-4955-v10.patch, HBASE-4955-v10.patch, HBASE-4955-v10.patch We currently use private versions for Surefire JUnit since HBASE-4763. This JIRA traks what we need to move to official versions. Surefire 2.11 is just out, but, after some tests, it does not contain all what we need. JUnit. Could be for JUnit 4.11. Issue to monitor: https://github.com/KentBeck/junit/issues/359: fixed in our version, no feedback for an integration on trunk Surefire: Could be for Surefire 2.12. Issues to monitor are: 329 (category support): fixed, we use the official implementation from the trunk 786 (@Category with forkMode=always): fixed, we use the official implementation from the trunk 791 (incorrect elapsed time on test failure): fixed, we use the official implementation from the trunk 793 (incorrect time in the XML report): Not fixed (reopen) on trunk, fixed on our version. 760 (does not take into account the test method): fixed in trunk, not fixed in our version 798 (print immediately the test class name): not fixed in trunk, not fixed in our version 799 (Allow test parallelization when forkMode=always): not fixed in trunk, not fixed in our version 800 (redirectTestOutputToFile not taken into account): not yet fix on trunk, fixed on our version 800 793 are the more important to monitor, it's the only ones that are fixed in our version but not on trunk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12247) Use a user managed connection in TableInputFormat
[ https://issues.apache.org/jira/browse/HBASE-12247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171356#comment-14171356 ] stack commented on HBASE-12247: --- [~sduskis] i'd say that even in 0.93, the JVM just exited before the close was called. Don't spend much time on it boss. Use a user managed connection in TableInputFormat - Key: HBASE-12247 URL: https://issues.apache.org/jira/browse/HBASE-12247 Project: HBase Issue Type: Bug Affects Versions: 2.0.0, 0.99.2 Reporter: Solomon Duskis Assignee: Solomon Duskis TableInputFormat uses a new HTable() that manages its own connection. Switch that use to a manually created ConnectionFactory and then close it at the end of the operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12147) Porting Online Config Change from 89-fb
[ https://issues.apache.org/jira/browse/HBASE-12147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171359#comment-14171359 ] Hudson commented on HBASE-12147: SUCCESS: Integrated in HBase-1.0 #311 (See [https://builds.apache.org/job/HBase-1.0/311/]) HBASE-12147 Porting Online Config Change from 89-fb -- JAVADOC FIX ADDENDUM -- REVERT (stack: rev 0ebb98d7b38337d226d1d210204c965acddf93be) * hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java HBASE-12147 Porting Online Config Change from 89-fb -- REVERT (stack: rev 2822a289f9bd59156fb5ea8305da26944fe5152e) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/test/java/org/apache/hadoop/hbase/conf/TestConfigurationManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java * hbase-server/src/main/java/org/apache/hadoop/hbase/conf/PropagatingConfigurationObserver.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerOnlineConfigChange.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionConfiguration.java * hbase-server/src/main/java/org/apache/hadoop/hbase/conf/ConfigurationManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/conf/ConfigurationObserver.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/OffPeakHours.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionPolicy.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestOffPeakHours.java Porting Online Config Change from 89-fb --- Key: HBASE-12147 URL: https://issues.apache.org/jira/browse/HBASE-12147 Project: HBase Issue Type: Sub-task Affects Versions: 1.0.0 Reporter: Manukranth Kolloju Assignee: Manukranth Kolloju Fix For: 0.99.2 Attachments: 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch Original Estimate: 168h Remaining Estimate: 168h This jira is to track the forward port of HBASE-8805 and HBASE-8544 implemented by [~gaurav.menghani] in 89-fb. This improves operational efficiency in managing clusters that are serving production traffic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12147) Porting Online Config Change from 89-fb
[ https://issues.apache.org/jira/browse/HBASE-12147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171364#comment-14171364 ] Manukranth Kolloju commented on HBASE-12147: There was a change in the Bytes class to remove Writables and replace them with bytes. And as part of that some function names were changed as well. Hence this merge conflict. This is the change needed to fix the patch : https://reviews.facebook.net/F223347. I will attach the patch for branch-1 Porting Online Config Change from 89-fb --- Key: HBASE-12147 URL: https://issues.apache.org/jira/browse/HBASE-12147 Project: HBase Issue Type: Sub-task Affects Versions: 1.0.0 Reporter: Manukranth Kolloju Assignee: Manukranth Kolloju Fix For: 0.99.2 Attachments: 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch Original Estimate: 168h Remaining Estimate: 168h This jira is to track the forward port of HBASE-8805 and HBASE-8544 implemented by [~gaurav.menghani] in 89-fb. This improves operational efficiency in managing clusters that are serving production traffic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12252) IntegrationTestBulkLoad fails with illegal partition error
[
https://issues.apache.org/jira/browse/HBASE-12252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dima Spivak updated HBASE-12252:
Attachment: HBASE-12252.patch
Simple fix; ran IntegrationTestBulkLoad successfully for 40 minutes after
applying.
IntegrationTestBulkLoad fails with illegal partition error
--
Key: HBASE-12252
URL: https://issues.apache.org/jira/browse/HBASE-12252
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.98.6
Reporter: Dima Spivak
Assignee: Dima Spivak
Attachments: HBASE-12252.patch
While running {{IntegrationTestBulkLoad}} for a while, I encountered the
following error:
{code}
14/10/14 05:57:12 INFO mapreduce.IntegrationTestBulkLoad: Running check
14/10/14 05:57:12 INFO client.RMProxy: Connecting to ResourceManager at
node-1.internal/10.0.0.3:8032
14/10/14 05:57:16 INFO util.RegionSizeCalculator: Calculating region sizes
for table IntegrationTestBulkLoad.
14/10/14 05:57:16 INFO mapreduce.JobSubmitter: number of splits:16
14/10/14 05:57:17 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1413261370136_0012
14/10/14 05:57:17 INFO impl.YarnClientImpl: Submitted application
application_1413261370136_0012
14/10/14 05:57:17 INFO mapreduce.Job: The url to track the job:
http://node-1.internal:8088/proxy/application_1413261370136_0012/
14/10/14 05:57:17 INFO mapreduce.Job: Running job: job_1413261370136_0012
14/10/14 05:57:32 INFO mapreduce.Job: Job job_1413261370136_0012 running in
uber mode : false
14/10/14 05:57:32 INFO mapreduce.Job: map 0% reduce 0%
14/10/14 05:57:52 INFO mapreduce.Job: map 6% reduce 0%
14/10/14 05:57:53 INFO mapreduce.Job: Task Id :
attempt_1413261370136_0012_m_13_0, Status : FAILED
Error: java.io.IOException: Illegal partition for
org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad$LinkKey@133c4539
(-3)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1079)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:712)
at
org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
at
org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
at
org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad$LinkedListCheckingMapper.map(IntegrationTestBulkLoad.java:502)
at
org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad$LinkedListCheckingMapper.map(IntegrationTestBulkLoad.java:491)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
{code}
I'm pretty sure I've tracked this down to a partition generator that can
return a negative number; I'll post a fix once I've confirmed locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12252) IntegrationTestBulkLoad fails with illegal partition error
[
https://issues.apache.org/jira/browse/HBASE-12252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dima Spivak updated HBASE-12252:
Status: Patch Available (was: In Progress)
IntegrationTestBulkLoad fails with illegal partition error
--
Key: HBASE-12252
URL: https://issues.apache.org/jira/browse/HBASE-12252
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.98.6
Reporter: Dima Spivak
Assignee: Dima Spivak
Attachments: HBASE-12252.patch
While running {{IntegrationTestBulkLoad}} for a while, I encountered the
following error:
{code}
14/10/14 05:57:12 INFO mapreduce.IntegrationTestBulkLoad: Running check
14/10/14 05:57:12 INFO client.RMProxy: Connecting to ResourceManager at
node-1.internal/10.0.0.3:8032
14/10/14 05:57:16 INFO util.RegionSizeCalculator: Calculating region sizes
for table IntegrationTestBulkLoad.
14/10/14 05:57:16 INFO mapreduce.JobSubmitter: number of splits:16
14/10/14 05:57:17 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1413261370136_0012
14/10/14 05:57:17 INFO impl.YarnClientImpl: Submitted application
application_1413261370136_0012
14/10/14 05:57:17 INFO mapreduce.Job: The url to track the job:
http://node-1.internal:8088/proxy/application_1413261370136_0012/
14/10/14 05:57:17 INFO mapreduce.Job: Running job: job_1413261370136_0012
14/10/14 05:57:32 INFO mapreduce.Job: Job job_1413261370136_0012 running in
uber mode : false
14/10/14 05:57:32 INFO mapreduce.Job: map 0% reduce 0%
14/10/14 05:57:52 INFO mapreduce.Job: map 6% reduce 0%
14/10/14 05:57:53 INFO mapreduce.Job: Task Id :
attempt_1413261370136_0012_m_13_0, Status : FAILED
Error: java.io.IOException: Illegal partition for
org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad$LinkKey@133c4539
(-3)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1079)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:712)
at
org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
at
org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
at
org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad$LinkedListCheckingMapper.map(IntegrationTestBulkLoad.java:502)
at
org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad$LinkedListCheckingMapper.map(IntegrationTestBulkLoad.java:491)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
{code}
I'm pretty sure I've tracked this down to a partition generator that can
return a negative number; I'll post a fix once I've confirmed locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12243) HBaseFsck should auto set ignorePreCheckPermission to true if no fix option is set.
[ https://issues.apache.org/jira/browse/HBASE-12243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171380#comment-14171380 ] stack commented on HBASE-12243: --- +1 Patch LGTM. Javadoc warning not yours [~daviddengcn] Will fix in another issue. Will give [~jmhsieh] a chance to review before committing. HBaseFsck should auto set ignorePreCheckPermission to true if no fix option is set. --- Key: HBASE-12243 URL: https://issues.apache.org/jira/browse/HBASE-12243 Project: HBase Issue Type: Improvement Components: hbck Affects Versions: 1.0.0, 2.0.0, 0.98.6.1 Reporter: Yi Deng Assignee: Yi Deng Priority: Minor Labels: hbck Fix For: 2.0.0 Attachments: 0001-HBASE-12243-HBaseFsck-should-auto-set-ignorePreCheck.patch It does not make sense for HBaseFsck to require WRITE access if no fix option is specifiy. This will be much convinient for operation script to write. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12247) Replace setHTable() with initializeTable() in TableInputFormat.
[ https://issues.apache.org/jira/browse/HBASE-12247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Solomon Duskis updated HBASE-12247: --- Summary: Replace setHTable() with initializeTable() in TableInputFormat. (was: Use a user managed connection in TableInputFormat) Replace setHTable() with initializeTable() in TableInputFormat. --- Key: HBASE-12247 URL: https://issues.apache.org/jira/browse/HBASE-12247 Project: HBase Issue Type: Bug Affects Versions: 2.0.0, 0.99.2 Reporter: Solomon Duskis Assignee: Solomon Duskis TableInputFormat uses a new HTable() that manages its own connection. Switch that use to a manually created ConnectionFactory and then close it at the end of the operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12247) Replace setHTable() with initializeTable() in TableInputFormat.
[ https://issues.apache.org/jira/browse/HBASE-12247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Solomon Duskis updated HBASE-12247: --- Description: TableInputFormat uses a new HTable(). Switch that use to a manually created ConnectionFactory.createConnection() and call initializeTable(). (was: TableInputFormat uses a new HTable() that manages its own connection. Switch that use to a manually created ConnectionFactory and then close it at the end of the operation.) Replace setHTable() with initializeTable() in TableInputFormat. --- Key: HBASE-12247 URL: https://issues.apache.org/jira/browse/HBASE-12247 Project: HBase Issue Type: Bug Affects Versions: 2.0.0, 0.99.2 Reporter: Solomon Duskis Assignee: Solomon Duskis TableInputFormat uses a new HTable(). Switch that use to a manually created ConnectionFactory.createConnection() and call initializeTable(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12247) Replace setHTable() with initializeTable() in TableInputFormat.
[ https://issues.apache.org/jira/browse/HBASE-12247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Solomon Duskis updated HBASE-12247: --- Attachment: HBASE-12247.patch Changed the scope of this issue. This patch simply updates TableInputFormat to use the new interfaces and initializeTable(). Replace setHTable() with initializeTable() in TableInputFormat. --- Key: HBASE-12247 URL: https://issues.apache.org/jira/browse/HBASE-12247 Project: HBase Issue Type: Bug Affects Versions: 2.0.0, 0.99.2 Reporter: Solomon Duskis Assignee: Solomon Duskis Attachments: HBASE-12247.patch TableInputFormat uses a new HTable(). Switch that use to a manually created ConnectionFactory.createConnection() and call initializeTable(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12247) Replace setHTable() with initializeTable() in TableInputFormat.
[ https://issues.apache.org/jira/browse/HBASE-12247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Solomon Duskis updated HBASE-12247: --- Status: Patch Available (was: Open) Replace setHTable() with initializeTable() in TableInputFormat. --- Key: HBASE-12247 URL: https://issues.apache.org/jira/browse/HBASE-12247 Project: HBase Issue Type: Bug Affects Versions: 2.0.0, 0.99.2 Reporter: Solomon Duskis Assignee: Solomon Duskis Attachments: HBASE-12247.patch TableInputFormat uses a new HTable(). Switch that use to a manually created ConnectionFactory.createConnection() and call initializeTable(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12247) Replace setHTable() with initializeTable() in TableInputFormat.
[ https://issues.apache.org/jira/browse/HBASE-12247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171388#comment-14171388 ] stack commented on HBASE-12247: --- This one is a bit odd: * @returns A reference to this instance. If I read it right, this method is a void retrun so this javadoc is not right Otherwise patch is good. Waiting on hadoopqa. If you agree on above, I can fix on commit. Replace setHTable() with initializeTable() in TableInputFormat. --- Key: HBASE-12247 URL: https://issues.apache.org/jira/browse/HBASE-12247 Project: HBase Issue Type: Bug Affects Versions: 2.0.0, 0.99.2 Reporter: Solomon Duskis Assignee: Solomon Duskis Attachments: HBASE-12247.patch TableInputFormat uses a new HTable(). Switch that use to a manually created ConnectionFactory.createConnection() and call initializeTable(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12252) IntegrationTestBulkLoad fails with illegal partition error
[
https://issues.apache.org/jira/browse/HBASE-12252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171389#comment-14171389
]
Elliott Clark commented on HBASE-12252:
---
+1 lgtm
IntegrationTestBulkLoad fails with illegal partition error
--
Key: HBASE-12252
URL: https://issues.apache.org/jira/browse/HBASE-12252
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.98.6
Reporter: Dima Spivak
Assignee: Dima Spivak
Attachments: HBASE-12252.patch
While running {{IntegrationTestBulkLoad}} for a while, I encountered the
following error:
{code}
14/10/14 05:57:12 INFO mapreduce.IntegrationTestBulkLoad: Running check
14/10/14 05:57:12 INFO client.RMProxy: Connecting to ResourceManager at
node-1.internal/10.0.0.3:8032
14/10/14 05:57:16 INFO util.RegionSizeCalculator: Calculating region sizes
for table IntegrationTestBulkLoad.
14/10/14 05:57:16 INFO mapreduce.JobSubmitter: number of splits:16
14/10/14 05:57:17 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1413261370136_0012
14/10/14 05:57:17 INFO impl.YarnClientImpl: Submitted application
application_1413261370136_0012
14/10/14 05:57:17 INFO mapreduce.Job: The url to track the job:
http://node-1.internal:8088/proxy/application_1413261370136_0012/
14/10/14 05:57:17 INFO mapreduce.Job: Running job: job_1413261370136_0012
14/10/14 05:57:32 INFO mapreduce.Job: Job job_1413261370136_0012 running in
uber mode : false
14/10/14 05:57:32 INFO mapreduce.Job: map 0% reduce 0%
14/10/14 05:57:52 INFO mapreduce.Job: map 6% reduce 0%
14/10/14 05:57:53 INFO mapreduce.Job: Task Id :
attempt_1413261370136_0012_m_13_0, Status : FAILED
Error: java.io.IOException: Illegal partition for
org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad$LinkKey@133c4539
(-3)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1079)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:712)
at
org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
at
org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
at
org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad$LinkedListCheckingMapper.map(IntegrationTestBulkLoad.java:502)
at
org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad$LinkedListCheckingMapper.map(IntegrationTestBulkLoad.java:491)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
{code}
I'm pretty sure I've tracked this down to a partition generator that can
return a negative number; I'll post a fix once I've confirmed locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12247) Replace setHTable() with initializeTable() in TableInputFormat.
[ https://issues.apache.org/jira/browse/HBASE-12247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Solomon Duskis updated HBASE-12247: --- Attachment: HBASE-12247_v2.patch Removed the @return comment. Replace setHTable() with initializeTable() in TableInputFormat. --- Key: HBASE-12247 URL: https://issues.apache.org/jira/browse/HBASE-12247 Project: HBase Issue Type: Bug Affects Versions: 2.0.0, 0.99.2 Reporter: Solomon Duskis Assignee: Solomon Duskis Attachments: HBASE-12247.patch, HBASE-12247_v2.patch TableInputFormat uses a new HTable(). Switch that use to a manually created ConnectionFactory.createConnection() and call initializeTable(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12147) Porting Online Config Change from 89-fb
[ https://issues.apache.org/jira/browse/HBASE-12147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manukranth Kolloju updated HBASE-12147: --- Attachment: 0001-Online-config-change-Branch1.patch Patch for branch-1 I also had to fix the test classification for this one. Porting Online Config Change from 89-fb --- Key: HBASE-12147 URL: https://issues.apache.org/jira/browse/HBASE-12147 Project: HBase Issue Type: Sub-task Affects Versions: 1.0.0 Reporter: Manukranth Kolloju Assignee: Manukranth Kolloju Fix For: 0.99.2 Attachments: 0001-Online-config-change-Branch1.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch Original Estimate: 168h Remaining Estimate: 168h This jira is to track the forward port of HBASE-8805 and HBASE-8544 implemented by [~gaurav.menghani] in 89-fb. This improves operational efficiency in managing clusters that are serving production traffic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12147) Porting Online Config Change from 89-fb
[ https://issues.apache.org/jira/browse/HBASE-12147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171400#comment-14171400 ] Manukranth Kolloju commented on HBASE-12147: [~stack], I attached the patch that you've asked. Let me know if that works. I have checked : mvn clean -DskipTests package Porting Online Config Change from 89-fb --- Key: HBASE-12147 URL: https://issues.apache.org/jira/browse/HBASE-12147 Project: HBase Issue Type: Sub-task Affects Versions: 1.0.0 Reporter: Manukranth Kolloju Assignee: Manukranth Kolloju Fix For: 0.99.2 Attachments: 0001-Online-config-change-Branch1.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch Original Estimate: 168h Remaining Estimate: 168h This jira is to track the forward port of HBASE-8805 and HBASE-8544 implemented by [~gaurav.menghani] in 89-fb. This improves operational efficiency in managing clusters that are serving production traffic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12147) Porting Online Config Change from 89-fb
[ https://issues.apache.org/jira/browse/HBASE-12147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manukranth Kolloju updated HBASE-12147: --- Attachment: 0001-Online-config-change-Branch1.patch 0001-Online-config-change-Branch1.patch Porting Online Config Change from 89-fb --- Key: HBASE-12147 URL: https://issues.apache.org/jira/browse/HBASE-12147 Project: HBase Issue Type: Sub-task Affects Versions: 1.0.0 Reporter: Manukranth Kolloju Assignee: Manukranth Kolloju Fix For: 0.99.2 Attachments: 0001-Online-config-change-Branch1.patch, 0001-Online-config-change-Branch1.patch, 0001-Online-config-change-Branch1.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch Original Estimate: 168h Remaining Estimate: 168h This jira is to track the forward port of HBASE-8805 and HBASE-8544 implemented by [~gaurav.menghani] in 89-fb. This improves operational efficiency in managing clusters that are serving production traffic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12147) Porting Online Config Change from 89-fb
[
https://issues.apache.org/jira/browse/HBASE-12147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171433#comment-14171433
]
stack commented on HBASE-12147:
---
Pushed the branch-1 patch and the little javadoc fix again. Thanks
[~manukranthk] for fixup.
Lets revisit using Writables for this stuff even in branch-1 in another issue.
{code}
commit ca26d22b50773db4562752d12ba5eb4fdde59b20
Author: stack st...@apache.org
Date: Mon Oct 13 21:25:51 2014 -0700
HBASE-12147 Porting Online Config Change from 89-fb -- JAVADOC FIX ADDENDUM
commit dbd0ba36f53ce1adeaffc706090c390452f09f46
Author: stack st...@apache.org
Date: Tue Oct 14 12:51:49 2014 -0700
HBASE-12147 Porting Online Config Change from 89-fb -- REAPPLY
{code}
Porting Online Config Change from 89-fb
---
Key: HBASE-12147
URL: https://issues.apache.org/jira/browse/HBASE-12147
Project: HBase
Issue Type: Sub-task
Affects Versions: 1.0.0
Reporter: Manukranth Kolloju
Assignee: Manukranth Kolloju
Fix For: 0.99.2
Attachments: 0001-Online-config-change-Branch1.patch,
0001-Online-config-change-Branch1.patch,
0001-Online-config-change-Branch1.patch, 0001-Online-config-change.patch,
0001-Online-config-change.patch, 0001-Online-config-change.patch,
0001-Online-config-change.patch, 0001-Online-config-change.patch,
0001-Online-config-change.patch
Original Estimate: 168h
Remaining Estimate: 168h
This jira is to track the forward port of HBASE-8805 and HBASE-8544
implemented by [~gaurav.menghani] in 89-fb. This improves operational
efficiency in managing clusters that are serving production traffic.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-12257) TestAssignmentManager unsynchronized access to regionPlans
[
https://issues.apache.org/jira/browse/HBASE-12257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi reassigned HBASE-12257:
---
Assignee: Matteo Bertozzi
TestAssignmentManager unsynchronized access to regionPlans
--
Key: HBASE-12257
URL: https://issues.apache.org/jira/browse/HBASE-12257
Project: HBase
Issue Type: Bug
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
There is some code in TestMasterFailover that access regionPlans without
synchronized
{code}
Stack Trace:
java.lang.NullPointerException
at java.util.TreeMap.rotateLeft(TreeMap.java:2069)
at java.util.TreeMap.fixAfterInsertion(TreeMap.java:2127)
at java.util.TreeMap.put(TreeMap.java:574)
at
org.apache.hadoop.hbase.master.TestMasterFailover.testMasterFailoverWithMockedRITOnDeadRS(TestMasterFailover.java:596)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12257) TestAssignmentManager unsynchronized access to regionPlans
Matteo Bertozzi created HBASE-12257:
---
Summary: TestAssignmentManager unsynchronized access to regionPlans
Key: HBASE-12257
URL: https://issues.apache.org/jira/browse/HBASE-12257
Project: HBase
Issue Type: Bug
Reporter: Matteo Bertozzi
There is some code in TestMasterFailover that access regionPlans without
synchronized
{code}
Stack Trace:
java.lang.NullPointerException
at java.util.TreeMap.rotateLeft(TreeMap.java:2069)
at java.util.TreeMap.fixAfterInsertion(TreeMap.java:2127)
at java.util.TreeMap.put(TreeMap.java:574)
at
org.apache.hadoop.hbase.master.TestMasterFailover.testMasterFailoverWithMockedRITOnDeadRS(TestMasterFailover.java:596)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12257) TestAssignmentManager unsynchronized access to regionPlans
[
https://issues.apache.org/jira/browse/HBASE-12257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi updated HBASE-12257:
Attachment: HBASE-11257-98-v0.patch
TestAssignmentManager unsynchronized access to regionPlans
--
Key: HBASE-12257
URL: https://issues.apache.org/jira/browse/HBASE-12257
Project: HBase
Issue Type: Bug
Components: Region Assignment
Affects Versions: 0.98.7, 0.99.1
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Fix For: 0.98.8, 0.99.2
Attachments: HBASE-11257-98-v0.patch
There is some code in TestMasterFailover that access regionPlans without
synchronized
{code}
Stack Trace:
java.lang.NullPointerException
at java.util.TreeMap.rotateLeft(TreeMap.java:2069)
at java.util.TreeMap.fixAfterInsertion(TreeMap.java:2127)
at java.util.TreeMap.put(TreeMap.java:574)
at
org.apache.hadoop.hbase.master.TestMasterFailover.testMasterFailoverWithMockedRITOnDeadRS(TestMasterFailover.java:596)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12257) TestAssignmentManager unsynchronized access to regionPlans
[
https://issues.apache.org/jira/browse/HBASE-12257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi updated HBASE-12257:
Component/s: Region Assignment
Priority: Trivial (was: Major)
Affects Version/s: 0.98.7
0.99.1
Fix Version/s: 0.99.2
0.98.8
TestAssignmentManager unsynchronized access to regionPlans
--
Key: HBASE-12257
URL: https://issues.apache.org/jira/browse/HBASE-12257
Project: HBase
Issue Type: Bug
Components: Region Assignment
Affects Versions: 0.98.7, 0.99.1
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Trivial
Fix For: 0.98.8, 0.99.2
Attachments: HBASE-11257-98-v0.patch
There is some code in TestMasterFailover that access regionPlans without
synchronized
{code}
Stack Trace:
java.lang.NullPointerException
at java.util.TreeMap.rotateLeft(TreeMap.java:2069)
at java.util.TreeMap.fixAfterInsertion(TreeMap.java:2127)
at java.util.TreeMap.put(TreeMap.java:574)
at
org.apache.hadoop.hbase.master.TestMasterFailover.testMasterFailoverWithMockedRITOnDeadRS(TestMasterFailover.java:596)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12257) TestAssignmentManager unsynchronized access to regionPlans
[
https://issues.apache.org/jira/browse/HBASE-12257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171472#comment-14171472
]
stack commented on HBASE-12257:
---
+1
TestAssignmentManager unsynchronized access to regionPlans
--
Key: HBASE-12257
URL: https://issues.apache.org/jira/browse/HBASE-12257
Project: HBase
Issue Type: Bug
Components: Region Assignment
Affects Versions: 0.98.7, 0.99.1
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Trivial
Fix For: 0.98.8, 0.99.2
Attachments: HBASE-11257-98-v0.patch
There is some code in TestMasterFailover that access regionPlans without
synchronized
{code}
Stack Trace:
java.lang.NullPointerException
at java.util.TreeMap.rotateLeft(TreeMap.java:2069)
at java.util.TreeMap.fixAfterInsertion(TreeMap.java:2127)
at java.util.TreeMap.put(TreeMap.java:574)
at
org.apache.hadoop.hbase.master.TestMasterFailover.testMasterFailoverWithMockedRITOnDeadRS(TestMasterFailover.java:596)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12075) Preemptive Fast Fail
[ https://issues.apache.org/jira/browse/HBASE-12075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manukranth Kolloju updated HBASE-12075: --- Attachment: 0001-HBASE-12075-Implement-Preemptive-Fast-Fail.patch [~stack], [~tedyu], [~eclark], I have made changes to the ClusterConnection to expose a builder function which is going to be much more cleaner and would not add any new concepts into the Interface, which seems ok. This is one of the major features that we use at fb and I feel others can also take advantage of this feature and extend on top of it. I'd be glad to help speed up the review on this. Let me know what else you guys are waiting for me to do to get this in. Preemptive Fast Fail Key: HBASE-12075 URL: https://issues.apache.org/jira/browse/HBASE-12075 Project: HBase Issue Type: Sub-task Components: Client Affects Versions: 0.99.0, 2.0.0, 0.98.6.1 Reporter: Manukranth Kolloju Assignee: Manukranth Kolloju Attachments: 0001-Add-a-test-case-for-Preemptive-Fast-Fail.patch, 0001-HBASE-12075-Implement-Preemptive-Fast-Fail.patch, 0001-HBASE-12075-Implement-Preemptive-Fast-Fail.patch, 0001-HBASE-12075-Implement-Preemptive-Fast-Fail.patch, 0001-HBASE-12075-Implement-Preemptive-Fast-Fail.patch, 0001-HBASE-12075-Implement-Preemptive-Fast-Fail.patch, 0001-HBASE-12075-Implement-Preemptive-Fast-Fail.patch, 0001-Implement-Preemptive-Fast-Fail.patch, 0001-Implement-Preemptive-Fast-Fail.patch, 0001-Implement-Preemptive-Fast-Fail.patch, 0001-Implement-Preemptive-Fast-Fail.patch, 0001-Implement-Preemptive-Fast-Fail.patch In multi threaded clients, we use a feature developed on 0.89-fb branch called Preemptive Fast Fail. This allows the client threads which would potentially fail, fail fast. The idea behind this feature is that we allow, among the hundreds of client threads, one thread to try and establish connection with the regionserver and if that succeeds, we mark it as a live node again. Meanwhile, other threads which are trying to establish connection to the same server would ideally go into the timeouts which is effectively unfruitful. We can in those cases return appropriate exceptions to those clients instead of letting them retry. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-4955) Use the official versions of surefire junit
[ https://issues.apache.org/jira/browse/HBASE-4955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171487#comment-14171487 ] Hudson commented on HBASE-4955: --- FAILURE: Integrated in HBase-1.0 #312 (See [https://builds.apache.org/job/HBase-1.0/312/]) HBASE-4955 Use the official versions of surefire (Alex Newman) (eclark: rev 921d331fa326072c8df2dc5a870884aba3de9c77) * pom.xml * hbase-server/src/test/java/org/apache/hadoop/hbase/http/TestServletFilter.java Use the official versions of surefire junit - Key: HBASE-4955 URL: https://issues.apache.org/jira/browse/HBASE-4955 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0, 0.98.0, 0.96.0, 0.99.0 Environment: all Reporter: Nicolas Liochon Assignee: Alex Newman Priority: Critical Fix For: 2.0.0, 0.99.2 Attachments: 4955.v1.patch, 4955.v2.patch, 4955.v2.patch, 4955.v2.patch, 4955.v2.patch, 4955.v3.patch, 4955.v3.patch, 4955.v3.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v4.patch, 4955.v5.patch, 4955.v6.patch, 4955.v7.patch, 4955.v7.patch, 4955.v8.patch, 4955.v9.patch, 8204.v4.patch, HBASE-4955-v10.patch, HBASE-4955-v10.patch, HBASE-4955-v10.patch We currently use private versions for Surefire JUnit since HBASE-4763. This JIRA traks what we need to move to official versions. Surefire 2.11 is just out, but, after some tests, it does not contain all what we need. JUnit. Could be for JUnit 4.11. Issue to monitor: https://github.com/KentBeck/junit/issues/359: fixed in our version, no feedback for an integration on trunk Surefire: Could be for Surefire 2.12. Issues to monitor are: 329 (category support): fixed, we use the official implementation from the trunk 786 (@Category with forkMode=always): fixed, we use the official implementation from the trunk 791 (incorrect elapsed time on test failure): fixed, we use the official implementation from the trunk 793 (incorrect time in the XML report): Not fixed (reopen) on trunk, fixed on our version. 760 (does not take into account the test method): fixed in trunk, not fixed in our version 798 (print immediately the test class name): not fixed in trunk, not fixed in our version 799 (Allow test parallelization when forkMode=always): not fixed in trunk, not fixed in our version 800 (redirectTestOutputToFile not taken into account): not yet fix on trunk, fixed on our version 800 793 are the more important to monitor, it's the only ones that are fixed in our version but not on trunk. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12252) IntegrationTestBulkLoad fails with illegal partition error
[
https://issues.apache.org/jira/browse/HBASE-12252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171505#comment-14171505
]
Hadoop QA commented on HBASE-12252:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12674820/HBASE-12252.patch
against trunk revision .
ATTACHMENT ID: 12674820
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/11340//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11340//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11340//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11340//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11340//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11340//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11340//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11340//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11340//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11340//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11340//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11340//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11340//console
This message is automatically generated.
IntegrationTestBulkLoad fails with illegal partition error
--
Key: HBASE-12252
URL: https://issues.apache.org/jira/browse/HBASE-12252
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.98.6
Reporter: Dima Spivak
Assignee: Dima Spivak
Attachments: HBASE-12252.patch
While running {{IntegrationTestBulkLoad}} for a while, I encountered the
following error:
{code}
14/10/14 05:57:12 INFO mapreduce.IntegrationTestBulkLoad: Running check
14/10/14 05:57:12 INFO client.RMProxy: Connecting to ResourceManager at
node-1.internal/10.0.0.3:8032
14/10/14 05:57:16 INFO util.RegionSizeCalculator: Calculating region sizes
for table IntegrationTestBulkLoad.
14/10/14 05:57:16 INFO mapreduce.JobSubmitter: number of splits:16
14/10/14 05:57:17 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1413261370136_0012
14/10/14 05:57:17 INFO impl.YarnClientImpl: Submitted application
application_1413261370136_0012
14/10/14 05:57:17 INFO mapreduce.Job: The url to track the job:
http://node-1.internal:8088/proxy/application_1413261370136_0012/
14/10/14 05:57:17 INFO mapreduce.Job: Running job: job_1413261370136_0012
14/10/14 05:57:32 INFO mapreduce.Job: Job job_1413261370136_0012 running in
uber mode : false
14/10/14 05:57:32 INFO mapreduce.Job: map 0% reduce 0%
14/10/14 05:57:52 INFO mapreduce.Job: map 6% reduce 0%
14/10/14 05:57:53 INFO mapreduce.Job: Task Id :
attempt_1413261370136_0012_m_13_0, Status : FAILED
Error: java.io.IOException: Illegal partition for
[jira] [Commented] (HBASE-7211) Improve hbase ref guide for the testing part.
[ https://issues.apache.org/jira/browse/HBASE-7211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171510#comment-14171510 ] Misty Stanley-Jones commented on HBASE-7211: OK, 4995 is closed so I will close this JIRA too. Improve hbase ref guide for the testing part. - Key: HBASE-7211 URL: https://issues.apache.org/jira/browse/HBASE-7211 Project: HBase Issue Type: Bug Components: documentation Affects Versions: 0.95.2 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Minor Fix For: 2.0.0, 0.99.2 Attachments: hbase-7211-partial.patch Here is some stuff I saw. I will propose a fix in a week or so, please add the comment or issues you have in mind. ??15.6.1. Apache HBase Modules?? = We should be able to use categories in all modules. The default should be small; but any test manipulating the time needs to be in a specific jvm (hence medium), so it's not always related to minicluster. ??15.6.3.6. hbasetests.sh?? = We can remove this chapter, and the script The script is not totally useless, but I think nobody actually uses it. = Add a chapter on flakiness. Some tests are, unfortunately, flaky. While there number decreases, we still have some. Rules are: - don't write flaky tests! :-) - small tests cannot be flaky, as it blocks other test execution. Corollary: if you have an issue with a small test, it's either your environment either a severe issue. - rerun the test a few time to validate, check the ports and file descriptors used. ??mvn test -P localTests -Dtest=MyTest?? = We could actually activate the localTests profile whenever -Dtest is used. If we do that, we can remove the reference from localTests in the doc. ??mvn test -P runSmallTests?? ??mvn test -P runMediumTests?? = I'm not sure it's actually used. We could remove them from the pom.xml (and the doc). ??The HBase build uses a patched version of the maven surefire plugin?? = Hopefully, we will be able to remove this soon :-) ??Integration tests are described TODO: POINTER_TO_INTEGRATION_TEST_SECTION?? = Should be documented -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12257) TestAssignmentManager unsynchronized access to regionPlans
[
https://issues.apache.org/jira/browse/HBASE-12257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171508#comment-14171508
]
Enis Soztutar commented on HBASE-12257:
---
+1.
TestAssignmentManager unsynchronized access to regionPlans
--
Key: HBASE-12257
URL: https://issues.apache.org/jira/browse/HBASE-12257
Project: HBase
Issue Type: Bug
Components: Region Assignment
Affects Versions: 0.98.7, 0.99.1
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Trivial
Fix For: 2.0.0, 0.98.8, 0.99.2
Attachments: HBASE-11257-98-v0.patch
There is some code in TestMasterFailover that access regionPlans without
synchronized
{code}
Stack Trace:
java.lang.NullPointerException
at java.util.TreeMap.rotateLeft(TreeMap.java:2069)
at java.util.TreeMap.fixAfterInsertion(TreeMap.java:2127)
at java.util.TreeMap.put(TreeMap.java:574)
at
org.apache.hadoop.hbase.master.TestMasterFailover.testMasterFailoverWithMockedRITOnDeadRS(TestMasterFailover.java:596)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-7211) Improve hbase ref guide for the testing part.
[ https://issues.apache.org/jira/browse/HBASE-7211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones resolved HBASE-7211. Resolution: Fixed Fix Version/s: 0.99.2 2.0.0 Improve hbase ref guide for the testing part. - Key: HBASE-7211 URL: https://issues.apache.org/jira/browse/HBASE-7211 Project: HBase Issue Type: Bug Components: documentation Affects Versions: 0.95.2 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Minor Fix For: 2.0.0, 0.99.2 Attachments: hbase-7211-partial.patch Here is some stuff I saw. I will propose a fix in a week or so, please add the comment or issues you have in mind. ??15.6.1. Apache HBase Modules?? = We should be able to use categories in all modules. The default should be small; but any test manipulating the time needs to be in a specific jvm (hence medium), so it's not always related to minicluster. ??15.6.3.6. hbasetests.sh?? = We can remove this chapter, and the script The script is not totally useless, but I think nobody actually uses it. = Add a chapter on flakiness. Some tests are, unfortunately, flaky. While there number decreases, we still have some. Rules are: - don't write flaky tests! :-) - small tests cannot be flaky, as it blocks other test execution. Corollary: if you have an issue with a small test, it's either your environment either a severe issue. - rerun the test a few time to validate, check the ports and file descriptors used. ??mvn test -P localTests -Dtest=MyTest?? = We could actually activate the localTests profile whenever -Dtest is used. If we do that, we can remove the reference from localTests in the doc. ??mvn test -P runSmallTests?? ??mvn test -P runMediumTests?? = I'm not sure it's actually used. We could remove them from the pom.xml (and the doc). ??The HBase build uses a patched version of the maven surefire plugin?? = Hopefully, we will be able to remove this soon :-) ??Integration tests are described TODO: POINTER_TO_INTEGRATION_TEST_SECTION?? = Should be documented -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12257) TestAssignmentManager unsynchronized access to regionPlans
[
https://issues.apache.org/jira/browse/HBASE-12257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-12257:
--
Fix Version/s: 2.0.0
TestAssignmentManager unsynchronized access to regionPlans
--
Key: HBASE-12257
URL: https://issues.apache.org/jira/browse/HBASE-12257
Project: HBase
Issue Type: Bug
Components: Region Assignment
Affects Versions: 0.98.7, 0.99.1
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Trivial
Fix For: 2.0.0, 0.98.8, 0.99.2
Attachments: HBASE-11257-98-v0.patch
There is some code in TestMasterFailover that access regionPlans without
synchronized
{code}
Stack Trace:
java.lang.NullPointerException
at java.util.TreeMap.rotateLeft(TreeMap.java:2069)
at java.util.TreeMap.fixAfterInsertion(TreeMap.java:2127)
at java.util.TreeMap.put(TreeMap.java:574)
at
org.apache.hadoop.hbase.master.TestMasterFailover.testMasterFailoverWithMockedRITOnDeadRS(TestMasterFailover.java:596)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12247) Replace setHTable() with initializeTable() in TableInputFormat.
[
https://issues.apache.org/jira/browse/HBASE-12247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171515#comment-14171515
]
Hadoop QA commented on HBASE-12247:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12674823/HBASE-12247.patch
against trunk revision .
ATTACHMENT ID: 12674823
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/11341//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11341//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11341//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11341//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11341//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11341//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11341//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11341//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11341//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11341//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11341//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11341//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11341//console
This message is automatically generated.
Replace setHTable() with initializeTable() in TableInputFormat.
---
Key: HBASE-12247
URL: https://issues.apache.org/jira/browse/HBASE-12247
Project: HBase
Issue Type: Bug
Affects Versions: 2.0.0, 0.99.2
Reporter: Solomon Duskis
Assignee: Solomon Duskis
Attachments: HBASE-12247.patch, HBASE-12247_v2.patch
TableInputFormat uses a new HTable(). Switch that use to a manually created
ConnectionFactory.createConnection() and call initializeTable().
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12252) IntegrationTestBulkLoad fails with illegal partition error
[
https://issues.apache.org/jira/browse/HBASE-12252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-12252:
--
Resolution: Fixed
Fix Version/s: 0.99.2
0.98.8
2.0.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to 0.98+. Thanks [~dimaspivak]
IntegrationTestBulkLoad fails with illegal partition error
--
Key: HBASE-12252
URL: https://issues.apache.org/jira/browse/HBASE-12252
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.98.6
Reporter: Dima Spivak
Assignee: Dima Spivak
Fix For: 2.0.0, 0.98.8, 0.99.2
Attachments: HBASE-12252.patch
While running {{IntegrationTestBulkLoad}} for a while, I encountered the
following error:
{code}
14/10/14 05:57:12 INFO mapreduce.IntegrationTestBulkLoad: Running check
14/10/14 05:57:12 INFO client.RMProxy: Connecting to ResourceManager at
node-1.internal/10.0.0.3:8032
14/10/14 05:57:16 INFO util.RegionSizeCalculator: Calculating region sizes
for table IntegrationTestBulkLoad.
14/10/14 05:57:16 INFO mapreduce.JobSubmitter: number of splits:16
14/10/14 05:57:17 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1413261370136_0012
14/10/14 05:57:17 INFO impl.YarnClientImpl: Submitted application
application_1413261370136_0012
14/10/14 05:57:17 INFO mapreduce.Job: The url to track the job:
http://node-1.internal:8088/proxy/application_1413261370136_0012/
14/10/14 05:57:17 INFO mapreduce.Job: Running job: job_1413261370136_0012
14/10/14 05:57:32 INFO mapreduce.Job: Job job_1413261370136_0012 running in
uber mode : false
14/10/14 05:57:32 INFO mapreduce.Job: map 0% reduce 0%
14/10/14 05:57:52 INFO mapreduce.Job: map 6% reduce 0%
14/10/14 05:57:53 INFO mapreduce.Job: Task Id :
attempt_1413261370136_0012_m_13_0, Status : FAILED
Error: java.io.IOException: Illegal partition for
org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad$LinkKey@133c4539
(-3)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1079)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:712)
at
org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
at
org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
at
org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad$LinkedListCheckingMapper.map(IntegrationTestBulkLoad.java:502)
at
org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad$LinkedListCheckingMapper.map(IntegrationTestBulkLoad.java:491)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
{code}
I'm pretty sure I've tracked this down to a partition generator that can
return a negative number; I'll post a fix once I've confirmed locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-12257) TestAssignmentManager unsynchronized access to regionPlans
[
https://issues.apache.org/jira/browse/HBASE-12257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi resolved HBASE-12257.
-
Resolution: Fixed
Fix Version/s: (was: 2.0.0)
committed to branch-1 and 98, master doesn't seems to have that code
TestAssignmentManager unsynchronized access to regionPlans
--
Key: HBASE-12257
URL: https://issues.apache.org/jira/browse/HBASE-12257
Project: HBase
Issue Type: Bug
Components: Region Assignment
Affects Versions: 0.98.7, 0.99.1
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Trivial
Fix For: 0.98.8, 0.99.2
Attachments: HBASE-11257-98-v0.patch
There is some code in TestMasterFailover that access regionPlans without
synchronized
{code}
Stack Trace:
java.lang.NullPointerException
at java.util.TreeMap.rotateLeft(TreeMap.java:2069)
at java.util.TreeMap.fixAfterInsertion(TreeMap.java:2127)
at java.util.TreeMap.put(TreeMap.java:574)
at
org.apache.hadoop.hbase.master.TestMasterFailover.testMasterFailoverWithMockedRITOnDeadRS(TestMasterFailover.java:596)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12247) Replace setHTable() with initializeTable() in TableInputFormat.
[ https://issues.apache.org/jira/browse/HBASE-12247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-12247: -- Resolution: Fixed Fix Version/s: 0.99.2 2.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to branch-1+. Thanks for the patch [~sduskis] Replace setHTable() with initializeTable() in TableInputFormat. --- Key: HBASE-12247 URL: https://issues.apache.org/jira/browse/HBASE-12247 Project: HBase Issue Type: Bug Affects Versions: 2.0.0, 0.99.2 Reporter: Solomon Duskis Assignee: Solomon Duskis Fix For: 2.0.0, 0.99.2 Attachments: HBASE-12247.patch, HBASE-12247_v2.patch TableInputFormat uses a new HTable(). Switch that use to a manually created ConnectionFactory.createConnection() and call initializeTable(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12247) Replace setHTable() with initializeTable() in TableInputFormat.
[
https://issues.apache.org/jira/browse/HBASE-12247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171550#comment-14171550
]
Hadoop QA commented on HBASE-12247:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12674824/HBASE-12247_v2.patch
against trunk revision .
ATTACHMENT ID: 12674824
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/11342//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11342//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11342//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11342//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11342//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11342//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11342//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11342//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11342//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11342//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11342//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11342//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11342//console
This message is automatically generated.
Replace setHTable() with initializeTable() in TableInputFormat.
---
Key: HBASE-12247
URL: https://issues.apache.org/jira/browse/HBASE-12247
Project: HBase
Issue Type: Bug
Affects Versions: 2.0.0, 0.99.2
Reporter: Solomon Duskis
Assignee: Solomon Duskis
Fix For: 2.0.0, 0.99.2
Attachments: HBASE-12247.patch, HBASE-12247_v2.patch
TableInputFormat uses a new HTable(). Switch that use to a manually created
ConnectionFactory.createConnection() and call initializeTable().
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12247) Replace setHTable() with initializeTable() in TableInputFormat.
[ https://issues.apache.org/jira/browse/HBASE-12247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171554#comment-14171554 ] Enis Soztutar commented on HBASE-12247: --- IF has an interesting lifecycle in hadoop MR. The IF is configured statically, and the configuration is saved in the configuration object. Then MR initializes the class once, and passes the conf back in through setConf(). This instance is used for calling getSplits(). After that, there is no close method or anything that is called. Then for each map task, the IF is initialized again and passed the job configuration. Then createRecordReader() is called with one of the InputSplits that is passed. The returned record reader is closed properly. I think we can do this: IF.setConf() will create a Connection. IF.getSplits() will close that connection at the end in a finally close. Also the RecordReder.close() will close the connection in RR.close() method. It is similar to how DBInputFormat works as well https://github.com/apache/hadoop-common/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/db/DBInputFormat.java Replace setHTable() with initializeTable() in TableInputFormat. --- Key: HBASE-12247 URL: https://issues.apache.org/jira/browse/HBASE-12247 Project: HBase Issue Type: Bug Affects Versions: 2.0.0, 0.99.2 Reporter: Solomon Duskis Assignee: Solomon Duskis Fix For: 2.0.0, 0.99.2 Attachments: HBASE-12247.patch, HBASE-12247_v2.patch TableInputFormat uses a new HTable(). Switch that use to a manually created ConnectionFactory.createConnection() and call initializeTable(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12223) MultiTableInputFormatBase.getSplits is too slow
[ https://issues.apache.org/jira/browse/HBASE-12223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171557#comment-14171557 ] Ted Yu commented on HBASE-12223: Mind rebasing your patch ? Thanks MultiTableInputFormatBase.getSplits is too slow --- Key: HBASE-12223 URL: https://issues.apache.org/jira/browse/HBASE-12223 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.94.15 Reporter: shanwen Assignee: YuanBo Peng Priority: Minor Fix For: 2.0.0, 0.98.8, 0.94.25, 0.99.2 Attachments: HBASE-12223.patch when use Multiple scan,getSplits is too slow,800 scans take five minutes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12161) Add support for grant/revoke on namespaces in AccessControlClient
[ https://issues.apache.org/jira/browse/HBASE-12161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Srikanth Srungarapu updated HBASE-12161: Attachment: HBASE-12161_master_v2.patch [~mbertozzi] Thanks for pointing out my fundamental error. Rectified it, and also made changes to existing grant/revoke commands. Add support for grant/revoke on namespaces in AccessControlClient - Key: HBASE-12161 URL: https://issues.apache.org/jira/browse/HBASE-12161 Project: HBase Issue Type: Improvement Reporter: Srikanth Srungarapu Assignee: Srikanth Srungarapu Priority: Minor Attachments: HBASE-12161_0.98.patch, HBASE-12161_master.patch, HBASE-12161_master_v2.patch As per the description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12257) TestAssignmentManager unsynchronized access to regionPlans
[
https://issues.apache.org/jira/browse/HBASE-12257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171571#comment-14171571
]
Hudson commented on HBASE-12257:
FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #572 (See
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/572/])
HBASE-12257 TestAssignmentManager unsynchronized access to regionPlans
(matteo.bertozzi: rev ed4c8b7ec21c8d45cd64fd0760454d2367ab8544)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java
TestAssignmentManager unsynchronized access to regionPlans
--
Key: HBASE-12257
URL: https://issues.apache.org/jira/browse/HBASE-12257
Project: HBase
Issue Type: Bug
Components: Region Assignment
Affects Versions: 0.98.7, 0.99.1
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Trivial
Fix For: 0.98.8, 0.99.2
Attachments: HBASE-11257-98-v0.patch
There is some code in TestMasterFailover that access regionPlans without
synchronized
{code}
Stack Trace:
java.lang.NullPointerException
at java.util.TreeMap.rotateLeft(TreeMap.java:2069)
at java.util.TreeMap.fixAfterInsertion(TreeMap.java:2127)
at java.util.TreeMap.put(TreeMap.java:574)
at
org.apache.hadoop.hbase.master.TestMasterFailover.testMasterFailoverWithMockedRITOnDeadRS(TestMasterFailover.java:596)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12252) IntegrationTestBulkLoad fails with illegal partition error
[
https://issues.apache.org/jira/browse/HBASE-12252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171572#comment-14171572
]
Hudson commented on HBASE-12252:
FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #572 (See
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/572/])
HBASE-12252 IntegrationTestBulkLoad fails with illegal partition error (stack:
rev dc20003d12017a6e66e2aa35c1c393837511991c)
*
hbase-it/src/test/java/org/apache/hadoop/hbase/mapreduce/IntegrationTestBulkLoad.java
IntegrationTestBulkLoad fails with illegal partition error
--
Key: HBASE-12252
URL: https://issues.apache.org/jira/browse/HBASE-12252
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.98.6
Reporter: Dima Spivak
Assignee: Dima Spivak
Fix For: 2.0.0, 0.98.8, 0.99.2
Attachments: HBASE-12252.patch
While running {{IntegrationTestBulkLoad}} for a while, I encountered the
following error:
{code}
14/10/14 05:57:12 INFO mapreduce.IntegrationTestBulkLoad: Running check
14/10/14 05:57:12 INFO client.RMProxy: Connecting to ResourceManager at
node-1.internal/10.0.0.3:8032
14/10/14 05:57:16 INFO util.RegionSizeCalculator: Calculating region sizes
for table IntegrationTestBulkLoad.
14/10/14 05:57:16 INFO mapreduce.JobSubmitter: number of splits:16
14/10/14 05:57:17 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1413261370136_0012
14/10/14 05:57:17 INFO impl.YarnClientImpl: Submitted application
application_1413261370136_0012
14/10/14 05:57:17 INFO mapreduce.Job: The url to track the job:
http://node-1.internal:8088/proxy/application_1413261370136_0012/
14/10/14 05:57:17 INFO mapreduce.Job: Running job: job_1413261370136_0012
14/10/14 05:57:32 INFO mapreduce.Job: Job job_1413261370136_0012 running in
uber mode : false
14/10/14 05:57:32 INFO mapreduce.Job: map 0% reduce 0%
14/10/14 05:57:52 INFO mapreduce.Job: map 6% reduce 0%
14/10/14 05:57:53 INFO mapreduce.Job: Task Id :
attempt_1413261370136_0012_m_13_0, Status : FAILED
Error: java.io.IOException: Illegal partition for
org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad$LinkKey@133c4539
(-3)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1079)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:712)
at
org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
at
org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
at
org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad$LinkedListCheckingMapper.map(IntegrationTestBulkLoad.java:502)
at
org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad$LinkedListCheckingMapper.map(IntegrationTestBulkLoad.java:491)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
{code}
I'm pretty sure I've tracked this down to a partition generator that can
return a negative number; I'll post a fix once I've confirmed locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12161) Add support for grant/revoke on namespaces in AccessControlClient
[
https://issues.apache.org/jira/browse/HBASE-12161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171589#comment-14171589
]
Ted Yu commented on HBASE-12161:
lgtm
{code}
TableName.valueOf(NamespaceDescriptor.SYSTEM_NAMESPACE_NAME_STR, acl);
{code}
Referring to AccessControlConstants#OP_ATTRIBUTE_ACL would be better.
{code}
public void testAccessControlClientGrantRevokeOnNameSpace() throws
Exception {
{code}
nit: 'Namespace' (lower case s) should suffice.
Add support for grant/revoke on namespaces in AccessControlClient
-
Key: HBASE-12161
URL: https://issues.apache.org/jira/browse/HBASE-12161
Project: HBase
Issue Type: Improvement
Reporter: Srikanth Srungarapu
Assignee: Srikanth Srungarapu
Priority: Minor
Attachments: HBASE-12161_0.98.patch, HBASE-12161_master.patch,
HBASE-12161_master_v2.patch
As per the description.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12161) Add support for grant/revoke on namespaces in AccessControlClient
[ https://issues.apache.org/jira/browse/HBASE-12161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Srikanth Srungarapu updated HBASE-12161: Attachment: HBASE-12161_v3.patch Thanks [~te...@apache.org] for the feedback. Integrated your suggestions in the newly attached patch. Add support for grant/revoke on namespaces in AccessControlClient - Key: HBASE-12161 URL: https://issues.apache.org/jira/browse/HBASE-12161 Project: HBase Issue Type: Improvement Reporter: Srikanth Srungarapu Assignee: Srikanth Srungarapu Priority: Minor Attachments: HBASE-12161_0.98.patch, HBASE-12161_master.patch, HBASE-12161_master_v2.patch, HBASE-12161_v3.patch As per the description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12147) Porting Online Config Change from 89-fb
[ https://issues.apache.org/jira/browse/HBASE-12147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171604#comment-14171604 ] Hudson commented on HBASE-12147: SUCCESS: Integrated in HBase-1.0 #313 (See [https://builds.apache.org/job/HBase-1.0/313/]) HBASE-12147 Porting Online Config Change from 89-fb -- REAPPLY (stack: rev dbd0ba36f53ce1adeaffc706090c390452f09f46) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionPolicy.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerOnlineConfigChange.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionConfiguration.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/main/java/org/apache/hadoop/hbase/conf/ConfigurationObserver.java * hbase-server/src/main/java/org/apache/hadoop/hbase/conf/PropagatingConfigurationObserver.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestOffPeakHours.java * hbase-server/src/main/java/org/apache/hadoop/hbase/conf/ConfigurationManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/OffPeakHours.java * hbase-server/src/test/java/org/apache/hadoop/hbase/conf/TestConfigurationManager.java HBASE-12147 Porting Online Config Change from 89-fb -- JAVADOC FIX ADDENDUM (stack: rev ca26d22b50773db4562752d12ba5eb4fdde59b20) * hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java Porting Online Config Change from 89-fb --- Key: HBASE-12147 URL: https://issues.apache.org/jira/browse/HBASE-12147 Project: HBase Issue Type: Sub-task Affects Versions: 1.0.0 Reporter: Manukranth Kolloju Assignee: Manukranth Kolloju Fix For: 0.99.2 Attachments: 0001-Online-config-change-Branch1.patch, 0001-Online-config-change-Branch1.patch, 0001-Online-config-change-Branch1.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch, 0001-Online-config-change.patch Original Estimate: 168h Remaining Estimate: 168h This jira is to track the forward port of HBASE-8805 and HBASE-8544 implemented by [~gaurav.menghani] in 89-fb. This improves operational efficiency in managing clusters that are serving production traffic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12161) Add support for grant/revoke on namespaces in AccessControlClient
[ https://issues.apache.org/jira/browse/HBASE-12161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171609#comment-14171609 ] Matteo Bertozzi commented on HBASE-12161: - v3 looks good to me Add support for grant/revoke on namespaces in AccessControlClient - Key: HBASE-12161 URL: https://issues.apache.org/jira/browse/HBASE-12161 Project: HBase Issue Type: Improvement Reporter: Srikanth Srungarapu Assignee: Srikanth Srungarapu Priority: Minor Attachments: HBASE-12161_0.98.patch, HBASE-12161_master.patch, HBASE-12161_master_v2.patch, HBASE-12161_v3.patch As per the description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12247) Replace setHTable() with initializeTable() in TableInputFormat.
[ https://issues.apache.org/jira/browse/HBASE-12247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171623#comment-14171623 ] stack commented on HBASE-12247: --- Thanks [~enis] IF should just manage connections itself -- in splits and in RR? Just take a tablename rather than a settable? Replace setHTable() with initializeTable() in TableInputFormat. --- Key: HBASE-12247 URL: https://issues.apache.org/jira/browse/HBASE-12247 Project: HBase Issue Type: Bug Affects Versions: 2.0.0, 0.99.2 Reporter: Solomon Duskis Assignee: Solomon Duskis Fix For: 2.0.0, 0.99.2 Attachments: HBASE-12247.patch, HBASE-12247_v2.patch TableInputFormat uses a new HTable(). Switch that use to a manually created ConnectionFactory.createConnection() and call initializeTable(). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12252) IntegrationTestBulkLoad fails with illegal partition error
[
https://issues.apache.org/jira/browse/HBASE-12252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171639#comment-14171639
]
Hudson commented on HBASE-12252:
SUCCESS: Integrated in HBase-TRUNK #5663 (See
[https://builds.apache.org/job/HBase-TRUNK/5663/])
HBASE-12252 IntegrationTestBulkLoad fails with illegal partition error (stack:
rev dc860015232bf2055619d8c72abcb0161ba0bd57)
*
hbase-it/src/test/java/org/apache/hadoop/hbase/mapreduce/IntegrationTestBulkLoad.java
IntegrationTestBulkLoad fails with illegal partition error
--
Key: HBASE-12252
URL: https://issues.apache.org/jira/browse/HBASE-12252
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.98.6
Reporter: Dima Spivak
Assignee: Dima Spivak
Fix For: 2.0.0, 0.98.8, 0.99.2
Attachments: HBASE-12252.patch
While running {{IntegrationTestBulkLoad}} for a while, I encountered the
following error:
{code}
14/10/14 05:57:12 INFO mapreduce.IntegrationTestBulkLoad: Running check
14/10/14 05:57:12 INFO client.RMProxy: Connecting to ResourceManager at
node-1.internal/10.0.0.3:8032
14/10/14 05:57:16 INFO util.RegionSizeCalculator: Calculating region sizes
for table IntegrationTestBulkLoad.
14/10/14 05:57:16 INFO mapreduce.JobSubmitter: number of splits:16
14/10/14 05:57:17 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1413261370136_0012
14/10/14 05:57:17 INFO impl.YarnClientImpl: Submitted application
application_1413261370136_0012
14/10/14 05:57:17 INFO mapreduce.Job: The url to track the job:
http://node-1.internal:8088/proxy/application_1413261370136_0012/
14/10/14 05:57:17 INFO mapreduce.Job: Running job: job_1413261370136_0012
14/10/14 05:57:32 INFO mapreduce.Job: Job job_1413261370136_0012 running in
uber mode : false
14/10/14 05:57:32 INFO mapreduce.Job: map 0% reduce 0%
14/10/14 05:57:52 INFO mapreduce.Job: map 6% reduce 0%
14/10/14 05:57:53 INFO mapreduce.Job: Task Id :
attempt_1413261370136_0012_m_13_0, Status : FAILED
Error: java.io.IOException: Illegal partition for
org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad$LinkKey@133c4539
(-3)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1079)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:712)
at
org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
at
org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
at
org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad$LinkedListCheckingMapper.map(IntegrationTestBulkLoad.java:502)
at
org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad$LinkedListCheckingMapper.map(IntegrationTestBulkLoad.java:491)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
{code}
I'm pretty sure I've tracked this down to a partition generator that can
return a negative number; I'll post a fix once I've confirmed locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12075) Preemptive Fast Fail
[ https://issues.apache.org/jira/browse/HBASE-12075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171645#comment-14171645 ] Ted Yu commented on HBASE-12075: Somehow QA run was unsuccessful: https://builds.apache.org/job/PreCommit-HBASE-Build/11343/console Triggering one more run. Preemptive Fast Fail Key: HBASE-12075 URL: https://issues.apache.org/jira/browse/HBASE-12075 Project: HBase Issue Type: Sub-task Components: Client Affects Versions: 0.99.0, 2.0.0, 0.98.6.1 Reporter: Manukranth Kolloju Assignee: Manukranth Kolloju Attachments: 0001-Add-a-test-case-for-Preemptive-Fast-Fail.patch, 0001-HBASE-12075-Implement-Preemptive-Fast-Fail.patch, 0001-HBASE-12075-Implement-Preemptive-Fast-Fail.patch, 0001-HBASE-12075-Implement-Preemptive-Fast-Fail.patch, 0001-HBASE-12075-Implement-Preemptive-Fast-Fail.patch, 0001-HBASE-12075-Implement-Preemptive-Fast-Fail.patch, 0001-HBASE-12075-Implement-Preemptive-Fast-Fail.patch, 0001-Implement-Preemptive-Fast-Fail.patch, 0001-Implement-Preemptive-Fast-Fail.patch, 0001-Implement-Preemptive-Fast-Fail.patch, 0001-Implement-Preemptive-Fast-Fail.patch, 0001-Implement-Preemptive-Fast-Fail.patch In multi threaded clients, we use a feature developed on 0.89-fb branch called Preemptive Fast Fail. This allows the client threads which would potentially fail, fail fast. The idea behind this feature is that we allow, among the hundreds of client threads, one thread to try and establish connection with the regionserver and if that succeeds, we mark it as a live node again. Meanwhile, other threads which are trying to establish connection to the same server would ideally go into the timeouts which is effectively unfruitful. We can in those cases return appropriate exceptions to those clients instead of letting them retry. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12161) Add support for grant/revoke on namespaces in AccessControlClient
[ https://issues.apache.org/jira/browse/HBASE-12161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12161: --- Fix Version/s: 0.99.2 2.0.0 Hadoop Flags: Reviewed Add support for grant/revoke on namespaces in AccessControlClient - Key: HBASE-12161 URL: https://issues.apache.org/jira/browse/HBASE-12161 Project: HBase Issue Type: Improvement Reporter: Srikanth Srungarapu Assignee: Srikanth Srungarapu Priority: Minor Fix For: 2.0.0, 0.99.2 Attachments: HBASE-12161_0.98.patch, HBASE-12161_master.patch, HBASE-12161_master_v2.patch, HBASE-12161_v3.patch As per the description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12258) Make TestHBaseFsck less flaky
[ https://issues.apache.org/jira/browse/HBASE-12258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manukranth Kolloju updated HBASE-12258: --- Status: Patch Available (was: Open) Make TestHBaseFsck less flaky - Key: HBASE-12258 URL: https://issues.apache.org/jira/browse/HBASE-12258 Project: HBase Issue Type: Bug Components: test Affects Versions: 2.0.0 Reporter: Manukranth Kolloju Assignee: Manukranth Kolloju Priority: Minor Fix For: 2.0.0 Attachments: 0001-TestHbaseFsck-is-flaky-and-has-some-multithreading-i.patch Original Estimate: 168h Remaining Estimate: 168h TestHBaseFsck is flaky sometimes. This jira is to make it pass more predictably. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12258) Make TestHBaseFsck less flaky
[ https://issues.apache.org/jira/browse/HBASE-12258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manukranth Kolloju updated HBASE-12258: --- Attachment: 0001-TestHbaseFsck-is-flaky-and-has-some-multithreading-i.patch Make TestHBaseFsck less flaky - Key: HBASE-12258 URL: https://issues.apache.org/jira/browse/HBASE-12258 Project: HBase Issue Type: Bug Components: test Affects Versions: 2.0.0 Reporter: Manukranth Kolloju Assignee: Manukranth Kolloju Priority: Minor Fix For: 2.0.0 Attachments: 0001-TestHbaseFsck-is-flaky-and-has-some-multithreading-i.patch Original Estimate: 168h Remaining Estimate: 168h TestHBaseFsck is flaky sometimes. This jira is to make it pass more predictably. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12258) Make TestHBaseFsck less flaky
Manukranth Kolloju created HBASE-12258: -- Summary: Make TestHBaseFsck less flaky Key: HBASE-12258 URL: https://issues.apache.org/jira/browse/HBASE-12258 Project: HBase Issue Type: Bug Components: test Affects Versions: 2.0.0 Reporter: Manukranth Kolloju Assignee: Manukranth Kolloju Priority: Minor Fix For: 2.0.0 Attachments: 0001-TestHbaseFsck-is-flaky-and-has-some-multithreading-i.patch TestHBaseFsck is flaky sometimes. This jira is to make it pass more predictably. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12258) Make TestHBaseFsck less flaky
[ https://issues.apache.org/jira/browse/HBASE-12258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171657#comment-14171657 ] Elliott Clark commented on HBASE-12258: --- Nice find. Make TestHBaseFsck less flaky - Key: HBASE-12258 URL: https://issues.apache.org/jira/browse/HBASE-12258 Project: HBase Issue Type: Bug Components: test Affects Versions: 2.0.0 Reporter: Manukranth Kolloju Assignee: Manukranth Kolloju Priority: Minor Fix For: 2.0.0 Attachments: 0001-TestHbaseFsck-is-flaky-and-has-some-multithreading-i.patch Original Estimate: 168h Remaining Estimate: 168h TestHBaseFsck is flaky sometimes. This jira is to make it pass more predictably. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12258) Make TestHBaseFsck less flaky
[ https://issues.apache.org/jira/browse/HBASE-12258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171661#comment-14171661 ] Ted Yu commented on HBASE-12258: +1 Make TestHBaseFsck less flaky - Key: HBASE-12258 URL: https://issues.apache.org/jira/browse/HBASE-12258 Project: HBase Issue Type: Bug Components: test Affects Versions: 2.0.0 Reporter: Manukranth Kolloju Assignee: Manukranth Kolloju Priority: Minor Fix For: 2.0.0 Attachments: 0001-TestHbaseFsck-is-flaky-and-has-some-multithreading-i.patch Original Estimate: 168h Remaining Estimate: 168h TestHBaseFsck is flaky sometimes. This jira is to make it pass more predictably. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12252) IntegrationTestBulkLoad fails with illegal partition error
[
https://issues.apache.org/jira/browse/HBASE-12252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171683#comment-14171683
]
Hudson commented on HBASE-12252:
FAILURE: Integrated in HBase-0.98 #601 (See
[https://builds.apache.org/job/HBase-0.98/601/])
HBASE-12252 IntegrationTestBulkLoad fails with illegal partition error (stack:
rev dc20003d12017a6e66e2aa35c1c393837511991c)
*
hbase-it/src/test/java/org/apache/hadoop/hbase/mapreduce/IntegrationTestBulkLoad.java
IntegrationTestBulkLoad fails with illegal partition error
--
Key: HBASE-12252
URL: https://issues.apache.org/jira/browse/HBASE-12252
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.98.6
Reporter: Dima Spivak
Assignee: Dima Spivak
Fix For: 2.0.0, 0.98.8, 0.99.2
Attachments: HBASE-12252.patch
While running {{IntegrationTestBulkLoad}} for a while, I encountered the
following error:
{code}
14/10/14 05:57:12 INFO mapreduce.IntegrationTestBulkLoad: Running check
14/10/14 05:57:12 INFO client.RMProxy: Connecting to ResourceManager at
node-1.internal/10.0.0.3:8032
14/10/14 05:57:16 INFO util.RegionSizeCalculator: Calculating region sizes
for table IntegrationTestBulkLoad.
14/10/14 05:57:16 INFO mapreduce.JobSubmitter: number of splits:16
14/10/14 05:57:17 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1413261370136_0012
14/10/14 05:57:17 INFO impl.YarnClientImpl: Submitted application
application_1413261370136_0012
14/10/14 05:57:17 INFO mapreduce.Job: The url to track the job:
http://node-1.internal:8088/proxy/application_1413261370136_0012/
14/10/14 05:57:17 INFO mapreduce.Job: Running job: job_1413261370136_0012
14/10/14 05:57:32 INFO mapreduce.Job: Job job_1413261370136_0012 running in
uber mode : false
14/10/14 05:57:32 INFO mapreduce.Job: map 0% reduce 0%
14/10/14 05:57:52 INFO mapreduce.Job: map 6% reduce 0%
14/10/14 05:57:53 INFO mapreduce.Job: Task Id :
attempt_1413261370136_0012_m_13_0, Status : FAILED
Error: java.io.IOException: Illegal partition for
org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad$LinkKey@133c4539
(-3)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1079)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:712)
at
org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
at
org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
at
org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad$LinkedListCheckingMapper.map(IntegrationTestBulkLoad.java:502)
at
org.apache.hadoop.hbase.mapreduce.IntegrationTestBulkLoad$LinkedListCheckingMapper.map(IntegrationTestBulkLoad.java:491)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
{code}
I'm pretty sure I've tracked this down to a partition generator that can
return a negative number; I'll post a fix once I've confirmed locally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12257) TestAssignmentManager unsynchronized access to regionPlans
[
https://issues.apache.org/jira/browse/HBASE-12257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171682#comment-14171682
]
Hudson commented on HBASE-12257:
FAILURE: Integrated in HBase-0.98 #601 (See
[https://builds.apache.org/job/HBase-0.98/601/])
HBASE-12257 TestAssignmentManager unsynchronized access to regionPlans
(matteo.bertozzi: rev ed4c8b7ec21c8d45cd64fd0760454d2367ab8544)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java
TestAssignmentManager unsynchronized access to regionPlans
--
Key: HBASE-12257
URL: https://issues.apache.org/jira/browse/HBASE-12257
Project: HBase
Issue Type: Bug
Components: Region Assignment
Affects Versions: 0.98.7, 0.99.1
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Trivial
Fix For: 0.98.8, 0.99.2
Attachments: HBASE-11257-98-v0.patch
There is some code in TestMasterFailover that access regionPlans without
synchronized
{code}
Stack Trace:
java.lang.NullPointerException
at java.util.TreeMap.rotateLeft(TreeMap.java:2069)
at java.util.TreeMap.fixAfterInsertion(TreeMap.java:2127)
at java.util.TreeMap.put(TreeMap.java:574)
at
org.apache.hadoop.hbase.master.TestMasterFailover.testMasterFailoverWithMockedRITOnDeadRS(TestMasterFailover.java:596)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12161) Add support for grant/revoke on namespaces in AccessControlClient
[
https://issues.apache.org/jira/browse/HBASE-12161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171688#comment-14171688
]
Hadoop QA commented on HBASE-12161:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12674859/HBASE-12161_v3.patch
against trunk revision .
ATTACHMENT ID: 12674859
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 2
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/11345//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11345//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11345//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11345//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11345//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11345//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11345//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11345//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11345//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11345//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11345//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11345//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11345//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11345//console
This message is automatically generated.
Add support for grant/revoke on namespaces in AccessControlClient
-
Key: HBASE-12161
URL: https://issues.apache.org/jira/browse/HBASE-12161
Project: HBase
Issue Type: Improvement
Reporter: Srikanth Srungarapu
Assignee: Srikanth Srungarapu
Priority: Minor
Fix For: 2.0.0, 0.99.2
Attachments: HBASE-12161_0.98.patch, HBASE-12161_master.patch,
HBASE-12161_master_v2.patch, HBASE-12161_v3.patch
As per the description.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12075) Preemptive Fast Fail
[
https://issues.apache.org/jira/browse/HBASE-12075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171694#comment-14171694
]
Hadoop QA commented on HBASE-12075:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12674844/0001-HBASE-12075-Implement-Preemptive-Fast-Fail.patch
against trunk revision .
ATTACHMENT ID: 12674844
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestFastFailWithoutTestUtil
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/11346//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11346//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11346//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11346//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11346//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11346//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11346//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11346//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11346//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11346//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11346//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11346//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11346//console
This message is automatically generated.
Preemptive Fast Fail
Key: HBASE-12075
URL: https://issues.apache.org/jira/browse/HBASE-12075
Project: HBase
Issue Type: Sub-task
Components: Client
Affects Versions: 0.99.0, 2.0.0, 0.98.6.1
Reporter: Manukranth Kolloju
Assignee: Manukranth Kolloju
Attachments: 0001-Add-a-test-case-for-Preemptive-Fast-Fail.patch,
0001-HBASE-12075-Implement-Preemptive-Fast-Fail.patch,
0001-HBASE-12075-Implement-Preemptive-Fast-Fail.patch,
0001-HBASE-12075-Implement-Preemptive-Fast-Fail.patch,
0001-HBASE-12075-Implement-Preemptive-Fast-Fail.patch,
0001-HBASE-12075-Implement-Preemptive-Fast-Fail.patch,
0001-HBASE-12075-Implement-Preemptive-Fast-Fail.patch,
0001-Implement-Preemptive-Fast-Fail.patch,
0001-Implement-Preemptive-Fast-Fail.patch,
0001-Implement-Preemptive-Fast-Fail.patch,
0001-Implement-Preemptive-Fast-Fail.patch,
0001-Implement-Preemptive-Fast-Fail.patch
In multi threaded clients, we use a feature developed on 0.89-fb branch
called Preemptive Fast Fail. This allows the client threads which would
potentially fail, fail fast. The idea behind this feature is that we allow,
among the hundreds of client threads, one thread to try and establish
connection with the regionserver and if that succeeds, we mark it as a live
node again. Meanwhile, other threads which are trying to establish connection
to the same server would ideally go into the timeouts which is effectively
unfruitful. We can in those cases return appropriate
