[jira] [Commented] (HBASE-14394) Properly close the connection after reading records from table.
[ https://issues.apache.org/jira/browse/HBASE-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740272#comment-14740272 ] Ashish Singhi commented on HBASE-14394: --- OK. I got your point. What about the above comment, "Also the htable instance in TRRI is already closed as part of TRRI#close() call, so the newly added one is not required" ? > Properly close the connection after reading records from table. > --- > > Key: HBASE-14394 > URL: https://issues.apache.org/jira/browse/HBASE-14394 > Project: HBase > Issue Type: Bug >Reporter: Srikanth Srungarapu >Assignee: Srikanth Srungarapu >Priority: Minor > Attachments: HBASE-14394.patch, HBASE-14394_v2.patch > > > This was brought to notice by one of our observant customers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14394) Properly close the connection after reading records from table.
[ https://issues.apache.org/jira/browse/HBASE-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740287#comment-14740287 ] Srikanth Srungarapu commented on HBASE-14394: - Yeah, the table close method call is redundant. Uploaded new patch. > Properly close the connection after reading records from table. > --- > > Key: HBASE-14394 > URL: https://issues.apache.org/jira/browse/HBASE-14394 > Project: HBase > Issue Type: Bug >Reporter: Srikanth Srungarapu >Assignee: Srikanth Srungarapu >Priority: Minor > Attachments: HBASE-14394.patch, HBASE-14394_v2.patch, > HBASE-14394_v3.patch > > > This was brought to notice by one of our observant customers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-13538) Procedure v2 - client add/delete/modify column family sync (incompatible with branch-1.x)
[ https://issues.apache.org/jira/browse/HBASE-13538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi reassigned HBASE-13538: - Assignee: Ashish Singhi > Procedure v2 - client add/delete/modify column family sync (incompatible with > branch-1.x) > - > > Key: HBASE-13538 > URL: https://issues.apache.org/jira/browse/HBASE-13538 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Srikanth Srungarapu >Assignee: Ashish Singhi > > Client side part of HBASE-13209. > It uses the new procedure code to be know when the procedure is completed, > and have a proper sync/async behavior on add/modify/delete column family. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12298) Support BB usage in PrefixTree
[ https://issues.apache.org/jira/browse/HBASE-12298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740316#comment-14740316 ] Anoop Sam John commented on HBASE-12298: bq.readVariableBytesFromArray(ByteBuff buf, int offset) Need change the method name as per the change in parameter? bq.ByteBufferedCell cell = (ByteBufferedCell)ptSearcher.current(); Why such unconditional type casting? Why cant the type be Cell only? I see the special case just down. Pls add some comments why it is always BBCell appendCurrentTokenToRowBuffer Why cant do positional copy from Buf to Array? Same in ColumnNodeReader#prependTokenToBuffer // TODO : See if we can add only one API to do all this - I see... We dont have positional get API.. It will be easy to add IMO. Suggest do it along with this Jira. Will be simple API. bq.this.block.asSubByteBuffer(this.absoluteValueOffset, valueLength, pair); This pair comes from? The BB and offset is been properly taken from this? (where?) PrefixTreeCell - Why it has to keep a ref to ByteBuff block? Why not use asSubBuffer and keep ref to ByteBuffer for value rather than having a Pair and using it every time? > Support BB usage in PrefixTree > -- > > Key: HBASE-12298 > URL: https://issues.apache.org/jira/browse/HBASE-12298 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Reporter: Anoop Sam John >Assignee: ramkrishna.s.vasudevan > Attachments: HBASE-12298.patch, HBASE-12298_1.patch, > HBASE-12298_2.patch, HBASE-12298_3.patch, HBASE-12298_4 (1).patch, > HBASE-12298_4 (1).patch, HBASE-12298_4 (1).patch, HBASE-12298_4 (1).patch, > HBASE-12298_4 (1).patch, HBASE-12298_4.patch, HBASE-12298_4.patch, > HBASE-12298_4.patch, HBASE-12298_4.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14401) Stamp failed appends with sequenceid too.... Cleans up latches
[ https://issues.apache.org/jira/browse/HBASE-14401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14401: -- Attachment: 14401v3.txt No tests were run. "Rogue processes" Killed. > Stamp failed appends with sequenceid too Cleans up latches > -- > > Key: HBASE-14401 > URL: https://issues.apache.org/jira/browse/HBASE-14401 > Project: HBase > Issue Type: Sub-task > Components: test, wal >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 14401.txt, 14401v3.txt, 14401v3.txt > > > Looking in test output I see we can sometimes get stuck waiting on > sequenceid... The parent issues redo of our semantic makes it so we encounter > failed append more often around damaged WAL. > This patch makes it so we stamp sequenceid always, even if the append fails. > This way all sequenceids are accounted for but more important, the latch on > sequenceid down in WALKey will be cleared.. where before it was not being > cleared (there is no global list of outstanding WALKeys waiting on > sequenceids so no way to clean them up... we don't need such a list if we > ALWAYS stamp the sequenceid). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14408) Separate RPC pool for metadata and data related RPCs, for better HBCK use
Daisuke Kobayashi created HBASE-14408: - Summary: Separate RPC pool for metadata and data related RPCs, for better HBCK use Key: HBASE-14408 URL: https://issues.apache.org/jira/browse/HBASE-14408 Project: HBase Issue Type: Improvement Components: hbck, rpc Affects Versions: 1.0.0 Reporter: Daisuke Kobayashi While a regionserver is in busy state, such like plenty of responseTooSlow being thrown, hbck fails because it cannot reach out to the particular regionserver due to SocketTimeoutException. In such scenario, bunch of inconsistencies could be detected, while the regionserver itself doesn't go down, nor does inconsistencies exist actually. That should lead a lot of users off track and into a -repair frenzy. I'd be nice to have a separate RPC pool for such requests from Hbck. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14398) Create the fake keys required in the scan path to avoid copy to byte[]
[ https://issues.apache.org/jira/browse/HBASE-14398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740289#comment-14740289 ] Anoop Sam John commented on HBASE-14398: Why we need FirstOnRowColHintByteBufferedCell? How it is different from FirstOnRowColByteBufferedCell? > Create the fake keys required in the scan path to avoid copy to byte[] > -- > > Key: HBASE-14398 > URL: https://issues.apache.org/jira/browse/HBASE-14398 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-14398.patch > > > Already we have created some fake keys for the ByteBufferedCells so that we > can avoid the copy requried to create fake keys. This JIRA aims to fill up > all such places so that the Offheap BBs are not copied to onheap byte[]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14394) Properly close the connection after reading records from table.
[ https://issues.apache.org/jira/browse/HBASE-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740290#comment-14740290 ] Ashish Singhi commented on HBASE-14394: --- +1 (non-binding) > Properly close the connection after reading records from table. > --- > > Key: HBASE-14394 > URL: https://issues.apache.org/jira/browse/HBASE-14394 > Project: HBase > Issue Type: Bug >Reporter: Srikanth Srungarapu >Assignee: Srikanth Srungarapu >Priority: Minor > Attachments: HBASE-14394.patch, HBASE-14394_v2.patch, > HBASE-14394_v3.patch > > > This was brought to notice by one of our observant customers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14406) The dataframe datasource filter is wrong, and will result in data loss or unexpected behavior
[ https://issues.apache.org/jira/browse/HBASE-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-14406: Component/s: (was: hbase) spark > The dataframe datasource filter is wrong, and will result in data loss or > unexpected behavior > - > > Key: HBASE-14406 > URL: https://issues.apache.org/jira/browse/HBASE-14406 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Zhan Zhang >Priority: Blocker > > Following condition will result in the same filter. It will have data loss > with the current filter construction. > col1 > 4 && col2 < 3 > col1 > 4 || col2 < 3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14406) The dataframe datasource filter is wrong, and will result in data loss or unexpected behavior
[ https://issues.apache.org/jira/browse/HBASE-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-14406: Fix Version/s: 2.0.0 > The dataframe datasource filter is wrong, and will result in data loss or > unexpected behavior > - > > Key: HBASE-14406 > URL: https://issues.apache.org/jira/browse/HBASE-14406 > Project: HBase > Issue Type: Bug > Components: spark >Affects Versions: 2.0.0 >Reporter: Zhan Zhang >Priority: Blocker > Fix For: 2.0.0 > > > Following condition will result in the same filter. It will have data loss > with the current filter construction. > col1 > 4 && col2 < 3 > col1 > 4 || col2 < 3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14361) ReplicationSink should create Connection instances lazily
[
https://issues.apache.org/jira/browse/HBASE-14361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740293#comment-14740293
]
Hadoop QA commented on HBASE-14361:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12755303/HBASE-14361_v1.patch
against master branch at commit ae3176b9c0f63644deffc82dc424d219c1472818.
ATTACHMENT ID: 12755303
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15551//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15551//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15551//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15551//console
This message is automatically generated.
> ReplicationSink should create Connection instances lazily
> -
>
> Key: HBASE-14361
> URL: https://issues.apache.org/jira/browse/HBASE-14361
> Project: HBase
> Issue Type: Task
> Components: Replication
>Reporter: Nick Dimiduk
>Assignee: Heng Chen
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.15, 1.0.3, 1.1.3
>
> Attachments: HBASE-14361-0.98.patch, HBASE-14361.patch,
> HBASE-14361_v1.patch, hmaster.log
>
>
> Over on HBASE-12911 I have a patch that registers Connection instances with
> the metrics system. In both standalone server and tll client applications, I
> was surprised to see multiple connection objects showing up that are unused.
> These are pretty heavy objects, including lots of client threads for the
> batch pool. We should track these down and remove them -- if they're not some
> kind of phantom artifacts of my WIP patch over there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14406) The dataframe datasource filter is wrong, and will result in data loss or unexpected behavior
[ https://issues.apache.org/jira/browse/HBASE-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-14406: Affects Version/s: 2.0.0 > The dataframe datasource filter is wrong, and will result in data loss or > unexpected behavior > - > > Key: HBASE-14406 > URL: https://issues.apache.org/jira/browse/HBASE-14406 > Project: HBase > Issue Type: Bug > Components: spark >Affects Versions: 2.0.0 >Reporter: Zhan Zhang >Priority: Blocker > Fix For: 2.0.0 > > > Following condition will result in the same filter. It will have data loss > with the current filter construction. > col1 > 4 && col2 < 3 > col1 > 4 || col2 < 3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14392) [tests] TestLogRollingNoCluster fails on master from time to time
[ https://issues.apache.org/jira/browse/HBASE-14392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740255#comment-14740255 ] Hudson commented on HBASE-14392: SUCCESS: Integrated in HBase-1.2-IT #137 (See [https://builds.apache.org/job/HBase-1.2-IT/137/]) HBASE-14392 [tests] TestLogRollingNoCluster fails on master from time to time (stack: rev a2bca2748d5d310b1ea06674b106caa2c9b1b7d2) * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollingNoCluster.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogWriter.java > [tests] TestLogRollingNoCluster fails on master from time to time > - > > Key: HBASE-14392 > URL: https://issues.apache.org/jira/browse/HBASE-14392 > Project: HBase > Issue Type: Bug > Components: test >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 14392.txt, 14392.txt, 14392v2.txt, 14392v2.txt, > 14392v2.txt > > > TestLogRollingNoCluster fails from time to time on a rig I have running here. > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; > support was removed in 8.0 > Running org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.155 sec - > in org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster > Results : > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 > The test itself is a bit odd. We apparently have seen NPEs around close of a > WAL while trying to sync... which I suppose makes sense if two threads > involved. > Attached patch just fails the sync silently if stream is null... lets presume > it a close. Adds a sync on write of trailer too... > This patch seems to have gotten rid of the odd failure seen on a particular > box here if I keep cycling the test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740280#comment-14740280
]
Hiroshi Ikeda commented on HBASE-14331:
---
That seems a mere bug.
AnnotationReadingPriorityFunction.getDeadline
{code}
public long getDeadline(RequestHeader header, Message param) {
String methodName = header.getMethodName();
if (methodName.equalsIgnoreCase("scan")) {
ScanRequest request = (ScanRequest)param;
if (!request.hasScannerId()) {
return 0;
}
// get the 'virtual time' of the scanner, and applies sqrt() to get a
// nice curve for the delay. More a scanner is used the less priority it
gets.
// The weight is used to have more control on the delay.
long vtime = rpcServices.getScannerVirtualTime(request.getScannerId());
return Math.round(Math.sqrt(vtime * scanVirtualTimeWeight));
}
return 0;
}
{code}
> long vtime = rpcServices.getScannerVirtualTime(request.getScannerId());
doesn't give the 'virtual time' for the request. That should be:
long vtime = request.getNextCallSeq();
And It is subtle but seems not to be changed. (But I also feel that method is
not so trivial to be frequently used in a comparator...)
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331.patch,
> HBASE-14331.patch, SemaphoreBasedBlockingQueue.java,
> SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14396) audit log should record the operation object
[ https://issues.apache.org/jira/browse/HBASE-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740286#comment-14740286 ] Ashish Singhi commented on HBASE-14396: --- [~haitao-tony], I think you missed HBASE-13235. Your expectations are already handled as part of that. After checking it, please close this issue. > audit log should record the operation object > - > > Key: HBASE-14396 > URL: https://issues.apache.org/jira/browse/HBASE-14396 > Project: HBase > Issue Type: Bug >Reporter: sunhaitao > > Currently the hbase audit log only records the user and scope,we can't know > which table the user is operating on unless the scope is table. > It would be better to know what's going on if we record the exact table and > column family we are operating on besides the scope. > String logMessage = > "Access " + (result.isAllowed() ? "allowed" : "denied") + " for user " > + (result.getUser() != null ? result.getUser().getShortName() : > "UNKNOWN") > + "; reason: " + result.getReason() + "; remote address: " > + (remoteAddr != null ? remoteAddr : "") + "; request: " + > result.getRequest() > + "; context: " + result.toContextString(); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740304#comment-14740304
]
Hiroshi Ikeda commented on HBASE-14331:
---
Sorry I referred a old blanch, and string comparison has been replaced to type
check.
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> HBASE-14331-V2.patch, HBASE-14331-V3.patch, HBASE-14331.patch,
> HBASE-14331.patch, SemaphoreBasedBlockingQueue.java,
> SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14398) Create the fake keys required in the scan path to avoid copy to byte[]
[ https://issues.apache.org/jira/browse/HBASE-14398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-14398: --- Issue Type: Sub-task (was: Improvement) Parent: HBASE-11425 > Create the fake keys required in the scan path to avoid copy to byte[] > -- > > Key: HBASE-14398 > URL: https://issues.apache.org/jira/browse/HBASE-14398 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > > Already we have created some fake keys for the ByteBufferedCells so that we > can avoid the copy requried to create fake keys. This JIRA aims to fill up > all such places so that the Offheap BBs are not copied to onheap byte[]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14403) [0.98] Fix TestInterfaceAudienceAnnotations failures
[
https://issues.apache.org/jira/browse/HBASE-14403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740295#comment-14740295
]
Hadoop QA commented on HBASE-14403:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12755291/HBASE-14403-0.98.patch
against 0.98 branch at commit 411b516f5156730075558d91b69c3c3e09fb200d.
ATTACHMENT ID: 12755291
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
23 warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:red}-1 core zombie tests{color}. There are 5 zombie test(s):
at
org.apache.hadoop.hbase.mapreduce.TestImportExport.testWithFilter(TestImportExport.java:481)
at
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles.testSimpleLoad(TestLoadIncrementalHFiles.java:103)
at
org.apache.hadoop.hbase.mapreduce.TestTableMapReduceBase.testCombiner(TestTableMapReduceBase.java:106)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15547//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15547//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15547//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15547//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15547//console
This message is automatically generated.
> [0.98] Fix TestInterfaceAudienceAnnotations failures
>
>
> Key: HBASE-14403
> URL: https://issues.apache.org/jira/browse/HBASE-14403
> Project: HBase
> Issue Type: Task
>Reporter: Nick Dimiduk
>Assignee: Andrew Purtell
> Fix For: 0.98.15
>
> Attachments: HBASE-14403-0.98.patch
>
>
> HBASE-14382 backported {{TestInterfaceAudienceAnnotations}} to 0.98. This
> test now fails due to missing annotations.
> {noformat}
> 2015-09-10 16:21:41,705 DEBUG [main] hbase.ClassFinder(162): Will look for
> classes in
> /Users/ndimiduk/repos/hbase/hbase-client/target/classes/org/apache/hadoop/hbase
> 2015-09-10 16:21:41,710 DEBUG [main] hbase.ClassFinder(162): Will look for
> classes in
> /Users/ndimiduk/repos/hbase/hbase-annotations/target/classes/org/apache/hadoop/hbase
> 2015-09-10 16:21:41,710 DEBUG [main] hbase.ClassFinder(162): Will look for
> classes in
> /Users/ndimiduk/repos/hbase/hbase-common/target/classes/org/apache/hadoop/hbase
> 2015-09-10 16:21:41,710 DEBUG [main] hbase.ClassFinder(162): Will look for
> classes in
> /Users/ndimiduk/repos/hbase/hbase-protocol/target/classes/org/apache/hadoop/hbase
> 2015-09-10 16:21:43,471 INFO [main]
> hbase.TestInterfaceAudienceAnnotations(293): These are the classes that DO
> NOT have @InterfaceStability annotation:
> 2015-09-10 16:21:43,471 INFO [main]
> hbase.TestInterfaceAudienceAnnotations(295): class
> org.apache.hadoop.hbase.replication.ReplicationException
> 2015-09-10 16:21:43,471 INFO [main]
> hbase.TestInterfaceAudienceAnnotations(295): class
> org.apache.hadoop.hbase.security.visibility.VisibilityControllerNotReadyException
> 2015-09-10 16:21:43,472 INFO [main]
> hbase.TestInterfaceAudienceAnnotations(295): class
> org.apache.hadoop.hbase.snapshot.ExportSnapshotException
> 2015-09-10 16:21:43,476 DEBUG [main] hbase.ClassFinder(162): Will
[jira] [Updated] (HBASE-14398) Create the fake keys required in the scan path to avoid copy to byte[]
[ https://issues.apache.org/jira/browse/HBASE-14398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-14398: --- Status: Patch Available (was: Open) > Create the fake keys required in the scan path to avoid copy to byte[] > -- > > Key: HBASE-14398 > URL: https://issues.apache.org/jira/browse/HBASE-14398 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-14398.patch > > > Already we have created some fake keys for the ByteBufferedCells so that we > can avoid the copy requried to create fake keys. This JIRA aims to fill up > all such places so that the Offheap BBs are not copied to onheap byte[]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14398) Create the fake keys required in the scan path to avoid copy to byte[]
[ https://issues.apache.org/jira/browse/HBASE-14398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-14398: --- Attachment: HBASE-14398.patch This patch tries to avoid copies while creating fake keys particularly when we have more cols and we try to explicitly add columns. > Create the fake keys required in the scan path to avoid copy to byte[] > -- > > Key: HBASE-14398 > URL: https://issues.apache.org/jira/browse/HBASE-14398 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-14398.patch > > > Already we have created some fake keys for the ByteBufferedCells so that we > can avoid the copy requried to create fake keys. This JIRA aims to fill up > all such places so that the Offheap BBs are not copied to onheap byte[]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14394) Properly close the connection after reading records from table.
[ https://issues.apache.org/jira/browse/HBASE-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Srikanth Srungarapu updated HBASE-14394: Attachment: HBASE-14394_v3.patch > Properly close the connection after reading records from table. > --- > > Key: HBASE-14394 > URL: https://issues.apache.org/jira/browse/HBASE-14394 > Project: HBase > Issue Type: Bug >Reporter: Srikanth Srungarapu >Assignee: Srikanth Srungarapu >Priority: Minor > Attachments: HBASE-14394.patch, HBASE-14394_v2.patch, > HBASE-14394_v3.patch > > > This was brought to notice by one of our observant customers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14407) NotServingRegion: hbase region closed forever
[ https://issues.apache.org/jira/browse/HBASE-14407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shuaifeng Zhou updated HBASE-14407: --- Attachment: hs4.log master.log attached is logs on master and regionserver > NotServingRegion: hbase region closed forever > - > > Key: HBASE-14407 > URL: https://issues.apache.org/jira/browse/HBASE-14407 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.10, 1.1.2 >Reporter: Shuaifeng Zhou >Assignee: Shuaifeng Zhou > Attachments: hs4.log, master.log > > > I found a situation may cause region closed forever, and this situation > happend usually on my cluster, version is 0.98.10, but 1.1.2 also have the > problem: > 1, master send region open to regionserver > 2, rs open a handler do openregion > 3, rs return resopnse to master > 3, master not received the response, or timeout, send open region again > 4, rs already opened the region > 5, master processAlreadyOpenedRegion, update regionstate open in master > memory > 6, master received zk message region opened(for some reason late, eg: net > work), and triger update regionstate open, but find that region already > opened, ERROR! > 7, master send close region, and region be closed forever. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14306) Refine RegionGroupingProvider: fix issues and make it more scalable
[
https://issues.apache.org/jira/browse/HBASE-14306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740300#comment-14740300
]
Hadoop QA commented on HBASE-14306:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12755305/HBASE-14306_v3.patch
against master branch at commit ae3176b9c0f63644deffc82dc424d219c1472818.
ATTACHMENT ID: 12755305
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn post-site goal
to fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.util.TestProcessBasedCluster
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15550//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15550//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15550//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15550//console
This message is automatically generated.
> Refine RegionGroupingProvider: fix issues and make it more scalable
> ---
>
> Key: HBASE-14306
> URL: https://issues.apache.org/jira/browse/HBASE-14306
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Affects Versions: 2.0.0, 1.1.2
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14306.patch, HBASE-14306_v2.patch,
> HBASE-14306_v3.patch
>
>
> There're multiple issues in RegionGroupingProvider, including:
> * The provider cache in it is using byte array as the key of
> ConcurrentHashMap, which is not right (the reason is
> [here|http://stackoverflow.com/questions/1058149/using-a-byte-array-as-hashmap-key-java])
> * It's using IdentityGroupingStrategy to get group and use it as key of the
> cache, which means the cache will include an entry for each region. This is
> especially unnecessary when using BoundedRegionGroupingProvider
> Besides fixing the above issues, I suggest to change
> BoundedRegionGroupingProvider from a *provider* to a pluggable *strategy*,
> which will make the whole picture much more clear.
> For more details, please refer to the patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14380) Correct data also getting skipped along with bad data in importTsv bulk load thru TsvImporterTextMapper
[
https://issues.apache.org/jira/browse/HBASE-14380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741024#comment-14741024
]
Ted Yu commented on HBASE-14380:
Test result was good:
{code}
Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 487.604 sec -
in org.apache.hadoop.hbase.mapreduce.TestImportTsv
{code}
8 minutes for a test ? Looks like we should speed up or break TestImportTsv
into multiple, smaller tests.
Will attach a patch that addresses line length warnings.
> Correct data also getting skipped along with bad data in importTsv bulk load
> thru TsvImporterTextMapper
> ---
>
> Key: HBASE-14380
> URL: https://issues.apache.org/jira/browse/HBASE-14380
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Bhupendra Kumar Jain
>Assignee: Bhupendra Kumar Jain
> Attachments: 0001-HBASE-14380.patch, HBASE-14380_v1.patch
>
>
> Cosider the input data is as below
> ROWKEY, TIEMSTAMP, Col_Value
> r1,1,v1 >> Correct line
> r1 >> Bad line
> r1,3,v3 >> Correct line
> r1,4,v4 >> Correct line
> When data is bulk loaded using importTsv with mapper as TsvImporterTextMapper
> , All the lines are getting ignored even though skipBadLines is set to true.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14380) Correct data gets skipped along with bad data in importTsv bulk load thru TsvImporterTextMapper
[ https://issues.apache.org/jira/browse/HBASE-14380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741221#comment-14741221 ] Hudson commented on HBASE-14380: SUCCESS: Integrated in HBase-1.3-IT #147 (See [https://builds.apache.org/job/HBase-1.3-IT/147/]) HBASE-14380 Correct data gets skipped along with bad data in importTsv bulk load thru TsvImporterTextMapper (Bhupendra Kumar Jain) (tedyu: rev 84dbe39f5d424159d7d57ed63f8a654a922104eb) * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TextSortReducer.java * hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTsv.java > Correct data gets skipped along with bad data in importTsv bulk load thru > TsvImporterTextMapper > --- > > Key: HBASE-14380 > URL: https://issues.apache.org/jira/browse/HBASE-14380 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Bhupendra Kumar Jain >Assignee: Bhupendra Kumar Jain > Fix For: 2.0.0, 1.3.0 > > Attachments: 0001-HBASE-14380.patch, 14380-v2.txt, > HBASE-14380_v1.patch > > > Cosider the input data is as below > ROWKEY, TIEMSTAMP, Col_Value > r1,1,v1 >> Correct line > r1 >> Bad line > r1,3,v3 >> Correct line > r1,4,v4 >> Correct line > When data is bulk loaded using importTsv with mapper as TsvImporterTextMapper > , All the lines are getting ignored even though skipBadLines is set to true. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-6617) ReplicationSourceManager should be able to track multiple WAL paths
[ https://issues.apache.org/jira/browse/HBASE-6617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741278#comment-14741278 ] Yu Li commented on HBASE-6617: -- Done. Please let me know if any changes required for the release note. [~tedyu], thanks for all the efforts on review and coordination! [~zjushch] and [~busbey], thanks for the review, and please feel free to let me know if you found any addendum issue need to be resolved later. > ReplicationSourceManager should be able to track multiple WAL paths > --- > > Key: HBASE-6617 > URL: https://issues.apache.org/jira/browse/HBASE-6617 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Ted Yu >Assignee: Yu Li > Fix For: 2.0.0, 1.3.0 > > Attachments: 6617-v11.patch, HBASE-6617.branch-1.patch, > HBASE-6617.branch-1.v2.patch, HBASE-6617.branch-1.v2.patch, > HBASE-6617.branch-1.v2.patch, HBASE-6617.patch, HBASE-6617_v10.patch, > HBASE-6617_v11.patch, HBASE-6617_v12.patch, HBASE-6617_v2.patch, > HBASE-6617_v3.patch, HBASE-6617_v4.patch, HBASE-6617_v7.patch, > HBASE-6617_v9.patch > > > Currently ReplicationSourceManager uses logRolled() to receive notification > about new HLog and remembers it in latestPath. > When region server has multiple WAL support, we need to keep track of > multiple Path's in ReplicationSourceManager -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14398) Create the fake keys required in the scan path to avoid copy to byte[]
[ https://issues.apache.org/jira/browse/HBASE-14398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741090#comment-14741090 ] stack commented on HBASE-14398: --- Are we doing something wrong that we need all these specialized types? > Create the fake keys required in the scan path to avoid copy to byte[] > -- > > Key: HBASE-14398 > URL: https://issues.apache.org/jira/browse/HBASE-14398 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-14398.patch, HBASE-14398_1.patch > > > Already we have created some fake keys for the ByteBufferedCells so that we > can avoid the copy requried to create fake keys. This JIRA aims to fill up > all such places so that the Offheap BBs are not copied to onheap byte[]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14370) Use separate thread for calling ZKPermissionWatcher#refreshNodes()
[
https://issues.apache.org/jira/browse/HBASE-14370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741104#comment-14741104
]
Ted Yu commented on HBASE-14370:
QA environment issue:
{code}
testRegionCrossingHFileSplitRowBloom(org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesUseSecurityEndPoint)
Time elapsed: 1.308 sec <<< ERROR!
org.apache.hadoop.ipc.RemoteException: unable to create new native thread
{code}
I don't think any of the above test failure is involved with enabling ACL.
I am running the tests locally to make sure they pass.
> Use separate thread for calling ZKPermissionWatcher#refreshNodes()
> --
>
> Key: HBASE-14370
> URL: https://issues.apache.org/jira/browse/HBASE-14370
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.15, 1.0.3, 1.1.3
>
> Attachments: 14370-branch-1-v10.txt, 14370-branch-1-v10.txt,
> 14370-v1.txt, 14370-v10.txt, 14370-v3.txt, 14370-v5.txt, 14370-v7.txt,
> 14370-v8.txt, 14370-wait-nofity-v2.txt, 14370-wait-nofity.txt,
> hbase-14370_v4.patch, test-acl3-branch-1.stack
>
>
> I came off a support case (0.98.0) where main zk thread was seen doing the
> following:
> {code}
> at
> org.apache.hadoop.hbase.security.access.ZKPermissionWatcher.refreshAuthManager(ZKPermissionWatcher.java:152)
> at
> org.apache.hadoop.hbase.security.access.ZKPermissionWatcher.refreshNodes(ZKPermissionWatcher.java:135)
> at
> org.apache.hadoop.hbase.security.access.ZKPermissionWatcher.nodeChildrenChanged(ZKPermissionWatcher.java:121)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:348)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
> {code}
> There were 62000 nodes under /acl due to lack of fix from HBASE-12635,
> leading to slowness in table creation because zk notification for region
> offline was blocked by the above.
> The attached patch separates refreshNodes() call into its own thread.
> Thanks to Enis and Devaraj for offline discussion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-6617) ReplicationSourceManager should be able to track multiple WAL paths
[ https://issues.apache.org/jira/browse/HBASE-6617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated HBASE-6617: - Release Note: ReplicationSourceManager now could track multiple wal paths. Notice that although most changes are internal and all metrics names remain the same, signature of below methods in MetricsSource are changed: 1. refreshAgeOfLastShippedOp now requires a String parameter which indicates the wal group id of the reporter 2. setAgeOfLastShippedOp also adds a String parameter for wal group id > ReplicationSourceManager should be able to track multiple WAL paths > --- > > Key: HBASE-6617 > URL: https://issues.apache.org/jira/browse/HBASE-6617 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Ted Yu >Assignee: Yu Li > Fix For: 2.0.0, 1.3.0 > > Attachments: 6617-v11.patch, HBASE-6617.branch-1.patch, > HBASE-6617.branch-1.v2.patch, HBASE-6617.branch-1.v2.patch, > HBASE-6617.branch-1.v2.patch, HBASE-6617.patch, HBASE-6617_v10.patch, > HBASE-6617_v11.patch, HBASE-6617_v12.patch, HBASE-6617_v2.patch, > HBASE-6617_v3.patch, HBASE-6617_v4.patch, HBASE-6617_v7.patch, > HBASE-6617_v9.patch > > > Currently ReplicationSourceManager uses logRolled() to receive notification > about new HLog and remembers it in latestPath. > When region server has multiple WAL support, we need to keep track of > multiple Path's in ReplicationSourceManager -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14307) Incorrect use of positional read api in HFileBlock
[
https://issues.apache.org/jira/browse/HBASE-14307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741021#comment-14741021
]
Hudson commented on HBASE-14307:
FAILURE: Integrated in HBase-1.0 #1046 (See
[https://builds.apache.org/job/HBase-1.0/1046/])
HBASE-14307 Addendum fixes compilation for TestHFileBlockPositionalRead (tedyu:
rev 677b28ab056bb2cdbc41a04b60ea031303c1f327)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockPositionalRead.java
> Incorrect use of positional read api in HFileBlock
> --
>
> Key: HBASE-14307
> URL: https://issues.apache.org/jira/browse/HBASE-14307
> Project: HBase
> Issue Type: Bug
>Reporter: Shradha Revankar
>Assignee: Chris Nauroth
> Fix For: 2.0.0, 0.98.15, 1.2.1, 1.0.3, 1.1.3
>
> Attachments: 14307-branch-1.addendum, HBASE-14307.001.master.patch,
> HBASE-14307.002.master.patch, HBASE-14307.003.patch
>
>
> Considering that {{read()}} is not guaranteed to read all bytes,
> I'm interested to understand this particular piece of code and why is partial
> read treated as an error :
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java#L1446-L1450
> Particularly, if hbase were to use a different filesystem, say
> WebhdfsFileSystem, this would not work, please also see
> https://issues.apache.org/jira/browse/HDFS-8943 for discussion around this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-6617) ReplicationSourceManager should be able to track multiple WAL paths
[ https://issues.apache.org/jira/browse/HBASE-6617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741068#comment-14741068 ] Sean Busbey commented on HBASE-6617: Would still like a release note for LimitedPrivate things that broke compatibility. (i.e. replication.regionserver.MetricsSource) > ReplicationSourceManager should be able to track multiple WAL paths > --- > > Key: HBASE-6617 > URL: https://issues.apache.org/jira/browse/HBASE-6617 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Ted Yu >Assignee: Yu Li > Fix For: 2.0.0, 1.3.0 > > Attachments: 6617-v11.patch, HBASE-6617.branch-1.patch, > HBASE-6617.branch-1.v2.patch, HBASE-6617.branch-1.v2.patch, > HBASE-6617.branch-1.v2.patch, HBASE-6617.patch, HBASE-6617_v10.patch, > HBASE-6617_v11.patch, HBASE-6617_v12.patch, HBASE-6617_v2.patch, > HBASE-6617_v3.patch, HBASE-6617_v4.patch, HBASE-6617_v7.patch, > HBASE-6617_v9.patch > > > Currently ReplicationSourceManager uses logRolled() to receive notification > about new HLog and remembers it in latestPath. > When region server has multiple WAL support, we need to keep track of > multiple Path's in ReplicationSourceManager -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14401) Stamp failed appends with sequenceid too.... Cleans up latches
[
https://issues.apache.org/jira/browse/HBASE-14401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741224#comment-14741224
]
Hadoop QA commented on HBASE-14401:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12755419/14401v3.txt
against master branch at commit c438052cc2280121727d4ae0883f0b76cad5816a.
ATTACHMENT ID: 12755419
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 3 zombie test(s):
at
org.apache.hadoop.hbase.client.TestSnapshotCloneIndependence.testOnlineSnapshotRegionOperationsIndependent(TestSnapshotCloneIndependence.java:172)
at
org.apache.hadoop.hbase.client.TestFromClientSide.testUpdatesWithMajorCompaction(TestFromClientSide.java:3581)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15563//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15563//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15563//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15563//console
This message is automatically generated.
> Stamp failed appends with sequenceid too Cleans up latches
> --
>
> Key: HBASE-14401
> URL: https://issues.apache.org/jira/browse/HBASE-14401
> Project: HBase
> Issue Type: Sub-task
> Components: test, wal
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14401.txt, 14401v3.txt, 14401v3.txt, 14401v3.txt
>
>
> Looking in test output I see we can sometimes get stuck waiting on
> sequenceid... The parent issues redo of our semantic makes it so we encounter
> failed append more often around damaged WAL.
> This patch makes it so we stamp sequenceid always, even if the append fails.
> This way all sequenceids are accounted for but more important, the latch on
> sequenceid down in WALKey will be cleared.. where before it was not being
> cleared (there is no global list of outstanding WALKeys waiting on
> sequenceids so no way to clean them up... we don't need such a list if we
> ALWAYS stamp the sequenceid).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-6617) ReplicationSourceManager should be able to track multiple WAL paths
[ https://issues.apache.org/jira/browse/HBASE-6617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741220#comment-14741220 ] Hudson commented on HBASE-6617: --- SUCCESS: Integrated in HBase-1.3-IT #147 (See [https://builds.apache.org/job/HBase-1.3-IT/147/]) HBASE-6617 ReplicationSourceManager should be able to track multiple WAL paths (Yu Li) (tedyu: rev be96bb6adf77f4bbcc1513cb407a705ac4624a0f) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/multiwal/TestReplicationEndpointWithMultipleWAL.java * hbase-server/src/main/java/org/apache/hadoop/hbase/wal/DefaultWALProvider.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationWALReaderManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSource.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java * hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALFactory.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationEndpoint.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/multiwal/TestReplicationKillMasterRSCompressedWithMultipleWAL.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/multiwal/TestReplicationSyncUpToolWithMultipleWAL.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpointNoMaster.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/ReplicationEndpoint.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HBaseInterClusterReplicationEndpoint.java > ReplicationSourceManager should be able to track multiple WAL paths > --- > > Key: HBASE-6617 > URL: https://issues.apache.org/jira/browse/HBASE-6617 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Ted Yu >Assignee: Yu Li > Fix For: 2.0.0, 1.3.0 > > Attachments: 6617-v11.patch, HBASE-6617.branch-1.patch, > HBASE-6617.branch-1.v2.patch, HBASE-6617.branch-1.v2.patch, > HBASE-6617.branch-1.v2.patch, HBASE-6617.patch, HBASE-6617_v10.patch, > HBASE-6617_v11.patch, HBASE-6617_v12.patch, HBASE-6617_v2.patch, > HBASE-6617_v3.patch, HBASE-6617_v4.patch, HBASE-6617_v7.patch, > HBASE-6617_v9.patch > > > Currently ReplicationSourceManager uses logRolled() to receive notification > about new HLog and remembers it in latestPath. > When region server has multiple WAL support, we need to keep track of > multiple Path's in ReplicationSourceManager -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12751) Allow RowLock to be reader writer
[
https://issues.apache.org/jira/browse/HBASE-12751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741287#comment-14741287
]
Hadoop QA commented on HBASE-12751:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12755418/12751.rebased.v35.txt
against master branch at commit c438052cc2280121727d4ae0883f0b76cad5816a.
ATTACHMENT ID: 12755418
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 99 new
or modified tests.
{color:red}-1 Anti-pattern{color}. The patch appears to
have anti-pattern where BYTES_COMPARATOR was omitted:
-getRegionInfo(), -1, new TreeMap());.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 5 zombie test(s):
at
org.apache.hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpoint.testRegionReplicaReplicationIgnoresDisabledTables(TestRegionReplicaReplicationEndpoint.java:350)
at
org.apache.hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpoint.testRegionReplicaReplicationIgnoresDroppedTables(TestRegionReplicaReplicationEndpoint.java:336)
at
org.apache.hadoop.hbase.replication.TestMasterReplication.testCyclicReplication3(TestMasterReplication.java:220)
at
org.apache.hadoop.hbase.TestIOFencing.testFencingAroundCompaction(TestIOFencing.java:229)
at
org.apache.hadoop.hbase.replication.TestPerTableCFReplication.testPerTableCFReplication(TestPerTableCFReplication.java:334)
at
org.apache.hadoop.hbase.replication.regionserver.TestReplicationWALReaderManager.test(TestReplicationWALReaderManager.java:184)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15564//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15564//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15564//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15564//console
This message is automatically generated.
> Allow RowLock to be reader writer
> -
>
> Key: HBASE-12751
> URL: https://issues.apache.org/jira/browse/HBASE-12751
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.3.0
>
> Attachments: 12751.rebased.v25.txt, 12751.rebased.v26.txt,
> 12751.rebased.v26.txt, 12751.rebased.v27.txt, 12751.rebased.v29.txt,
> 12751.rebased.v31.txt, 12751.rebased.v32.txt, 12751.rebased.v32.txt,
> 12751.rebased.v33.txt, 12751.rebased.v34.txt, 12751.rebased.v35.txt,
> 12751.rebased.v35.txt, 12751v22.txt, 12751v23.txt, 12751v23.txt,
> 12751v23.txt, 12751v23.txt, HBASE-12751-v1.patch, HBASE-12751-v10.patch,
> HBASE-12751-v10.patch, HBASE-12751-v11.patch, HBASE-12751-v12.patch,
> HBASE-12751-v13.patch, HBASE-12751-v14.patch, HBASE-12751-v15.patch,
> HBASE-12751-v16.patch, HBASE-12751-v17.patch, HBASE-12751-v18.patch,
> HBASE-12751-v19 (1).patch, HBASE-12751-v19.patch, HBASE-12751-v2.patch,
> HBASE-12751-v20.patch, HBASE-12751-v20.patch, HBASE-12751-v21.patch,
> HBASE-12751-v3.patch, HBASE-12751-v4.patch, HBASE-12751-v5.patch,
> HBASE-12751-v6.patch, HBASE-12751-v7.patch, HBASE-12751-v8.patch,
> HBASE-12751-v9.patch, HBASE-12751.patch
>
>
> Right now every write operation grabs a row lock. This is to prevent values
> from changing during a read modify write operation (increment
[jira] [Updated] (HBASE-14380) Correct data gets skipped along with bad data in importTsv bulk load thru TsvImporterTextMapper
[ https://issues.apache.org/jira/browse/HBASE-14380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14380: --- Summary: Correct data gets skipped along with bad data in importTsv bulk load thru TsvImporterTextMapper (was: Correct data also getting skipped along with bad data in importTsv bulk load thru TsvImporterTextMapper) > Correct data gets skipped along with bad data in importTsv bulk load thru > TsvImporterTextMapper > --- > > Key: HBASE-14380 > URL: https://issues.apache.org/jira/browse/HBASE-14380 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Bhupendra Kumar Jain >Assignee: Bhupendra Kumar Jain > Attachments: 0001-HBASE-14380.patch, 14380-v2.txt, > HBASE-14380_v1.patch > > > Cosider the input data is as below > ROWKEY, TIEMSTAMP, Col_Value > r1,1,v1 >> Correct line > r1 >> Bad line > r1,3,v3 >> Correct line > r1,4,v4 >> Correct line > When data is bulk loaded using importTsv with mapper as TsvImporterTextMapper > , All the lines are getting ignored even though skipBadLines is set to true. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-6617) ReplicationSourceManager should be able to track multiple WAL paths
[ https://issues.apache.org/jira/browse/HBASE-6617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741052#comment-14741052 ] Sean Busbey commented on HBASE-6617: that sounds reasonable. presuming this is going to 1.3+ and folks are reasonably certain any issues I've already brought up were addressed. > ReplicationSourceManager should be able to track multiple WAL paths > --- > > Key: HBASE-6617 > URL: https://issues.apache.org/jira/browse/HBASE-6617 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Ted Yu >Assignee: Yu Li > Fix For: 2.0.0, 1.3.0 > > Attachments: 6617-v11.patch, HBASE-6617.branch-1.patch, > HBASE-6617.branch-1.v2.patch, HBASE-6617.branch-1.v2.patch, > HBASE-6617.branch-1.v2.patch, HBASE-6617.patch, HBASE-6617_v10.patch, > HBASE-6617_v11.patch, HBASE-6617_v12.patch, HBASE-6617_v2.patch, > HBASE-6617_v3.patch, HBASE-6617_v4.patch, HBASE-6617_v7.patch, > HBASE-6617_v9.patch > > > Currently ReplicationSourceManager uses logRolled() to receive notification > about new HLog and remembers it in latestPath. > When region server has multiple WAL support, we need to keep track of > multiple Path's in ReplicationSourceManager -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-6617) ReplicationSourceManager should be able to track multiple WAL paths
[ https://issues.apache.org/jira/browse/HBASE-6617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741073#comment-14741073 ] Ted Yu commented on HBASE-6617: --- [~carp84]: Please add release note > ReplicationSourceManager should be able to track multiple WAL paths > --- > > Key: HBASE-6617 > URL: https://issues.apache.org/jira/browse/HBASE-6617 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Ted Yu >Assignee: Yu Li > Fix For: 2.0.0, 1.3.0 > > Attachments: 6617-v11.patch, HBASE-6617.branch-1.patch, > HBASE-6617.branch-1.v2.patch, HBASE-6617.branch-1.v2.patch, > HBASE-6617.branch-1.v2.patch, HBASE-6617.patch, HBASE-6617_v10.patch, > HBASE-6617_v11.patch, HBASE-6617_v12.patch, HBASE-6617_v2.patch, > HBASE-6617_v3.patch, HBASE-6617_v4.patch, HBASE-6617_v7.patch, > HBASE-6617_v9.patch > > > Currently ReplicationSourceManager uses logRolled() to receive notification > about new HLog and remembers it in latestPath. > When region server has multiple WAL support, we need to keep track of > multiple Path's in ReplicationSourceManager -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14400) Fix HBase RPC protection documentation
[ https://issues.apache.org/jira/browse/HBASE-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-14400: Release Note: To use rpc protection in HBase, set the value of 'hbase.rpc.protection' to: 'authentication' : simple authentication using kerberos 'integrity' : authentication and integrity 'privacy' : authentication and confidentiality Earlier, HBase reference guide erroneously mentioned in some places to set the value to 'auth-conf'. This patch fixes the guide and adds temporary support for erroneously recommended values. > Fix HBase RPC protection documentation > -- > > Key: HBASE-14400 > URL: https://issues.apache.org/jira/browse/HBASE-14400 > Project: HBase > Issue Type: Bug > Components: encryption, rpc, security >Reporter: Apekshit Sharma >Assignee: Apekshit Sharma >Priority: Critical > Fix For: 2.0.0, 1.2.1, 1.0.3, 1.1.3 > > Attachments: HBASE-14400-branch-1.0.patch, > HBASE-14400-branch-1.1.patch, HBASE-14400-branch-1.2.patch, > HBASE-14400-master-v2.patch, HBASE-14400-master.patch > > > HBase configuration 'hbase.rpc.protection' can be set to 'authentication', > 'integrity' or 'privacy'. > "authentication means authentication only and no integrity or privacy; > integrity implies > authentication and integrity are enabled; and privacy implies all of > authentication, integrity and privacy are enabled." > However hbase ref guide incorrectly suggests in some places to set the value > to 'auth-conf' instead of 'privacy'. Setting value to 'auth-conf' doesn't > provide rpc encryption which is what user wants. > This jira will fix: > - documentation: change 'auth-conf' references to 'privacy' > - SaslUtil to support both set of values (privacy/integrity/authentication > and auth-conf/auth-int/auth) to be backward compatible with what was being > suggested till now. > - change 'hbase.thrift.security.qop' to be consistent with other similar > configurations by using same set of values (privacy/integrity/authentication). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14398) Create the fake keys required in the scan path to avoid copy to byte[]
[ https://issues.apache.org/jira/browse/HBASE-14398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741087#comment-14741087 ] stack commented on HBASE-14398: --- bq. Why we need FirstOnRowColHintByteBufferedCell? How it is different from FirstOnRowColByteBufferedCell? lol > Create the fake keys required in the scan path to avoid copy to byte[] > -- > > Key: HBASE-14398 > URL: https://issues.apache.org/jira/browse/HBASE-14398 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-14398.patch, HBASE-14398_1.patch > > > Already we have created some fake keys for the ByteBufferedCells so that we > can avoid the copy requried to create fake keys. This JIRA aims to fill up > all such places so that the Offheap BBs are not copied to onheap byte[]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14401) Stamp failed appends with sequenceid too.... Cleans up latches
[ https://issues.apache.org/jira/browse/HBASE-14401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741283#comment-14741283 ] stack commented on HBASE-14401: --- Says: kalashnikov:hbase.git stack$ python ./dev-support/findHangingTests.py https://builds.apache.org/job/PreCommit-HBASE-Build/15563//consoleText Fetching the console output from the URL Printing hanging tests Hanging test : org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient Hanging test : org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas Hanging test : org.apache.hadoop.hbase.client.TestReplicasClient Hanging test : org.apache.hadoop.hbase.client.TestAdmin2 Hanging test : org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat2 Hanging test : org.apache.hadoop.hbase.mapred.TestTableMapReduceUtil Hanging test : org.apache.hadoop.hbase.mapred.TestTableSnapshotInputFormat Hanging test : org.apache.hadoop.hbase.client.TestFromClientSide Hanging test : org.apache.hadoop.hbase.client.TestCloneSnapshotFromClient Hanging test : org.apache.hadoop.hbase.client.TestMobSnapshotFromClient Hanging test : org.apache.hadoop.hbase.client.TestMobSnapshotCloneIndependence Hanging test : org.apache.hadoop.hbase.util.TestHBaseFsck Hanging test : org.apache.hadoop.hbase.TestZooKeeper Printing Failing tests Failing test : org.apache.hadoop.hbase.master.TestDistributedLogSplitting Failing test : org.apache.hadoop.hbase.regionserver.TestWALLockup Looking at the test output it says: --- Test set: org.apache.hadoop.hbase.regionserver.TestWALLockup --- Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.952 sec - in org.apache.hadoop.hbase.regionserver.TestWALLockup > Stamp failed appends with sequenceid too Cleans up latches > -- > > Key: HBASE-14401 > URL: https://issues.apache.org/jira/browse/HBASE-14401 > Project: HBase > Issue Type: Sub-task > Components: test, wal >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 14401.txt, 14401v3.txt, 14401v3.txt, 14401v3.txt > > > Looking in test output I see we can sometimes get stuck waiting on > sequenceid... The parent issues redo of our semantic makes it so we encounter > failed append more often around damaged WAL. > This patch makes it so we stamp sequenceid always, even if the append fails. > This way all sequenceids are accounted for but more important, the latch on > sequenceid down in WALKey will be cleared.. where before it was not being > cleared (there is no global list of outstanding WALKeys waiting on > sequenceids so no way to clean them up... we don't need such a list if we > ALWAYS stamp the sequenceid). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14380) Correct data gets skipped along with bad data in importTsv bulk load thru TsvImporterTextMapper
[ https://issues.apache.org/jira/browse/HBASE-14380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14380: --- Hadoop Flags: Reviewed Fix Version/s: 1.3.0 2.0.0 Applied to master and branch-1 Thanks for the patch, Bhupendra Thanks for the review, Anoop. For branch-1.2, I got: 5 out of 6 hunks FAILED -- saving rejects to file hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTsv.java.rej Bhupendra: Do you mind attaching patch for branch-1.2 ? > Correct data gets skipped along with bad data in importTsv bulk load thru > TsvImporterTextMapper > --- > > Key: HBASE-14380 > URL: https://issues.apache.org/jira/browse/HBASE-14380 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Bhupendra Kumar Jain >Assignee: Bhupendra Kumar Jain > Fix For: 2.0.0, 1.3.0 > > Attachments: 0001-HBASE-14380.patch, 14380-v2.txt, > HBASE-14380_v1.patch > > > Cosider the input data is as below > ROWKEY, TIEMSTAMP, Col_Value > r1,1,v1 >> Correct line > r1 >> Bad line > r1,3,v3 >> Correct line > r1,4,v4 >> Correct line > When data is bulk loaded using importTsv with mapper as TsvImporterTextMapper > , All the lines are getting ignored even though skipBadLines is set to true. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14361) ReplicationSink should create Connection instances lazily
[
https://issues.apache.org/jira/browse/HBASE-14361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741094#comment-14741094
]
stack commented on HBASE-14361:
---
Do you want to take lock and set to null here:
216 if (this.sharedHtableCon != null) {
217 this.sharedHtableCon.close();
218 }
> ReplicationSink should create Connection instances lazily
> -
>
> Key: HBASE-14361
> URL: https://issues.apache.org/jira/browse/HBASE-14361
> Project: HBase
> Issue Type: Task
> Components: Replication
>Reporter: Nick Dimiduk
>Assignee: Heng Chen
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.15, 1.0.3, 1.1.3
>
> Attachments: HBASE-14361-0.98.patch, HBASE-14361.patch,
> HBASE-14361_v1.patch, hmaster.log
>
>
> Over on HBASE-12911 I have a patch that registers Connection instances with
> the metrics system. In both standalone server and tll client applications, I
> was surprised to see multiple connection objects showing up that are unused.
> These are pretty heavy objects, including lots of client threads for the
> batch pool. We should track these down and remove them -- if they're not some
> kind of phantom artifacts of my WIP patch over there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14261) Enhance Chaos Monkey framework by adding zookeeper and datanode fault injections.
[
https://issues.apache.org/jira/browse/HBASE-14261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741290#comment-14741290
]
Ted Yu commented on HBASE-14261:
The following compilation error is reproducible on Mac and Linux for hadoop-1.1:
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile
(default-testCompile) on project hbase-it: Compilation failure: Compilation
failure:
[ERROR]
/Users/tyu/98/hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/RestartRandomDataNodeAction.java:[27,38]
error: cannot find symbol
[ERROR] symbol: class HdfsConstants
[ERROR] location: package org.apache.hadoop.hdfs.protocol
[ERROR]
/Users/tyu/98/hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/RestartRandomDataNodeAction.java:[53,70]
error: package HdfsConstants does not exist
{code}
> Enhance Chaos Monkey framework by adding zookeeper and datanode fault
> injections.
> -
>
> Key: HBASE-14261
> URL: https://issues.apache.org/jira/browse/HBASE-14261
> Project: HBase
> Issue Type: Improvement
>Reporter: Srikanth Srungarapu
>Assignee: Srikanth Srungarapu
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.15, 1.0.3, 1.1.3
>
> Attachments: HBASE-14261-addendum.patch, HBASE-14261-branch-1.patch,
> HBASE-14261-branch-1_v3.patch, HBASE-14261-branch-1_v4.patch,
> HBASE-14261.branch-1_v2.patch, HBASE-14261.patch
>
>
> One of the shortcomings of existing ChaosMonkey framework is lack of fault
> injections for hbase dependencies like zookeeper, hdfs etc. This patch
> attempts to solve this problem partially by adding datanode and zk node fault
> injections.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14380) Correct data also getting skipped along with bad data in importTsv bulk load thru TsvImporterTextMapper
[ https://issues.apache.org/jira/browse/HBASE-14380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14380: --- Attachment: 14380-v2.txt > Correct data also getting skipped along with bad data in importTsv bulk load > thru TsvImporterTextMapper > --- > > Key: HBASE-14380 > URL: https://issues.apache.org/jira/browse/HBASE-14380 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Bhupendra Kumar Jain >Assignee: Bhupendra Kumar Jain > Attachments: 0001-HBASE-14380.patch, 14380-v2.txt, > HBASE-14380_v1.patch > > > Cosider the input data is as below > ROWKEY, TIEMSTAMP, Col_Value > r1,1,v1 >> Correct line > r1 >> Bad line > r1,3,v3 >> Correct line > r1,4,v4 >> Correct line > When data is bulk loaded using importTsv with mapper as TsvImporterTextMapper > , All the lines are getting ignored even though skipBadLines is set to true. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14380) Correct data gets skipped along with bad data in importTsv bulk load thru TsvImporterTextMapper
[ https://issues.apache.org/jira/browse/HBASE-14380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14380: --- Status: Open (was: Patch Available) > Correct data gets skipped along with bad data in importTsv bulk load thru > TsvImporterTextMapper > --- > > Key: HBASE-14380 > URL: https://issues.apache.org/jira/browse/HBASE-14380 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Bhupendra Kumar Jain >Assignee: Bhupendra Kumar Jain > Fix For: 2.0.0, 1.3.0 > > Attachments: 0001-HBASE-14380.patch, 14380-v2.txt, > HBASE-14380_v1.patch > > > Cosider the input data is as below > ROWKEY, TIEMSTAMP, Col_Value > r1,1,v1 >> Correct line > r1 >> Bad line > r1,3,v3 >> Correct line > r1,4,v4 >> Correct line > When data is bulk loaded using importTsv with mapper as TsvImporterTextMapper > , All the lines are getting ignored even though skipBadLines is set to true. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14307) Incorrect use of positional read api in HFileBlock
[
https://issues.apache.org/jira/browse/HBASE-14307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741065#comment-14741065
]
Hudson commented on HBASE-14307:
FAILURE: Integrated in HBase-1.1 #657 (See
[https://builds.apache.org/job/HBase-1.1/657/])
HBASE-14307 Addendum fixes compilation for TestHFileBlockPositionalRead (tedyu:
rev 856d2400190e7a07f94e0fb8afa1e66accc0b7d7)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockPositionalRead.java
> Incorrect use of positional read api in HFileBlock
> --
>
> Key: HBASE-14307
> URL: https://issues.apache.org/jira/browse/HBASE-14307
> Project: HBase
> Issue Type: Bug
>Reporter: Shradha Revankar
>Assignee: Chris Nauroth
> Fix For: 2.0.0, 0.98.15, 1.2.1, 1.0.3, 1.1.3
>
> Attachments: 14307-branch-1.addendum, HBASE-14307.001.master.patch,
> HBASE-14307.002.master.patch, HBASE-14307.003.patch
>
>
> Considering that {{read()}} is not guaranteed to read all bytes,
> I'm interested to understand this particular piece of code and why is partial
> read treated as an error :
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java#L1446-L1450
> Particularly, if hbase were to use a different filesystem, say
> WebhdfsFileSystem, this would not work, please also see
> https://issues.apache.org/jira/browse/HDFS-8943 for discussion around this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14261) Enhance Chaos Monkey framework by adding zookeeper and datanode fault injections.
[ https://issues.apache.org/jira/browse/HBASE-14261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741333#comment-14741333 ] Ted Yu commented on HBASE-14261: My hands are full at the moment. > Enhance Chaos Monkey framework by adding zookeeper and datanode fault > injections. > - > > Key: HBASE-14261 > URL: https://issues.apache.org/jira/browse/HBASE-14261 > Project: HBase > Issue Type: Improvement >Reporter: Srikanth Srungarapu >Assignee: Srikanth Srungarapu > Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.15, 1.0.3, 1.1.3 > > Attachments: HBASE-14261-addendum.patch, HBASE-14261-branch-1.patch, > HBASE-14261-branch-1_v3.patch, HBASE-14261-branch-1_v4.patch, > HBASE-14261.branch-1_v2.patch, HBASE-14261.patch > > > One of the shortcomings of existing ChaosMonkey framework is lack of fault > injections for hbase dependencies like zookeeper, hdfs etc. This patch > attempts to solve this problem partially by adding datanode and zk node fault > injections. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14401) Stamp failed appends with sequenceid too.... Cleans up latches
[
https://issues.apache.org/jira/browse/HBASE-14401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740381#comment-14740381
]
Hadoop QA commented on HBASE-14401:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12755336/14401v3.txt
against master branch at commit fda317cebb5d306cabf1899e05cedb0225b2b62b.
ATTACHMENT ID: 12755336
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15557//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15557//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15557//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15557//console
This message is automatically generated.
> Stamp failed appends with sequenceid too Cleans up latches
> --
>
> Key: HBASE-14401
> URL: https://issues.apache.org/jira/browse/HBASE-14401
> Project: HBase
> Issue Type: Sub-task
> Components: test, wal
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14401.txt, 14401v3.txt, 14401v3.txt
>
>
> Looking in test output I see we can sometimes get stuck waiting on
> sequenceid... The parent issues redo of our semantic makes it so we encounter
> failed append more often around damaged WAL.
> This patch makes it so we stamp sequenceid always, even if the append fails.
> This way all sequenceids are accounted for but more important, the latch on
> sequenceid down in WALKey will be cleared.. where before it was not being
> cleared (there is no global list of outstanding WALKeys waiting on
> sequenceids so no way to clean them up... we don't need such a list if we
> ALWAYS stamp the sequenceid).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12298) Support BB usage in PrefixTree
[
https://issues.apache.org/jira/browse/HBASE-12298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740396#comment-14740396
]
Hadoop QA commented on HBASE-12298:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12755327/HBASE-12298_4%20%281%29.patch
against master branch at commit fda317cebb5d306cabf1899e05cedb0225b2b62b.
ATTACHMENT ID: 12755327
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 28 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestDistributedLogSplitting
{color:red}-1 core zombie tests{color}. There are 2 zombie test(s):
at
org.apache.hadoop.hbase.regionserver.TestScannerRetriableFailure.testFaultyScanner(TestScannerRetriableFailure.java:118)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15553//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15553//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15553//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15553//console
This message is automatically generated.
> Support BB usage in PrefixTree
> --
>
> Key: HBASE-12298
> URL: https://issues.apache.org/jira/browse/HBASE-12298
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver, Scanners
>Reporter: Anoop Sam John
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-12298.patch, HBASE-12298_1.patch,
> HBASE-12298_2.patch, HBASE-12298_3.patch, HBASE-12298_4 (1).patch,
> HBASE-12298_4 (1).patch, HBASE-12298_4 (1).patch, HBASE-12298_4 (1).patch,
> HBASE-12298_4 (1).patch, HBASE-12298_4.patch, HBASE-12298_4.patch,
> HBASE-12298_4.patch, HBASE-12298_4.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14394) Properly close the connection after reading records from table.
[
https://issues.apache.org/jira/browse/HBASE-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740502#comment-14740502
]
Hadoop QA commented on HBASE-14394:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12755337/HBASE-14394_v3.patch
against master branch at commit fda317cebb5d306cabf1899e05cedb0225b2b62b.
ATTACHMENT ID: 12755337
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan1
org.apache.hadoop.hbase.mapreduce.TestSyncTable
org.apache.hadoop.hbase.mob.mapreduce.TestMobSweeper
org.apache.hadoop.hbase.mapreduce.TestHashTable
org.apache.hadoop.hbase.mapreduce.TestCellCounter
org.apache.hadoop.hbase.replication.TestReplicationSmallTests
org.apache.hadoop.hbase.mapreduce.TestTableInputFormat
org.apache.hadoop.hbase.TestStochasticBalancerJmxMetrics
{color:red}-1 core zombie tests{color}. There are 5 zombie test(s):
at
org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScanBase.testScan(TestTableInputFormatScanBase.java:243)
at
org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan2.testScanOBBToQPP(TestTableInputFormatScan2.java:58)
at
org.apache.hadoop.hbase.mapreduce.TestImportExport.testMetaExport(TestImportExport.java:216)
at
org.apache.hadoop.hbase.mapreduce.TestTableInputFormat.testInputFormat(TestTableInputFormat.java:373)
at
org.apache.hadoop.hbase.mapreduce.TestTableInputFormat.testDeprecatedExtensionOfTableInputFormatBase(TestTableInputFormat.java:361)
at
org.apache.hadoop.hbase.mapreduce.TestCellCounter.testCellCounteEndTimeRange(TestCellCounter.java:179)
at
org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScanBase.testScan(TestTableInputFormatScanBase.java:243)
at
org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan1.testScanEmptyToBBB(TestTableInputFormatScan1.java:84)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15556//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15556//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15556//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15556//console
This message is automatically generated.
> Properly close the connection after reading records from table.
> ---
>
> Key: HBASE-14394
> URL: https://issues.apache.org/jira/browse/HBASE-14394
> Project: HBase
> Issue Type: Bug
>Reporter: Srikanth Srungarapu
>Assignee: Srikanth Srungarapu
>Priority: Minor
> Attachments: HBASE-14394.patch, HBASE-14394_v2.patch,
> HBASE-14394_v3.patch
>
>
> This was brought to notice by one of our observant customers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-14396) audit log should record the operation object
[ https://issues.apache.org/jira/browse/HBASE-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi resolved HBASE-14396. --- Resolution: Not A Problem > audit log should record the operation object > - > > Key: HBASE-14396 > URL: https://issues.apache.org/jira/browse/HBASE-14396 > Project: HBase > Issue Type: Bug >Reporter: sunhaitao > > Currently the hbase audit log only records the user and scope,we can't know > which table the user is operating on unless the scope is table. > It would be better to know what's going on if we record the exact table and > column family we are operating on besides the scope. > String logMessage = > "Access " + (result.isAllowed() ? "allowed" : "denied") + " for user " > + (result.getUser() != null ? result.getUser().getShortName() : > "UNKNOWN") > + "; reason: " + result.getReason() + "; remote address: " > + (remoteAddr != null ? remoteAddr : "") + "; request: " + > result.getRequest() > + "; context: " + result.toContextString(); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-14396) audit log should record the operation object
[ https://issues.apache.org/jira/browse/HBASE-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi reopened HBASE-14396: --- > audit log should record the operation object > - > > Key: HBASE-14396 > URL: https://issues.apache.org/jira/browse/HBASE-14396 > Project: HBase > Issue Type: Bug >Reporter: sunhaitao > > Currently the hbase audit log only records the user and scope,we can't know > which table the user is operating on unless the scope is table. > It would be better to know what's going on if we record the exact table and > column family we are operating on besides the scope. > String logMessage = > "Access " + (result.isAllowed() ? "allowed" : "denied") + " for user " > + (result.getUser() != null ? result.getUser().getShortName() : > "UNKNOWN") > + "; reason: " + result.getReason() + "; remote address: " > + (remoteAddr != null ? remoteAddr : "") + "; request: " + > result.getRequest() > + "; context: " + result.toContextString(); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14398) Create the fake keys required in the scan path to avoid copy to byte[]
[
https://issues.apache.org/jira/browse/HBASE-14398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740498#comment-14740498
]
Hadoop QA commented on HBASE-14398:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12755332/HBASE-14398.patch
against master branch at commit fda317cebb5d306cabf1899e05cedb0225b2b62b.
ATTACHMENT ID: 12755332
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/1//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1//console
This message is automatically generated.
> Create the fake keys required in the scan path to avoid copy to byte[]
> --
>
> Key: HBASE-14398
> URL: https://issues.apache.org/jira/browse/HBASE-14398
> Project: HBase
> Issue Type: Sub-task
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-14398.patch
>
>
> Already we have created some fake keys for the ByteBufferedCells so that we
> can avoid the copy requried to create fake keys. This JIRA aims to fill up
> all such places so that the Offheap BBs are not copied to onheap byte[].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14395) Short circuit last byte check in CellUtil#matchingXXX methods for ByteBufferedCells
[ https://issues.apache.org/jira/browse/HBASE-14395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740364#comment-14740364 ] Hudson commented on HBASE-14395: FAILURE: Integrated in HBase-TRUNK #6795 (See [https://builds.apache.org/job/HBase-TRUNK/6795/]) HBASE-14395 Short circuit last byte check in CellUtil#matchingXXX methods for ByteBufferedCells. (anoopsamjohn: rev ae3176b9c0f63644deffc82dc424d219c1472818) * hbase-common/src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java * hbase-common/src/main/java/org/apache/hadoop/hbase/CellUtil.java * hbase-common/src/main/java/org/apache/hadoop/hbase/OffheapKeyValue.java > Short circuit last byte check in CellUtil#matchingXXX methods for > ByteBufferedCells > --- > > Key: HBASE-14395 > URL: https://issues.apache.org/jira/browse/HBASE-14395 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: 2.0.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-14395.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14396) audit log should record the operation object
[ https://issues.apache.org/jira/browse/HBASE-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740366#comment-14740366 ] sunhaitao commented on HBASE-14396: --- I see it ,thanks > audit log should record the operation object > - > > Key: HBASE-14396 > URL: https://issues.apache.org/jira/browse/HBASE-14396 > Project: HBase > Issue Type: Bug >Reporter: sunhaitao > > Currently the hbase audit log only records the user and scope,we can't know > which table the user is operating on unless the scope is table. > It would be better to know what's going on if we record the exact table and > column family we are operating on besides the scope. > String logMessage = > "Access " + (result.isAllowed() ? "allowed" : "denied") + " for user " > + (result.getUser() != null ? result.getUser().getShortName() : > "UNKNOWN") > + "; reason: " + result.getReason() + "; remote address: " > + (remoteAddr != null ? remoteAddr : "") + "; request: " + > result.getRequest() > + "; context: " + result.toContextString(); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14378) Get TestAccessController* passing again on branch-1
[
https://issues.apache.org/jira/browse/HBASE-14378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740378#comment-14740378
]
Hadoop QA commented on HBASE-14378:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12755318/14386.branch-1.v3%20%281%29.txt
against branch-1 branch at commit fda317cebb5d306cabf1899e05cedb0225b2b62b.
ATTACHMENT ID: 12755318
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 27 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 2 zombie test(s):
at
org.apache.hadoop.hbase.TestClassFinder.testClassFinderFiltersByClassInJar(TestClassFinder.java:165)
at
org.apache.hadoop.hbase.security.access.TestAccessController.testAccessControllerUserPermsRegexHandling(TestAccessController.java:2591)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15554//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15554//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15554//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15554//console
This message is automatically generated.
> Get TestAccessController* passing again on branch-1
> ---
>
> Key: HBASE-14378
> URL: https://issues.apache.org/jira/browse/HBASE-14378
> Project: HBase
> Issue Type: Sub-task
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14378.branch-1.txt, 14378.branch-1.v2.txt,
> 14378.branch-1.v2.txt, 14378.branch-1.v2.txt, 14386.branch-1.v3 (1).txt,
> 14386.branch-1.v3.txt, 14386.branch-1.v3.txt, 14386.branch-1.v4.do.nothing.txt
>
>
> TestAccessController* are failing reliably on branch-1. They go zombie. I
> learned that setting the junit test timeout facility on the class doesn't
> make the zombie timeout nor does setting a timeout on each test turn zombies
> to test failures; the test goes zombie on the way out in the tear down of the
> cluster.
> Digging, we are out of handlers... all are occupied.
> 3dacee6 HBASE-14290 Spin up less threads in tests cut the default thread
> count to 3 from 10. Putting the value back on these tests seems to make them
> pass reliably when I run locally. For good measure, I'll add in the timeouts
> .
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12751) Allow RowLock to be reader writer
[
https://issues.apache.org/jira/browse/HBASE-12751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740426#comment-14740426
]
Hadoop QA commented on HBASE-12751:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12755323/12751.rebased.v35.txt
against master branch at commit fda317cebb5d306cabf1899e05cedb0225b2b62b.
ATTACHMENT ID: 12755323
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 99 new
or modified tests.
{color:red}-1 Anti-pattern{color}. The patch appears to
have anti-pattern where BYTES_COMPARATOR was omitted:
-getRegionInfo(), -1, new TreeMap());.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.util.TestProcessBasedCluster
org.apache.hadoop.hbase.mapreduce.TestImportExport
{color:red}-1 core zombie tests{color}. There are 4 zombie test(s):
at
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDefaultVisLabelService.testListLabelsWithRegEx(TestVisibilityLabelsWithDefaultVisLabelService.java:220)
at
org.apache.hadoop.hbase.TestIOFencing.testFencingAroundCompaction(TestIOFencing.java:229)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15552//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15552//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15552//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15552//console
This message is automatically generated.
> Allow RowLock to be reader writer
> -
>
> Key: HBASE-12751
> URL: https://issues.apache.org/jira/browse/HBASE-12751
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 2.0.0, 1.3.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.3.0
>
> Attachments: 12751.rebased.v25.txt, 12751.rebased.v26.txt,
> 12751.rebased.v26.txt, 12751.rebased.v27.txt, 12751.rebased.v29.txt,
> 12751.rebased.v31.txt, 12751.rebased.v32.txt, 12751.rebased.v32.txt,
> 12751.rebased.v33.txt, 12751.rebased.v34.txt, 12751.rebased.v35.txt,
> 12751v22.txt, 12751v23.txt, 12751v23.txt, 12751v23.txt, 12751v23.txt,
> HBASE-12751-v1.patch, HBASE-12751-v10.patch, HBASE-12751-v10.patch,
> HBASE-12751-v11.patch, HBASE-12751-v12.patch, HBASE-12751-v13.patch,
> HBASE-12751-v14.patch, HBASE-12751-v15.patch, HBASE-12751-v16.patch,
> HBASE-12751-v17.patch, HBASE-12751-v18.patch, HBASE-12751-v19 (1).patch,
> HBASE-12751-v19.patch, HBASE-12751-v2.patch, HBASE-12751-v20.patch,
> HBASE-12751-v20.patch, HBASE-12751-v21.patch, HBASE-12751-v3.patch,
> HBASE-12751-v4.patch, HBASE-12751-v5.patch, HBASE-12751-v6.patch,
> HBASE-12751-v7.patch, HBASE-12751-v8.patch, HBASE-12751-v9.patch,
> HBASE-12751.patch
>
>
> Right now every write operation grabs a row lock. This is to prevent values
> from changing during a read modify write operation (increment or check and
> put). However it limits parallelism in several different scenarios.
> If there are several puts to the same row but different columns or stores
> then this is very limiting.
> If there are puts to the same column then mvcc number should ensure a
> consistent ordering. So locking is not needed.
> However locking for check and put or increment is still needed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14392) [tests] TestLogRollingNoCluster fails on master from time to time
[ https://issues.apache.org/jira/browse/HBASE-14392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740445#comment-14740445 ] Hudson commented on HBASE-14392: SUCCESS: Integrated in HBase-1.3-IT #145 (See [https://builds.apache.org/job/HBase-1.3-IT/145/]) HBASE-14392 [tests] TestLogRollingNoCluster fails on master from time to time (stack: rev d9bc84b1fcc2c4f8673a57f4aba3b1d9aae94cb9) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogWriter.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollingNoCluster.java > [tests] TestLogRollingNoCluster fails on master from time to time > - > > Key: HBASE-14392 > URL: https://issues.apache.org/jira/browse/HBASE-14392 > Project: HBase > Issue Type: Bug > Components: test >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 14392.txt, 14392.txt, 14392v2.txt, 14392v2.txt, > 14392v2.txt > > > TestLogRollingNoCluster fails from time to time on a rig I have running here. > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; > support was removed in 8.0 > Running org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.155 sec - > in org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster > Results : > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 > The test itself is a bit odd. We apparently have seen NPEs around close of a > WAL while trying to sync... which I suppose makes sense if two threads > involved. > Attached patch just fails the sync silently if stream is null... lets presume > it a close. Adds a sync on write of trailer too... > This patch seems to have gotten rid of the odd failure seen on a particular > box here if I keep cycling the test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14392) [tests] TestLogRollingNoCluster fails on master from time to time
[ https://issues.apache.org/jira/browse/HBASE-14392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740324#comment-14740324 ] Hudson commented on HBASE-14392: FAILURE: Integrated in HBase-1.2 #162 (See [https://builds.apache.org/job/HBase-1.2/162/]) HBASE-14392 [tests] TestLogRollingNoCluster fails on master from time to time (stack: rev a2bca2748d5d310b1ea06674b106caa2c9b1b7d2) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogWriter.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollingNoCluster.java > [tests] TestLogRollingNoCluster fails on master from time to time > - > > Key: HBASE-14392 > URL: https://issues.apache.org/jira/browse/HBASE-14392 > Project: HBase > Issue Type: Bug > Components: test >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 14392.txt, 14392.txt, 14392v2.txt, 14392v2.txt, > 14392v2.txt > > > TestLogRollingNoCluster fails from time to time on a rig I have running here. > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; > support was removed in 8.0 > Running org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.155 sec - > in org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster > Results : > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 > The test itself is a bit odd. We apparently have seen NPEs around close of a > WAL while trying to sync... which I suppose makes sense if two threads > involved. > Attached patch just fails the sync silently if stream is null... lets presume > it a close. Adds a sync on write of trailer too... > This patch seems to have gotten rid of the odd failure seen on a particular > box here if I keep cycling the test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14392) [tests] TestLogRollingNoCluster fails on master from time to time
[ https://issues.apache.org/jira/browse/HBASE-14392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740369#comment-14740369 ] Hudson commented on HBASE-14392: FAILURE: Integrated in HBase-1.3 #161 (See [https://builds.apache.org/job/HBase-1.3/161/]) HBASE-14392 [tests] TestLogRollingNoCluster fails on master from time to time (stack: rev d9bc84b1fcc2c4f8673a57f4aba3b1d9aae94cb9) * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollingNoCluster.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogWriter.java > [tests] TestLogRollingNoCluster fails on master from time to time > - > > Key: HBASE-14392 > URL: https://issues.apache.org/jira/browse/HBASE-14392 > Project: HBase > Issue Type: Bug > Components: test >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 14392.txt, 14392.txt, 14392v2.txt, 14392v2.txt, > 14392v2.txt > > > TestLogRollingNoCluster fails from time to time on a rig I have running here. > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; > support was removed in 8.0 > Running org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.155 sec - > in org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster > Results : > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 > The test itself is a bit odd. We apparently have seen NPEs around close of a > WAL while trying to sync... which I suppose makes sense if two threads > involved. > Attached patch just fails the sync silently if stream is null... lets presume > it a close. Adds a sync on write of trailer too... > This patch seems to have gotten rid of the odd failure seen on a particular > box here if I keep cycling the test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14361) ReplicationSink should create Connection instances lazily
[
https://issues.apache.org/jira/browse/HBASE-14361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740326#comment-14740326
]
Heng Chen commented on HBASE-14361:
---
QA console shows testcase was killed..
{quote}
Running org.apache.hadoop.hbase.mob.TestDefaultMobStoreFlusher
Killed
Killed
{quote}
> ReplicationSink should create Connection instances lazily
> -
>
> Key: HBASE-14361
> URL: https://issues.apache.org/jira/browse/HBASE-14361
> Project: HBase
> Issue Type: Task
> Components: Replication
>Reporter: Nick Dimiduk
>Assignee: Heng Chen
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.15, 1.0.3, 1.1.3
>
> Attachments: HBASE-14361-0.98.patch, HBASE-14361.patch,
> HBASE-14361_v1.patch, hmaster.log
>
>
> Over on HBASE-12911 I have a patch that registers Connection instances with
> the metrics system. In both standalone server and tll client applications, I
> was surprised to see multiple connection objects showing up that are unused.
> These are pretty heavy objects, including lots of client threads for the
> batch pool. We should track these down and remove them -- if they're not some
> kind of phantom artifacts of my WIP patch over there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14392) [tests] TestLogRollingNoCluster fails on master from time to time
[ https://issues.apache.org/jira/browse/HBASE-14392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740365#comment-14740365 ] Hudson commented on HBASE-14392: FAILURE: Integrated in HBase-TRUNK #6795 (See [https://builds.apache.org/job/HBase-TRUNK/6795/]) HBASE-14392 [tests] TestLogRollingNoCluster fails on master from time to time (stack: rev 411b516f5156730075558d91b69c3c3e09fb200d) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogWriter.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollingNoCluster.java > [tests] TestLogRollingNoCluster fails on master from time to time > - > > Key: HBASE-14392 > URL: https://issues.apache.org/jira/browse/HBASE-14392 > Project: HBase > Issue Type: Bug > Components: test >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: 14392.txt, 14392.txt, 14392v2.txt, 14392v2.txt, > 14392v2.txt > > > TestLogRollingNoCluster fails from time to time on a rig I have running here. > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; > support was removed in 8.0 > Running org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.155 sec - > in org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster > Results : > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 > The test itself is a bit odd. We apparently have seen NPEs around close of a > WAL while trying to sync... which I suppose makes sense if two threads > involved. > Attached patch just fails the sync silently if stream is null... lets presume > it a close. Adds a sync on write of trailer too... > This patch seems to have gotten rid of the odd failure seen on a particular > box here if I keep cycling the test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14396) audit log should record the operation object
[ https://issues.apache.org/jira/browse/HBASE-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunhaitao resolved HBASE-14396. --- Resolution: Fixed > audit log should record the operation object > - > > Key: HBASE-14396 > URL: https://issues.apache.org/jira/browse/HBASE-14396 > Project: HBase > Issue Type: Bug >Reporter: sunhaitao > > Currently the hbase audit log only records the user and scope,we can't know > which table the user is operating on unless the scope is table. > It would be better to know what's going on if we record the exact table and > column family we are operating on besides the scope. > String logMessage = > "Access " + (result.isAllowed() ? "allowed" : "denied") + " for user " > + (result.getUser() != null ? result.getUser().getShortName() : > "UNKNOWN") > + "; reason: " + result.getReason() + "; remote address: " > + (remoteAddr != null ? remoteAddr : "") + "; request: " + > result.getRequest() > + "; context: " + result.toContextString(); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14261) Enhance Chaos Monkey framework by adding zookeeper and datanode fault injections.
[ https://issues.apache.org/jira/browse/HBASE-14261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741346#comment-14741346 ] Andrew Purtell commented on HBASE-14261: Ok, I'll fix it > Enhance Chaos Monkey framework by adding zookeeper and datanode fault > injections. > - > > Key: HBASE-14261 > URL: https://issues.apache.org/jira/browse/HBASE-14261 > Project: HBase > Issue Type: Improvement >Reporter: Srikanth Srungarapu >Assignee: Srikanth Srungarapu > Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.15, 1.0.3, 1.1.3 > > Attachments: HBASE-14261-addendum.patch, HBASE-14261-branch-1.patch, > HBASE-14261-branch-1_v3.patch, HBASE-14261-branch-1_v4.patch, > HBASE-14261.branch-1_v2.patch, HBASE-14261.patch > > > One of the shortcomings of existing ChaosMonkey framework is lack of fault > injections for hbase dependencies like zookeeper, hdfs etc. This patch > attempts to solve this problem partially by adding datanode and zk node fault > injections. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14400) Fix HBase RPC protection documentation
[ https://issues.apache.org/jira/browse/HBASE-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741364#comment-14741364 ] Apekshit Sharma commented on HBASE-14400: - *attached patch for it now. > Fix HBase RPC protection documentation > -- > > Key: HBASE-14400 > URL: https://issues.apache.org/jira/browse/HBASE-14400 > Project: HBase > Issue Type: Bug > Components: encryption, rpc, security >Reporter: Apekshit Sharma >Assignee: Apekshit Sharma >Priority: Critical > Fix For: 2.0.0, 1.2.1, 1.0.3, 1.1.3 > > Attachments: HBASE-14400-branch-0.98.patch, > HBASE-14400-branch-1.0.patch, HBASE-14400-branch-1.1.patch, > HBASE-14400-branch-1.2.patch, HBASE-14400-master-v2.patch, > HBASE-14400-master.patch > > > HBase configuration 'hbase.rpc.protection' can be set to 'authentication', > 'integrity' or 'privacy'. > "authentication means authentication only and no integrity or privacy; > integrity implies > authentication and integrity are enabled; and privacy implies all of > authentication, integrity and privacy are enabled." > However hbase ref guide incorrectly suggests in some places to set the value > to 'auth-conf' instead of 'privacy'. Setting value to 'auth-conf' doesn't > provide rpc encryption which is what user wants. > This jira will fix: > - documentation: change 'auth-conf' references to 'privacy' > - SaslUtil to support both set of values (privacy/integrity/authentication > and auth-conf/auth-int/auth) to be backward compatible with what was being > suggested till now. > - change 'hbase.thrift.security.qop' to be consistent with other similar > configurations by using same set of values (privacy/integrity/authentication). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14400) Fix HBase RPC protection documentation
[ https://issues.apache.org/jira/browse/HBASE-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741363#comment-14741363 ] Apekshit Sharma commented on HBASE-14400: - Yes. attached patch for it. > Fix HBase RPC protection documentation > -- > > Key: HBASE-14400 > URL: https://issues.apache.org/jira/browse/HBASE-14400 > Project: HBase > Issue Type: Bug > Components: encryption, rpc, security >Reporter: Apekshit Sharma >Assignee: Apekshit Sharma >Priority: Critical > Fix For: 2.0.0, 1.2.1, 1.0.3, 1.1.3 > > Attachments: HBASE-14400-branch-0.98.patch, > HBASE-14400-branch-1.0.patch, HBASE-14400-branch-1.1.patch, > HBASE-14400-branch-1.2.patch, HBASE-14400-master-v2.patch, > HBASE-14400-master.patch > > > HBase configuration 'hbase.rpc.protection' can be set to 'authentication', > 'integrity' or 'privacy'. > "authentication means authentication only and no integrity or privacy; > integrity implies > authentication and integrity are enabled; and privacy implies all of > authentication, integrity and privacy are enabled." > However hbase ref guide incorrectly suggests in some places to set the value > to 'auth-conf' instead of 'privacy'. Setting value to 'auth-conf' doesn't > provide rpc encryption which is what user wants. > This jira will fix: > - documentation: change 'auth-conf' references to 'privacy' > - SaslUtil to support both set of values (privacy/integrity/authentication > and auth-conf/auth-int/auth) to be backward compatible with what was being > suggested till now. > - change 'hbase.thrift.security.qop' to be consistent with other similar > configurations by using same set of values (privacy/integrity/authentication). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14380) Correct data gets skipped along with bad data in importTsv bulk load thru TsvImporterTextMapper
[ https://issues.apache.org/jira/browse/HBASE-14380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741432#comment-14741432 ] Hudson commented on HBASE-14380: FAILURE: Integrated in HBase-1.3 #164 (See [https://builds.apache.org/job/HBase-1.3/164/]) HBASE-14380 Correct data gets skipped along with bad data in importTsv bulk load thru TsvImporterTextMapper (Bhupendra Kumar Jain) (tedyu: rev 84dbe39f5d424159d7d57ed63f8a654a922104eb) * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TextSortReducer.java * hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTsv.java > Correct data gets skipped along with bad data in importTsv bulk load thru > TsvImporterTextMapper > --- > > Key: HBASE-14380 > URL: https://issues.apache.org/jira/browse/HBASE-14380 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Bhupendra Kumar Jain >Assignee: Bhupendra Kumar Jain > Fix For: 2.0.0, 1.3.0 > > Attachments: 0001-HBASE-14380.patch, 14380-v2.txt, > HBASE-14380_v1.patch > > > Cosider the input data is as below > ROWKEY, TIEMSTAMP, Col_Value > r1,1,v1 >> Correct line > r1 >> Bad line > r1,3,v3 >> Correct line > r1,4,v4 >> Correct line > When data is bulk loaded using importTsv with mapper as TsvImporterTextMapper > , All the lines are getting ignored even though skipBadLines is set to true. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14378) Get TestAccessController* passing again on branch-1
[
https://issues.apache.org/jira/browse/HBASE-14378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741485#comment-14741485
]
Hadoop QA commented on HBASE-14378:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12755438/14386.branch-1.v5.txt
against branch-1 branch at commit c94d10952fe44f73096027cc9083cee993983940.
ATTACHMENT ID: 12755438
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 27 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 2 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15566//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15566//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15566//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15566//console
This message is automatically generated.
> Get TestAccessController* passing again on branch-1
> ---
>
> Key: HBASE-14378
> URL: https://issues.apache.org/jira/browse/HBASE-14378
> Project: HBase
> Issue Type: Sub-task
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14378.branch-1.txt, 14378.branch-1.v2.txt,
> 14378.branch-1.v2.txt, 14378.branch-1.v2.txt, 14386.branch-1.v3 (1).txt,
> 14386.branch-1.v3.txt, 14386.branch-1.v3.txt,
> 14386.branch-1.v4.do.nothing.txt, 14386.branch-1.v5.txt
>
>
> TestAccessController* are failing reliably on branch-1. They go zombie. I
> learned that setting the junit test timeout facility on the class doesn't
> make the zombie timeout nor does setting a timeout on each test turn zombies
> to test failures; the test goes zombie on the way out in the tear down of the
> cluster.
> Digging, we are out of handlers... all are occupied.
> 3dacee6 HBASE-14290 Spin up less threads in tests cut the default thread
> count to 3 from 10. Putting the value back on these tests seems to make them
> pass reliably when I run locally. For good measure, I'll add in the timeouts
> .
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14403) [0.98] Fix TestInterfaceAudienceAnnotations failures
[
https://issues.apache.org/jira/browse/HBASE-14403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741506#comment-14741506
]
Hudson commented on HBASE-14403:
FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1072 (See
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1072/])
HBASE-14403 [0.98] Fix TestInterfaceAudienceAnnotations failures (apurtell: rev
501d5c484114b0dab859194b7cd0c2911be422ec)
* hbase-common/src/main/java/org/apache/hadoop/hbase/io/LimitInputStream.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityControllerNotReadyException.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/MetaMutationAnnotation.java
* hbase-protocol/src/main/java/org/apache/hadoop/hbase/util/ByteStringer.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/DelegatingPayloadCarryingRpcController.java
*
hbase-common/src/main/java/org/apache/hadoop/hbase/io/hadoopbackport/ThrottledInputStream.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/ServerRpcController.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/util/ConcatenatedLists.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
*
hbase-common/src/main/java/org/apache/hadoop/hbase/io/ImmutableBytesWritable.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/UnknownProtocolException.java
*
hbase-annotations/src/main/java/org/apache/hadoop/hbase/classification/tools/IncludePublicAnnotationsStandardDoclet.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/LockTimeoutException.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/types/PBType.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/FailedSanityCheckException.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/Batch.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/util/PrettyPrinter.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/DelegatingRetryingCallable.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationException.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/io/crypto/Encryption.java
*
hbase-annotations/src/main/java/org/apache/hadoop/hbase/classification/tools/ExcludePrivateAnnotationsStandardDoclet.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RpcControllerFactory.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/filter/LongComparator.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/KeepDeletedCells.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/util/ChecksumFactory.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshotException.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/util/Base64.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/BaseConfigurable.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RetriesExhaustedException.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/IPCUtil.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/NamespaceDescriptor.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/security/access/Permission.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/util/ExceptionUtil.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/filter/RegexStringComparator.java
> [0.98] Fix TestInterfaceAudienceAnnotations failures
>
>
> Key: HBASE-14403
> URL: https://issues.apache.org/jira/browse/HBASE-14403
> Project: HBase
> Issue Type: Task
>Reporter: Nick Dimiduk
>Assignee: Andrew Purtell
> Fix For: 0.98.15
>
> Attachments: HBASE-14403-0.98.patch
>
>
> HBASE-14382 backported {{TestInterfaceAudienceAnnotations}} to 0.98. This
> test now fails due to missing annotations.
> {noformat}
> 2015-09-10 16:21:41,705 DEBUG [main] hbase.ClassFinder(162): Will look for
> classes in
> /Users/ndimiduk/repos/hbase/hbase-client/target/classes/org/apache/hadoop/hbase
> 2015-09-10 16:21:41,710 DEBUG [main] hbase.ClassFinder(162): Will look for
> classes in
> /Users/ndimiduk/repos/hbase/hbase-annotations/target/classes/org/apache/hadoop/hbase
> 2015-09-10 16:21:41,710 DEBUG [main] hbase.ClassFinder(162): Will look for
> classes in
> /Users/ndimiduk/repos/hbase/hbase-common/target/classes/org/apache/hadoop/hbase
> 2015-09-10 16:21:41,710 DEBUG [main] hbase.ClassFinder(162): Will look for
> classes in
>
[jira] [Commented] (HBASE-14261) Enhance Chaos Monkey framework by adding zookeeper and datanode fault injections.
[ https://issues.apache.org/jira/browse/HBASE-14261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741372#comment-14741372 ] Srikanth Srungarapu commented on HBASE-14261: - Thanks! > Enhance Chaos Monkey framework by adding zookeeper and datanode fault > injections. > - > > Key: HBASE-14261 > URL: https://issues.apache.org/jira/browse/HBASE-14261 > Project: HBase > Issue Type: Improvement >Reporter: Srikanth Srungarapu >Assignee: Srikanth Srungarapu > Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.15, 1.0.3, 1.1.3 > > Attachments: HBASE-14261-addendum.patch, HBASE-14261-branch-1.patch, > HBASE-14261-branch-1_v3.patch, HBASE-14261-branch-1_v4.patch, > HBASE-14261.branch-1_v2.patch, HBASE-14261.patch > > > One of the shortcomings of existing ChaosMonkey framework is lack of fault > injections for hbase dependencies like zookeeper, hdfs etc. This patch > attempts to solve this problem partially by adding datanode and zk node fault > injections. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14412) remove_peer of replication peers with hyphens in name doesn't remove ZK entries
[
https://issues.apache.org/jira/browse/HBASE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Biju Nair updated HBASE-14412:
--
Description:
- Created replication peer POB3-HBASE-LL-A. By mistake used hyphens in the
replication peer name.
- Did {{remove_peer POB3-HBASE-LL-A}}
- Created a new peer POB3_HBASE_LL_A
- Verified zk for entries
{noformat}
ls /hbase/replication/peers
[POB3_HBASE_LL_A]
ls /hbase/replication/rs
[r2n14,60200,1441836473749, r1n16,60200,1441836441250,
r1n14,60200,1441836352689, r3n15,60200,1441836547607,
r2n15,60200,1441836551992, r3n14,60200,1441836230275,
r1n15,60200,1441836355145]
ls /hbase/replication/rs/r2n14,60200,1441836473749
[POB3_HBASE_LL_A, POB3-HBASE-LL-A]
ls /hbase/replication/rs/r2n14,60200,1441836473749/POB3-HBASE-LL-A
[r2n14%2C60200%2C1441836473749.1441865284930,
r2n14%2C60200%2C1441836473749.1441847284451,
r2n14%2C60200%2C1441836473749.1441890379203,
r2n14%2C60200%2C1441836473749.1441890384725,
r2n14%2C60200%2C1441836473749.1441840084237,
r2n14%2C60200%2C1441836473749.1441876085231,
r2n14%2C60200%2C1441836473749.1441890367559,
r2n14%2C60200%2C1441836473749.1441883285466,
r2n14%2C60200%2C1441836473749.1441890370829,
r2n14%2C60200%2C1441836473749.1441879685329,
r2n14%2C60200%2C1441836473749.1441890382514,
r2n14%2C60200%2C1441836473749.1441890381514,
r2n14%2C60200%2C1441836473749.1441890380286,
r2n14%2C60200%2C1441836473749.1441890378164,
r2n14%2C60200%2C1441836473749.1441872485132,
r2n14%2C60200%2C1441836473749.1441868885039,
r2n14%2C60200%2C1441836473749.1441890374603,
r2n14%2C60200%2C1441836473749.1441890377072,
r2n14%2C60200%2C1441836473749.1441858084739,
r2n14%2C60200%2C1441836473749.1441890369005,
r2n14%2C60200%2C1441836473749.1441850884549,
r2n14%2C60200%2C1441836473749.1441886885548,
r2n14%2C60200%2C1441836473749.1441890372063,
r2n14%2C60200%2C1441836473749.1441890375866,
r2n14%2C60200%2C1441836473749.1441890373369,
r2n14%2C60200%2C1441836473749.1441890383608,
r2n14%2C60200%2C1441836473749.1441836483297,
r2n14%2C60200%2C1441836473749.1441843684350,
r2n14%2C60200%2C1441836473749.1441890366118,
r2n14%2C60200%2C1441836473749.1441861684834,
r2n14%2C60200%2C1441836473749.1441854484648]
{noformat}
Even though the peer got removed, there are still some ZK entries left which
need to be cleaned.
was:
- Created replication peer POB3-HBASE-LL-A. By mistake used hyphens in the
replication peer name.
- Did {{remove_peer POB3-HBASE-LL-A}}
- Created a new peer POB3_HBASE_LL_A
- Verified zk for entries
{quote}
ls /hbase/replication/peers
[POB3_HBASE_LL_A]
ls /hbase/replication/rs
[r2n14,60200,1441836473749, r1n16,60200,1441836441250,
r1n14,60200,1441836352689, r3n15,60200,1441836547607,
r2n15,60200,1441836551992, r3n14,60200,1441836230275,
r1n15,60200,1441836355145]
ls /hbase/replication/rs/r2n14,60200,1441836473749
[POB3_HBASE_LL_A, POB3-HBASE-LL-A]
ls /hbase/replication/rs/r2n14,60200,1441836473749/POB3-HBASE-LL-A
[r2n14%2C60200%2C1441836473749.1441865284930,
r2n14%2C60200%2C1441836473749.1441847284451,
r2n14%2C60200%2C1441836473749.1441890379203,
r2n14%2C60200%2C1441836473749.1441890384725,
r2n14%2C60200%2C1441836473749.1441840084237,
r2n14%2C60200%2C1441836473749.1441876085231,
r2n14%2C60200%2C1441836473749.1441890367559,
r2n14%2C60200%2C1441836473749.1441883285466,
r2n14%2C60200%2C1441836473749.1441890370829,
r2n14%2C60200%2C1441836473749.1441879685329,
r2n14%2C60200%2C1441836473749.1441890382514,
r2n14%2C60200%2C1441836473749.1441890381514,
r2n14%2C60200%2C1441836473749.1441890380286,
r2n14%2C60200%2C1441836473749.1441890378164,
r2n14%2C60200%2C1441836473749.1441872485132,
r2n14%2C60200%2C1441836473749.1441868885039,
r2n14%2C60200%2C1441836473749.1441890374603,
r2n14%2C60200%2C1441836473749.1441890377072,
r2n14%2C60200%2C1441836473749.1441858084739,
r2n14%2C60200%2C1441836473749.1441890369005,
r2n14%2C60200%2C1441836473749.1441850884549,
r2n14%2C60200%2C1441836473749.1441886885548,
r2n14%2C60200%2C1441836473749.1441890372063,
r2n14%2C60200%2C1441836473749.1441890375866,
r2n14%2C60200%2C1441836473749.1441890373369,
r2n14%2C60200%2C1441836473749.1441890383608,
r2n14%2C60200%2C1441836473749.1441836483297,
r2n14%2C60200%2C1441836473749.1441843684350,
r2n14%2C60200%2C1441836473749.1441890366118,
r2n14%2C60200%2C1441836473749.1441861684834,
r2n14%2C60200%2C1441836473749.1441854484648]
{quote}
Even though the peer got removed, there are still some ZK entries left which
need to be cleaned.
> remove_peer of replication peers with hyphens in name doesn't remove ZK
> entries
> ---
>
> Key: HBASE-14412
> URL: https://issues.apache.org/jira/browse/HBASE-14412
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Biju Nair
>Priority: Minor
>
> -
[jira] [Commented] (HBASE-14307) Incorrect use of positional read api in HFileBlock
[
https://issues.apache.org/jira/browse/HBASE-14307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741434#comment-14741434
]
Chris Nauroth commented on HBASE-14307:
---
Everyone, thank you for the reviews, commits and addendums.
> Incorrect use of positional read api in HFileBlock
> --
>
> Key: HBASE-14307
> URL: https://issues.apache.org/jira/browse/HBASE-14307
> Project: HBase
> Issue Type: Bug
>Reporter: Shradha Revankar
>Assignee: Chris Nauroth
> Fix For: 2.0.0, 0.98.15, 1.2.1, 1.0.3, 1.1.3
>
> Attachments: 14307-branch-1.addendum, HBASE-14307.001.master.patch,
> HBASE-14307.002.master.patch, HBASE-14307.003.patch
>
>
> Considering that {{read()}} is not guaranteed to read all bytes,
> I'm interested to understand this particular piece of code and why is partial
> read treated as an error :
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java#L1446-L1450
> Particularly, if hbase were to use a different filesystem, say
> WebhdfsFileSystem, this would not work, please also see
> https://issues.apache.org/jira/browse/HDFS-8943 for discussion around this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14412) remove_peer of replication peers with hyphens in name doesn't remove ZK entries
Biju Nair created HBASE-14412:
-
Summary: remove_peer of replication peers with hyphens in name
doesn't remove ZK entries
Key: HBASE-14412
URL: https://issues.apache.org/jira/browse/HBASE-14412
Project: HBase
Issue Type: Bug
Components: Replication
Reporter: Biju Nair
Priority: Minor
- Created replication peer POB3-HBASE-LL-A. By mistake used hyphens in the
replication peer name.
- Did {{remove_peer POB3-HBASE-LL-A}}
- Created a new peer POB3_HBASE_LL_A
- Verified zk for entries
{{
ls /hbase/replication/peers
[POB3_HBASE_LL_A]
ls /hbase/replication/rs
[r2n14,60200,1441836473749, r1n16,60200,1441836441250,
r1n14,60200,1441836352689, r3n15,60200,1441836547607,
r2n15,60200,1441836551992, r3n14,60200,1441836230275,
r1n15,60200,1441836355145]
ls /hbase/replication/rs/r2n14,60200,1441836473749
[POB3_HBASE_LL_A, POB3-HBASE-LL-A]
ls /hbase/replication/rs/r2n14,60200,1441836473749/POB3-HBASE-LL-A
[r2n14%2C60200%2C1441836473749.1441865284930,
r2n14%2C60200%2C1441836473749.1441847284451,
r2n14%2C60200%2C1441836473749.1441890379203,
r2n14%2C60200%2C1441836473749.1441890384725,
r2n14%2C60200%2C1441836473749.1441840084237,
r2n14%2C60200%2C1441836473749.1441876085231,
r2n14%2C60200%2C1441836473749.1441890367559,
r2n14%2C60200%2C1441836473749.1441883285466,
r2n14%2C60200%2C1441836473749.1441890370829,
r2n14%2C60200%2C1441836473749.1441879685329,
r2n14%2C60200%2C1441836473749.1441890382514,
r2n14%2C60200%2C1441836473749.1441890381514,
r2n14%2C60200%2C1441836473749.1441890380286,
r2n14%2C60200%2C1441836473749.1441890378164,
r2n14%2C60200%2C1441836473749.1441872485132,
r2n14%2C60200%2C1441836473749.1441868885039,
r2n14%2C60200%2C1441836473749.1441890374603,
r2n14%2C60200%2C1441836473749.1441890377072,
r2n14%2C60200%2C1441836473749.1441858084739,
r2n14%2C60200%2C1441836473749.1441890369005,
r2n14%2C60200%2C1441836473749.1441850884549,
r2n14%2C60200%2C1441836473749.1441886885548,
r2n14%2C60200%2C1441836473749.1441890372063,
r2n14%2C60200%2C1441836473749.1441890375866,
r2n14%2C60200%2C1441836473749.1441890373369,
r2n14%2C60200%2C1441836473749.1441890383608,
r2n14%2C60200%2C1441836473749.1441836483297,
r2n14%2C60200%2C1441836473749.1441843684350,
r2n14%2C60200%2C1441836473749.1441890366118,
r2n14%2C60200%2C1441836473749.1441861684834,
r2n14%2C60200%2C1441836473749.1441854484648]
}}
Even though the peer got removed, there are still some ZK entries left which
need to be cleaned.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14411) Fix UT failures when using multiwal as default provider
[
https://issues.apache.org/jira/browse/HBASE-14411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-14411:
--
Description:
If we set hbase.wal.provider to multiwal in
hbase-server/src/test/resources/hbase-site.xml which allows us to use
BoundedRegionGroupingProvider in UT, we will observe below failures in current
code base:
{noformat}
Failed tests:
TestHLogRecordReader>TestWALRecordReader.testPartialRead:164 expected:<1> but
was:<2>
TestHLogRecordReader>TestWALRecordReader.testWALRecordReader:216 expected:<2>
but was:<3>
TestWALRecordReader.testPartialRead:164 expected:<1> but was:<2>
TestWALRecordReader.testWALRecordReader:216 expected:<2> but was:<3>
TestDistributedLogSplitting.testRecoveredEdits:276 edits dir should have more
than a single file in it. instead has 1
TestAtomicOperation.testMultiRowMutationMultiThreads:499 expected:<0> but
was:<1>
TestHRegionServerBulkLoad.testAtomicBulkLoad:307
Expected: is
but: was
TestLogRolling.testCompactionRecordDoesntBlockRolling:611 Should have WAL;
one table is not flushed expected:<1> but was:<0>
TestLogRolling.testLogRollOnDatanodeDeath:359 null
TestLogRolling.testLogRollOnPipelineRestart:472 Missing datanode should've
triggered a log roll
TestReplicationSourceManager.testLogRoll:237 expected:<6> but was:<7>
TestReplicationWALReaderManager.test:155 null
TestReplicationWALReaderManager.test:155 null
TestReplicationWALReaderManager.test:155 null
TestReplicationWALReaderManager.test:155 null
TestReplicationWALReaderManager.test:155 null
TestReplicationWALReaderManager.test:155 null
TestReplicationWALReaderManager.test:155 null
TestReplicationWALReaderManager.test:155 null
TestWALSplit.testCorruptedLogFilesSkipErrorsFalseDoesNotTouchLogs:594 if
skip.errors is false all files should remain in place expected:<11> but was:<12>
TestWALSplit.testLogsGetArchivedAfterSplit:649 wrong number of files in the
archive log expected:<11> but was:<12>
TestWALSplit.testMovedWALDuringRecovery:810->retryOverHdfsProblem:793
expected:<11> but was:<12>
TestWALSplit.testRetryOpenDuringRecovery:838->retryOverHdfsProblem:793
expected:<11> but was:<12>
TestWALSplitCompressed>TestWALSplit.testCorruptedLogFilesSkipErrorsFalseDoesNotTouchLogs:594
if skip.errors is false all files should remain in place expected:<11> but
was:<12>
TestWALSplitCompressed>TestWALSplit.testLogsGetArchivedAfterSplit:649 wrong
number of files in the archive log expected:<11> but was:<12>
TestWALSplitCompressed>TestWALSplit.testMovedWALDuringRecovery:810->TestWALSplit.retryOverHdfsProblem:793
expected:<11> but was:<12>
TestWALSplitCompressed>TestWALSplit.testRetryOpenDuringRecovery:838->TestWALSplit.retryOverHdfsProblem:793
expected:<11> but was:<12>
{noformat}
While patch for HBASE-14306 could resolve failures of TestHLogRecordReader,
TestReplicationSourceManager and TestReplicationWALReaderManager, this JIRA
will focus on resolving the others
was:
If we set hbase.wal.provider to multiwal in
hbase-server/src/test/resources/hbase-site.xml which allows us to use
BoundedRegionGroupingProvider in UT, we will observe below failures in current
code base:
{noformat}
Failed tests:
TestHLogRecordReader>TestWALRecordReader.testPartialRead:164 expected:<1> but
was:<2>
TestHLogRecordReader>TestWALRecordReader.testWALRecordReader:216 expected:<2>
but was:<3>
TestWALRecordReader.testPartialRead:164 expected:<1> but was:<2>
TestWALRecordReader.testWALRecordReader:216 expected:<2> but was:<3>
TestDistributedLogSplitting.testRecoveredEdits:276 edits dir should have more
than a single file in it. instead has 1
TestAtomicOperation.testMultiRowMutationMultiThreads:499 expected:<0> but
was:<1>
TestHRegionServerBulkLoad.testAtomicBulkLoad:307
Expected: is
but: was
TestLogRolling.testCompactionRecordDoesntBlockRolling:611 Should have WAL;
one table is not flushed expected:<1> but was:<0>
TestLogRolling.testLogRollOnDatanodeDeath:359 null
TestLogRolling.testLogRollOnPipelineRestart:472 Missing datanode should've
triggered a log roll
TestReplicationSourceManager.testLogRoll:237 expected:<6> but was:<7>
TestReplicationWALReaderManager.test:155 null
TestReplicationWALReaderManager.test:155 null
TestReplicationWALReaderManager.test:155 null
TestReplicationWALReaderManager.test:155 null
TestReplicationWALReaderManager.test:155 null
TestReplicationWALReaderManager.test:155 null
TestReplicationWALReaderManager.test:155 null
TestReplicationWALReaderManager.test:155 null
TestWALSplit.testCorruptedLogFilesSkipErrorsFalseDoesNotTouchLogs:594 if
skip.errors is false all files should remain in place expected:<11> but was:<12>
TestWALSplit.testLogsGetArchivedAfterSplit:649 wrong number of files in the
archive log expected:<11> but was:<12>
TestWALSplit.testMovedWALDuringRecovery:810->retryOverHdfsProblem:793
expected:<11>
[jira] [Updated] (HBASE-6617) ReplicationSourceManager should be able to track multiple WAL paths
[ https://issues.apache.org/jira/browse/HBASE-6617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-6617: -- Resolution: Fixed Status: Resolved (was: Patch Available) Thanks Yu for the contribution. Thanks for the reviews Sean and Chunhui. > ReplicationSourceManager should be able to track multiple WAL paths > --- > > Key: HBASE-6617 > URL: https://issues.apache.org/jira/browse/HBASE-6617 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Ted Yu >Assignee: Yu Li > Fix For: 2.0.0, 1.3.0 > > Attachments: 6617-v11.patch, HBASE-6617.branch-1.patch, > HBASE-6617.branch-1.v2.patch, HBASE-6617.branch-1.v2.patch, > HBASE-6617.branch-1.v2.patch, HBASE-6617.patch, HBASE-6617_v10.patch, > HBASE-6617_v11.patch, HBASE-6617_v12.patch, HBASE-6617_v2.patch, > HBASE-6617_v3.patch, HBASE-6617_v4.patch, HBASE-6617_v7.patch, > HBASE-6617_v9.patch > > > Currently ReplicationSourceManager uses logRolled() to receive notification > about new HLog and remembers it in latestPath. > When region server has multiple WAL support, we need to keep track of > multiple Path's in ReplicationSourceManager -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14380) Correct data gets skipped along with bad data in importTsv bulk load thru TsvImporterTextMapper
[ https://issues.apache.org/jira/browse/HBASE-14380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741404#comment-14741404 ] Hudson commented on HBASE-14380: FAILURE: Integrated in HBase-TRUNK #6798 (See [https://builds.apache.org/job/HBase-TRUNK/6798/]) HBASE-14380 Correct data gets skipped along with bad data in importTsv bulk load thru TsvImporterTextMapper (Bhupendra Kumar Jain) (tedyu: rev a8730c28390bf15cd68b728cfefd28817e918295) * hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTsv.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TextSortReducer.java > Correct data gets skipped along with bad data in importTsv bulk load thru > TsvImporterTextMapper > --- > > Key: HBASE-14380 > URL: https://issues.apache.org/jira/browse/HBASE-14380 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Bhupendra Kumar Jain >Assignee: Bhupendra Kumar Jain > Fix For: 2.0.0, 1.3.0 > > Attachments: 0001-HBASE-14380.patch, 14380-v2.txt, > HBASE-14380_v1.patch > > > Cosider the input data is as below > ROWKEY, TIEMSTAMP, Col_Value > r1,1,v1 >> Correct line > r1 >> Bad line > r1,3,v3 >> Correct line > r1,4,v4 >> Correct line > When data is bulk loaded using importTsv with mapper as TsvImporterTextMapper > , All the lines are getting ignored even though skipBadLines is set to true. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14400) Fix HBase RPC protection documentation
[ https://issues.apache.org/jira/browse/HBASE-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apekshit Sharma updated HBASE-14400: Attachment: HBASE-14400-branch-0.98.patch > Fix HBase RPC protection documentation > -- > > Key: HBASE-14400 > URL: https://issues.apache.org/jira/browse/HBASE-14400 > Project: HBase > Issue Type: Bug > Components: encryption, rpc, security >Reporter: Apekshit Sharma >Assignee: Apekshit Sharma >Priority: Critical > Fix For: 2.0.0, 1.2.1, 1.0.3, 1.1.3 > > Attachments: HBASE-14400-branch-0.98.patch, > HBASE-14400-branch-1.0.patch, HBASE-14400-branch-1.1.patch, > HBASE-14400-branch-1.2.patch, HBASE-14400-master-v2.patch, > HBASE-14400-master.patch > > > HBase configuration 'hbase.rpc.protection' can be set to 'authentication', > 'integrity' or 'privacy'. > "authentication means authentication only and no integrity or privacy; > integrity implies > authentication and integrity are enabled; and privacy implies all of > authentication, integrity and privacy are enabled." > However hbase ref guide incorrectly suggests in some places to set the value > to 'auth-conf' instead of 'privacy'. Setting value to 'auth-conf' doesn't > provide rpc encryption which is what user wants. > This jira will fix: > - documentation: change 'auth-conf' references to 'privacy' > - SaslUtil to support both set of values (privacy/integrity/authentication > and auth-conf/auth-int/auth) to be backward compatible with what was being > suggested till now. > - change 'hbase.thrift.security.qop' to be consistent with other similar > configurations by using same set of values (privacy/integrity/authentication). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14411) Fix UT failures when using multiwal as default provider
Yu Li created HBASE-14411:
-
Summary: Fix UT failures when using multiwal as default provider
Key: HBASE-14411
URL: https://issues.apache.org/jira/browse/HBASE-14411
Project: HBase
Issue Type: Bug
Reporter: Yu Li
Assignee: Yu Li
Fix For: 2.0.0
If we set hbase.wal.provider to multiwal in
hbase-server/src/test/resources/hbase-site.xml which allows us to use
BoundedRegionGroupingProvider in UT, we will observe below failures in current
code base:
{noformat}
Failed tests:
TestHLogRecordReader>TestWALRecordReader.testPartialRead:164 expected:<1> but
was:<2>
TestHLogRecordReader>TestWALRecordReader.testWALRecordReader:216 expected:<2>
but was:<3>
TestWALRecordReader.testPartialRead:164 expected:<1> but was:<2>
TestWALRecordReader.testWALRecordReader:216 expected:<2> but was:<3>
TestDistributedLogSplitting.testRecoveredEdits:276 edits dir should have more
than a single file in it. instead has 1
TestAtomicOperation.testMultiRowMutationMultiThreads:499 expected:<0> but
was:<1>
TestHRegionServerBulkLoad.testAtomicBulkLoad:307
Expected: is
but: was
TestLogRolling.testCompactionRecordDoesntBlockRolling:611 Should have WAL;
one table is not flushed expected:<1> but was:<0>
TestLogRolling.testLogRollOnDatanodeDeath:359 null
TestLogRolling.testLogRollOnPipelineRestart:472 Missing datanode should've
triggered a log roll
TestReplicationSourceManager.testLogRoll:237 expected:<6> but was:<7>
TestReplicationWALReaderManager.test:155 null
TestReplicationWALReaderManager.test:155 null
TestReplicationWALReaderManager.test:155 null
TestReplicationWALReaderManager.test:155 null
TestReplicationWALReaderManager.test:155 null
TestReplicationWALReaderManager.test:155 null
TestReplicationWALReaderManager.test:155 null
TestReplicationWALReaderManager.test:155 null
TestWALSplit.testCorruptedLogFilesSkipErrorsFalseDoesNotTouchLogs:594 if
skip.errors is false all files should remain in place expected:<11> but was:<12>
TestWALSplit.testLogsGetArchivedAfterSplit:649 wrong number of files in the
archive log expected:<11> but was:<12>
TestWALSplit.testMovedWALDuringRecovery:810->retryOverHdfsProblem:793
expected:<11> but was:<12>
TestWALSplit.testRetryOpenDuringRecovery:838->retryOverHdfsProblem:793
expected:<11> but was:<12>
TestWALSplitCompressed>TestWALSplit.testCorruptedLogFilesSkipErrorsFalseDoesNotTouchLogs:594
if skip.errors is false all files should remain in place expected:<11> but
was:<12>
TestWALSplitCompressed>TestWALSplit.testLogsGetArchivedAfterSplit:649 wrong
number of files in the archive log expected:<11> but was:<12>
TestWALSplitCompressed>TestWALSplit.testMovedWALDuringRecovery:810->TestWALSplit.retryOverHdfsProblem:793
expected:<11> but was:<12>
TestWALSplitCompressed>TestWALSplit.testRetryOpenDuringRecovery:838->TestWALSplit.retryOverHdfsProblem:793
expected:<11> but was:<12>
{noformat}
While patch for HBASE-14306 could resolve failures of TestHLogRecordReader,
TestReplicationSourceManager and TestReplicationWALReaderManager, this JIRA
will focus of resolving the others
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14411) Fix UT failures when using multiwal as default provider
[
https://issues.apache.org/jira/browse/HBASE-14411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741385#comment-14741385
]
Yu Li commented on HBASE-14411:
---
One thing to mention here is that most left UT failures are caused by lacking
of consideration on using multiple wal rather than fatal issue in current
mutiwal implementation, so the fixes would mainly on refining UT codes.
Meanwhile, fixing UT is important since only this way we could make sure
multiwal function works well in perspective of developer.
> Fix UT failures when using multiwal as default provider
> ---
>
> Key: HBASE-14411
> URL: https://issues.apache.org/jira/browse/HBASE-14411
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0
>
>
> If we set hbase.wal.provider to multiwal in
> hbase-server/src/test/resources/hbase-site.xml which allows us to use
> BoundedRegionGroupingProvider in UT, we will observe below failures in
> current code base:
> {noformat}
> Failed tests:
> TestHLogRecordReader>TestWALRecordReader.testPartialRead:164 expected:<1>
> but was:<2>
> TestHLogRecordReader>TestWALRecordReader.testWALRecordReader:216
> expected:<2> but was:<3>
> TestWALRecordReader.testPartialRead:164 expected:<1> but was:<2>
> TestWALRecordReader.testWALRecordReader:216 expected:<2> but was:<3>
> TestDistributedLogSplitting.testRecoveredEdits:276 edits dir should have
> more than a single file in it. instead has 1
> TestAtomicOperation.testMultiRowMutationMultiThreads:499 expected:<0> but
> was:<1>
> TestHRegionServerBulkLoad.testAtomicBulkLoad:307
> Expected: is
> but: was
> TestLogRolling.testCompactionRecordDoesntBlockRolling:611 Should have WAL;
> one table is not flushed expected:<1> but was:<0>
> TestLogRolling.testLogRollOnDatanodeDeath:359 null
> TestLogRolling.testLogRollOnPipelineRestart:472 Missing datanode should've
> triggered a log roll
> TestReplicationSourceManager.testLogRoll:237 expected:<6> but was:<7>
> TestReplicationWALReaderManager.test:155 null
> TestReplicationWALReaderManager.test:155 null
> TestReplicationWALReaderManager.test:155 null
> TestReplicationWALReaderManager.test:155 null
> TestReplicationWALReaderManager.test:155 null
> TestReplicationWALReaderManager.test:155 null
> TestReplicationWALReaderManager.test:155 null
> TestReplicationWALReaderManager.test:155 null
> TestWALSplit.testCorruptedLogFilesSkipErrorsFalseDoesNotTouchLogs:594 if
> skip.errors is false all files should remain in place expected:<11> but
> was:<12>
> TestWALSplit.testLogsGetArchivedAfterSplit:649 wrong number of files in the
> archive log expected:<11> but was:<12>
> TestWALSplit.testMovedWALDuringRecovery:810->retryOverHdfsProblem:793
> expected:<11> but was:<12>
> TestWALSplit.testRetryOpenDuringRecovery:838->retryOverHdfsProblem:793
> expected:<11> but was:<12>
>
> TestWALSplitCompressed>TestWALSplit.testCorruptedLogFilesSkipErrorsFalseDoesNotTouchLogs:594
> if skip.errors is false all files should remain in place expected:<11> but
> was:<12>
> TestWALSplitCompressed>TestWALSplit.testLogsGetArchivedAfterSplit:649 wrong
> number of files in the archive log expected:<11> but was:<12>
>
> TestWALSplitCompressed>TestWALSplit.testMovedWALDuringRecovery:810->TestWALSplit.retryOverHdfsProblem:793
> expected:<11> but was:<12>
>
> TestWALSplitCompressed>TestWALSplit.testRetryOpenDuringRecovery:838->TestWALSplit.retryOverHdfsProblem:793
> expected:<11> but was:<12>
> {noformat}
> While patch for HBASE-14306 could resolve failures of TestHLogRecordReader,
> TestReplicationSourceManager and TestReplicationWALReaderManager, this JIRA
> will focus on resolving the others
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14400) Fix HBase RPC protection documentation
[
https://issues.apache.org/jira/browse/HBASE-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741487#comment-14741487
]
Hadoop QA commented on HBASE-14400:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12755434/HBASE-14400-branch-1.2.patch
against branch-1.2 branch at commit c94d10952fe44f73096027cc9083cee993983940.
ATTACHMENT ID: 12755434
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 7 zombie test(s):
at
org.apache.hadoop.hbase.security.access.TestScanEarlyTermination.testEarlyScanTermination(TestScanEarlyTermination.java:249)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15565//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15565//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15565//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15565//console
This message is automatically generated.
> Fix HBase RPC protection documentation
> --
>
> Key: HBASE-14400
> URL: https://issues.apache.org/jira/browse/HBASE-14400
> Project: HBase
> Issue Type: Bug
> Components: encryption, rpc, security
>Reporter: Apekshit Sharma
>Assignee: Apekshit Sharma
>Priority: Critical
> Fix For: 2.0.0, 1.2.1, 1.0.3, 1.1.3
>
> Attachments: HBASE-14400-branch-0.98.patch,
> HBASE-14400-branch-1.0.patch, HBASE-14400-branch-1.1.patch,
> HBASE-14400-branch-1.2.patch, HBASE-14400-master-v2.patch,
> HBASE-14400-master.patch
>
>
> HBase configuration 'hbase.rpc.protection' can be set to 'authentication',
> 'integrity' or 'privacy'.
> "authentication means authentication only and no integrity or privacy;
> integrity implies
> authentication and integrity are enabled; and privacy implies all of
> authentication, integrity and privacy are enabled."
> However hbase ref guide incorrectly suggests in some places to set the value
> to 'auth-conf' instead of 'privacy'. Setting value to 'auth-conf' doesn't
> provide rpc encryption which is what user wants.
> This jira will fix:
> - documentation: change 'auth-conf' references to 'privacy'
> - SaslUtil to support both set of values (privacy/integrity/authentication
> and auth-conf/auth-int/auth) to be backward compatible with what was being
> suggested till now.
> - change 'hbase.thrift.security.qop' to be consistent with other similar
> configurations by using same set of values (privacy/integrity/authentication).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14370) Use separate thread for calling ZKPermissionWatcher#refreshNodes()
[
https://issues.apache.org/jira/browse/HBASE-14370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741503#comment-14741503
]
Ted Yu commented on HBASE-14370:
For 0.98 QA run:
{code}
fht https://builds.apache.org/job/PreCommit-HBASE-Build/15567/console
Fetching the console output from the URL
Printing hanging tests
Hanging test : org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan1
Hanging test : org.apache.hadoop.hbase.mapreduce.TestMultiTableInputFormat
Printing Failing tests
{code}
The above hanging tests were not related to the patch.
Planning to commit in the near future.
> Use separate thread for calling ZKPermissionWatcher#refreshNodes()
> --
>
> Key: HBASE-14370
> URL: https://issues.apache.org/jira/browse/HBASE-14370
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.15, 1.0.3, 1.1.3
>
> Attachments: 14370-0.98-v10.txt, 14370-branch-1-v10.txt,
> 14370-branch-1-v10.txt, 14370-v1.txt, 14370-v10.txt, 14370-v3.txt,
> 14370-v5.txt, 14370-v7.txt, 14370-v8.txt, 14370-wait-nofity-v2.txt,
> 14370-wait-nofity.txt, hbase-14370_v4.patch, test-acl3-branch-1.stack
>
>
> I came off a support case (0.98.0) where main zk thread was seen doing the
> following:
> {code}
> at
> org.apache.hadoop.hbase.security.access.ZKPermissionWatcher.refreshAuthManager(ZKPermissionWatcher.java:152)
> at
> org.apache.hadoop.hbase.security.access.ZKPermissionWatcher.refreshNodes(ZKPermissionWatcher.java:135)
> at
> org.apache.hadoop.hbase.security.access.ZKPermissionWatcher.nodeChildrenChanged(ZKPermissionWatcher.java:121)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:348)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
> {code}
> There were 62000 nodes under /acl due to lack of fix from HBASE-12635,
> leading to slowness in table creation because zk notification for region
> offline was blocked by the above.
> The attached patch separates refreshNodes() call into its own thread.
> Thanks to Enis and Devaraj for offline discussion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14412) remove_peer of replication peers with hyphens in name doesn't remove ZK entries
[
https://issues.apache.org/jira/browse/HBASE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Biju Nair updated HBASE-14412:
--
Description:
- Created replication peer POB3-HBASE-LL-A. By mistake used hyphens in the
replication peer name.
- Did {{remove_peer POB3-HBASE-LL-A}}
- Created a new peer POB3_HBASE_LL_A
- Verified zk for entries
```
ls /hbase/replication/peers
[POB3_HBASE_LL_A]
ls /hbase/replication/rs
[r2n14,60200,1441836473749, r1n16,60200,1441836441250,
r1n14,60200,1441836352689, r3n15,60200,1441836547607,
r2n15,60200,1441836551992, r3n14,60200,1441836230275,
r1n15,60200,1441836355145]
ls /hbase/replication/rs/r2n14,60200,1441836473749
[POB3_HBASE_LL_A, POB3-HBASE-LL-A]
ls /hbase/replication/rs/r2n14,60200,1441836473749/POB3-HBASE-LL-A
[r2n14%2C60200%2C1441836473749.1441865284930,
r2n14%2C60200%2C1441836473749.1441847284451,
r2n14%2C60200%2C1441836473749.1441890379203,
r2n14%2C60200%2C1441836473749.1441890384725,
r2n14%2C60200%2C1441836473749.1441840084237,
r2n14%2C60200%2C1441836473749.1441876085231,
r2n14%2C60200%2C1441836473749.1441890367559,
r2n14%2C60200%2C1441836473749.1441883285466,
r2n14%2C60200%2C1441836473749.1441890370829,
r2n14%2C60200%2C1441836473749.1441879685329,
r2n14%2C60200%2C1441836473749.1441890382514,
r2n14%2C60200%2C1441836473749.1441890381514,
r2n14%2C60200%2C1441836473749.1441890380286,
r2n14%2C60200%2C1441836473749.1441890378164,
r2n14%2C60200%2C1441836473749.1441872485132,
r2n14%2C60200%2C1441836473749.1441868885039,
r2n14%2C60200%2C1441836473749.1441890374603,
r2n14%2C60200%2C1441836473749.1441890377072,
r2n14%2C60200%2C1441836473749.1441858084739,
r2n14%2C60200%2C1441836473749.1441890369005,
r2n14%2C60200%2C1441836473749.1441850884549,
r2n14%2C60200%2C1441836473749.1441886885548,
r2n14%2C60200%2C1441836473749.1441890372063,
r2n14%2C60200%2C1441836473749.1441890375866,
r2n14%2C60200%2C1441836473749.1441890373369,
r2n14%2C60200%2C1441836473749.1441890383608,
r2n14%2C60200%2C1441836473749.1441836483297,
r2n14%2C60200%2C1441836473749.1441843684350,
r2n14%2C60200%2C1441836473749.1441890366118,
r2n14%2C60200%2C1441836473749.1441861684834,
r2n14%2C60200%2C1441836473749.1441854484648]
```
Even though the peer got removed, there are still some ZK entries left which
need to be cleaned.
was:
- Created replication peer POB3-HBASE-LL-A. By mistake used hyphens in the
replication peer name.
- Did {{remove_peer POB3-HBASE-LL-A}}
- Created a new peer POB3_HBASE_LL_A
- Verified zk for entries
{{
ls /hbase/replication/peers
[POB3_HBASE_LL_A]
ls /hbase/replication/rs
[r2n14,60200,1441836473749, r1n16,60200,1441836441250,
r1n14,60200,1441836352689, r3n15,60200,1441836547607,
r2n15,60200,1441836551992, r3n14,60200,1441836230275,
r1n15,60200,1441836355145]
ls /hbase/replication/rs/r2n14,60200,1441836473749
[POB3_HBASE_LL_A, POB3-HBASE-LL-A]
ls /hbase/replication/rs/r2n14,60200,1441836473749/POB3-HBASE-LL-A
[r2n14%2C60200%2C1441836473749.1441865284930,
r2n14%2C60200%2C1441836473749.1441847284451,
r2n14%2C60200%2C1441836473749.1441890379203,
r2n14%2C60200%2C1441836473749.1441890384725,
r2n14%2C60200%2C1441836473749.1441840084237,
r2n14%2C60200%2C1441836473749.1441876085231,
r2n14%2C60200%2C1441836473749.1441890367559,
r2n14%2C60200%2C1441836473749.1441883285466,
r2n14%2C60200%2C1441836473749.1441890370829,
r2n14%2C60200%2C1441836473749.1441879685329,
r2n14%2C60200%2C1441836473749.1441890382514,
r2n14%2C60200%2C1441836473749.1441890381514,
r2n14%2C60200%2C1441836473749.1441890380286,
r2n14%2C60200%2C1441836473749.1441890378164,
r2n14%2C60200%2C1441836473749.1441872485132,
r2n14%2C60200%2C1441836473749.1441868885039,
r2n14%2C60200%2C1441836473749.1441890374603,
r2n14%2C60200%2C1441836473749.1441890377072,
r2n14%2C60200%2C1441836473749.1441858084739,
r2n14%2C60200%2C1441836473749.1441890369005,
r2n14%2C60200%2C1441836473749.1441850884549,
r2n14%2C60200%2C1441836473749.1441886885548,
r2n14%2C60200%2C1441836473749.1441890372063,
r2n14%2C60200%2C1441836473749.1441890375866,
r2n14%2C60200%2C1441836473749.1441890373369,
r2n14%2C60200%2C1441836473749.1441890383608,
r2n14%2C60200%2C1441836473749.1441836483297,
r2n14%2C60200%2C1441836473749.1441843684350,
r2n14%2C60200%2C1441836473749.1441890366118,
r2n14%2C60200%2C1441836473749.1441861684834,
r2n14%2C60200%2C1441836473749.1441854484648]
}}
Even though the peer got removed, there are still some ZK entries left which
need to be cleaned.
> remove_peer of replication peers with hyphens in name doesn't remove ZK
> entries
> ---
>
> Key: HBASE-14412
> URL: https://issues.apache.org/jira/browse/HBASE-14412
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Biju Nair
>Priority: Minor
>
> - Created replication
[jira] [Updated] (HBASE-14394) Properly close the connection after reading records from table.
[ https://issues.apache.org/jira/browse/HBASE-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Srikanth Srungarapu updated HBASE-14394: Attachment: HBASE-14394_v4.patch Adding null check before closing the connection. > Properly close the connection after reading records from table. > --- > > Key: HBASE-14394 > URL: https://issues.apache.org/jira/browse/HBASE-14394 > Project: HBase > Issue Type: Bug >Reporter: Srikanth Srungarapu >Assignee: Srikanth Srungarapu >Priority: Minor > Attachments: HBASE-14394.patch, HBASE-14394_v2.patch, > HBASE-14394_v3.patch, HBASE-14394_v4.patch > > > This was brought to notice by one of our observant customers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14401) Stamp failed appends with sequenceid too.... Cleans up latches
[
https://issues.apache.org/jira/browse/HBASE-14401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-14401:
--
Attachment: 14401v6.txt
v3 removed this
142 // Don't countdown latch until someone waiting on it.
143 while (this.latch.getCount() <= 0) {
144 Threads.sleep(10);
145 }
... from test. Put it back.
It is needed to make sure we go through the test in the right order... i.e.
hold up the roll of the log after it has taken out latch and before it sends in
its sync... only then let the errant sync processing of zigzag happen -- attain
safe point.
The idea is that a sync other than the roll wal sync comes in before the roll
wal... in the past it used to hang us up...
Not doing the wait above, had the errant sync cleaning up the zigzag latch
BEFORE the wall roll got to the test latch meant no one to clear the test
latch. We were hung.
> Stamp failed appends with sequenceid too Cleans up latches
> --
>
> Key: HBASE-14401
> URL: https://issues.apache.org/jira/browse/HBASE-14401
> Project: HBase
> Issue Type: Sub-task
> Components: test, wal
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14401.txt, 14401v3.txt, 14401v3.txt, 14401v3.txt,
> 14401v6.txt
>
>
> Looking in test output I see we can sometimes get stuck waiting on
> sequenceid... The parent issues redo of our semantic makes it so we encounter
> failed append more often around damaged WAL.
> This patch makes it so we stamp sequenceid always, even if the append fails.
> This way all sequenceids are accounted for but more important, the latch on
> sequenceid down in WALKey will be cleared.. where before it was not being
> cleared (there is no global list of outstanding WALKeys waiting on
> sequenceids so no way to clean them up... we don't need such a list if we
> ALWAYS stamp the sequenceid).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14412) remove_peer of replication peers with hyphens in name doesn't remove ZK entries
[
https://issues.apache.org/jira/browse/HBASE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Biju Nair updated HBASE-14412:
--
Description:
- Created replication peer POB3-HBASE-LL-A. By mistake used hyphens in the
replication peer name.
- Did {{remove_peer POB3-HBASE-LL-A}}
- Created a new peer POB3_HBASE_LL_A
- Verified zk for entries
{quote}
ls /hbase/replication/peers
[POB3_HBASE_LL_A]
ls /hbase/replication/rs
[r2n14,60200,1441836473749, r1n16,60200,1441836441250,
r1n14,60200,1441836352689, r3n15,60200,1441836547607,
r2n15,60200,1441836551992, r3n14,60200,1441836230275,
r1n15,60200,1441836355145]
ls /hbase/replication/rs/r2n14,60200,1441836473749
[POB3_HBASE_LL_A, POB3-HBASE-LL-A]
ls /hbase/replication/rs/r2n14,60200,1441836473749/POB3-HBASE-LL-A
[r2n14%2C60200%2C1441836473749.1441865284930,
r2n14%2C60200%2C1441836473749.1441847284451,
r2n14%2C60200%2C1441836473749.1441890379203,
r2n14%2C60200%2C1441836473749.1441890384725,
r2n14%2C60200%2C1441836473749.1441840084237,
r2n14%2C60200%2C1441836473749.1441876085231,
r2n14%2C60200%2C1441836473749.1441890367559,
r2n14%2C60200%2C1441836473749.1441883285466,
r2n14%2C60200%2C1441836473749.1441890370829,
r2n14%2C60200%2C1441836473749.1441879685329,
r2n14%2C60200%2C1441836473749.1441890382514,
r2n14%2C60200%2C1441836473749.1441890381514,
r2n14%2C60200%2C1441836473749.1441890380286,
r2n14%2C60200%2C1441836473749.1441890378164,
r2n14%2C60200%2C1441836473749.1441872485132,
r2n14%2C60200%2C1441836473749.1441868885039,
r2n14%2C60200%2C1441836473749.1441890374603,
r2n14%2C60200%2C1441836473749.1441890377072,
r2n14%2C60200%2C1441836473749.1441858084739,
r2n14%2C60200%2C1441836473749.1441890369005,
r2n14%2C60200%2C1441836473749.1441850884549,
r2n14%2C60200%2C1441836473749.1441886885548,
r2n14%2C60200%2C1441836473749.1441890372063,
r2n14%2C60200%2C1441836473749.1441890375866,
r2n14%2C60200%2C1441836473749.1441890373369,
r2n14%2C60200%2C1441836473749.1441890383608,
r2n14%2C60200%2C1441836473749.1441836483297,
r2n14%2C60200%2C1441836473749.1441843684350,
r2n14%2C60200%2C1441836473749.1441890366118,
r2n14%2C60200%2C1441836473749.1441861684834,
r2n14%2C60200%2C1441836473749.1441854484648]
{quote}
Even though the peer got removed, there are still some ZK entries left which
need to be cleaned.
was:
- Created replication peer POB3-HBASE-LL-A. By mistake used hyphens in the
replication peer name.
- Did {{remove_peer POB3-HBASE-LL-A}}
- Created a new peer POB3_HBASE_LL_A
- Verified zk for entries
```
ls /hbase/replication/peers
[POB3_HBASE_LL_A]
ls /hbase/replication/rs
[r2n14,60200,1441836473749, r1n16,60200,1441836441250,
r1n14,60200,1441836352689, r3n15,60200,1441836547607,
r2n15,60200,1441836551992, r3n14,60200,1441836230275,
r1n15,60200,1441836355145]
ls /hbase/replication/rs/r2n14,60200,1441836473749
[POB3_HBASE_LL_A, POB3-HBASE-LL-A]
ls /hbase/replication/rs/r2n14,60200,1441836473749/POB3-HBASE-LL-A
[r2n14%2C60200%2C1441836473749.1441865284930,
r2n14%2C60200%2C1441836473749.1441847284451,
r2n14%2C60200%2C1441836473749.1441890379203,
r2n14%2C60200%2C1441836473749.1441890384725,
r2n14%2C60200%2C1441836473749.1441840084237,
r2n14%2C60200%2C1441836473749.1441876085231,
r2n14%2C60200%2C1441836473749.1441890367559,
r2n14%2C60200%2C1441836473749.1441883285466,
r2n14%2C60200%2C1441836473749.1441890370829,
r2n14%2C60200%2C1441836473749.1441879685329,
r2n14%2C60200%2C1441836473749.1441890382514,
r2n14%2C60200%2C1441836473749.1441890381514,
r2n14%2C60200%2C1441836473749.1441890380286,
r2n14%2C60200%2C1441836473749.1441890378164,
r2n14%2C60200%2C1441836473749.1441872485132,
r2n14%2C60200%2C1441836473749.1441868885039,
r2n14%2C60200%2C1441836473749.1441890374603,
r2n14%2C60200%2C1441836473749.1441890377072,
r2n14%2C60200%2C1441836473749.1441858084739,
r2n14%2C60200%2C1441836473749.1441890369005,
r2n14%2C60200%2C1441836473749.1441850884549,
r2n14%2C60200%2C1441836473749.1441886885548,
r2n14%2C60200%2C1441836473749.1441890372063,
r2n14%2C60200%2C1441836473749.1441890375866,
r2n14%2C60200%2C1441836473749.1441890373369,
r2n14%2C60200%2C1441836473749.1441890383608,
r2n14%2C60200%2C1441836473749.1441836483297,
r2n14%2C60200%2C1441836473749.1441843684350,
r2n14%2C60200%2C1441836473749.1441890366118,
r2n14%2C60200%2C1441836473749.1441861684834,
r2n14%2C60200%2C1441836473749.1441854484648]
```
Even though the peer got removed, there are still some ZK entries left which
need to be cleaned.
> remove_peer of replication peers with hyphens in name doesn't remove ZK
> entries
> ---
>
> Key: HBASE-14412
> URL: https://issues.apache.org/jira/browse/HBASE-14412
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Biju Nair
>Priority: Minor
>
> - Created
[jira] [Commented] (HBASE-14370) Use separate thread for calling ZKPermissionWatcher#refreshNodes()
[
https://issues.apache.org/jira/browse/HBASE-14370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741495#comment-14741495
]
Hadoop QA commented on HBASE-14370:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12755457/14370-0.98-v10.txt
against 0.98 branch at commit c94d10952fe44f73096027cc9083cee993983940.
ATTACHMENT ID: 12755457
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 14 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
22 warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
3869 checkstyle errors (more than the master's current 3868 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15567//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15567//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15567//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15567//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15567//console
This message is automatically generated.
> Use separate thread for calling ZKPermissionWatcher#refreshNodes()
> --
>
> Key: HBASE-14370
> URL: https://issues.apache.org/jira/browse/HBASE-14370
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.15, 1.0.3, 1.1.3
>
> Attachments: 14370-0.98-v10.txt, 14370-branch-1-v10.txt,
> 14370-branch-1-v10.txt, 14370-v1.txt, 14370-v10.txt, 14370-v3.txt,
> 14370-v5.txt, 14370-v7.txt, 14370-v8.txt, 14370-wait-nofity-v2.txt,
> 14370-wait-nofity.txt, hbase-14370_v4.patch, test-acl3-branch-1.stack
>
>
> I came off a support case (0.98.0) where main zk thread was seen doing the
> following:
> {code}
> at
> org.apache.hadoop.hbase.security.access.ZKPermissionWatcher.refreshAuthManager(ZKPermissionWatcher.java:152)
> at
> org.apache.hadoop.hbase.security.access.ZKPermissionWatcher.refreshNodes(ZKPermissionWatcher.java:135)
> at
> org.apache.hadoop.hbase.security.access.ZKPermissionWatcher.nodeChildrenChanged(ZKPermissionWatcher.java:121)
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:348)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
> {code}
> There were 62000 nodes under /acl due to lack of fix from HBASE-12635,
> leading to slowness in table creation because zk notification for region
> offline was blocked by the above.
> The attached patch separates refreshNodes() call into its own thread.
> Thanks to Enis and Devaraj for offline discussion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14400) Fix HBase RPC protection documentation
[
https://issues.apache.org/jira/browse/HBASE-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741494#comment-14741494
]
Hadoop QA commented on HBASE-14400:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12755467/HBASE-14400-branch-0.98.patch
against 0.98 branch at commit c94d10952fe44f73096027cc9083cee993983940.
ATTACHMENT ID: 12755467
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:red}-1 findbugs{color}. The patch appears to cause Findbugs
(version 2.0.3) to fail.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn post-site goal
to fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15568//testReport/
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15568//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15568//console
This message is automatically generated.
> Fix HBase RPC protection documentation
> --
>
> Key: HBASE-14400
> URL: https://issues.apache.org/jira/browse/HBASE-14400
> Project: HBase
> Issue Type: Bug
> Components: encryption, rpc, security
>Reporter: Apekshit Sharma
>Assignee: Apekshit Sharma
>Priority: Critical
> Fix For: 2.0.0, 1.2.1, 1.0.3, 1.1.3
>
> Attachments: HBASE-14400-branch-0.98.patch,
> HBASE-14400-branch-1.0.patch, HBASE-14400-branch-1.1.patch,
> HBASE-14400-branch-1.2.patch, HBASE-14400-master-v2.patch,
> HBASE-14400-master.patch
>
>
> HBase configuration 'hbase.rpc.protection' can be set to 'authentication',
> 'integrity' or 'privacy'.
> "authentication means authentication only and no integrity or privacy;
> integrity implies
> authentication and integrity are enabled; and privacy implies all of
> authentication, integrity and privacy are enabled."
> However hbase ref guide incorrectly suggests in some places to set the value
> to 'auth-conf' instead of 'privacy'. Setting value to 'auth-conf' doesn't
> provide rpc encryption which is what user wants.
> This jira will fix:
> - documentation: change 'auth-conf' references to 'privacy'
> - SaslUtil to support both set of values (privacy/integrity/authentication
> and auth-conf/auth-int/auth) to be backward compatible with what was being
> suggested till now.
> - change 'hbase.thrift.security.qop' to be consistent with other similar
> configurations by using same set of values (privacy/integrity/authentication).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12751) Allow RowLock to be reader writer
[ https://issues.apache.org/jira/browse/HBASE-12751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741504#comment-14741504 ] stack commented on HBASE-12751: --- All this fails: 2015-09-11 13:08:01.896 Vim[4556:17136812] Can't allocate a new block for a pasteboard. Creation of a new Pasteboard will fail. kalashnikov:hbase.git stack$ python dev-support/findHangingTests.py https://builds.apache.org/job/PreCommit-HBASE-Build/15564//consoleText Fetching the console output from the URL Printing hanging tests Hanging test : org.apache.hadoop.hbase.replication.TestMasterReplication Hanging test : org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol Hanging test : org.apache.hadoop.hbase.master.handler.TestCreateTableHandler Hanging test : org.apache.hadoop.hbase.TestIOFencing Hanging test : org.apache.hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpoint Hanging test : org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient Hanging test : org.apache.hadoop.hbase.master.TestDistributedLogSplitting Hanging test : org.apache.hadoop.hbase.io.hfile.TestHFileBlock Hanging test : org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancer Hanging test : org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures Hanging test : org.apache.hadoop.hbase.io.hfile.TestScannerSelectionUsingTTL Hanging test : org.apache.hadoop.hbase.master.procedure.TestServerCrashProcedure Hanging test : org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster Hanging test : org.apache.hadoop.hbase.master.snapshot.TestSnapshotFileCache Hanging test : org.apache.hadoop.hbase.master.TestMasterFailover Hanging test : org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner Hanging test : org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover Hanging test : org.apache.hadoop.hbase.replication.regionserver.TestReplicationWALReaderManager Hanging test : org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancer2 Hanging test : org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite Hanging test : org.apache.hadoop.hbase.regionserver.compactions.TestCompactionWithThroughputController Hanging test : org.apache.hadoop.hbase.master.TestMasterShutdown Hanging test : org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDistributedLogReplay Hanging test : org.apache.hadoop.hbase.snapshot.TestExportSnapshot Hanging test : org.apache.hadoop.hbase.regionserver.TestRegionReplicas Printing Failing tests Failing test : org.apache.hadoop.hbase.client.TestFastFail Failing test : org.apache.hadoop.hbase.master.procedure.TestWALProcedureStoreOnHDFS > Allow RowLock to be reader writer > - > > Key: HBASE-12751 > URL: https://issues.apache.org/jira/browse/HBASE-12751 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 2.0.0, 1.3.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.3.0 > > Attachments: 12751.rebased.v25.txt, 12751.rebased.v26.txt, > 12751.rebased.v26.txt, 12751.rebased.v27.txt, 12751.rebased.v29.txt, > 12751.rebased.v31.txt, 12751.rebased.v32.txt, 12751.rebased.v32.txt, > 12751.rebased.v33.txt, 12751.rebased.v34.txt, 12751.rebased.v35.txt, > 12751.rebased.v35.txt, 12751v22.txt, 12751v23.txt, 12751v23.txt, > 12751v23.txt, 12751v23.txt, HBASE-12751-v1.patch, HBASE-12751-v10.patch, > HBASE-12751-v10.patch, HBASE-12751-v11.patch, HBASE-12751-v12.patch, > HBASE-12751-v13.patch, HBASE-12751-v14.patch, HBASE-12751-v15.patch, > HBASE-12751-v16.patch, HBASE-12751-v17.patch, HBASE-12751-v18.patch, > HBASE-12751-v19 (1).patch, HBASE-12751-v19.patch, HBASE-12751-v2.patch, > HBASE-12751-v20.patch, HBASE-12751-v20.patch, HBASE-12751-v21.patch, > HBASE-12751-v3.patch, HBASE-12751-v4.patch, HBASE-12751-v5.patch, > HBASE-12751-v6.patch, HBASE-12751-v7.patch, HBASE-12751-v8.patch, > HBASE-12751-v9.patch, HBASE-12751.patch > > > Right now every write operation grabs a row lock. This is to prevent values > from changing during a read modify write operation (increment or check and > put). However it limits parallelism in several different scenarios. > If there are several puts to the same row but different columns or stores > then this is very limiting. > If there are puts to the same column then mvcc number should ensure a > consistent ordering. So locking is not needed. > However locking for check and put or increment is still needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14412) remove_peer of replication peers with hyphens in name doesn't remove ZK entries
[
https://issues.apache.org/jira/browse/HBASE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741587#comment-14741587
]
Enis Soztutar commented on HBASE-14412:
---
Biju, HBASE-11394 is already committed. In never code bases, you cannot create
a peer with hyphen in the name. Resolve this?
> remove_peer of replication peers with hyphens in name doesn't remove ZK
> entries
> ---
>
> Key: HBASE-14412
> URL: https://issues.apache.org/jira/browse/HBASE-14412
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Biju Nair
>Priority: Minor
>
> - Created replication peer POB3-HBASE-LL-A. By mistake used hyphens in the
> replication peer name.
> - Did {{remove_peer POB3-HBASE-LL-A}}
> - Created a new peer POB3_HBASE_LL_A
> - Verified zk for entries
> {noformat}
> ls /hbase/replication/peers
> [POB3_HBASE_LL_A]
> ls /hbase/replication/rs
> [r2n14,60200,1441836473749, r1n16,60200,1441836441250,
> r1n14,60200,1441836352689, r3n15,60200,1441836547607,
> r2n15,60200,1441836551992, r3n14,60200,1441836230275,
> r1n15,60200,1441836355145]
> ls /hbase/replication/rs/r2n14,60200,1441836473749
> [POB3_HBASE_LL_A, POB3-HBASE-LL-A]
> ls /hbase/replication/rs/r2n14,60200,1441836473749/POB3-HBASE-LL-A
> [r2n14%2C60200%2C1441836473749.1441865284930,
> r2n14%2C60200%2C1441836473749.1441847284451,
> r2n14%2C60200%2C1441836473749.1441890379203,
> r2n14%2C60200%2C1441836473749.1441890384725,
> r2n14%2C60200%2C1441836473749.1441840084237,
> r2n14%2C60200%2C1441836473749.1441876085231,
> r2n14%2C60200%2C1441836473749.1441890367559,
> r2n14%2C60200%2C1441836473749.1441883285466,
> r2n14%2C60200%2C1441836473749.1441890370829,
> r2n14%2C60200%2C1441836473749.1441879685329,
> r2n14%2C60200%2C1441836473749.1441890382514,
> r2n14%2C60200%2C1441836473749.1441890381514,
> r2n14%2C60200%2C1441836473749.1441890380286,
> r2n14%2C60200%2C1441836473749.1441890378164,
> r2n14%2C60200%2C1441836473749.1441872485132,
> r2n14%2C60200%2C1441836473749.1441868885039,
> r2n14%2C60200%2C1441836473749.1441890374603,
> r2n14%2C60200%2C1441836473749.1441890377072,
> r2n14%2C60200%2C1441836473749.1441858084739,
> r2n14%2C60200%2C1441836473749.1441890369005,
> r2n14%2C60200%2C1441836473749.1441850884549,
> r2n14%2C60200%2C1441836473749.1441886885548,
> r2n14%2C60200%2C1441836473749.1441890372063,
> r2n14%2C60200%2C1441836473749.1441890375866,
> r2n14%2C60200%2C1441836473749.1441890373369,
> r2n14%2C60200%2C1441836473749.1441890383608,
> r2n14%2C60200%2C1441836473749.1441836483297,
> r2n14%2C60200%2C1441836473749.1441843684350,
> r2n14%2C60200%2C1441836473749.1441890366118,
> r2n14%2C60200%2C1441836473749.1441861684834,
> r2n14%2C60200%2C1441836473749.1441854484648]
> {noformat}
> Even though the peer got removed, there are still some ZK entries left which
> need to be cleaned.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14413) HBase snapshots v2
Vladimir Rodionov created HBASE-14413: - Summary: HBase snapshots v2 Key: HBASE-14413 URL: https://issues.apache.org/jira/browse/HBASE-14413 Project: HBase Issue Type: New Feature Reporter: Vladimir Rodionov Fix For: 2.0.0 We need new implementation of snapshot feature that is more robust and performant. Ideally, it will work with multiple tables as well. The possible areas of improvements: # Must be flushless. Coordinated memstore flushes across a cluster are bad. # Verification phase must be distributed, done in parallel and not on Master. In theory, the only info we need to record snapshot of a table: list of WAL files, list of HFiles and max sequence id of an edit which has been flushed per Region. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14413) HBase snapshots v2
[ https://issues.apache.org/jira/browse/HBASE-14413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741617#comment-14741617 ] Vladimir Rodionov commented on HBASE-14413: --- Ping [~mbertozzi] > HBase snapshots v2 > -- > > Key: HBASE-14413 > URL: https://issues.apache.org/jira/browse/HBASE-14413 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov > Fix For: 2.0.0 > > > We need new implementation of snapshot feature that is more robust and > performant. Ideally, it will work with multiple tables as well. The possible > areas of improvements: > # Must be flushless. Coordinated memstore flushes across a cluster are bad. > # Verification phase must be distributed, done in parallel and not on Master. > In theory, the only info we need to record snapshot of a table: list of WAL > files, list of HFiles and max sequence id of an edit which has been flushed > per Region. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14415) Full backup based on Snapshot v2
Vladimir Rodionov created HBASE-14415: - Summary: Full backup based on Snapshot v2 Key: HBASE-14415 URL: https://issues.apache.org/jira/browse/HBASE-14415 Project: HBase Issue Type: New Feature Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-6617) ReplicationSourceManager should be able to track multiple WAL paths
[ https://issues.apache.org/jira/browse/HBASE-6617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741658#comment-14741658 ] Hudson commented on HBASE-6617: --- FAILURE: Integrated in HBase-1.3 #165 (See [https://builds.apache.org/job/HBase-1.3/165/]) HBASE-6617 ReplicationSourceManager should be able to track multiple WAL paths (Yu Li) (tedyu: rev be96bb6adf77f4bbcc1513cb407a705ac4624a0f) * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/ReplicationEndpoint.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HBaseInterClusterReplicationEndpoint.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationWALReaderManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/wal/DefaultWALProvider.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/multiwal/TestReplicationKillMasterRSCompressedWithMultipleWAL.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/multiwal/TestReplicationSyncUpToolWithMultipleWAL.java * hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALFactory.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSource.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationEndpoint.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/multiwal/TestReplicationEndpointWithMultipleWAL.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpointNoMaster.java > ReplicationSourceManager should be able to track multiple WAL paths > --- > > Key: HBASE-6617 > URL: https://issues.apache.org/jira/browse/HBASE-6617 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Ted Yu >Assignee: Yu Li > Fix For: 2.0.0, 1.3.0 > > Attachments: 6617-v11.patch, HBASE-6617.branch-1.patch, > HBASE-6617.branch-1.v2.patch, HBASE-6617.branch-1.v2.patch, > HBASE-6617.branch-1.v2.patch, HBASE-6617.patch, HBASE-6617_v10.patch, > HBASE-6617_v11.patch, HBASE-6617_v12.patch, HBASE-6617_v2.patch, > HBASE-6617_v3.patch, HBASE-6617_v4.patch, HBASE-6617_v7.patch, > HBASE-6617_v9.patch > > > Currently ReplicationSourceManager uses logRolled() to receive notification > about new HLog and remembers it in latestPath. > When region server has multiple WAL support, we need to keep track of > multiple Path's in ReplicationSourceManager -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14261) Enhance Chaos Monkey framework by adding zookeeper and datanode fault injections.
[
https://issues.apache.org/jira/browse/HBASE-14261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Srikanth Srungarapu updated HBASE-14261:
Release Note:
This change augments existing chaos monkey framework with actions for
restarting underlying zookeeper quorum and hdfs nodes of distributed hbase
cluster. One assumption made while creating zk actions are that zookeper
ensemble is an independent external service and won't be managed by hbase
cluster. For these actions to work as expected, the following parameters need
to be configured appropriately.
{code}
hbase.it.clustermanager.hadoop.home
$HADOOP_HOME
hbase.it.clustermanager.zookeeper.home
$ZOOKEEPER_HOME
hbase.it.clustermanager.hbase.user
hbase
hbase.it.clustermanager.hadoop.hdfs.user
hdfs
hbase.it.clustermanager.zookeeper.user
zookeeper
{code}
The service user related configurations are newly introduced since in prod/test
environments each service is managed by different user. Once the above
parameters are configured properly, you can start using them as needed. An
example usage for invoking these new actions is:
{{./hbase org.apache.hadoop.hbase.IntegrationTestAcidGuarantees -m
serverAndDependenciesKilling}}
> Enhance Chaos Monkey framework by adding zookeeper and datanode fault
> injections.
> -
>
> Key: HBASE-14261
> URL: https://issues.apache.org/jira/browse/HBASE-14261
> Project: HBase
> Issue Type: Improvement
>Reporter: Srikanth Srungarapu
>Assignee: Srikanth Srungarapu
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.15, 1.0.3, 1.1.3
>
> Attachments: HBASE-14261-addendum.patch, HBASE-14261-branch-1.patch,
> HBASE-14261-branch-1_v3.patch, HBASE-14261-branch-1_v4.patch,
> HBASE-14261.branch-1_v2.patch, HBASE-14261.patch
>
>
> One of the shortcomings of existing ChaosMonkey framework is lack of fault
> injections for hbase dependencies like zookeeper, hdfs etc. This patch
> attempts to solve this problem partially by adding datanode and zk node fault
> injections.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14412) remove_peer of replication peers with hyphens in name doesn't remove ZK entries
[
https://issues.apache.org/jira/browse/HBASE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741615#comment-14741615
]
Biju Nair commented on HBASE-14412:
---
[~enis] Thanks. Please close the issue.
> remove_peer of replication peers with hyphens in name doesn't remove ZK
> entries
> ---
>
> Key: HBASE-14412
> URL: https://issues.apache.org/jira/browse/HBASE-14412
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Biju Nair
>Priority: Minor
>
> - Created replication peer POB3-HBASE-LL-A. By mistake used hyphens in the
> replication peer name.
> - Did {{remove_peer POB3-HBASE-LL-A}}
> - Created a new peer POB3_HBASE_LL_A
> - Verified zk for entries
> {noformat}
> ls /hbase/replication/peers
> [POB3_HBASE_LL_A]
> ls /hbase/replication/rs
> [r2n14,60200,1441836473749, r1n16,60200,1441836441250,
> r1n14,60200,1441836352689, r3n15,60200,1441836547607,
> r2n15,60200,1441836551992, r3n14,60200,1441836230275,
> r1n15,60200,1441836355145]
> ls /hbase/replication/rs/r2n14,60200,1441836473749
> [POB3_HBASE_LL_A, POB3-HBASE-LL-A]
> ls /hbase/replication/rs/r2n14,60200,1441836473749/POB3-HBASE-LL-A
> [r2n14%2C60200%2C1441836473749.1441865284930,
> r2n14%2C60200%2C1441836473749.1441847284451,
> r2n14%2C60200%2C1441836473749.1441890379203,
> r2n14%2C60200%2C1441836473749.1441890384725,
> r2n14%2C60200%2C1441836473749.1441840084237,
> r2n14%2C60200%2C1441836473749.1441876085231,
> r2n14%2C60200%2C1441836473749.1441890367559,
> r2n14%2C60200%2C1441836473749.1441883285466,
> r2n14%2C60200%2C1441836473749.1441890370829,
> r2n14%2C60200%2C1441836473749.1441879685329,
> r2n14%2C60200%2C1441836473749.1441890382514,
> r2n14%2C60200%2C1441836473749.1441890381514,
> r2n14%2C60200%2C1441836473749.1441890380286,
> r2n14%2C60200%2C1441836473749.1441890378164,
> r2n14%2C60200%2C1441836473749.1441872485132,
> r2n14%2C60200%2C1441836473749.1441868885039,
> r2n14%2C60200%2C1441836473749.1441890374603,
> r2n14%2C60200%2C1441836473749.1441890377072,
> r2n14%2C60200%2C1441836473749.1441858084739,
> r2n14%2C60200%2C1441836473749.1441890369005,
> r2n14%2C60200%2C1441836473749.1441850884549,
> r2n14%2C60200%2C1441836473749.1441886885548,
> r2n14%2C60200%2C1441836473749.1441890372063,
> r2n14%2C60200%2C1441836473749.1441890375866,
> r2n14%2C60200%2C1441836473749.1441890373369,
> r2n14%2C60200%2C1441836473749.1441890383608,
> r2n14%2C60200%2C1441836473749.1441836483297,
> r2n14%2C60200%2C1441836473749.1441843684350,
> r2n14%2C60200%2C1441836473749.1441890366118,
> r2n14%2C60200%2C1441836473749.1441861684834,
> r2n14%2C60200%2C1441836473749.1441854484648]
> {noformat}
> Even though the peer got removed, there are still some ZK entries left which
> need to be cleaned.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14414) HBase Backup/Restore Phase 3
Vladimir Rodionov created HBASE-14414: - Summary: HBase Backup/Restore Phase 3 Key: HBASE-14414 URL: https://issues.apache.org/jira/browse/HBASE-14414 Project: HBase Issue Type: New Feature Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 2.0.0 Umbrella ticket for Phase 3 of development -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14417) Incremental backup and bulk loading
Vladimir Rodionov created HBASE-14417: - Summary: Incremental backup and bulk loading Key: HBASE-14417 URL: https://issues.apache.org/jira/browse/HBASE-14417 Project: HBase Issue Type: New Feature Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 2.0.0 Currently, incremental backup is based on WAL files. Bulk data loading bypasses WALs for obvious reasons, breaking incremental backups. The only way to continue backups after bulk loading is to create new full backup of a table. This may not be feasible for customers who do bulk loading regularly (say, every day). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14397) PrefixFilter fail to filter all remainings if the prefix is longer than compared rowkey
[
https://issues.apache.org/jira/browse/HBASE-14397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-14397:
---
Assignee: cuijianwei
Hadoop Flags: Reviewed
Fix Version/s: 1.3.0
2.0.0
> PrefixFilter fail to filter all remainings if the prefix is longer than
> compared rowkey
> ---
>
> Key: HBASE-14397
> URL: https://issues.apache.org/jira/browse/HBASE-14397
> Project: HBase
> Issue Type: Improvement
> Components: Filters
>Affects Versions: 2.0.0
>Reporter: cuijianwei
>Assignee: cuijianwei
>Priority: Minor
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14397-trunk-v1.patch
>
>
> The PrefixFilter will filter rowkey as:
> {code}
> public boolean filterRowKey(Cell firstRowCell) {
> ...
> int length = firstRowCell.getRowLength();
> if (length < prefix.length) return true; // ===> return directly if the
> prefix is longer
>
> if ((!isReversed() && cmp > 0) || (isReversed() && cmp < 0)) {
> passedPrefix = true;
> }
> filterRow = (cmp != 0);
> return filterRow;
> }
> {code}
> If the prefix is longer than the current rowkey, PrefixFilter#filterRowKey
> will filter the rowkey directly without comparing, so that won't set
> 'passedPrefix' flag even the current row is larger than the prefix.
> For example, if there are three rows 'a', 'b' and 'c' in the table, and we
> issue a scan request as:
> {code}
> hbase(main):001:0> scan 'test_table', {STARTROW => 'a', FILTER =>
> "(PrefixFilter ('aa'))"}
> {code}
> The region server will check the three rows before returning. In our
> production, the user issue a scan with a PrefixFilter. The prefix is longer
> than the rowkeys of following millions of rows, so the region server will
> continue to check rows until hit a rowkey longer than the prefix. This make
> the client easily timeout. To fix this case, it seems we need to compare the
> prefix with the rowkey every serveral rows even when the prefix is longer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14307) Incorrect use of positional read api in HFileBlock
[
https://issues.apache.org/jira/browse/HBASE-14307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-14307:
--
Fix Version/s: (was: 1.2.1)
1.2.0
> Incorrect use of positional read api in HFileBlock
> --
>
> Key: HBASE-14307
> URL: https://issues.apache.org/jira/browse/HBASE-14307
> Project: HBase
> Issue Type: Bug
>Reporter: Shradha Revankar
>Assignee: Chris Nauroth
> Fix For: 2.0.0, 1.2.0, 0.98.15, 1.0.3, 1.1.3
>
> Attachments: 14307-branch-1.addendum, HBASE-14307.001.master.patch,
> HBASE-14307.002.master.patch, HBASE-14307.003.patch
>
>
> Considering that {{read()}} is not guaranteed to read all bytes,
> I'm interested to understand this particular piece of code and why is partial
> read treated as an error :
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java#L1446-L1450
> Particularly, if hbase were to use a different filesystem, say
> WebhdfsFileSystem, this would not work, please also see
> https://issues.apache.org/jira/browse/HDFS-8943 for discussion around this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14416) HBase Backup/Restore Phase 2: Create plugin hooks for Backup tools
Geoffrey Jacoby created HBASE-14416: --- Summary: HBase Backup/Restore Phase 2: Create plugin hooks for Backup tools Key: HBASE-14416 URL: https://issues.apache.org/jira/browse/HBASE-14416 Project: HBase Issue Type: Sub-task Reporter: Geoffrey Jacoby Different organizations may already have their own backup tools for HBase, or wish to develop them in the future for their own particular use cases. It would be useful if HBase supported hooks to integrate those tools so that they could be configured and run in a standard way. In particular, the administrative interface to start a backup, restore, or merge should be decoupled from any particular implementation as much as possible, so that any implementation with similar capabilities can be substituted via configuration without needing to fork or modify the code. Ideally, it will also be possible to decouple the various components so that implementations can be mixed and matched. (For example, one could use the standard backup's functionality to track what needs to be backed up, but use a custom plugin to change the logic or storage format of the backup operation itself.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
