[jira] [Updated] (HBASE-15709) Handle large edits for asynchronous WAL
[ https://issues.apache.org/jira/browse/HBASE-15709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-15709: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master. Thanks all for reviewing. > Handle large edits for asynchronous WAL > --- > > Key: HBASE-15709 > URL: https://issues.apache.org/jira/browse/HBASE-15709 > Project: HBase > Issue Type: Sub-task > Components: io, wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-15709-v1.patch, HBASE-15709-v2.patch, > HBASE-15709.patch > > > First, FanOutOneBlockAsyncDFSOutput can not work if the buffered data is > larger than PacketReceiver.MAX_PACKET_SIZE(16MB). > Second, since we only allow one block here, we need to make sure we do not > exceed the block size after writing a large chunk of data. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[
https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604290#comment-15604290
]
Duo Zhang commented on HBASE-16890:
---
No magic.
{code}
protected synchronized int flushBuffer(boolean keep,
boolean flushPartial) throws IOException {
int bufLen = count;
int partialLen = bufLen % sum.getBytesPerChecksum();
int lenToFlush = flushPartial ? bufLen : bufLen - partialLen;
if (lenToFlush != 0) {
writeChecksumChunks(buf, 0, lenToFlush);
if (!flushPartial || keep) {
count = partialLen;
System.arraycopy(buf, bufLen - count, buf, 0, count);
} else {
count = 0;
}
}
// total bytes left minus unflushed bytes left
return count - (bufLen - lenToFlush);
}
{code}
And in DFSOutputStream.flushOrSync
{code}
// flush checksum buffer, but keep checksum buffer intact if we do not
// need to end the current block
int numKept = flushBuffer(!endBlock, true);
{code}
Just a simple copy...
> Analyze the performance of AsyncWAL and fix the same
>
>
> Key: HBASE-16890
> URL: https://issues.apache.org/jira/browse/HBASE-16890
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: contention.png, contention_defaultWAL.png
>
>
> Tests reveal that AsyncWAL under load in single node cluster performs slower
> than the Default WAL. This task is to analyze and see if we could fix it.
> See some discussions in the tail of JIRA HBASE-15536.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[ https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604276#comment-15604276 ] ramkrishna.s.vasudevan commented on HBASE-16890: Yes the check sum alignment could be a problem because in Dfsclient case i think there is some way to align the data before actually cerating the packet header to be written to DN. Will be back here after some code reading. > Analyze the performance of AsyncWAL and fix the same > > > Key: HBASE-16890 > URL: https://issues.apache.org/jira/browse/HBASE-16890 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: contention.png, contention_defaultWAL.png > > > Tests reveal that AsyncWAL under load in single node cluster performs slower > than the Default WAL. This task is to analyze and see if we could fix it. > See some discussions in the tail of JIRA HBASE-15536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15709) Handle large edits for asynchronous WAL
[
https://issues.apache.org/jira/browse/HBASE-15709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604266#comment-15604266
]
Hadoop QA commented on HBASE-15709:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 49s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 22s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 2s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 125m 53s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.master.procedure.TestEnableTableProcedure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12835062/HBASE-15709-v2.patch |
| JIRA Issue | HBASE-15709 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 34550fa2e1cb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 4533bb6 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4174/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4174/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4174/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4174/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4174/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This
[jira] [Commented] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[ https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604261#comment-15604261 ] Duo Zhang commented on HBASE-16890: --- The only possible way that AsyncFSWAL does more work on checksuming is that, it syncs more than FSHLog. The packets sent to datanode need to be aligned with checksum boundary. So if the data is not aligned, we will copy the unaligned data to the beginning of the buffer and its checksum will be calucated again next time. So if AsyncFSWAL syncs more, it is more likely that the checksum of the same piece of data will also be calculated more times. > Analyze the performance of AsyncWAL and fix the same > > > Key: HBASE-16890 > URL: https://issues.apache.org/jira/browse/HBASE-16890 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: contention.png, contention_defaultWAL.png > > > Tests reveal that AsyncWAL under load in single node cluster performs slower > than the Default WAL. This task is to analyze and see if we could fix it. > See some discussions in the tail of JIRA HBASE-15536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[ https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604235#comment-15604235 ] ramkrishna.s.vasudevan commented on HBASE-16890: bq. We used to do bulk of checksum calculation at append time rather than inline w/ the sync call. That might make a difference. I think so this is the case. I have installed the hadoop code to read it. Thanks Stack for the help here. I think I can also share my inputs here. > Analyze the performance of AsyncWAL and fix the same > > > Key: HBASE-16890 > URL: https://issues.apache.org/jira/browse/HBASE-16890 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: contention.png, contention_defaultWAL.png > > > Tests reveal that AsyncWAL under load in single node cluster performs slower > than the Default WAL. This task is to analyze and see if we could fix it. > See some discussions in the tail of JIRA HBASE-15536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[ https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604232#comment-15604232 ] Duo Zhang commented on HBASE-16890: --- There is no contention for the synchronization in DFSClient#DFSOS for our FSHLog. We always call it in the consumer thread of Disruptor... The payload queue in AsyncFSWAL acts as the ringbuffer in FSHLog. And I still do not think calculate checksum when syncing cause any difference. For both FSHLog and AsyncFSWAL, the append and sync are executed in one thread, no concurrency at all. The FSHLog.append only puts the WALEntry into the ringbuffer, not appends it to DFSClient#DFSOS. It does not make sense that calcuate the checksum of a big piece of data chunk by chunk could be faster? The calculated amount does not change I think? Thanks. > Analyze the performance of AsyncWAL and fix the same > > > Key: HBASE-16890 > URL: https://issues.apache.org/jira/browse/HBASE-16890 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: contention.png, contention_defaultWAL.png > > > Tests reveal that AsyncWAL under load in single node cluster performs slower > than the Default WAL. This task is to analyze and see if we could fix it. > See some discussions in the tail of JIRA HBASE-15536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15789) PB related changes to work with offheap
[ https://issues.apache.org/jira/browse/HBASE-15789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604208#comment-15604208 ] Anoop Sam John commented on HBASE-15789: Thanks Stack. > PB related changes to work with offheap > --- > > Key: HBASE-15789 > URL: https://issues.apache.org/jira/browse/HBASE-15789 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-15789.master.001.patch, HBASE-15789.patch, > HBASE-15789_V2.patch > > > This is an issue to brainstorm. Whether we go with pb 2.x or pb 3.0 and also > depends on the shading of protobuf classes. > We should also decide if we are going to fork the PB classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16880) Correct the javadoc/behaviour of the APIs in ByteBufferUtils
[
https://issues.apache.org/jira/browse/HBASE-16880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604207#comment-15604207
]
Anoop Sam John commented on HBASE-16880:
+1
> Correct the javadoc/behaviour of the APIs in ByteBufferUtils
>
>
> Key: HBASE-16880
> URL: https://issues.apache.org/jira/browse/HBASE-16880
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-16880.patch
>
>
> There are some issues either with the javadoc or the actual behaviour of some
> APIs in BBUtils.
> For eg,
> BBUtil#copyFromBufferToBuffer() says
> {code}
> /**
>* Copy one buffer's whole data to another. Write starts at the current
> position of 'out' buffer.
>* Note : This will advance the position marker of {@code out} but not
> change the position maker
>* for {@code in}. The position and limit of the {@code in} buffer to be
> set properly by caller.
>* @param in source buffer
>* @param out destination buffer
> {code}
> But this is true in case of UNSAFE.
> {code}
> if (UNSAFE_AVAIL) {
> int length = in.remaining();
> UnsafeAccess.copy(in, in.position(), out, out.position(), length);
> out.position(out.position() + length);
> } else {
> out.put(in);
> }
> {code}
> But in other case where we move the else - the 'in' is also advanced. So we
> need to either correct the behaviour or change the doc and see all the used
> places. This JIRA can be used to correct all the APIs in this util class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[ https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604204#comment-15604204 ] Duo Zhang commented on HBASE-16890: --- But I think we should be able to load the libhadoop.so when testing performance? > Analyze the performance of AsyncWAL and fix the same > > > Key: HBASE-16890 > URL: https://issues.apache.org/jira/browse/HBASE-16890 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: contention.png, contention_defaultWAL.png > > > Tests reveal that AsyncWAL under load in single node cluster performs slower > than the Default WAL. This task is to analyze and see if we could fix it. > See some discussions in the tail of JIRA HBASE-15536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16783) Use ByteBufferPool for the header and message during Rpc response
[
https://issues.apache.org/jira/browse/HBASE-16783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604202#comment-15604202
]
Anoop Sam John edited comment on HBASE-16783 at 10/25/16 5:02 AM:
--
+1.
Some minor comments. Can fix on commit
bq.ByteBuffer possiblePBBuf = (cellBlockSize > 0) ?
cellBlock.get(cellBlock.size() - 1) : null;
Ya some comments above this line what we are trying to do/optimize here. This
is irrsepective of the BBPool. We made the cellblock any way and of there is
space left in the buffer of that, make use of that rather than creating a temp
one.
bq.possiblePBBuf.remaining() + totalPBSize <= possiblePBBuf.capacity()
Better to do possiblePBBuf.limit() + totalPBSize <= possiblePBBuf.capacity().
No issue for above also as the pos will be always zero. But this will be better
in generic way. Even down we do the similar only
bq. pbBuf.limit(totalPBSize + limit);
{code}
if (possiblePBBuf != null) {
542 // Only if the last buffer has enough space for header use it.
Else allocate
543 // a new buffer. Assume they are all flipped
544 if (possiblePBBuf.remaining() + totalPBSize <=
possiblePBBuf.capacity()) {
{code}
Can we have both checks in single if? Wont be looking so complex also. Then u
can avoid this duplicated else blocks
{code}
} else {
560 return createHeaderAndMessageBytes(result, header, totalSize,
totalPBSize);
561 }
562 } else {
563 return createHeaderAndMessageBytes(result, header, totalSize,
totalPBSize);
564 }
{code}
was (Author: anoop.hbase):
+1.
Some minor comments. Can fix on commit
bq.ByteBuffer possiblePBBuf = (cellBlockSize > 0) ?
cellBlock.get(cellBlock.size() - 1) : null;
Ya some comments above this line what we are trying to do/optimize here. This
is irrsepective of the BBPool. We made the cellblock any way and of there is
space left in the buffer of that, make use of that rather than creating a temp
one.
bq.possiblePBBuf.remaining() + totalPBSize <= possiblePBBuf.capacity()
Better to do possiblePBBuf.limit() + totalPBSize <= possiblePBBuf.capacity().
No issue for above also as the pos will be always zero. But this will be better
in generic way. Even down we do the similar only
bq. pbBuf.limit(totalPBSize + limit);
{quote}
if (possiblePBBuf != null) {
542 // Only if the last buffer has enough space for header use it.
Else allocate
543 // a new buffer. Assume they are all flipped
544 if (possiblePBBuf.remaining() + totalPBSize <=
possiblePBBuf.capacity()) {
{quote}
Can we have both checks in single if? Wont be looking so complex also. Then u
can avoid this duplicated else blocks
{quote}
} else {
560 return createHeaderAndMessageBytes(result, header, totalSize,
totalPBSize);
561 }
562 } else {
563 return createHeaderAndMessageBytes(result, header, totalSize,
totalPBSize);
564 }
{quote}
> Use ByteBufferPool for the header and message during Rpc response
> -
>
> Key: HBASE-16783
> URL: https://issues.apache.org/jira/browse/HBASE-16783
> Project: HBase
> Issue Type: Improvement
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
> Attachments: HBASE-16783.patch, HBASE-16783_1.patch,
> HBASE-16783_2.patch, HBASE-16783_3.patch, HBASE-16783_4.patch,
> HBASE-16783_5.patch, HBASE-16783_6.patch
>
>
> With ByteBufferPool in place we could avoid the byte[] creation in
> RpcServer#createHeaderAndMessageBytes and try using the Buffer from the pool
> rather than creating byte[] every time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16783) Use ByteBufferPool for the header and message during Rpc response
[
https://issues.apache.org/jira/browse/HBASE-16783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604202#comment-15604202
]
Anoop Sam John commented on HBASE-16783:
+1.
Some minor comments. Can fix on commit
bq.ByteBuffer possiblePBBuf = (cellBlockSize > 0) ?
cellBlock.get(cellBlock.size() - 1) : null;
Ya some comments above this line what we are trying to do/optimize here. This
is irrsepective of the BBPool. We made the cellblock any way and of there is
space left in the buffer of that, make use of that rather than creating a temp
one.
bq.possiblePBBuf.remaining() + totalPBSize <= possiblePBBuf.capacity()
Better to do possiblePBBuf.limit() + totalPBSize <= possiblePBBuf.capacity().
No issue for above also as the pos will be always zero. But this will be better
in generic way. Even down we do the similar only
bq. pbBuf.limit(totalPBSize + limit);
{quote}
if (possiblePBBuf != null) {
542 // Only if the last buffer has enough space for header use it.
Else allocate
543 // a new buffer. Assume they are all flipped
544 if (possiblePBBuf.remaining() + totalPBSize <=
possiblePBBuf.capacity()) {
{quote}
Can we have both checks in single if? Wont be looking so complex also. Then u
can avoid this duplicated else blocks
{quote}
} else {
560 return createHeaderAndMessageBytes(result, header, totalSize,
totalPBSize);
561 }
562 } else {
563 return createHeaderAndMessageBytes(result, header, totalSize,
totalPBSize);
564 }
{quote}
> Use ByteBufferPool for the header and message during Rpc response
> -
>
> Key: HBASE-16783
> URL: https://issues.apache.org/jira/browse/HBASE-16783
> Project: HBase
> Issue Type: Improvement
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
> Attachments: HBASE-16783.patch, HBASE-16783_1.patch,
> HBASE-16783_2.patch, HBASE-16783_3.patch, HBASE-16783_4.patch,
> HBASE-16783_5.patch, HBASE-16783_6.patch
>
>
> With ByteBufferPool in place we could avoid the byte[] creation in
> RpcServer#createHeaderAndMessageBytes and try using the Buffer from the pool
> rather than creating byte[] every time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16891) Try copying to the Netty ByteBuf directly from the WALEdit
[
https://issues.apache.org/jira/browse/HBASE-16891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604201#comment-15604201
]
ramkrishna.s.vasudevan commented on HBASE-16891:
The hack that I tried here. Just pasting the snippet. I am already working on a
full fledged patch\
In AsyncProtobufWriter#append
{code}
length.addAndGet(buf.size());
output.write(buf.getBuffer(), 0, buf.size());
buf.reset();
for (Cell cell : entry.getEdit().getCells()) {
length.addAndGet(((KeyValue)cell).getLength());
output.write(((KeyValue)cell).getBuffer(),
((KeyValue)cell).getOffset(), ((KeyValue)cell).getLength());
}
{code}
And inside the Asyncoutput write I tried writing the length and the data
together at one shot.
Something like this
{code}
buf.ensureWritable(len + Bytes.SIZEOF_INT);
if (cryptoCodec == null) {
buf.writeInt(len);
buf.writeBytes(b, off, len);
} else {
ByteBuffer inBuffer = ByteBuffer.wrap(b, off, len);
cryptoCodec.encrypt(inBuffer, buf.nioBuffer(buf.writerIndex(), len));
buf.writerIndex(buf.writerIndex() + len);
}
{code}
> Try copying to the Netty ByteBuf directly from the WALEdit
> --
>
> Key: HBASE-16891
> URL: https://issues.apache.org/jira/browse/HBASE-16891
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
>
> -> The FanOutOneBlockAsyncDFSOutput is a much sophisticated dfs client model
> that works with Netty ByteBuf. Here we hold on connection to the datanodes
> using
> Netty Channels. And the idea is to write data direclty to these channels.
> AsyncHLog gets an append call. The AysncWAL uses the HBase's
> ByteArrayOutputSTream and so the content of the cell is written to this BAOS
> and that is again
> copied to the netty Bytebuf in the FanOutOneBlockAsyncDFSOutput.
> So when the sync call happens this FanoutDFSoutput does the checksum
> calcualtion itself and then writes the content of this buffer direclty to the
> DN channel.
> -> In case of FSHLOg this is different. When an append call comes we direclty
> write the content to the FSDataOutputStream (it is copied to this stream).
> Then here internally there is a checkSum calculation that happens. when a
> sync call happens there is noth ing to do except to notify the NN to flush
> the latest
> data.
> AS we can see from the above that there are two copies in AsyncWAL
> -> From the Cell to the BAOS
> -> From the BAOS to the Netty byte buf
> -> On sync() call, do check sum and finally flush the netty byte buf to the
> DN channel
> In case of FSHLog
> -> From cell to the FSDataoutputstream. data is copied. Check sum happens
> here.
> -> Sync call just tries to notify the NN.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[
https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604192#comment-15604192
]
ramkrishna.s.vasudevan commented on HBASE-16890:
I just verified. So it does go to this piece of code
{code}
data.mark();
checksums.mark();
try {
byte[] buf = new byte[bytesPerChecksum];
while (data.remaining() > 0) {
int n = Math.min(data.remaining(), bytesPerChecksum);
data.get(buf, 0, n);
summer.reset();
summer.update(buf, 0, n);
checksums.putInt((int)summer.getValue());
}
} finally {
data.reset();
checksums.reset();
}
{code}
So may be I don't have the native library in my system.
> Analyze the performance of AsyncWAL and fix the same
>
>
> Key: HBASE-16890
> URL: https://issues.apache.org/jira/browse/HBASE-16890
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: contention.png, contention_defaultWAL.png
>
>
> Tests reveal that AsyncWAL under load in single node cluster performs slower
> than the Default WAL. This task is to analyze and see if we could fix it.
> See some discussions in the tail of JIRA HBASE-15536.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16783) Use ByteBufferPool for the header and message during Rpc response
[
https://issues.apache.org/jira/browse/HBASE-16783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604166#comment-15604166
]
ramkrishna.s.vasudevan commented on HBASE-16783:
bq.(cellBlockSize > 0) ? cellBlock.get(cellBlock.size() - 1) : null;
If I don't pass this here then i need to pass the cellBlock and do the above
check inside createHeaderAndMessageBytes.
bq.if (possiblePBBuf != null) {
Sure will add comment. Thanks Stack for the review.
> Use ByteBufferPool for the header and message during Rpc response
> -
>
> Key: HBASE-16783
> URL: https://issues.apache.org/jira/browse/HBASE-16783
> Project: HBase
> Issue Type: Improvement
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
> Attachments: HBASE-16783.patch, HBASE-16783_1.patch,
> HBASE-16783_2.patch, HBASE-16783_3.patch, HBASE-16783_4.patch,
> HBASE-16783_5.patch, HBASE-16783_6.patch
>
>
> With ByteBufferPool in place we could avoid the byte[] creation in
> RpcServer#createHeaderAndMessageBytes and try using the Buffer from the pool
> rather than creating byte[] every time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16463) Improve transparent table/CF encryption with Commons Crypto
[ https://issues.apache.org/jira/browse/HBASE-16463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604163#comment-15604163 ] ramkrishna.s.vasudevan commented on HBASE-16463: Thank you [~apurtell] and thanks [~dapengsun]. > Improve transparent table/CF encryption with Commons Crypto > --- > > Key: HBASE-16463 > URL: https://issues.apache.org/jira/browse/HBASE-16463 > Project: HBase > Issue Type: New Feature > Components: encryption >Affects Versions: 2.0.0 >Reporter: Dapeng Sun >Assignee: Dapeng Sun > Fix For: 2.0.0 > > Attachments: HBASE-16463.001.patch, HBASE-16463.002.patch, > HBASE-16463.003.patch, HBASE-16463.004.patch, HBASE-16463.004.patch, > HBASE-16463.005.patch > > > Apache Commons Crypto > (https://commons.apache.org/proper/commons-crypto/index.html) is a > cryptographic library optimized with AES-NI. > HBASE-7544 introduces a framework for transparent encryption feature for > protecting HFile and WAL data at rest. Currently JCE cipher is used bu > default, the improvement will use Commons Crypto to accelerate the > transparent encryption of HBase. new crypto provider with Commons CRYPTO will > be provided for Transparent encryption. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16880) Correct the javadoc/behaviour of the APIs in ByteBufferUtils
[
https://issues.apache.org/jira/browse/HBASE-16880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604159#comment-15604159
]
ramkrishna.s.vasudevan commented on HBASE-16880:
Can this be committed? Ping [~anoop.hbase]
> Correct the javadoc/behaviour of the APIs in ByteBufferUtils
>
>
> Key: HBASE-16880
> URL: https://issues.apache.org/jira/browse/HBASE-16880
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-16880.patch
>
>
> There are some issues either with the javadoc or the actual behaviour of some
> APIs in BBUtils.
> For eg,
> BBUtil#copyFromBufferToBuffer() says
> {code}
> /**
>* Copy one buffer's whole data to another. Write starts at the current
> position of 'out' buffer.
>* Note : This will advance the position marker of {@code out} but not
> change the position maker
>* for {@code in}. The position and limit of the {@code in} buffer to be
> set properly by caller.
>* @param in source buffer
>* @param out destination buffer
> {code}
> But this is true in case of UNSAFE.
> {code}
> if (UNSAFE_AVAIL) {
> int length = in.remaining();
> UnsafeAccess.copy(in, in.position(), out, out.position(), length);
> out.position(out.position() + length);
> } else {
> out.put(in);
> }
> {code}
> But in other case where we move the else - the 'in' is also advanced. So we
> need to either correct the behaviour or change the doc and see all the used
> places. This JIRA can be used to correct all the APIs in this util class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[
https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604156#comment-15604156
]
ramkrishna.s.vasudevan commented on HBASE-16890:
{code}
if (NativeCrc32.isAvailable()) {
NativeCrc32.calculateChunkedSumsByteArray(bytesPerChecksum, type.id,
sums, sumsOffset, data, dataOffset, dataLength);
return;
}
{code}
I see. I thought this is a NativeCrc32 that needs to be available in my system.
And I was not sure if that was working for me. I can verify that once. But if
that works then DBB is not an issue. Thanks @Duo.
> Analyze the performance of AsyncWAL and fix the same
>
>
> Key: HBASE-16890
> URL: https://issues.apache.org/jira/browse/HBASE-16890
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: contention.png, contention_defaultWAL.png
>
>
> Tests reveal that AsyncWAL under load in single node cluster performs slower
> than the Default WAL. This task is to analyze and see if we could fix it.
> See some discussions in the tail of JIRA HBASE-15536.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16938) TableCFsUpdater maybe failed due to no write permission on peerNode
[
https://issues.apache.org/jira/browse/HBASE-16938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604136#comment-15604136
]
Hadoop QA commented on HBASE-16938:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 40s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 8s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 95m 29s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 134m 59s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.mapred.TestTableInputFormat |
| | org.apache.hadoop.hbase.filter.TestFilterWrapper |
| | org.apache.hadoop.hbase.TestNamespace |
| |
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes |
| | org.apache.hadoop.hbase.filter.TestMultiRowRangeFilter |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12835055/HBASE-16938.patch |
| JIRA Issue | HBASE-16938 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux a3bfd7642b86 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 4533bb6 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4171/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4171/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
[jira] [Commented] (HBASE-16886) hbase-client: scanner with reversed=true and small=true gets no result
[
https://issues.apache.org/jira/browse/HBASE-16886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604134#comment-15604134
]
Hadoop QA commented on HBASE-16886:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 38s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 4s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 2s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 87m 20s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 132m 2s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.client.TestFromClientSide |
| | org.apache.hadoop.hbase.client.TestHCM |
| | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient |
| | org.apache.hadoop.hbase.client.TestMobSnapshotCloneIndependence |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12834903/HBASE-16886.v2.patch |
| JIRA Issue | HBASE-16886 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 74c4d26241bf 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 4533bb6 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| findbugs |
[jira] [Commented] (HBASE-15789) PB related changes to work with offheap
[ https://issues.apache.org/jira/browse/HBASE-15789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604113#comment-15604113 ] Hudson commented on HBASE-15789: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #1849 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1849/]) Revert "Revert "HBASE-15789 PB related changes to work with offheap"" (stack: rev 4533bb63cf24b13ba3bdb00cb2aa65f1e42cf1a8) * (add) hbase-protocol-shaded/src/main/patches/HBASE-15789_V2.patch * (edit) hbase-server/pom.xml * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/com/google/protobuf/Utf8.java * (edit) hbase-protocol-shaded/README.txt * (add) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/com/google/protobuf/ByteInputByteString.java * (edit) hbase-assembly/pom.xml * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/com/google/protobuf/CodedInputStream.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/com/google/protobuf/ByteString.java * (add) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/com/google/protobuf/ByteInput.java * (edit) hbase-protocol/pom.xml * (edit) hbase-protocol-shaded/pom.xml * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/com/google/protobuf/ByteBufferWriter.java > PB related changes to work with offheap > --- > > Key: HBASE-15789 > URL: https://issues.apache.org/jira/browse/HBASE-15789 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-15789.master.001.patch, HBASE-15789.patch, > HBASE-15789_V2.patch > > > This is an issue to brainstorm. Whether we go with pb 2.x or pb 3.0 and also > depends on the shading of protobuf classes. > We should also decide if we are going to fork the PB classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16939) ExportSnapshot: set owner and permission on right directory
[
https://issues.apache.org/jira/browse/HBASE-16939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604095#comment-15604095
]
Guanghao Zhang commented on HBASE-16939:
Checked with branch-1, branch-1.3, branch-1.2, branch-1.1 and passed. But apply
to 0.98 brnach has conflict, I will upload a new one for 0.98.
> ExportSnapshot: set owner and permission on right directory
> ---
>
> Key: HBASE-16939
> URL: https://issues.apache.org/jira/browse/HBASE-16939
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Attachments: HBASE-16939-v1.patch, HBASE-16939.patch
>
>
> {code}
> FileUtil.copy(inputFs, snapshotDir, outputFs, initialOutputSnapshotDir,
> false, false, conf);
> if (filesUser != null || filesGroup != null) {
> setOwner(outputFs, snapshotTmpDir, filesUser, filesGroup, true);
> }
> if (filesMode > 0) {
> setPermission(outputFs, snapshotTmpDir, (short)filesMode, true);
> }
> {code}
> It copy snapshot manifest to initialOutputSnapshotDir, but it set owner on
> snapshotTmpDir. They are different directory when skipTmp is true.
> Another problem is new cluster doesn't have .hbase-snapshot directory. So
> after export snapshot, it should set owner on .hbase-snapshot directory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16939) ExportSnapshot: set owner and permission on right directory
[
https://issues.apache.org/jira/browse/HBASE-16939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604077#comment-15604077
]
Matteo Bertozzi commented on HBASE-16939:
-
+1. I guess we can apply the patch to every active branch, right?
> ExportSnapshot: set owner and permission on right directory
> ---
>
> Key: HBASE-16939
> URL: https://issues.apache.org/jira/browse/HBASE-16939
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Attachments: HBASE-16939-v1.patch, HBASE-16939.patch
>
>
> {code}
> FileUtil.copy(inputFs, snapshotDir, outputFs, initialOutputSnapshotDir,
> false, false, conf);
> if (filesUser != null || filesGroup != null) {
> setOwner(outputFs, snapshotTmpDir, filesUser, filesGroup, true);
> }
> if (filesMode > 0) {
> setPermission(outputFs, snapshotTmpDir, (short)filesMode, true);
> }
> {code}
> It copy snapshot manifest to initialOutputSnapshotDir, but it set owner on
> snapshotTmpDir. They are different directory when skipTmp is true.
> Another problem is new cluster doesn't have .hbase-snapshot directory. So
> after export snapshot, it should set owner on .hbase-snapshot directory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16939) ExportSnapshot: set owner and permission on right directory
[
https://issues.apache.org/jira/browse/HBASE-16939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-16939:
---
Attachment: HBASE-16939-v1.patch
Add log when change owner or group.
> ExportSnapshot: set owner and permission on right directory
> ---
>
> Key: HBASE-16939
> URL: https://issues.apache.org/jira/browse/HBASE-16939
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Attachments: HBASE-16939-v1.patch, HBASE-16939.patch
>
>
> {code}
> FileUtil.copy(inputFs, snapshotDir, outputFs, initialOutputSnapshotDir,
> false, false, conf);
> if (filesUser != null || filesGroup != null) {
> setOwner(outputFs, snapshotTmpDir, filesUser, filesGroup, true);
> }
> if (filesMode > 0) {
> setPermission(outputFs, snapshotTmpDir, (short)filesMode, true);
> }
> {code}
> It copy snapshot manifest to initialOutputSnapshotDir, but it set owner on
> snapshotTmpDir. They are different directory when skipTmp is true.
> Another problem is new cluster doesn't have .hbase-snapshot directory. So
> after export snapshot, it should set owner on .hbase-snapshot directory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16838) Implement scan
[
https://issues.apache.org/jira/browse/HBASE-16838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604061#comment-15604061
]
Duo Zhang commented on HBASE-16838:
---
{quote}
You mean a whole region's worth of results or you mean the batch?
{quote}
I mean the whole result of a scan, not the data of the whole region...
And the observer API is borrowed from grpc, but It is not suitable for some
scenarios, for example, MR. And it calls the onNext method directly in the
event loop thread so the method can not be blocked. And if users use another
thread to process the result and return immediately from the onNext method,
then it may cause OOM as we do not have flow control. A flow controled scan
could be built on top of the observe API, but to be honest, I do not have a
good API to expose the 'flow controlled' scan out yet...
Thanks.
> Implement scan
> --
>
> Key: HBASE-16838
> URL: https://issues.apache.org/jira/browse/HBASE-16838
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-16838.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15709) Handle large edits for asynchronous WAL
[ https://issues.apache.org/jira/browse/HBASE-15709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-15709: -- Attachment: HBASE-15709-v2.patch > Handle large edits for asynchronous WAL > --- > > Key: HBASE-15709 > URL: https://issues.apache.org/jira/browse/HBASE-15709 > Project: HBase > Issue Type: Sub-task > Components: io, wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-15709-v1.patch, HBASE-15709-v2.patch, > HBASE-15709.patch > > > First, FanOutOneBlockAsyncDFSOutput can not work if the buffered data is > larger than PacketReceiver.MAX_PACKET_SIZE(16MB). > Second, since we only allow one block here, we need to make sure we do not > exceed the block size after writing a large chunk of data. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16939) ExportSnapshot: set owner and permission on right directory
[
https://issues.apache.org/jira/browse/HBASE-16939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604011#comment-15604011
]
Matteo Bertozzi commented on HBASE-16939:
-
can you add a LOG.warn() or LOG.info() informing that we are creating and
setting permission on the .hbase-snapshot dir? I'd like to see in the log that
export snapshot created that dir.
other than that little addition, patch looks good to me.
> ExportSnapshot: set owner and permission on right directory
> ---
>
> Key: HBASE-16939
> URL: https://issues.apache.org/jira/browse/HBASE-16939
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Attachments: HBASE-16939.patch
>
>
> {code}
> FileUtil.copy(inputFs, snapshotDir, outputFs, initialOutputSnapshotDir,
> false, false, conf);
> if (filesUser != null || filesGroup != null) {
> setOwner(outputFs, snapshotTmpDir, filesUser, filesGroup, true);
> }
> if (filesMode > 0) {
> setPermission(outputFs, snapshotTmpDir, (short)filesMode, true);
> }
> {code}
> It copy snapshot manifest to initialOutputSnapshotDir, but it set owner on
> snapshotTmpDir. They are different directory when skipTmp is true.
> Another problem is new cluster doesn't have .hbase-snapshot directory. So
> after export snapshot, it should set owner on .hbase-snapshot directory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16939) ExportSnapshot: set owner and permission on right directory
[
https://issues.apache.org/jira/browse/HBASE-16939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603998#comment-15603998
]
Guanghao Zhang commented on HBASE-16939:
Yeah. In our use case, we use different kerberos for online serve cluster and
offline processing cluster. And we use a unify global admin kerberos to manage
cluster. Then when we export snapshot from online cluster to offline cluster,
we use global admin user and change directory owner to offline cluster user.
> ExportSnapshot: set owner and permission on right directory
> ---
>
> Key: HBASE-16939
> URL: https://issues.apache.org/jira/browse/HBASE-16939
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Attachments: HBASE-16939.patch
>
>
> {code}
> FileUtil.copy(inputFs, snapshotDir, outputFs, initialOutputSnapshotDir,
> false, false, conf);
> if (filesUser != null || filesGroup != null) {
> setOwner(outputFs, snapshotTmpDir, filesUser, filesGroup, true);
> }
> if (filesMode > 0) {
> setPermission(outputFs, snapshotTmpDir, (short)filesMode, true);
> }
> {code}
> It copy snapshot manifest to initialOutputSnapshotDir, but it set owner on
> snapshotTmpDir. They are different directory when skipTmp is true.
> Another problem is new cluster doesn't have .hbase-snapshot directory. So
> after export snapshot, it should set owner on .hbase-snapshot directory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15347) Update CHANGES.txt for 1.3
[ https://issues.apache.org/jira/browse/HBASE-15347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-15347: Attachment: HBASE-15347-v2.branch-1.3.patch updated CHANGES.txt for 1.3 > Update CHANGES.txt for 1.3 > -- > > Key: HBASE-15347 > URL: https://issues.apache.org/jira/browse/HBASE-15347 > Project: HBase > Issue Type: Sub-task > Components: documentation >Affects Versions: 1.3.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov > Fix For: 1.3.0 > > Attachments: HBASE-15347-branch-1.3.v1.patch, > HBASE-15347-v2.branch-1.3.patch > > > Going to post the steps in preparing changes file for 1.3 here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[
https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603994#comment-15603994
]
Duo Zhang commented on HBASE-16890:
---
{quote}
Now the amount of data to be synced is much bigger and I feel the CPU spends
more time there.
{quote}
But the total data does not increase right? So the total time we spends on
calucating checksum will not increase either? Or increase? Why?
> Analyze the performance of AsyncWAL and fix the same
>
>
> Key: HBASE-16890
> URL: https://issues.apache.org/jira/browse/HBASE-16890
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: contention.png, contention_defaultWAL.png
>
>
> Tests reveal that AsyncWAL under load in single node cluster performs slower
> than the Default WAL. This task is to analyze and see if we could fix it.
> See some discussions in the tail of JIRA HBASE-15536.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[
https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603988#comment-15603988
]
Duo Zhang commented on HBASE-16890:
---
{quote}
Next is to use onheap buffers instead of DBB because the checkSum calculation
API does not deal with DBB directly.
{quote}
Really? I think the native part is used to deal with DBB?
{code}
if (data.hasArray() && checksums.hasArray()) {
calculateChunkedSums(data.array(), data.arrayOffset() + data.position(),
data.remaining(),
checksums.array(), checksums.arrayOffset() + checksums.position());
return;
}
if (NativeCrc32.isAvailable()) {
NativeCrc32.calculateChunkedSums(bytesPerChecksum, type.id,
checksums, data);
return;
}
{code}
> Analyze the performance of AsyncWAL and fix the same
>
>
> Key: HBASE-16890
> URL: https://issues.apache.org/jira/browse/HBASE-16890
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: contention.png, contention_defaultWAL.png
>
>
> Tests reveal that AsyncWAL under load in single node cluster performs slower
> than the Default WAL. This task is to analyze and see if we could fix it.
> See some discussions in the tail of JIRA HBASE-15536.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[ https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603977#comment-15603977 ] ramkrishna.s.vasudevan commented on HBASE-16890: bq.Which means that when the amount of data to be checksummed is more in async wal case we do spend lot of CPU there. Yes [~Apache9]. See my previous comment. It is sync that is heavy now. Previously the amount of data to be synced was less in size and once the required size was reached we handed it over to the dfs client to queue those synced data to form the packet header. Am still to read this part of code but i just got inputs from HDFS guys here. Now the amount of data to be synced is much bigger and I feel the CPU spends more time there. The append synchronization cost I think is something we cannot avoid because that is where we sync the append/sync operations and the consumer uses that lock to actual do the consumption logic. [~saint.ack[~saint@gmail.com] I have few patches to directly copy to Asyncoutput (it is hacky for now). Next is to use onheap buffers instead of DBB because the checkSum calculation API does not deal with DBB directly. But before that let me finish the hadoop code and the logic here to actually calculate the checksum. > Analyze the performance of AsyncWAL and fix the same > > > Key: HBASE-16890 > URL: https://issues.apache.org/jira/browse/HBASE-16890 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: contention.png, contention_defaultWAL.png > > > Tests reveal that AsyncWAL under load in single node cluster performs slower > than the Default WAL. This task is to analyze and see if we could fix it. > See some discussions in the tail of JIRA HBASE-15536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16939) ExportSnapshot: set owner and permission on right directory
[
https://issues.apache.org/jira/browse/HBASE-16939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603974#comment-15603974
]
Matteo Bertozzi commented on HBASE-16939:
-
if the .hbase-snapshot dir does not exist, and we are exporting to another
cluster will the cluster be able to take snapshots since the dir permission is
set to the export user and not to the hbase user?
> ExportSnapshot: set owner and permission on right directory
> ---
>
> Key: HBASE-16939
> URL: https://issues.apache.org/jira/browse/HBASE-16939
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Attachments: HBASE-16939.patch
>
>
> {code}
> FileUtil.copy(inputFs, snapshotDir, outputFs, initialOutputSnapshotDir,
> false, false, conf);
> if (filesUser != null || filesGroup != null) {
> setOwner(outputFs, snapshotTmpDir, filesUser, filesGroup, true);
> }
> if (filesMode > 0) {
> setPermission(outputFs, snapshotTmpDir, (short)filesMode, true);
> }
> {code}
> It copy snapshot manifest to initialOutputSnapshotDir, but it set owner on
> snapshotTmpDir. They are different directory when skipTmp is true.
> Another problem is new cluster doesn't have .hbase-snapshot directory. So
> after export snapshot, it should set owner on .hbase-snapshot directory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16940) Review 52748: Miscellaneous
Vladimir Rodionov created HBASE-16940: - Summary: Review 52748: Miscellaneous Key: HBASE-16940 URL: https://issues.apache.org/jira/browse/HBASE-16940 Project: HBase Issue Type: Sub-task Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Review 52748 remaining issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16939) ExportSnapshot: set owner and permission on right directory
[
https://issues.apache.org/jira/browse/HBASE-16939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-16939:
---
Assignee: Guanghao Zhang
Status: Patch Available (was: Open)
> ExportSnapshot: set owner and permission on right directory
> ---
>
> Key: HBASE-16939
> URL: https://issues.apache.org/jira/browse/HBASE-16939
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Attachments: HBASE-16939.patch
>
>
> {code}
> FileUtil.copy(inputFs, snapshotDir, outputFs, initialOutputSnapshotDir,
> false, false, conf);
> if (filesUser != null || filesGroup != null) {
> setOwner(outputFs, snapshotTmpDir, filesUser, filesGroup, true);
> }
> if (filesMode > 0) {
> setPermission(outputFs, snapshotTmpDir, (short)filesMode, true);
> }
> {code}
> It copy snapshot manifest to initialOutputSnapshotDir, but it set owner on
> snapshotTmpDir. They are different directory when skipTmp is true.
> Another problem is new cluster doesn't have .hbase-snapshot directory. So
> after export snapshot, it should set owner on .hbase-snapshot directory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16939) ExportSnapshot: set owner and permission on right directory
[
https://issues.apache.org/jira/browse/HBASE-16939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-16939:
---
Attachment: HBASE-16939.patch
> ExportSnapshot: set owner and permission on right directory
> ---
>
> Key: HBASE-16939
> URL: https://issues.apache.org/jira/browse/HBASE-16939
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Priority: Minor
> Attachments: HBASE-16939.patch
>
>
> {code}
> FileUtil.copy(inputFs, snapshotDir, outputFs, initialOutputSnapshotDir,
> false, false, conf);
> if (filesUser != null || filesGroup != null) {
> setOwner(outputFs, snapshotTmpDir, filesUser, filesGroup, true);
> }
> if (filesMode > 0) {
> setPermission(outputFs, snapshotTmpDir, (short)filesMode, true);
> }
> {code}
> It copy snapshot manifest to initialOutputSnapshotDir, but it set owner on
> snapshotTmpDir. They are different directory when skipTmp is true.
> Another problem is new cluster doesn't have .hbase-snapshot directory. So
> after export snapshot, it should set owner on .hbase-snapshot directory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16886) hbase-client: scanner with reversed=true and small=true gets no result
[
https://issues.apache.org/jira/browse/HBASE-16886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603923#comment-15603923
]
Ted Yu commented on HBASE-16886:
{code}
+ public static class ReversedScannerCallablePrepareHandler{
{code}
ReversedScannerCallablePrepareHandler can be package private.
> hbase-client: scanner with reversed=true and small=true gets no result
> --
>
> Key: HBASE-16886
> URL: https://issues.apache.org/jira/browse/HBASE-16886
> Project: HBase
> Issue Type: Bug
>Reporter: huzheng
> Labels: patch
> Attachments: HBASE-16886.v0.patch, HBASE-16886.v1.patch,
> HBASE-16886.v2.patch, TestReversedSmallScan.java
>
>
> Assume HBase have four regions (-oo, b), [b, c), [c, d), [d,+oo) , and all
> rowKeys are located in region [d, +oo). using a Reversed Small Scanner will
> get no result.
> Attached file show this failed test case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16939) ExportSnapshot: set owner and permission on right directory
Guanghao Zhang created HBASE-16939:
--
Summary: ExportSnapshot: set owner and permission on right
directory
Key: HBASE-16939
URL: https://issues.apache.org/jira/browse/HBASE-16939
Project: HBase
Issue Type: Bug
Reporter: Guanghao Zhang
Priority: Minor
{code}
FileUtil.copy(inputFs, snapshotDir, outputFs, initialOutputSnapshotDir, false,
false, conf);
if (filesUser != null || filesGroup != null) {
setOwner(outputFs, snapshotTmpDir, filesUser, filesGroup, true);
}
if (filesMode > 0) {
setPermission(outputFs, snapshotTmpDir, (short)filesMode, true);
}
{code}
It copy snapshot manifest to initialOutputSnapshotDir, but it set owner on
snapshotTmpDir. They are different directory when skipTmp is true.
Another problem is new cluster doesn't have .hbase-snapshot directory. So after
export snapshot, it should set owner on .hbase-snapshot directory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15709) Handle large edits for asynchronous WAL
[ https://issues.apache.org/jira/browse/HBASE-15709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603908#comment-15603908 ] Duo Zhang commented on HBASE-15709: --- Yeah the datanode only aceept a packet starting from a checksum boundary, so if we break at a point which is not the checksum boundary, then we need to also include the data to the previous checksum boundary when sending the next packet. And thanks for the remind, I think I need to make sure the MAX_DATA_LEN is aligned with the checksum chunk size. Will upload a new patch and commit it if the precommit result is OK. Thanks. > Handle large edits for asynchronous WAL > --- > > Key: HBASE-15709 > URL: https://issues.apache.org/jira/browse/HBASE-15709 > Project: HBase > Issue Type: Sub-task > Components: io, wal >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-15709-v1.patch, HBASE-15709.patch > > > First, FanOutOneBlockAsyncDFSOutput can not work if the buffered data is > larger than PacketReceiver.MAX_PACKET_SIZE(16MB). > Second, since we only allow one block here, we need to make sure we do not > exceed the block size after writing a large chunk of data. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16938) TableCFsUpdater maybe failed due to no write permission on peerNode
[ https://issues.apache.org/jira/browse/HBASE-16938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-16938: --- Attachment: HBASE-16938.patch Attach a little patch for this. > TableCFsUpdater maybe failed due to no write permission on peerNode > --- > > Key: HBASE-16938 > URL: https://issues.apache.org/jira/browse/HBASE-16938 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.4.0 >Reporter: Guanghao Zhang > Attachments: HBASE-16938.patch > > > After HBASE-11393, replication table-cfs use a PB object. So it need copy the > old string config to new PB object when upgrade cluster. In our use case, we > have different kerberos for different cluster, etc. online serve cluster and > offline processing cluster. And we use a unify global admin kerberos for all > clusters. The peer node is created by client. So only global admin has the > write permission for it. When upgrade cluster, HMaster doesn't has the write > permission on peer node, it maybe failed to copy old table-cfs string to new > PB Object. I thought it need a tool for client to do this copy job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16938) TableCFsUpdater maybe failed due to no write permission on peerNode
[ https://issues.apache.org/jira/browse/HBASE-16938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-16938: --- Assignee: Guanghao Zhang Status: Patch Available (was: Open) > TableCFsUpdater maybe failed due to no write permission on peerNode > --- > > Key: HBASE-16938 > URL: https://issues.apache.org/jira/browse/HBASE-16938 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.4.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Attachments: HBASE-16938.patch > > > After HBASE-11393, replication table-cfs use a PB object. So it need copy the > old string config to new PB object when upgrade cluster. In our use case, we > have different kerberos for different cluster, etc. online serve cluster and > offline processing cluster. And we use a unify global admin kerberos for all > clusters. The peer node is created by client. So only global admin has the > write permission for it. When upgrade cluster, HMaster doesn't has the write > permission on peer node, it maybe failed to copy old table-cfs string to new > PB Object. I thought it need a tool for client to do this copy job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[ https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603895#comment-15603895 ] Duo Zhang commented on HBASE-16890: --- In my opinion, there are two things may impact performance a lot. One is the contention when append as [~ram_krish] said, and the second is the aggregation of sync request as [~stack] said. The former one is hard to optimize as we have some other logics in the critial section, it is not only a queue offering... The latter one is a trade off between latency and throughput. I have already done the aggregation work in the consumer of AsyncFSWAL so first we need to confirm that it works correctly... And second, if we add some delay before the actual sync, it may help increasing the throughput. This is a common tech to increase throughput by also increasing latency. Of course, it does not always increase latency. Here sync is a heavy operation, so choosing a suitable delay may also decrease the latency. I think we could have a try. Thanks. > Analyze the performance of AsyncWAL and fix the same > > > Key: HBASE-16890 > URL: https://issues.apache.org/jira/browse/HBASE-16890 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: contention.png, contention_defaultWAL.png > > > Tests reveal that AsyncWAL under load in single node cluster performs slower > than the Default WAL. This task is to analyze and see if we could fix it. > See some discussions in the tail of JIRA HBASE-15536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16938) TableCFsUpdater maybe failed due to no write permission on peerNode
[ https://issues.apache.org/jira/browse/HBASE-16938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603884#comment-15603884 ] Guanghao Zhang commented on HBASE-16938: FYI [~chenheng] [~enis] > TableCFsUpdater maybe failed due to no write permission on peerNode > --- > > Key: HBASE-16938 > URL: https://issues.apache.org/jira/browse/HBASE-16938 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.4.0 >Reporter: Guanghao Zhang > > After HBASE-11393, replication table-cfs use a PB object. So it need copy the > old string config to new PB object when upgrade cluster. In our use case, we > have different kerberos for different cluster, etc. online serve cluster and > offline processing cluster. And we use a unify global admin kerberos for all > clusters. The peer node is created by client. So only global admin has the > write permission for it. When upgrade cluster, HMaster doesn't has the write > permission on peer node, it maybe failed to copy old table-cfs string to new > PB Object. I thought it need a tool for client to do this copy job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16938) TableCFsUpdater maybe failed due to no write permission on peerNode
Guanghao Zhang created HBASE-16938: -- Summary: TableCFsUpdater maybe failed due to no write permission on peerNode Key: HBASE-16938 URL: https://issues.apache.org/jira/browse/HBASE-16938 Project: HBase Issue Type: Bug Components: Replication Affects Versions: 2.0.0, 1.4.0 Reporter: Guanghao Zhang After HBASE-11393, replication table-cfs use a PB object. So it need copy the old string config to new PB object when upgrade cluster. In our use case, we have different kerberos for different cluster, etc. online serve cluster and offline processing cluster. And we use a unify global admin kerberos for all clusters. The peer node is created by client. So only global admin has the write permission for it. When upgrade cluster, HMaster doesn't has the write permission on peer node, it maybe failed to copy old table-cfs string to new PB Object. I thought it need a tool for client to do this copy job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[ https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603867#comment-15603867 ] Duo Zhang commented on HBASE-16890: --- The DFSOutputStream works like this 1. Copy data into the buf in FSOutputSummer 2. If the size exceeds limit(checksumChunkSize * BUFFER_NUM_CHUNKS), then compute the checksum and call writeChunk method in DFSOutputStream 3. Serialize the chunk data into currentPacket field in DFSOutputStream. 4. When flushing, compute the checksum of the remaining data in buf if any, call writeChunk, and then write the currentPacket out. And for FanOutOneBlockAsyncDFSOutput, the processing is straight forward. Buffer all data in buf, and do all other things such as compute checksum, construct packet when flushing. I do not think this will cause much difference unless we write very large data at the same time that we maybe blocked in the JNI call for a long time which has some bad impact on gc. > Analyze the performance of AsyncWAL and fix the same > > > Key: HBASE-16890 > URL: https://issues.apache.org/jira/browse/HBASE-16890 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: contention.png, contention_defaultWAL.png > > > Tests reveal that AsyncWAL under load in single node cluster performs slower > than the Default WAL. This task is to analyze and see if we could fix it. > See some discussions in the tail of JIRA HBASE-15536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16463) Improve transparent table/CF encryption with Commons Crypto
[ https://issues.apache.org/jira/browse/HBASE-16463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603858#comment-15603858 ] Dapeng Sun commented on HBASE-16463: Thank [~ram_krish] and [~apurtell] for your help, I have updated the RELEASE NOTE. > Improve transparent table/CF encryption with Commons Crypto > --- > > Key: HBASE-16463 > URL: https://issues.apache.org/jira/browse/HBASE-16463 > Project: HBase > Issue Type: New Feature > Components: encryption >Affects Versions: 2.0.0 >Reporter: Dapeng Sun >Assignee: Dapeng Sun > Fix For: 2.0.0 > > Attachments: HBASE-16463.001.patch, HBASE-16463.002.patch, > HBASE-16463.003.patch, HBASE-16463.004.patch, HBASE-16463.004.patch, > HBASE-16463.005.patch > > > Apache Commons Crypto > (https://commons.apache.org/proper/commons-crypto/index.html) is a > cryptographic library optimized with AES-NI. > HBASE-7544 introduces a framework for transparent encryption feature for > protecting HFile and WAL data at rest. Currently JCE cipher is used bu > default, the improvement will use Commons Crypto to accelerate the > transparent encryption of HBase. new crypto provider with Commons CRYPTO will > be provided for Transparent encryption. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16463) Improve transparent table/CF encryption with Commons Crypto
[ https://issues.apache.org/jira/browse/HBASE-16463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dapeng Sun updated HBASE-16463: --- Release Note: Improve transparent table/CF encryption with Commons Crypto. The change introduces a new optional CryptoCipherProvider (CommonsCryptoAES) for transparent table/CF encryption. And the encryption performance would be accelerated by hardware in modern CPU (AES-NI). This feature could be enabled by updating the configuration "hbase.crypto.cipherprovider" to "org.apache.hadoop.hbase.io.crypto.CryptoCipherProvider" in hbase-site.xml. For detailed information about transparent table/CF encryption including configuration examples see the Security section of the HBase manual. (was: Release note) > Improve transparent table/CF encryption with Commons Crypto > --- > > Key: HBASE-16463 > URL: https://issues.apache.org/jira/browse/HBASE-16463 > Project: HBase > Issue Type: New Feature > Components: encryption >Affects Versions: 2.0.0 >Reporter: Dapeng Sun >Assignee: Dapeng Sun > Fix For: 2.0.0 > > Attachments: HBASE-16463.001.patch, HBASE-16463.002.patch, > HBASE-16463.003.patch, HBASE-16463.004.patch, HBASE-16463.004.patch, > HBASE-16463.005.patch > > > Apache Commons Crypto > (https://commons.apache.org/proper/commons-crypto/index.html) is a > cryptographic library optimized with AES-NI. > HBASE-7544 introduces a framework for transparent encryption feature for > protecting HFile and WAL data at rest. Currently JCE cipher is used bu > default, the improvement will use Commons Crypto to accelerate the > transparent encryption of HBase. new crypto provider with Commons CRYPTO will > be provided for Transparent encryption. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16463) Improve transparent table/CF encryption with Commons Crypto
[ https://issues.apache.org/jira/browse/HBASE-16463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dapeng Sun updated HBASE-16463: --- Release Note: Release note > Improve transparent table/CF encryption with Commons Crypto > --- > > Key: HBASE-16463 > URL: https://issues.apache.org/jira/browse/HBASE-16463 > Project: HBase > Issue Type: New Feature > Components: encryption >Affects Versions: 2.0.0 >Reporter: Dapeng Sun >Assignee: Dapeng Sun > Fix For: 2.0.0 > > Attachments: HBASE-16463.001.patch, HBASE-16463.002.patch, > HBASE-16463.003.patch, HBASE-16463.004.patch, HBASE-16463.004.patch, > HBASE-16463.005.patch > > > Apache Commons Crypto > (https://commons.apache.org/proper/commons-crypto/index.html) is a > cryptographic library optimized with AES-NI. > HBASE-7544 introduces a framework for transparent encryption feature for > protecting HFile and WAL data at rest. Currently JCE cipher is used bu > default, the improvement will use Commons Crypto to accelerate the > transparent encryption of HBase. new crypto provider with Commons CRYPTO will > be provided for Transparent encryption. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16892) Use TableName instead of String in SnapshotDescription
[
https://issues.apache.org/jira/browse/HBASE-16892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603746#comment-15603746
]
Hadoop QA commented on HBASE-16892:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
6s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 43s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 58s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 53s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 85m 16s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 130m 34s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.security.token.TestZKSecretWatcher |
| Timed out junit tests | org.apache.hadoop.hbase.client.TestReplicaWithCluster
|
| | org.apache.hadoop.hbase.client.TestHCM |
| | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12835030/HBASE-16892-v2.patch |
| JIRA Issue | HBASE-16892 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 399a2b35bca5 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 988d1f9 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| findbugs |
[jira] [Updated] (HBASE-16827) Backup.proto refactoring
[ https://issues.apache.org/jira/browse/HBASE-16827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16827: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.0.0 Status: Resolved (was: Patch Available) Backup tests passed. Thanks for the patch, Vlad. > Backup.proto refactoring > > > Key: HBASE-16827 > URL: https://issues.apache.org/jira/browse/HBASE-16827 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-16827-v1.patch, HBASE-16827-v2.patch, > HBASE-16827-v3.patch, HBASE-16827-v4.patch, HBASE-16827-v5.patch, > org.apache.hadoop.hbase.backup.TestIncrementalBackup-output.txt > > > # Make server_name type ServerName (from HBase.proto) > # Make all fields optional > # Remove duplicate fields from Backup.proto > # Removed BackupManifest from .proto -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15789) PB related changes to work with offheap
[ https://issues.apache.org/jira/browse/HBASE-15789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15789: -- Release Note: This issue adds a patch to our checked in internal, shaded protobuf, but it also adds a general means of apply patches to our version of protobuf. Patches found in the new src/main/patches directory are all applied as the last task when you run a build with the -Pcompile-protobuf profile under the hbase-protocol-shaded module. This commit also includes our first patch to protobuf; it adds ByteInput to mimic pb3.1's ByteOutput (src/main/patches/HBASE-15789_V2.patch attached here). was:Adds means of patching our checked-in protobuf as last part of our offline -Pcompile-protobuf step. Patches found in the new src/main/patches directory are all applied as the last task when you run a build with the -Pcompile-protobuf profile. This commit also includes an actual patch that adds ByteInput to mimic pb3.1's ByteOutput (src/main/patches/HBASE-15789_V2.patch attached here). > PB related changes to work with offheap > --- > > Key: HBASE-15789 > URL: https://issues.apache.org/jira/browse/HBASE-15789 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-15789.master.001.patch, HBASE-15789.patch, > HBASE-15789_V2.patch > > > This is an issue to brainstorm. Whether we go with pb 2.x or pb 3.0 and also > depends on the shading of protobuf classes. > We should also decide if we are going to fork the PB classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15789) PB related changes to work with offheap
[ https://issues.apache.org/jira/browse/HBASE-15789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15789: -- Release Note: Adds means of patching our checked-in protobuf as last part of our offline -Pcompile-protobuf step. Patches found in the new src/main/patches directory are all applied as the last task when you run a build with the -Pcompile-protobuf profile. This commit also includes an actual patch that adds ByteInput to mimic pb3.1's ByteOutput (src/main/patches/HBASE-15789_V2.patch attached here). (was: Adds means of patching our checked-in protobuf as last part of our offline -Pgenerate-shaded-classes step. Patches found in the new src/main/patches directory are all applied as the last step when you run a build with the -Pgenerate-shaded-classes profile. This commit also includes an actual patch that adds ByteInput to mimic pb3.1's ByteOutput (src/main/patches/HBASE-15789_V2.patch attached here).) > PB related changes to work with offheap > --- > > Key: HBASE-15789 > URL: https://issues.apache.org/jira/browse/HBASE-15789 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-15789.master.001.patch, HBASE-15789.patch, > HBASE-15789_V2.patch > > > This is an issue to brainstorm. Whether we go with pb 2.x or pb 3.0 and also > depends on the shading of protobuf classes. > We should also decide if we are going to fork the PB classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-15789) PB related changes to work with offheap
[ https://issues.apache.org/jira/browse/HBASE-15789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-15789. --- Resolution: Fixed Reapplied (FYI [~anoop.hbase]). Add a clean of generated and com/google/protobuf dirs as first step so when comes time for the patch to run, it'll work even if the src is already patched. Seems to fix build. > PB related changes to work with offheap > --- > > Key: HBASE-15789 > URL: https://issues.apache.org/jira/browse/HBASE-15789 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-15789.master.001.patch, HBASE-15789.patch, > HBASE-15789_V2.patch > > > This is an issue to brainstorm. Whether we go with pb 2.x or pb 3.0 and also > depends on the shading of protobuf classes. > We should also decide if we are going to fork the PB classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16830) RSRpcServices#openRegion() should handle the case where table descriptor is null
[
https://issues.apache.org/jira/browse/HBASE-16830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603609#comment-15603609
]
Hudson commented on HBASE-16830:
FAILURE: Integrated in Jenkins build HBase-1.1-JDK7 #1803 (See
[https://builds.apache.org/job/HBase-1.1-JDK7/1803/])
HBASE-16830 RSRpcServices#openRegion() should handle the case where (tedyu: rev
3203e27245bb68dddcd1331e38a79bcf5c44912b)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
> RSRpcServices#openRegion() should handle the case where table descriptor is
> null
>
>
> Key: HBASE-16830
> URL: https://issues.apache.org/jira/browse/HBASE-16830
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 1.3.0, 1.2.5, 1.1.8
>
> Attachments: 16830.branch-1.3.txt, 16830.branch-1.3.txt,
> testClusterId-output.txt
>
>
> Over in HBASE-16349, [~mantonov] pointed me to the failed TestClusterId in
> Jenkins 1.3 build.
> I ran the test locally with Java 8 and consistently stumbled over timeout in
> starting the mini cluster:
> {code}
> 2016-10-13 15:41:05,276 INFO
> [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233]
> regionserver.RSRpcServices(1620): Open hbase:namespace,,1476398463993.
>7ab1d0b3b7be8b9a7ce9af61c29d5342.
> 2016-10-13 15:41:05,277 ERROR
> [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233]
> ipc.RpcServer(2320): Unexpected throwable object
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.openRegion(RSRpcServices.java:1679)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22737)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
> {code}
> It turned out that htd for hbase:namespace table was null.
> Once the NPE is fixed, the test passes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16934) Revert ITBLL misconfiguration check introduced in HBASE-16562
[ https://issues.apache.org/jira/browse/HBASE-16934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603608#comment-15603608 ] Hudson commented on HBASE-16934: FAILURE: Integrated in Jenkins build HBase-1.1-JDK7 #1803 (See [https://builds.apache.org/job/HBase-1.1-JDK7/1803/]) HBASE-16934 Revert ITBLL misconfiguration check that results in NPE. (busbey: rev 51522c14bc5488e0a549fe87c450591b0a69c845) * (edit) hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java > Revert ITBLL misconfiguration check introduced in HBASE-16562 > - > > Key: HBASE-16934 > URL: https://issues.apache.org/jira/browse/HBASE-16934 > Project: HBase > Issue Type: Bug > Components: integration tests >Affects Versions: 1.1.7, 0.98.23 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Fix For: 0.98.24, 1.1.8 > > > The misconfiguration check in HBASE-16562 does insufficient input validation > / handling, resulting in a NPE during normal operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16830) RSRpcServices#openRegion() should handle the case where table descriptor is null
[
https://issues.apache.org/jira/browse/HBASE-16830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603533#comment-15603533
]
Hudson commented on HBASE-16830:
FAILURE: Integrated in Jenkins build HBase-1.1-JDK8 #1887 (See
[https://builds.apache.org/job/HBase-1.1-JDK8/1887/])
HBASE-16830 RSRpcServices#openRegion() should handle the case where (tedyu: rev
3203e27245bb68dddcd1331e38a79bcf5c44912b)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
> RSRpcServices#openRegion() should handle the case where table descriptor is
> null
>
>
> Key: HBASE-16830
> URL: https://issues.apache.org/jira/browse/HBASE-16830
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 1.3.0, 1.2.5, 1.1.8
>
> Attachments: 16830.branch-1.3.txt, 16830.branch-1.3.txt,
> testClusterId-output.txt
>
>
> Over in HBASE-16349, [~mantonov] pointed me to the failed TestClusterId in
> Jenkins 1.3 build.
> I ran the test locally with Java 8 and consistently stumbled over timeout in
> starting the mini cluster:
> {code}
> 2016-10-13 15:41:05,276 INFO
> [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233]
> regionserver.RSRpcServices(1620): Open hbase:namespace,,1476398463993.
>7ab1d0b3b7be8b9a7ce9af61c29d5342.
> 2016-10-13 15:41:05,277 ERROR
> [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233]
> ipc.RpcServer(2320): Unexpected throwable object
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.openRegion(RSRpcServices.java:1679)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22737)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
> {code}
> It turned out that htd for hbase:namespace table was null.
> Once the NPE is fixed, the test passes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16934) Revert ITBLL misconfiguration check introduced in HBASE-16562
[ https://issues.apache.org/jira/browse/HBASE-16934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603532#comment-15603532 ] Hudson commented on HBASE-16934: FAILURE: Integrated in Jenkins build HBase-1.1-JDK8 #1887 (See [https://builds.apache.org/job/HBase-1.1-JDK8/1887/]) HBASE-16934 Revert ITBLL misconfiguration check that results in NPE. (busbey: rev 51522c14bc5488e0a549fe87c450591b0a69c845) * (edit) hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java > Revert ITBLL misconfiguration check introduced in HBASE-16562 > - > > Key: HBASE-16934 > URL: https://issues.apache.org/jira/browse/HBASE-16934 > Project: HBase > Issue Type: Bug > Components: integration tests >Affects Versions: 1.1.7, 0.98.23 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Fix For: 0.98.24, 1.1.8 > > > The misconfiguration check in HBASE-16562 does insufficient input validation > / handling, resulting in a NPE during normal operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16608) Introducing the ability to merge ImmutableSegments without copy-compaction or SQM usage
[ https://issues.apache.org/jira/browse/HBASE-16608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603535#comment-15603535 ] Hudson commented on HBASE-16608: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #1847 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1847/]) HBASE-16608 Introducing the ability to merge ImmutableSegments without (anoopsamjohn: rev 988d1f9bc955de53b9bf520b69be5bedc51f4f7e) * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreMergerSegmentsIterator.java * (delete) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreCompactorIterator.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactingMemStore.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactingToCellArrayMapMemStore.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionPipeline.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactingMemStore.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestWalAndCompactingMemStoreFlush.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSegmentsIterator.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SegmentFactory.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestPerColumnFamilyFlush.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ImmutableSegment.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ImmutableMemStoreLAB.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreCompactor.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HeapMemStoreLAB.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/VersionedSegmentsList.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreCompactorSegmentsIterator.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java > Introducing the ability to merge ImmutableSegments without copy-compaction or > SQM usage > --- > > Key: HBASE-16608 > URL: https://issues.apache.org/jira/browse/HBASE-16608 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > Attachments: HBASE-16417-V02.patch, HBASE-16417-V04.patch, > HBASE-16417-V06.patch, HBASE-16417-V07.patch, HBASE-16417-V08.patch, > HBASE-16417-V10.patch, HBASE-16608-Final.patch, HBASE-16608-Final.patch, > HBASE-16608-V01.patch, HBASE-16608-V03.patch, HBASE-16608-V04.patch, > HBASE-16608-V08.patch, HBASE-16608-V09.patch, HBASE-16608-V09.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16827) Backup.proto refactoring
[
https://issues.apache.org/jira/browse/HBASE-16827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603471#comment-15603471
]
Hadoop QA commented on HBASE-16827:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s {color}
| {color:red} HBASE-16827 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12835032/HBASE-16827-v5.patch |
| JIRA Issue | HBASE-16827 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4170/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Backup.proto refactoring
>
>
> Key: HBASE-16827
> URL: https://issues.apache.org/jira/browse/HBASE-16827
> Project: HBase
> Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-16827-v1.patch, HBASE-16827-v2.patch,
> HBASE-16827-v3.patch, HBASE-16827-v4.patch, HBASE-16827-v5.patch,
> org.apache.hadoop.hbase.backup.TestIncrementalBackup-output.txt
>
>
> # Make server_name type ServerName (from HBase.proto)
> # Make all fields optional
> # Remove duplicate fields from Backup.proto
> # Removed BackupManifest from .proto
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16827) Backup.proto refactoring
[ https://issues.apache.org/jira/browse/HBASE-16827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16827: -- Attachment: HBASE-16827-v5.patch v5. Removed BackupManifest completely from .proto definition. > Backup.proto refactoring > > > Key: HBASE-16827 > URL: https://issues.apache.org/jira/browse/HBASE-16827 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16827-v1.patch, HBASE-16827-v2.patch, > HBASE-16827-v3.patch, HBASE-16827-v4.patch, HBASE-16827-v5.patch, > org.apache.hadoop.hbase.backup.TestIncrementalBackup-output.txt > > > # Make server_name type ServerName (from HBase.proto) > # Make all fields optional > # Remove duplicate fields from Backup.proto > # Removed BackupManifest from .proto -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16827) Backup.proto refactoring
[ https://issues.apache.org/jira/browse/HBASE-16827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16827: -- Description: # Make server_name type ServerName (from HBase.proto) # Make all fields optional # Remove duplicate fields from Backup.proto # Removed BackupManifest from .proto was: # Make server_name type ServerName (from HBase.proto) # Make all fields optional # Remove duplicate fields from Backup.proto > Backup.proto refactoring > > > Key: HBASE-16827 > URL: https://issues.apache.org/jira/browse/HBASE-16827 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16827-v1.patch, HBASE-16827-v2.patch, > HBASE-16827-v3.patch, HBASE-16827-v4.patch, > org.apache.hadoop.hbase.backup.TestIncrementalBackup-output.txt > > > # Make server_name type ServerName (from HBase.proto) > # Make all fields optional > # Remove duplicate fields from Backup.proto > # Removed BackupManifest from .proto -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16892) Use TableName instead of String in SnapshotDescription
[ https://issues.apache.org/jira/browse/HBASE-16892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16892: Attachment: HBASE-16892-v2.patch > Use TableName instead of String in SnapshotDescription > -- > > Key: HBASE-16892 > URL: https://issues.apache.org/jira/browse/HBASE-16892 > Project: HBase > Issue Type: Sub-task > Components: snapshots >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-16892-v0.patch, HBASE-16892-v1.patch, > HBASE-16892-v2.patch > > > mostly find & replace work: > deprecate the SnapshotDescription constructors with the String argument in > favor of the TableName ones. > Replace the TableName.valueOf() around with the new getTableName() > Replace the TableName.getNameAsString() by just passing the TableName -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16937) Replace SnapshotType protobuf conversion when we can directly use the pojo object
[ https://issues.apache.org/jira/browse/HBASE-16937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16937: Status: Open (was: Patch Available) > Replace SnapshotType protobuf conversion when we can directly use the pojo > object > - > > Key: HBASE-16937 > URL: https://issues.apache.org/jira/browse/HBASE-16937 > Project: HBase > Issue Type: Sub-task >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-16937-v0.patch, HBASE-16937-v1.patch > > > mostly find & replace work: > replace the back and forth protobuf conversion when we can just use the > client SnapshotType enum. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16937) Replace SnapshotType protobuf conversion when we can directly use the pojo object
[ https://issues.apache.org/jira/browse/HBASE-16937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603419#comment-15603419 ] Matteo Bertozzi commented on HBASE-16937: - patch depends on HBASE-16892 > Replace SnapshotType protobuf conversion when we can directly use the pojo > object > - > > Key: HBASE-16937 > URL: https://issues.apache.org/jira/browse/HBASE-16937 > Project: HBase > Issue Type: Sub-task >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-16937-v0.patch, HBASE-16937-v1.patch > > > mostly find & replace work: > replace the back and forth protobuf conversion when we can just use the > client SnapshotType enum. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16937) Replace SnapshotType protobuf conversion when we can directly use the pojo object
[ https://issues.apache.org/jira/browse/HBASE-16937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16937: Attachment: HBASE-16937-v1.patch > Replace SnapshotType protobuf conversion when we can directly use the pojo > object > - > > Key: HBASE-16937 > URL: https://issues.apache.org/jira/browse/HBASE-16937 > Project: HBase > Issue Type: Sub-task >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-16937-v0.patch, HBASE-16937-v1.patch > > > mostly find & replace work: > replace the back and forth protobuf conversion when we can just use the > client SnapshotType enum. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16830) RSRpcServices#openRegion() should handle the case where table descriptor is null
[
https://issues.apache.org/jira/browse/HBASE-16830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603368#comment-15603368
]
Hudson commented on HBASE-16830:
SUCCESS: Integrated in Jenkins build HBase-1.2-JDK8 #49 (See
[https://builds.apache.org/job/HBase-1.2-JDK8/49/])
HBASE-16830 RSRpcServices#openRegion() should handle the case where (tedyu: rev
044bbddf2f182b0cba1668349671f9a23c1977e0)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
> RSRpcServices#openRegion() should handle the case where table descriptor is
> null
>
>
> Key: HBASE-16830
> URL: https://issues.apache.org/jira/browse/HBASE-16830
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 1.3.0, 1.2.5, 1.1.8
>
> Attachments: 16830.branch-1.3.txt, 16830.branch-1.3.txt,
> testClusterId-output.txt
>
>
> Over in HBASE-16349, [~mantonov] pointed me to the failed TestClusterId in
> Jenkins 1.3 build.
> I ran the test locally with Java 8 and consistently stumbled over timeout in
> starting the mini cluster:
> {code}
> 2016-10-13 15:41:05,276 INFO
> [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233]
> regionserver.RSRpcServices(1620): Open hbase:namespace,,1476398463993.
>7ab1d0b3b7be8b9a7ce9af61c29d5342.
> 2016-10-13 15:41:05,277 ERROR
> [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233]
> ipc.RpcServer(2320): Unexpected throwable object
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.openRegion(RSRpcServices.java:1679)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22737)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
> {code}
> It turned out that htd for hbase:namespace table was null.
> Once the NPE is fixed, the test passes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16562) ITBLL should fail to start if misconfigured
[ https://issues.apache.org/jira/browse/HBASE-16562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-16562: Fix Version/s: 1.3.1 > ITBLL should fail to start if misconfigured > --- > > Key: HBASE-16562 > URL: https://issues.apache.org/jira/browse/HBASE-16562 > Project: HBase > Issue Type: Improvement > Components: integration tests >Reporter: Andrew Purtell >Assignee: Heng Chen > Fix For: 1.0.4, 1.3.1, 1.1.7, 0.98.23 > > Attachments: HBASE-16562-branch-1.2.patch, > HBASE-16562-branch-1.2.v1.patch, HBASE-16562.patch, HBASE-16562.v1.patch, > HBASE-16562.v1.patch-addendum > > > The number of nodes in ITBLL must a multiple of width*wrap (defaults to 25M, > but can be configured by adding two more args to the test invocation) or else > verification will fail. This can be very expensive in terms of time or hourly > billing for on demand test resources. Check the sanity of test parameters > before launching any MR jobs and fail fast if invariants aren't met with an > indication what parameter(s) need fixing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16359) NullPointerException in RSRpcServices.openRegion()
[
https://issues.apache.org/jira/browse/HBASE-16359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Antonov updated HBASE-16359:
Fix Version/s: (was: 1.3.1)
> NullPointerException in RSRpcServices.openRegion()
> --
>
> Key: HBASE-16359
> URL: https://issues.apache.org/jira/browse/HBASE-16359
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.2
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 16359.addendum, 16359.v2.txt
>
>
> I was investigating why some region failed to move out of transition within
> timeout 12ms and found the following in region server log:
> {code}
> 2016-08-04 09:19:52,616 INFO
> [B.priority.fifo.QRpcServer.handler=12,queue=0,port=16020]
> regionserver.RSRpcServices: Open hbck_table_772674,,1470302211047.
> da859880bb51bc0fd25979798a96c444.
> 2016-08-04 09:19:52,620 ERROR
> [B.priority.fifo.QRpcServer.handler=12,queue=0,port=16020] ipc.RpcServer:
> Unexpected throwable object
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.openRegion(RSRpcServices.java:1530)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22737)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2127)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> {code}
> Here is related code - NPE was thrown from the last line:
> {code}
> htd = htds.get(region.getTable());
> if (htd == null) {
> htd = regionServer.tableDescriptors.get(region.getTable());
> htds.put(region.getTable(), htd);
> }
> ...
> if (region.isMetaRegion()) {
> regionServer.service.submit(new OpenMetaHandler(
> regionServer, regionServer, region, htd, masterSystemTime,
> coordination, ord));
> } else {
>
> regionServer.updateRegionFavoredNodesMapping(region.getEncodedName(),
> regionOpenInfo.getFavoredNodesList());
> if (htd.getPriority() >= HConstants.ADMIN_QOS ||
> region.getTable().isSystemTable()) {
> {code}
> region.getTable() shouldn't be null since it is called via
> htds.get(region.getTable()) unconditionally.
> It seems htd was null.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16937) Replace SnapshotType protobuf conversion when we can directly use the pojo object
[
https://issues.apache.org/jira/browse/HBASE-16937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603279#comment-15603279
]
Hadoop QA commented on HBASE-16937:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s {color}
| {color:red} HBASE-16937 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12835015/HBASE-16937-v0.patch |
| JIRA Issue | HBASE-16937 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4168/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Replace SnapshotType protobuf conversion when we can directly use the pojo
> object
> -
>
> Key: HBASE-16937
> URL: https://issues.apache.org/jira/browse/HBASE-16937
> Project: HBase
> Issue Type: Sub-task
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Trivial
> Fix For: 2.0.0
>
> Attachments: HBASE-16937-v0.patch
>
>
> mostly find & replace work:
> replace the back and forth protobuf conversion when we can just use the
> client SnapshotType enum.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16937) Replace SnapshotType protobuf conversion when we can directly use the pojo object
[ https://issues.apache.org/jira/browse/HBASE-16937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16937: Attachment: HBASE-16937-v0.patch > Replace SnapshotType protobuf conversion when we can directly use the pojo > object > - > > Key: HBASE-16937 > URL: https://issues.apache.org/jira/browse/HBASE-16937 > Project: HBase > Issue Type: Sub-task >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-16937-v0.patch > > > mostly find & replace work: > replace the back and forth protobuf conversion when we can just use the > client SnapshotType enum. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16937) Replace SnapshotType protobuf conversion when we can directly use the pojo object
Matteo Bertozzi created HBASE-16937: --- Summary: Replace SnapshotType protobuf conversion when we can directly use the pojo object Key: HBASE-16937 URL: https://issues.apache.org/jira/browse/HBASE-16937 Project: HBase Issue Type: Sub-task Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Trivial Fix For: 2.0.0 Attachments: HBASE-16937-v0.patch mostly find & replace work: replace the back and forth protobuf conversion when we can just use the client SnapshotType enum. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16937) Replace SnapshotType protobuf conversion when we can directly use the pojo object
[ https://issues.apache.org/jira/browse/HBASE-16937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16937: Status: Patch Available (was: Open) > Replace SnapshotType protobuf conversion when we can directly use the pojo > object > - > > Key: HBASE-16937 > URL: https://issues.apache.org/jira/browse/HBASE-16937 > Project: HBase > Issue Type: Sub-task >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-16937-v0.patch > > > mostly find & replace work: > replace the back and forth protobuf conversion when we can just use the > client SnapshotType enum. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-16936) TestRateLimiter#testOverconsumptionFixedIntervalRefillStrategy is flaky
[
https://issues.apache.org/jira/browse/HBASE-16936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huaxiang sun reassigned HBASE-16936:
Assignee: huaxiang sun
> TestRateLimiter#testOverconsumptionFixedIntervalRefillStrategy is flaky
> ---
>
> Key: HBASE-16936
> URL: https://issues.apache.org/jira/browse/HBASE-16936
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Dima Spivak
>Assignee: huaxiang sun
>
> Seeing this once every month or two in-house. Looks like it's a timing-based
> test, which makes it prone to flakiness, but I've noticed that whenever it
> fails, it fails with the same {{AssertionError}} (including values), so it'd
> be worth digging into. In our case:
> {noformat}
> expected:<1000> but was:<999>
> Stack Trace:
> java.lang.AssertionError: expected:<1000> but was:<999>
> at
> org.apache.hadoop.hbase.quotas.TestRateLimiter.testOverconsumptionFixedIntervalRefillStrategy(TestRateLimiter.java:119)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16877) Remove directory layout/ filesystem references form Master
[
https://issues.apache.org/jira/browse/HBASE-16877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603042#comment-15603042
]
Umesh Agashe commented on HBASE-16877:
--
I have partial patch for this. But this requires a few other issues to be fixed
before work can be continued on this.
> Remove directory layout/ filesystem references form Master
> --
>
> Key: HBASE-16877
> URL: https://issues.apache.org/jira/browse/HBASE-16877
> Project: HBase
> Issue Type: Sub-task
> Components: Filesystem Integration
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
>
> Remove directory layout/ filesystem references form Master and add required
> APIs to MasterStorage/ RegionStorage
> {code}
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterStatusServlet.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterWalManager.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionPlacementMaintainer.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileLinkCleaner.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/MasterProcedureEnv.java
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HBASE-16898) remove directory layout / fs references from MOB
[
https://issues.apache.org/jira/browse/HBASE-16898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-16898 started by Umesh Agashe.
> remove directory layout / fs references from MOB
>
>
> Key: HBASE-16898
> URL: https://issues.apache.org/jira/browse/HBASE-16898
> Project: HBase
> Issue Type: Sub-task
> Components: Filesystem Integration
>Reporter: Sean Busbey
>Assignee: Umesh Agashe
>
> update mob related classes and utilities to use MasterStorage / RegionStorage
> and update those apis as needed to do so.
> {code}
> hbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobUtils.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/mob/compactions/MobCompactor.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/mob/mapreduce/SweepJob.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/mob/mapreduce/SweepReducer.java
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16877) Remove directory layout/ filesystem references form Master
[
https://issues.apache.org/jira/browse/HBASE-16877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603042#comment-15603042
]
Umesh Agashe edited comment on HBASE-16877 at 10/24/16 8:04 PM:
I have partial patch for this. But this requires a few other issues to be fixed
before work can be continued on this. Putting this on hold for a while.
was (Author: uagashe):
I have partial patch for this. But this requires a few other issues to be fixed
before work can be continued on this.
> Remove directory layout/ filesystem references form Master
> --
>
> Key: HBASE-16877
> URL: https://issues.apache.org/jira/browse/HBASE-16877
> Project: HBase
> Issue Type: Sub-task
> Components: Filesystem Integration
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
>
> Remove directory layout/ filesystem references form Master and add required
> APIs to MasterStorage/ RegionStorage
> {code}
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterStatusServlet.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterWalManager.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionPlacementMaintainer.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileLinkCleaner.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/MasterProcedureEnv.java
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-16898) remove directory layout / fs references from MOB
[
https://issues.apache.org/jira/browse/HBASE-16898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Umesh Agashe reassigned HBASE-16898:
Assignee: Umesh Agashe
> remove directory layout / fs references from MOB
>
>
> Key: HBASE-16898
> URL: https://issues.apache.org/jira/browse/HBASE-16898
> Project: HBase
> Issue Type: Sub-task
> Components: Filesystem Integration
>Reporter: Sean Busbey
>Assignee: Umesh Agashe
>
> update mob related classes and utilities to use MasterStorage / RegionStorage
> and update those apis as needed to do so.
> {code}
> hbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobUtils.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/mob/compactions/MobCompactor.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/mob/mapreduce/SweepJob.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/mob/mapreduce/SweepReducer.java
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16608) Introducing the ability to merge ImmutableSegments without copy-compaction or SQM usage
[
https://issues.apache.org/jira/browse/HBASE-16608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603024#comment-15603024
]
Hadoop QA commented on HBASE-16608:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 22s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
9s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 40s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 37s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 98m 13s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 138m 18s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.snapshot.TestMobExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestMobRestoreFlushSnapshotFromClient |
| | org.apache.hadoop.hbase.snapshot.TestRestoreFlushSnapshotFromClient |
| | org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12834979/HBASE-16608-Final.patch
|
| JIRA Issue | HBASE-16608 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 98163d5b16cc 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 1b12a60 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4164/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html
|
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/4164/artifact/patchprocess/whitespace-eol.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4164/artifact/patchprocess/patch-unit-hbase-server.txt
|
|
[jira] [Commented] (HBASE-16562) ITBLL should fail to start if misconfigured
[ https://issues.apache.org/jira/browse/HBASE-16562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603023#comment-15603023 ] Hudson commented on HBASE-16562: SUCCESS: Integrated in Jenkins build HBase-1.3-JDK7 #50 (See [https://builds.apache.org/job/HBase-1.3-JDK7/50/]) Revert "HBASE-16562 ITBLL should fail to start if misconfigured, (busbey: rev 01c2a3176709ffbcc2ad94804e821fe62f138d29) * (edit) hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java Revert "HBASE-16562 ITBLL should fail to start if misconfigured" (busbey: rev 1b4b6109c8e8479e9edbec1d1027dfd37059c6b3) * (edit) hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java > ITBLL should fail to start if misconfigured > --- > > Key: HBASE-16562 > URL: https://issues.apache.org/jira/browse/HBASE-16562 > Project: HBase > Issue Type: Improvement > Components: integration tests >Reporter: Andrew Purtell >Assignee: Heng Chen > Fix For: 1.0.4, 1.1.7, 0.98.23 > > Attachments: HBASE-16562-branch-1.2.patch, > HBASE-16562-branch-1.2.v1.patch, HBASE-16562.patch, HBASE-16562.v1.patch, > HBASE-16562.v1.patch-addendum > > > The number of nodes in ITBLL must a multiple of width*wrap (defaults to 25M, > but can be configured by adding two more args to the test invocation) or else > verification will fail. This can be very expensive in terms of time or hourly > billing for on demand test resources. Check the sanity of test parameters > before launching any MR jobs and fail fast if invariants aren't met with an > indication what parameter(s) need fixing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16828) Make backup command-line tool interface consistent
[
https://issues.apache.org/jira/browse/HBASE-16828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16828:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Make backup command-line tool interface consistent
> --
>
> Key: HBASE-16828
> URL: https://issues.apache.org/jira/browse/HBASE-16828
> Project: HBase
> Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
> Attachments: HBASE-16828-v1.patch, HBASE-16828-v2.patch,
> HBASE-16828-v3.patch, HBASE-16828-v4.patch, HBASE-16828-v5.patch
>
>
> From Stacks review:
> {quote}
> This is interesting... if I pass a set and a table as command lets me do, it
> runs the backup twice. I was sort of expecting one backup dir with all
> content in it. No biggie. Here is the command I ran (the -set included the
> table so the tool behavior avoids backup overwrite).
> ./hbase/bin/hbase --config ~/conf_hbase backup create full
> hdfs://ve0524.halxg.cloudera.com:8020/backup clicks -set s
> {quote}
> {quote}
> The command I ran was $ ./hbase/bin/hbase --config ~/conf_hbase backup create
> full clicks I'm missing the backup location which probably explains the
> above. I'd think the tool would verify the command before going ahead and
> running a mapreduce job?
> {quote}
> {quote}
> stack@ve0524:~$ ./hbase/bin/hbase --config ~/conf_hbase backup create
> ... does the tables 'option' not take a hyphen but all others do? Most of the
> time options are delimited by space but not always
> {quote}
> {quote}
> When required args are not supplied, dumping help/usage rather than ERROR is
> usual and user friendly.
> {quote}
> {quote}
> What about consistency in errors, that sometimes commands are single letter,
> other times words, that descriptions in usage are not what is on the Option,
> that we do Options but don't use it to do Usage/help, and others.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16898) remove directory layout / fs references from MOB
[
https://issues.apache.org/jira/browse/HBASE-16898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Umesh Agashe updated HBASE-16898:
-
Description:
update mob related classes and utilities to use MasterStorage / RegionStorage
and update those apis as needed to do so.
{code}
hbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobUtils.java
hbase-server/src/main/java/org/apache/hadoop/hbase/mob/compactions/MobCompactor.java
hbase-server/src/main/java/org/apache/hadoop/hbase/mob/mapreduce/SweepJob.java
hbase-server/src/main/java/org/apache/hadoop/hbase/mob/mapreduce/SweepReducer.java
{code}
was:
update mob related classes and utilities to use MasterStorage / RegionStorage
and update those apis as needed to do so.
hbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobUtils.java
hbase-server/src/main/java/org/apache/hadoop/hbase/mob/compactions/MobCompactor.java
hbase-server/src/main/java/org/apache/hadoop/hbase/mob/mapreduce/SweepJob.java
hbase-server/src/main/java/org/apache/hadoop/hbase/mob/mapreduce/SweepReducer.java
> remove directory layout / fs references from MOB
>
>
> Key: HBASE-16898
> URL: https://issues.apache.org/jira/browse/HBASE-16898
> Project: HBase
> Issue Type: Sub-task
> Components: Filesystem Integration
>Reporter: Sean Busbey
>
> update mob related classes and utilities to use MasterStorage / RegionStorage
> and update those apis as needed to do so.
> {code}
> hbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobUtils.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/mob/compactions/MobCompactor.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/mob/mapreduce/SweepJob.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/mob/mapreduce/SweepReducer.java
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16898) remove directory layout / fs references from MOB
[
https://issues.apache.org/jira/browse/HBASE-16898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603021#comment-15603021
]
Umesh Agashe commented on HBASE-16898:
--
Following two files are removed from master:
{code}
hbase-server/src/main/java/org/apache/hadoop/hbase/mob/mapreduce/SweepJob.java
hbase-server/src/main/java/org/apache/hadoop/hbase/mob/mapreduce/SweepReducer.java
{code}
> remove directory layout / fs references from MOB
>
>
> Key: HBASE-16898
> URL: https://issues.apache.org/jira/browse/HBASE-16898
> Project: HBase
> Issue Type: Sub-task
> Components: Filesystem Integration
>Reporter: Sean Busbey
>
> update mob related classes and utilities to use MasterStorage / RegionStorage
> and update those apis as needed to do so.
> {code}
> hbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobUtils.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/mob/compactions/MobCompactor.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/mob/mapreduce/SweepJob.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/mob/mapreduce/SweepReducer.java
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16838) Implement scan
[ https://issues.apache.org/jira/browse/HBASE-16838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603006#comment-15603006 ] stack commented on HBASE-16838: --- bq. The reason why we introduce a small scan is that we want to get the whole result with one rpc call, so I think it is reasonable to have the same API with get? That seems reasonable. Just seems weird having two ways to do a Scan, one way when small and another when not-small. The method should be called smallScan? Or do you think that the the API with a return that is a CompletableFuture is indicator enough that this is a short-lived Scan? Maybe it is. bq. So I think a scan method that returns the whole results is acceptable? You mean a whole region's worth of results or you mean the batch? If the former, no, we should not return whole regions' worth (I don't think you meant this) and if the latter, where is the user getting an Async API? They have to block even though the Scan maybe returning them chunks of the Scan or later, we could be streaming Cells back to the API. CompleteableFuture is all or nothing with its Done? There is no way to get out Cell at a time? Is that a different API? On AsyncResultScanner, https://github.com/jurmous/async-hbase-client/blob/master/src/main/java/org/apache/hadoop/hbase/client/AsyncResultScanner.java, there is a use case where you'd want to perhaps prefetch or do parallel non-sequenced access to a Scan but that'd be tough since we don't know where the next 'next' should start till the currently running one has returned. Thanks for turning up AsyncResultScanner. I don't think it applicable given where we are now after changes you've done [~Apache9] > Implement scan > -- > > Key: HBASE-16838 > URL: https://issues.apache.org/jira/browse/HBASE-16838 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16838.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[ https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15602972#comment-15602972 ] stack commented on HBASE-16890: --- Looking at the histograms, if to be believed, then we are sync'ing more in asyncwal; less aggregation. That synchronize on payloads will be popular. Looks like plenty of opportunity for speedup. Want to put up a patch w/ yours so far [~ram_krish]? > Analyze the performance of AsyncWAL and fix the same > > > Key: HBASE-16890 > URL: https://issues.apache.org/jira/browse/HBASE-16890 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: contention.png, contention_defaultWAL.png > > > Tests reveal that AsyncWAL under load in single node cluster performs slower > than the Default WAL. This task is to analyze and see if we could fix it. > See some discussions in the tail of JIRA HBASE-15536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16747) Track memstore data size and heap overhead separately
[
https://issues.apache.org/jira/browse/HBASE-16747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15602938#comment-15602938
]
Hadoop QA commented on HBASE-16747:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
6s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 48s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 22s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 46s
{color} | {color:red} hbase-common generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 4s
{color} | {color:red} hbase-server generated 2 new + 0 unchanged - 1 fixed = 2
total (was 1) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 30s
{color} | {color:red} hbase-server generated 2 new + 1 unchanged - 0 fixed = 3
total (was 1) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 50s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 7s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 62m 41s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-common |
| | Uninitialized read of cell in new
org.apache.hadoop.hbase.CellUtil$TagRewriteCell(Cell, byte[]) At
CellUtil.java:new org.apache.hadoop.hbase.CellUtil$TagRewriteCell(Cell, byte[])
At CellUtil.java:[line 395] |
| FindBugs | module:hbase-server |
| |
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion$BatchOperation)
does not release lock on all paths At HRegion.java:on all paths At
HRegion.java:[line 3273] |
| | Integral division result cast to double or float in
org.apache.hadoop.hbase.regionserver.HeapMemoryManager$HeapMemoryTunerChore.tune()
At HeapMemoryManager.java:double or float in
org.apache.hadoop.hbase.regionserver.HeapMemoryManager$HeapMemoryTunerChore.tune()
At HeapMemoryManager.java:[line
[jira] [Commented] (HBASE-16828) Make backup command-line tool interface consistent
[
https://issues.apache.org/jira/browse/HBASE-16828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15602912#comment-15602912
]
Hadoop QA commented on HBASE-16828:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 11s {color}
| {color:red} HBASE-16828 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12834993/HBASE-16828-v5.patch |
| JIRA Issue | HBASE-16828 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4167/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Make backup command-line tool interface consistent
> --
>
> Key: HBASE-16828
> URL: https://issues.apache.org/jira/browse/HBASE-16828
> Project: HBase
> Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
> Attachments: HBASE-16828-v1.patch, HBASE-16828-v2.patch,
> HBASE-16828-v3.patch, HBASE-16828-v4.patch, HBASE-16828-v5.patch
>
>
> From Stacks review:
> {quote}
> This is interesting... if I pass a set and a table as command lets me do, it
> runs the backup twice. I was sort of expecting one backup dir with all
> content in it. No biggie. Here is the command I ran (the -set included the
> table so the tool behavior avoids backup overwrite).
> ./hbase/bin/hbase --config ~/conf_hbase backup create full
> hdfs://ve0524.halxg.cloudera.com:8020/backup clicks -set s
> {quote}
> {quote}
> The command I ran was $ ./hbase/bin/hbase --config ~/conf_hbase backup create
> full clicks I'm missing the backup location which probably explains the
> above. I'd think the tool would verify the command before going ahead and
> running a mapreduce job?
> {quote}
> {quote}
> stack@ve0524:~$ ./hbase/bin/hbase --config ~/conf_hbase backup create
> ... does the tables 'option' not take a hyphen but all others do? Most of the
> time options are delimited by space but not always
> {quote}
> {quote}
> When required args are not supplied, dumping help/usage rather than ERROR is
> usual and user friendly.
> {quote}
> {quote}
> What about consistency in errors, that sometimes commands are single letter,
> other times words, that descriptions in usage are not what is on the Option,
> that we do Options but don't use it to do Usage/help, and others.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16828) Make backup command-line tool interface consistent
[
https://issues.apache.org/jira/browse/HBASE-16828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15602920#comment-15602920
]
Vladimir Rodionov edited comment on HBASE-16828 at 10/24/16 7:13 PM:
-
All tests pass.
was (Author: vrodionov):
All test pass.
> Make backup command-line tool interface consistent
> --
>
> Key: HBASE-16828
> URL: https://issues.apache.org/jira/browse/HBASE-16828
> Project: HBase
> Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
> Attachments: HBASE-16828-v1.patch, HBASE-16828-v2.patch,
> HBASE-16828-v3.patch, HBASE-16828-v4.patch, HBASE-16828-v5.patch
>
>
> From Stacks review:
> {quote}
> This is interesting... if I pass a set and a table as command lets me do, it
> runs the backup twice. I was sort of expecting one backup dir with all
> content in it. No biggie. Here is the command I ran (the -set included the
> table so the tool behavior avoids backup overwrite).
> ./hbase/bin/hbase --config ~/conf_hbase backup create full
> hdfs://ve0524.halxg.cloudera.com:8020/backup clicks -set s
> {quote}
> {quote}
> The command I ran was $ ./hbase/bin/hbase --config ~/conf_hbase backup create
> full clicks I'm missing the backup location which probably explains the
> above. I'd think the tool would verify the command before going ahead and
> running a mapreduce job?
> {quote}
> {quote}
> stack@ve0524:~$ ./hbase/bin/hbase --config ~/conf_hbase backup create
> ... does the tables 'option' not take a hyphen but all others do? Most of the
> time options are delimited by space but not always
> {quote}
> {quote}
> When required args are not supplied, dumping help/usage rather than ERROR is
> usual and user friendly.
> {quote}
> {quote}
> What about consistency in errors, that sometimes commands are single letter,
> other times words, that descriptions in usage are not what is on the Option,
> that we do Options but don't use it to do Usage/help, and others.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16828) Make backup command-line tool interface consistent
[
https://issues.apache.org/jira/browse/HBASE-16828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15602920#comment-15602920
]
Vladimir Rodionov commented on HBASE-16828:
---
All test pass.
> Make backup command-line tool interface consistent
> --
>
> Key: HBASE-16828
> URL: https://issues.apache.org/jira/browse/HBASE-16828
> Project: HBase
> Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
> Attachments: HBASE-16828-v1.patch, HBASE-16828-v2.patch,
> HBASE-16828-v3.patch, HBASE-16828-v4.patch, HBASE-16828-v5.patch
>
>
> From Stacks review:
> {quote}
> This is interesting... if I pass a set and a table as command lets me do, it
> runs the backup twice. I was sort of expecting one backup dir with all
> content in it. No biggie. Here is the command I ran (the -set included the
> table so the tool behavior avoids backup overwrite).
> ./hbase/bin/hbase --config ~/conf_hbase backup create full
> hdfs://ve0524.halxg.cloudera.com:8020/backup clicks -set s
> {quote}
> {quote}
> The command I ran was $ ./hbase/bin/hbase --config ~/conf_hbase backup create
> full clicks I'm missing the backup location which probably explains the
> above. I'd think the tool would verify the command before going ahead and
> running a mapreduce job?
> {quote}
> {quote}
> stack@ve0524:~$ ./hbase/bin/hbase --config ~/conf_hbase backup create
> ... does the tables 'option' not take a hyphen but all others do? Most of the
> time options are delimited by space but not always
> {quote}
> {quote}
> When required args are not supplied, dumping help/usage rather than ERROR is
> usual and user friendly.
> {quote}
> {quote}
> What about consistency in errors, that sometimes commands are single letter,
> other times words, that descriptions in usage are not what is on the Option,
> that we do Options but don't use it to do Usage/help, and others.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16828) Make backup command-line tool interface consistent
[
https://issues.apache.org/jira/browse/HBASE-16828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladimir Rodionov updated HBASE-16828:
--
Attachment: HBASE-16828-v5.patch
v5. Rebased. Ready to be committed. [~tedyu]
> Make backup command-line tool interface consistent
> --
>
> Key: HBASE-16828
> URL: https://issues.apache.org/jira/browse/HBASE-16828
> Project: HBase
> Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
> Attachments: HBASE-16828-v1.patch, HBASE-16828-v2.patch,
> HBASE-16828-v3.patch, HBASE-16828-v4.patch, HBASE-16828-v5.patch
>
>
> From Stacks review:
> {quote}
> This is interesting... if I pass a set and a table as command lets me do, it
> runs the backup twice. I was sort of expecting one backup dir with all
> content in it. No biggie. Here is the command I ran (the -set included the
> table so the tool behavior avoids backup overwrite).
> ./hbase/bin/hbase --config ~/conf_hbase backup create full
> hdfs://ve0524.halxg.cloudera.com:8020/backup clicks -set s
> {quote}
> {quote}
> The command I ran was $ ./hbase/bin/hbase --config ~/conf_hbase backup create
> full clicks I'm missing the backup location which probably explains the
> above. I'd think the tool would verify the command before going ahead and
> running a mapreduce job?
> {quote}
> {quote}
> stack@ve0524:~$ ./hbase/bin/hbase --config ~/conf_hbase backup create
> ... does the tables 'option' not take a hyphen but all others do? Most of the
> time options are delimited by space but not always
> {quote}
> {quote}
> When required args are not supplied, dumping help/usage rather than ERROR is
> usual and user friendly.
> {quote}
> {quote}
> What about consistency in errors, that sometimes commands are single letter,
> other times words, that descriptions in usage are not what is on the Option,
> that we do Options but don't use it to do Usage/help, and others.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16936) TestRateLimiter#testOverconsumptionFixedIntervalRefillStrategy is flaky
[
https://issues.apache.org/jira/browse/HBASE-16936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15602914#comment-15602914
]
huaxiang sun commented on HBASE-16936:
--
Hi [~dimaspivak], mind I take this one? I had another Throttle flakey test to
take a look and this is related, thanks.
> TestRateLimiter#testOverconsumptionFixedIntervalRefillStrategy is flaky
> ---
>
> Key: HBASE-16936
> URL: https://issues.apache.org/jira/browse/HBASE-16936
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Dima Spivak
>
> Seeing this once every month or two in-house. Looks like it's a timing-based
> test, which makes it prone to flakiness, but I've noticed that whenever it
> fails, it fails with the same {{AssertionError}} (including values), so it'd
> be worth digging into. In our case:
> {noformat}
> expected:<1000> but was:<999>
> Stack Trace:
> java.lang.AssertionError: expected:<1000> but was:<999>
> at
> org.apache.hadoop.hbase.quotas.TestRateLimiter.testOverconsumptionFixedIntervalRefillStrategy(TestRateLimiter.java:119)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16936) TestRateLimiter#testOverconsumptionFixedIntervalRefillStrategy is flaky
Dima Spivak created HBASE-16936:
---
Summary:
TestRateLimiter#testOverconsumptionFixedIntervalRefillStrategy is flaky
Key: HBASE-16936
URL: https://issues.apache.org/jira/browse/HBASE-16936
Project: HBase
Issue Type: Bug
Components: test
Reporter: Dima Spivak
Seeing this once every month or two in-house. Looks like it's a timing-based
test, which makes it prone to flakiness, but I've noticed that whenever it
fails, it fails with the same {{AssertionError}} (including values), so it'd be
worth digging into. In our case:
{noformat}
expected:<1000> but was:<999>
Stack Trace:
java.lang.AssertionError: expected:<1000> but was:<999>
at
org.apache.hadoop.hbase.quotas.TestRateLimiter.testOverconsumptionFixedIntervalRefillStrategy(TestRateLimiter.java:119)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16830) RSRpcServices#openRegion() should handle the case where table descriptor is null
[
https://issues.apache.org/jira/browse/HBASE-16830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15602868#comment-15602868
]
Hudson commented on HBASE-16830:
SUCCESS: Integrated in Jenkins build HBase-1.2-JDK7 #54 (See
[https://builds.apache.org/job/HBase-1.2-JDK7/54/])
HBASE-16830 RSRpcServices#openRegion() should handle the case where (tedyu: rev
044bbddf2f182b0cba1668349671f9a23c1977e0)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
> RSRpcServices#openRegion() should handle the case where table descriptor is
> null
>
>
> Key: HBASE-16830
> URL: https://issues.apache.org/jira/browse/HBASE-16830
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 1.3.0, 1.2.5, 1.1.8
>
> Attachments: 16830.branch-1.3.txt, 16830.branch-1.3.txt,
> testClusterId-output.txt
>
>
> Over in HBASE-16349, [~mantonov] pointed me to the failed TestClusterId in
> Jenkins 1.3 build.
> I ran the test locally with Java 8 and consistently stumbled over timeout in
> starting the mini cluster:
> {code}
> 2016-10-13 15:41:05,276 INFO
> [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233]
> regionserver.RSRpcServices(1620): Open hbase:namespace,,1476398463993.
>7ab1d0b3b7be8b9a7ce9af61c29d5342.
> 2016-10-13 15:41:05,277 ERROR
> [RpcServer.FifoWFPBQ.priority.handler=5,queue=1,port=59233]
> ipc.RpcServer(2320): Unexpected throwable object
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.openRegion(RSRpcServices.java:1679)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22737)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2270)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
> {code}
> It turned out that htd for hbase:namespace table was null.
> Once the NPE is fixed, the test passes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16748) Release 1.2.4
[ https://issues.apache.org/jira/browse/HBASE-16748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15602867#comment-15602867 ] Hudson commented on HBASE-16748: SUCCESS: Integrated in Jenkins build HBase-1.2-JDK7 #54 (See [https://builds.apache.org/job/HBase-1.2-JDK7/54/]) HBASE-16748 update CHANGES.txt for RC1. (busbey: rev da7b1fc87dd9b81e84e7c2815d91a5b7c23ce9b6) * (edit) CHANGES.txt > Release 1.2.4 > - > > Key: HBASE-16748 > URL: https://issues.apache.org/jira/browse/HBASE-16748 > Project: HBase > Issue Type: Task >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 1.2.4 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16562) ITBLL should fail to start if misconfigured
[ https://issues.apache.org/jira/browse/HBASE-16562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15602869#comment-15602869 ] Hudson commented on HBASE-16562: SUCCESS: Integrated in Jenkins build HBase-1.2-JDK7 #54 (See [https://builds.apache.org/job/HBase-1.2-JDK7/54/]) Revert "HBASE-16562 ITBLL should fail to start if misconfigured" (busbey: rev 9fc3b94b6704764980552feec15ad6191817fa05) * (edit) hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java > ITBLL should fail to start if misconfigured > --- > > Key: HBASE-16562 > URL: https://issues.apache.org/jira/browse/HBASE-16562 > Project: HBase > Issue Type: Improvement > Components: integration tests >Reporter: Andrew Purtell >Assignee: Heng Chen > Fix For: 1.0.4, 1.1.7, 0.98.23 > > Attachments: HBASE-16562-branch-1.2.patch, > HBASE-16562-branch-1.2.v1.patch, HBASE-16562.patch, HBASE-16562.v1.patch, > HBASE-16562.v1.patch-addendum > > > The number of nodes in ITBLL must a multiple of width*wrap (defaults to 25M, > but can be configured by adding two more args to the test invocation) or else > verification will fail. This can be very expensive in terms of time or hourly > billing for on demand test resources. Check the sanity of test parameters > before launching any MR jobs and fail fast if invariants aren't met with an > indication what parameter(s) need fixing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15532) core favored nodes enhancements
[ https://issues.apache.org/jira/browse/HBASE-15532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15602859#comment-15602859 ] Thiruvel Thirumoolan commented on HBASE-15532: -- Thanks for looking at this [~devaraj]! Would the following work for you? Do you want to track them as part of this jira or handle them via sub-tasks? 1. Balancer and other enhancements (this includes all common changes to FNAssignmentHelper, FavoredNodeManager, etc). 2. Any other misc changes - like FavoredNodesRepairChore etc. 3. Split and merge code paths - this probably might be smaller. This is dependent on 1. 4. Just the interfaces for all the tools 5. Actual tools/client side code 6. Include all unit tests Also, how do you want to do this? Should we have a feature branch where we can keep merging stuff or do you want to merge this to the main branch after each step? > core favored nodes enhancements > --- > > Key: HBASE-15532 > URL: https://issues.apache.org/jira/browse/HBASE-15532 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Liu >Assignee: Thiruvel Thirumoolan > Fix For: 2.0.0 > > Attachments: HBASE-15532.master.000.patch, > HBASE-15532.master.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[ https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15602839#comment-15602839 ] stack commented on HBASE-16890: --- [~Apache9] You see Rams' notes above? > Analyze the performance of AsyncWAL and fix the same > > > Key: HBASE-16890 > URL: https://issues.apache.org/jira/browse/HBASE-16890 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: contention.png, contention_defaultWAL.png > > > Tests reveal that AsyncWAL under load in single node cluster performs slower > than the Default WAL. This task is to analyze and see if we could fix it. > See some discussions in the tail of JIRA HBASE-15536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[
https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15602833#comment-15602833
]
stack commented on HBASE-16890:
---
Hang on Rerunning. I had TRACE level logging going on for DFSClient...
thats not fair in compare. Sorry [~ram_krish] On rerun I see this:
{code}
5610 Performance counter stats for './hbase/bin/hbase --config
/home/stack/conf_hbase org.apache.hadoop.hbase.wal.WALPerformanceEvaluation
-threads 100 -iterat ions 25 -keySize 50 -valueSize 100 -qualifiers 10':
5611
5612 4036698.163447 task-clock (msec) #9.891 CPUs utilized
5613 41,344,568 context-switches #0.010 M/sec
5614 6,909,262 cpu-migrations#0.002 M/sec
5615 12,455,604 page-faults #0.003 M/sec
5616 7,548,061,036,706 cycles#1.870 GHz
5617 stalled-cycles-frontend
5618 stalled-cycles-backend
5619 2,788,283,330,940 instructions #0.37 insns per cycle
5620480,911,981,821 branches # 119.135 M/sec
5621 5,351,642,769 branch-misses #1.11% of all branches
5622
5623 408.109482919 seconds time elapsed
{code}
So, asyncwal is 30% slower than DFSClient#DFSOS
> Analyze the performance of AsyncWAL and fix the same
>
>
> Key: HBASE-16890
> URL: https://issues.apache.org/jira/browse/HBASE-16890
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: contention.png, contention_defaultWAL.png
>
>
> Tests reveal that AsyncWAL under load in single node cluster performs slower
> than the Default WAL. This task is to analyze and see if we could fix it.
> See some discussions in the tail of JIRA HBASE-15536.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16747) Track memstore data size and heap overhead separately
[ https://issues.apache.org/jira/browse/HBASE-16747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15602811#comment-15602811 ] Anoop Sam John commented on HBASE-16747: https://reviews.apache.org/r/53145/ > Track memstore data size and heap overhead separately > -- > > Key: HBASE-16747 > URL: https://issues.apache.org/jira/browse/HBASE-16747 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16747.patch, HBASE-16747.patch, > HBASE-16747_WIP.patch > > > We track the memstore size in 3 places. > 1. Global at RS level in RegionServerAccounting. This tracks all memstore's > size and used to calculate whether forced flushes needed because of global > heap pressure > 2. At region level in HRegion. This is sum of sizes of all memstores within > this region. This is used to decide whether region reaches flush size (128 MB) > 3. Segment level. This tracks the in memory flush/compaction decisions. > All these use the Cell's heap size which include the data bytes# as well as > Cell object heap overhead. Also we include the overhead because of addition > of Cells into Segment's data structures (Like CSLM). > Once we have off heap memstore, we will keep the cell data bytes in off heap > area. So we can not track both data size and heap overhead as one entity. We > need to separate them and track. > Proposal here is to track both cell data size and heap overhead separately at > global accounting layer. As of now we have only on heap memstore. So the > global memstore boundary checks will consider both (adds up and check against > global max memstore size) > Track cell data size alone (This can be on heap or off heap) in region level. > Region flushes use cell data size alone for the region flush decision. A > user configuring 128 MB as flush size, normally he will expect to get a 128MB > data flush size. But as we were including the heap overhead also, once the > flush happens, the actual data size getting flushed is way behind this 128 > MB. Now with this change we will behave more like what a user thinks. > Segment level in memory flush/compaction also considers cell data size alone. > But we will need to track the heap overhead also. (Once the in memory flush > or normal flush happens, we will have to adjust both cell data size and heap > overhead) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[ https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15602799#comment-15602799 ] ramkrishna.s.vasudevan commented on HBASE-16890: Am not sure how you take the above data. But am seeing the ops per sec that the tool prints. Excuse typos > Analyze the performance of AsyncWAL and fix the same > > > Key: HBASE-16890 > URL: https://issues.apache.org/jira/browse/HBASE-16890 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: contention.png, contention_defaultWAL.png > > > Tests reveal that AsyncWAL under load in single node cluster performs slower > than the Default WAL. This task is to analyze and see if we could fix it. > See some discussions in the tail of JIRA HBASE-15536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16931) Setting cell's seqId to zero in compaction flow might cause RS down.
[
https://issues.apache.org/jira/browse/HBASE-16931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15602755#comment-15602755
]
Hudson commented on HBASE-16931:
FAILURE: Integrated in Jenkins build HBase-1.4 #493 (See
[https://builds.apache.org/job/HBase-1.4/493/])
HBASE-16931 Setting cell's seqId to zero in compaction flow might cause (liyu:
rev 16823ff55e7da4822d82fb2a2108b4a253fd42f9)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java
> Setting cell's seqId to zero in compaction flow might cause RS down.
>
>
> Key: HBASE-16931
> URL: https://issues.apache.org/jira/browse/HBASE-16931
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 2.0.0
>Reporter: binlijin
>Assignee: binlijin
>Priority: Critical
> Attachments: HBASE-16931-master.patch, HBASE-16931.branch-1.patch,
> HBASE-16931.branch-1.v2.patch, HBASE-16931_master_v2.patch,
> HBASE-16931_master_v3.patch, HBASE-16931_master_v4.patch,
> HBASE-16931_master_v5.patch
>
>
> Compactor#performCompaction
> do {
> hasMore = scanner.next(cells, scannerContext);
> // output to writer:
> for (Cell c : cells) {
> if (cleanSeqId && c.getSequenceId() <= smallestReadPoint) {
> CellUtil.setSequenceId(c, 0);
> }
> writer.append(c);
> }
> cells.clear();
> } while (hasMore);
> scanner.next will choose at most "hbase.hstore.compaction.kv.max" kvs, the
> last cell still reference by StoreScanner.prevCell, so if cleanSeqId is
> called when the scanner.next call StoreScanner.checkScanOrder may throw
> exception and cause regionserver down.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
