[jira] [Commented] (HBASE-17869) UnsafeAvailChecker wrongly returns false on ppc
[
https://issues.apache.org/jira/browse/HBASE-17869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956341#comment-15956341
]
Anoop Sam John commented on HBASE-17869:
Pls raise another issue to track the FuzzyRowFilter.. As such +1 for this
issue commit.
bq.if ("ppc64".equals(arch) || "ppc64le".equals(arch))
We need a Spark issue kind of matching check than the exact string match? If u
are confident this is enough (After ur test in actual env) then go ahead.. Ur
call.
Thanks for digging into it.
> UnsafeAvailChecker wrongly returns false on ppc
> ---
>
> Key: HBASE-17869
> URL: https://issues.apache.org/jira/browse/HBASE-17869
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.4
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17869.patch
>
>
> On ppc64 arch, java.nio.Bits.unaligned() wrongly returns false due to a JDK
> bug.
> https://bugs.openjdk.java.net/browse/JDK-8165231
> This causes some problem for HBase. i.e. FuzzyRowFilter test fails.
> Fix it by providing a hard-code workaround for the JDK bug.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17849) PE tool random read is not totally random
[ https://issues.apache.org/jira/browse/HBASE-17849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17849: --- Summary: PE tool random read is not totally random (was: PE tool randomness is not totally random) Updated title as this JIRA targets randomReads and randomSeekScan alone. > PE tool random read is not totally random > - > > Key: HBASE-17849 > URL: https://issues.apache.org/jira/browse/HBASE-17849 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17849.patch, HBASE-17849.patch > > > Recently we were using the PE tool for doing some bucket cache related > performance tests. One thing that we noted was that the way the random read > works is not totally random. > Suppose we load 200G of data using --size param and then we use --rows=50 > to do the randomRead. The assumption was among the 200G of data it could > generate randomly 50 row keys to do the reads. > But it so happens that the PE tool generates random rows only on those set of > row keys which falls under the first 50 rows. > This was quite evident when we tried to use HBASE-15314 in our testing. > Suppose we split the bucket cache of size 200G into 2 files each 100G the > randomReads with --rows=50 always lands in the first file and not in the > 2nd file. Better to make PE purely random. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17849) PE tool randomness is not totally random
[ https://issues.apache.org/jira/browse/HBASE-17849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17849: --- Status: Patch Available (was: Open) > PE tool randomness is not totally random > > > Key: HBASE-17849 > URL: https://issues.apache.org/jira/browse/HBASE-17849 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17849.patch, HBASE-17849.patch > > > Recently we were using the PE tool for doing some bucket cache related > performance tests. One thing that we noted was that the way the random read > works is not totally random. > Suppose we load 200G of data using --size param and then we use --rows=50 > to do the randomRead. The assumption was among the 200G of data it could > generate randomly 50 row keys to do the reads. > But it so happens that the PE tool generates random rows only on those set of > row keys which falls under the first 50 rows. > This was quite evident when we tried to use HBASE-15314 in our testing. > Suppose we split the bucket cache of size 200G into 2 files each 100G the > randomReads with --rows=50 always lands in the first file and not in the > 2nd file. Better to make PE purely random. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17849) PE tool randomness is not totally random
[ https://issues.apache.org/jira/browse/HBASE-17849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17849: --- Attachment: HBASE-17849.patch Resubmit for QA. > PE tool randomness is not totally random > > > Key: HBASE-17849 > URL: https://issues.apache.org/jira/browse/HBASE-17849 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17849.patch, HBASE-17849.patch > > > Recently we were using the PE tool for doing some bucket cache related > performance tests. One thing that we noted was that the way the random read > works is not totally random. > Suppose we load 200G of data using --size param and then we use --rows=50 > to do the randomRead. The assumption was among the 200G of data it could > generate randomly 50 row keys to do the reads. > But it so happens that the PE tool generates random rows only on those set of > row keys which falls under the first 50 rows. > This was quite evident when we tried to use HBASE-15314 in our testing. > Suppose we split the bucket cache of size 200G into 2 files each 100G the > randomReads with --rows=50 always lands in the first file and not in the > 2nd file. Better to make PE purely random. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17849) PE tool randomness is not totally random
[ https://issues.apache.org/jira/browse/HBASE-17849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-17849: --- Status: Open (was: Patch Available) > PE tool randomness is not totally random > > > Key: HBASE-17849 > URL: https://issues.apache.org/jira/browse/HBASE-17849 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17849.patch > > > Recently we were using the PE tool for doing some bucket cache related > performance tests. One thing that we noted was that the way the random read > works is not totally random. > Suppose we load 200G of data using --size param and then we use --rows=50 > to do the randomRead. The assumption was among the 200G of data it could > generate randomly 50 row keys to do the reads. > But it so happens that the PE tool generates random rows only on those set of > row keys which falls under the first 50 rows. > This was quite evident when we tried to use HBASE-15314 in our testing. > Suppose we split the bucket cache of size 200G into 2 files each 100G the > randomReads with --rows=50 always lands in the first file and not in the > 2nd file. Better to make PE purely random. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956329#comment-15956329 ] ramkrishna.s.vasudevan commented on HBASE-16438: [~anastas] Pls have a look at RB and tell what you think. We can close this JIRA and move on to CellChunkMap. > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Attachments: > HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, > HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, > HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, > MemstoreChunkCell_trunk.patch > > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. > Why we need to embed the chunkid in the Cell is described by [~anastas] in > this remark over in parent issue > https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HBASE-17849) PE tool randomness is not totally random
[ https://issues.apache.org/jira/browse/HBASE-17849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan reassigned HBASE-17849: -- Assignee: ramkrishna.s.vasudevan > PE tool randomness is not totally random > > > Key: HBASE-17849 > URL: https://issues.apache.org/jira/browse/HBASE-17849 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17849.patch > > > Recently we were using the PE tool for doing some bucket cache related > performance tests. One thing that we noted was that the way the random read > works is not totally random. > Suppose we load 200G of data using --size param and then we use --rows=50 > to do the randomRead. The assumption was among the 200G of data it could > generate randomly 50 row keys to do the reads. > But it so happens that the PE tool generates random rows only on those set of > row keys which falls under the first 50 rows. > This was quite evident when we tried to use HBASE-15314 in our testing. > Suppose we split the bucket cache of size 200G into 2 files each 100G the > randomReads with --rows=50 always lands in the first file and not in the > 2nd file. Better to make PE purely random. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17854) Use StealJobQueue in HFileCleaner after HBASE-17215
[
https://issues.apache.org/jira/browse/HBASE-17854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956328#comment-15956328
]
huaxiang sun commented on HBASE-17854:
--
+1
> Use StealJobQueue in HFileCleaner after HBASE-17215
> ---
>
> Key: HBASE-17854
> URL: https://issues.apache.org/jira/browse/HBASE-17854
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17854.patch, HBASE-17854.v2.patch,
> HBASE-17854.v3.patch, HBASE-17854.v4.patch, HBASE-17854.v5.patch
>
>
> In HBASE-17215 we use specific threads for deleting large/small (archived)
> hfiles, and will improve it from below aspects in this JIRA:
> 1. Using {{StealJobQueue}} to allow large file deletion thread to steal jobs
> from small queue, based on the experience that in real world there'll be much
> more small hfiles
> 2. {{StealJobQueue}} is a kind of {{PriorityQueue}}, so we could also delete
> from the larger file in the queues.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17869) UnsafeAvailChecker wrongly returns false on ppc
[ https://issues.apache.org/jira/browse/HBASE-17869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956327#comment-15956327 ] ramkrishna.s.vasudevan commented on HBASE-17869: I think +1 to commit this and raise another issue to solve the Fuzzy row bug. I think we have some similar instances where ever our code path goes into UNSAFE 'true' case only. > UnsafeAvailChecker wrongly returns false on ppc > --- > > Key: HBASE-17869 > URL: https://issues.apache.org/jira/browse/HBASE-17869 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.4 >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17869.patch > > > On ppc64 arch, java.nio.Bits.unaligned() wrongly returns false due to a JDK > bug. > https://bugs.openjdk.java.net/browse/JDK-8165231 > This causes some problem for HBase. i.e. FuzzyRowFilter test fails. > Fix it by providing a hard-code workaround for the JDK bug. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17854) Use StealJobQueue in HFileCleaner after HBASE-17215
[
https://issues.apache.org/jira/browse/HBASE-17854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956324#comment-15956324
]
Hadoop QA commented on HBASE-17854:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
4s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 12s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 119m 11s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 159m 2s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12861995/HBASE-17854.v5.patch |
| JIRA Issue | HBASE-17854 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 552a8592100f 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 910b680 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/6318/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6318/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6318/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Use StealJobQueue in HFileCleaner after HBASE-17215
> ---
>
> Key: HBASE-17854
> URL: https://issues.apache.org/jira/browse/HBASE-17854
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
>
[jira] [Commented] (HBASE-17869) UnsafeAvailChecker wrongly returns false on ppc
[ https://issues.apache.org/jira/browse/HBASE-17869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956312#comment-15956312 ] Jerry He commented on HBASE-17869: -- I found out SPARK-1 does something similar. > UnsafeAvailChecker wrongly returns false on ppc > --- > > Key: HBASE-17869 > URL: https://issues.apache.org/jira/browse/HBASE-17869 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.4 >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17869.patch > > > On ppc64 arch, java.nio.Bits.unaligned() wrongly returns false due to a JDK > bug. > https://bugs.openjdk.java.net/browse/JDK-8165231 > This causes some problem for HBase. i.e. FuzzyRowFilter test fails. > Fix it by providing a hard-code workaround for the JDK bug. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17869) UnsafeAvailChecker wrongly returns false on ppc
[ https://issues.apache.org/jira/browse/HBASE-17869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956310#comment-15956310 ] Jerry He commented on HBASE-17869: -- Thanks for chiming in. Do you agree I can commit this one and open another issue to look into the false path with FuzzyRowFilter? > UnsafeAvailChecker wrongly returns false on ppc > --- > > Key: HBASE-17869 > URL: https://issues.apache.org/jira/browse/HBASE-17869 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.4 >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17869.patch > > > On ppc64 arch, java.nio.Bits.unaligned() wrongly returns false due to a JDK > bug. > https://bugs.openjdk.java.net/browse/JDK-8165231 > This causes some problem for HBase. i.e. FuzzyRowFilter test fails. > Fix it by providing a hard-code workaround for the JDK bug. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17872) Make BBU#copyFromBufferToBuffer thread-safe
[
https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956304#comment-15956304
]
Anoop Sam John commented on HBASE-17872:
Yes. We should not make the pos of any of the BBs affected by the ops in
BBUtils.. Unsafe way is ok as all are relative reads/writes. When unsafe is
not availble, we should take the path of BB.duplicate() if we need to change
the position so that the original BBs are unaffected wrt pos or limit. Nice
find.. All such cases u can unify. Will see once u attach patch.. Thanks so
much.
> Make BBU#copyFromBufferToBuffer thread-safe
> ---
>
> Key: HBASE-17872
> URL: https://issues.apache.org/jira/browse/HBASE-17872
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0
>
>
> A case is shown below.
> We will get the wrong position of buffer in multithreaded environment, so the
> method makes the invalid cell in MSLAB.
> {noformat}
> public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int
> sourceOffset,
> int destinationOffset, int length) {
> if (in.hasArray() && out.hasArray()) {
> // ...
> } else if (UNSAFE_AVAIL) {
> // ...
> } else {
> int outOldPos = out.position();
> out.position(destinationOffset);
> ByteBuffer inDup = in.duplicate();
> inDup.position(sourceOffset).limit(sourceOffset + length);
> out.put(inDup);
> out.position(outOldPos);
> }
> return destinationOffset + length;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[
https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956303#comment-15956303
]
Anoop Sam John commented on HBASE-17877:
Interesting. The main change here is to avoid reverseBytes calls. That is the
thing giving the perf boost?
{code}
if (minLength - offset >= SIZEOF_INT) {
..
1446 offset += SIZEOF_INT;
1447}
1448if (minLength - offset >= SIZEOF_SHORT) {
{code}
These were to avoid byte by byte read and check as much as possible after long
reads. So this is also a needed change?
> Replace/improve HBase's byte[] comparator
> -
>
> Key: HBASE-17877
> URL: https://issues.apache.org/jira/browse/HBASE-17877
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Vikas Vishwakarma
> Attachments: 17877-1.2.patch
>
>
> [~vik.karma] did some extensive tests and found that Hadoop's version is
> faster - dramatically faster in some cases.
> Patch forthcoming.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-12770) Don't transfer all the queued hlogs of a dead server to the same alive server
[ https://issues.apache.org/jira/browse/HBASE-12770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956279#comment-15956279 ] Hudson commented on HBASE-12770: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #19 (See [https://builds.apache.org/job/HBase-1.3-IT/19/]) HBASE-12770 Don't transfer all the queued hlogs of a dead server to the (antonov: rev fd297e280f25c26346c3343d6ea1be4f0362821e) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueues.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesZKImpl.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationStateBasic.java > Don't transfer all the queued hlogs of a dead server to the same alive server > - > > Key: HBASE-12770 > URL: https://issues.apache.org/jira/browse/HBASE-12770 > Project: HBase > Issue Type: Improvement > Components: Replication >Affects Versions: 2.0.0, 1.4.0 >Reporter: Jianwei Cui >Assignee: Phil Yang >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-12770-branch-1-v1.patch, > HBASE-12770-branch-1-v2.patch, HBASE-12770-branch-1-v3.patch, > HBASE-12770-branch-1-v3.patch, HBASE-12770-branch-1-v3.patch, > HBASE-12770-branch-1-v3.patch, HBASE-12770-trunk.patch, HBASE-12770-v1.patch, > HBASE-12770-v2.patch, HBASE-12770-v3.patch, HBASE-12770-v3.patch > > > When a region server is down(or the cluster restart), all the hlog queues > will be transferred by the same alive region server. In a shared cluster, we > might create several peers replicating data to different peer clusters. There > might be lots of hlogs queued for these peers caused by several reasons, such > as some peers might be disabled, or errors from peer cluster might prevent > the replication, or the replication sources may fail to read some hlog > because of hdfs problem. Then, if the server is down or restarted, another > alive server will take all the replication jobs of the dead server, this > might bring a big pressure to resources(network/disk read) of the alive > server and also is not fast enough to replicate the queued hlogs. And if the > alive server is down, all the replication jobs including that takes from > other dead servers will once again be totally transferred to another alive > server, this might cause a server have a large number of queued hlogs(in our > shared cluster, we find one server might have thousands of queued hlogs for > replication). As an optional way, is it reasonable that the alive server only > transfer one peer's hlogs from the dead server one time? Then, other alive > region servers might have the opportunity to transfer the hlogs of rest > peers. This may also help the queued hlogs be processed more fast. Any > discussion is welcome. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17870) Backport HBASE-12770 to branch-1.3
[ https://issues.apache.org/jira/browse/HBASE-17870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-17870: Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Backport HBASE-12770 to branch-1.3 > -- > > Key: HBASE-17870 > URL: https://issues.apache.org/jira/browse/HBASE-17870 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > Attachments: HBASE-17870.branch-1.3.001.patch > > > Based on discussion on HBASE-12770, let's backport it to branch-1.3. This > combined with zookeeper transport limit breaks replication quite often in > large clusters. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17870) Backport HBASE-12770 to branch-1.3
[ https://issues.apache.org/jira/browse/HBASE-17870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956277#comment-15956277 ] Mikhail Antonov commented on HBASE-17870: - Thank you, pushed to branch-1.3 cc [~ashu210890] > Backport HBASE-12770 to branch-1.3 > -- > > Key: HBASE-17870 > URL: https://issues.apache.org/jira/browse/HBASE-17870 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > Attachments: HBASE-17870.branch-1.3.001.patch > > > Based on discussion on HBASE-12770, let's backport it to branch-1.3. This > combined with zookeeper transport limit breaks replication quite often in > large clusters. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956274#comment-15956274 ] Lars Hofhansl commented on HBASE-17877: --- But it's certainly a change we can undo and leave it the way it was. > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-14141) HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits from backed up tables
[
https://issues.apache.org/jira/browse/HBASE-14141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956269#comment-15956269
]
Hudson commented on HBASE-14141:
SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2801 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/2801/])
HBASE-14141 HBase Backup/Restore Phase 3: Filter WALs on backup to (tedyu: rev
910b68082c8f200f0ba6395a76b7ee1c8917e401)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/impl/RestoreTablesClient.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/impl/BackupManager.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/mapreduce/MapReduceRestoreJob.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalTableBackupClient.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/WALInputFormat.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/wal/AbstractFSWALProvider.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/mapreduce/HFileSplitterJob.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/util/RestoreTool.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/impl/TableBackupClient.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalBackupManager.java
> HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits
> from backed up tables
> ---
>
> Key: HBASE-14141
> URL: https://issues.apache.org/jira/browse/HBASE-14141
> Project: HBase
> Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Blocker
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-14141.HBASE-14123.v1.patch, HBASE-14141.v1.patch,
> HBASE-14141.v2.patch, HBASE-14141.v4.patch, HBASE-14141.v5.patch,
> HBASE-14141.v6.patch
>
>
> h2. High level design overview
> * When incremental backup request comes for tables {t} we select all the
> tables already registered in a backup system - {T} and union them with {t},
> which results in a new table set - U(t, T)
> * For every table K from U(t,T) we perform the following:
> ** Convert new WAL files into HFile applying table filter K (only edits for
> table T will pass the filter)
> ** Move these HFile(s) to backup destination
> During restore (incremental):
> * We run full restore first
> * Then collect all HFiles from intermediate incremental images and run them
> through HFileSplitterJob, which splits files into a current tables region
> boundaries
> * Load these files using LoadIncrementalHFiles tool
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956266#comment-15956266 ] Lars Hofhansl edited comment on HBASE-17877 at 4/5/17 4:03 AM: --- [~Apache9] that's what the Hadoop comparator does, which tested faster. :) (Here: https://github.com/hanborq/hadoop/blob/master/src/core/org/apache/hadoop/io/FastByteComparisons.java#L161) As tests one can use PE and YCSB. (The test that I used was a simple scan test with 10-100m keys.) was (Author: lhofhansl): [~Apache9] that's what the Hadoop comparator does, which tested faster. :) As tests one can use PE and YCSB. (The test that I used was a simple scan test with 10-100m keys.) > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956266#comment-15956266 ] Lars Hofhansl edited comment on HBASE-17877 at 4/5/17 4:01 AM: --- [~Apache9] that's what the Hadoop comparator does, which tested faster. :) As tests one can use PE and YCSB. (The test that I used was a simple scan test with 10-100m keys.) was (Author: lhofhansl): [~Apache9] that's what the Hadoop comparator does, which tested faster. :) As tests one can use PE and YCSB. > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956266#comment-15956266 ] Lars Hofhansl commented on HBASE-17877: --- [~Apache9] that's what the Hadoop comparator does, which tested faster. :) As tests one can use PE and YCSB. > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17766) Generate Javadoc for hbase-spark module
[
https://issues.apache.org/jira/browse/HBASE-17766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956255#comment-15956255
]
Hadoop QA commented on HBASE-17766:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 31s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 52s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 21s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
32s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 6s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
28s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 17s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 3s
{color} | {color:green} hbase-spark in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 127m 50s
{color} | {color:green} root in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 184m 42s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12861985/HBASE-17766-V1.patch |
| JIRA Issue | HBASE-17766 |
| Optional Tests | asflicense javac javadoc unit xml compile findbugs
hadoopcheck hbaseanti checkstyle |
| uname | Linux 31c8e2b32e2a 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
|
[jira] [Commented] (HBASE-17869) UnsafeAvailChecker wrongly returns false on ppc
[ https://issues.apache.org/jira/browse/HBASE-17869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956253#comment-15956253 ] Anoop Sam John commented on HBASE-17869: Ya the false path might be problematic. Need to look into the test and then code deeply.. I do not know FuzzyRowFilter code path in deep. > UnsafeAvailChecker wrongly returns false on ppc > --- > > Key: HBASE-17869 > URL: https://issues.apache.org/jira/browse/HBASE-17869 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.4 >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17869.patch > > > On ppc64 arch, java.nio.Bits.unaligned() wrongly returns false due to a JDK > bug. > https://bugs.openjdk.java.net/browse/JDK-8165231 > This causes some problem for HBase. i.e. FuzzyRowFilter test fails. > Fix it by providing a hard-code workaround for the JDK bug. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17863) Procedure V2: Some cleanup around Procedure.isFinished() and procedure executor
[
https://issues.apache.org/jira/browse/HBASE-17863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956246#comment-15956246
]
Hadoop QA commented on HBASE-17863:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 33s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 52s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 42s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
44s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 10s
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 27s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 56s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 31s
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 54s
{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 119m 59s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
58s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 189m 4s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.coprocessor.TestCoprocessorMetrics |
| | org.apache.hadoop.hbase.regionserver.wal.TestLogRolling |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12861983/HBASE-17863.v4.patch |
| JIRA Issue | HBASE-17863 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
[jira] [Updated] (HBASE-16775) Flakey test with TestExportSnapshot#testExportRetry and TestMobExportSnapshot#testExportRetry
[
https://issues.apache.org/jira/browse/HBASE-16775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Appy updated HBASE-16775:
-
Attachment: HBASE-16775.master.005.patch
> Flakey test with TestExportSnapshot#testExportRetry and
> TestMobExportSnapshot#testExportRetry
> --
>
> Key: HBASE-16775
> URL: https://issues.apache.org/jira/browse/HBASE-16775
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: Appy
> Attachments: disable.patch, HBASE-16775.master.001.patch,
> HBASE-16775.master.002.patch, HBASE-16775.master.003.patch,
> HBASE-16775.master.004.patch, HBASE-16775.master.005.patch
>
>
> The root cause is that conf.setInt("mapreduce.map.maxattempts", 10) is not
> taken by the mapper job, so the retry is actually 0. Debugging to see why
> this is the case.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17854) Use StealJobQueue in HFileCleaner after HBASE-17215
[
https://issues.apache.org/jira/browse/HBASE-17854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-17854:
--
Attachment: HBASE-17854.v5.patch
Fix the awkward error introduced by v4 patch that caused UT failure...
> Use StealJobQueue in HFileCleaner after HBASE-17215
> ---
>
> Key: HBASE-17854
> URL: https://issues.apache.org/jira/browse/HBASE-17854
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17854.patch, HBASE-17854.v2.patch,
> HBASE-17854.v3.patch, HBASE-17854.v4.patch, HBASE-17854.v5.patch
>
>
> In HBASE-17215 we use specific threads for deleting large/small (archived)
> hfiles, and will improve it from below aspects in this JIRA:
> 1. Using {{StealJobQueue}} to allow large file deletion thread to steal jobs
> from small queue, based on the experience that in real world there'll be much
> more small hfiles
> 2. {{StealJobQueue}} is a kind of {{PriorityQueue}}, so we could also delete
> from the larger file in the queues.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[
https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956166#comment-15956166
]
Duo Zhang commented on HBASE-17877:
---
{noformat}
-int j = minWords << 3; // Same as minWords * SIZEOF_LONG
-for (int i = 0; i < j; i += SIZEOF_LONG) {
+for (int i = 0; i < minWords * SIZEOF_LONG; i += SIZEOF_LONG) {
{noformat}
Why this change?
And mind sharing the code of your perf tests?
Thanks.
> Replace/improve HBase's byte[] comparator
> -
>
> Key: HBASE-17877
> URL: https://issues.apache.org/jira/browse/HBASE-17877
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Vikas Vishwakarma
> Attachments: 17877-1.2.patch
>
>
> [~vik.karma] did some extensive tests and found that Hadoop's version is
> faster - dramatically faster in some cases.
> Patch forthcoming.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17872) Make BBU#copyFromBufferToBuffer thread-safe
[
https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956156#comment-15956156
]
Chia-Ping Tsai commented on HBASE-17872:
BBU#copy*ToBuffer is widely used. For example, all of CellUtil#copy*To((Cell,
ByteBuffer, int) call it.
It is an ineffective way to fix all of them. We should define that the
BBU#copy*ToBuffer WON'T affect the position of any of the buffers.
> Make BBU#copyFromBufferToBuffer thread-safe
> ---
>
> Key: HBASE-17872
> URL: https://issues.apache.org/jira/browse/HBASE-17872
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0
>
>
> A case is shown below.
> We will get the wrong position of buffer in multithreaded environment, so the
> method makes the invalid cell in MSLAB.
> {noformat}
> public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int
> sourceOffset,
> int destinationOffset, int length) {
> if (in.hasArray() && out.hasArray()) {
> // ...
> } else if (UNSAFE_AVAIL) {
> // ...
> } else {
> int outOldPos = out.position();
> out.position(destinationOffset);
> ByteBuffer inDup = in.duplicate();
> inDup.position(sourceOffset).limit(sourceOffset + length);
> out.put(inDup);
> out.position(outOldPos);
> }
> return destinationOffset + length;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17872) Make BBU#copyFromBufferToBuffer thread-safe
[
https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955097#comment-15955097
]
Chia-Ping Tsai edited comment on HBASE-17872 at 4/5/17 1:26 AM:
The unsafe doesn't affect the position in BBU, but the others may affect the
position. We should unify the logic of all methods in BBU.
was (Author: chia7712):
The unsafe doesn't effect the position in BBU, but the others may effect the
position. We should unify the logic of all methods in BBU.
> Make BBU#copyFromBufferToBuffer thread-safe
> ---
>
> Key: HBASE-17872
> URL: https://issues.apache.org/jira/browse/HBASE-17872
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0
>
>
> A case is shown below.
> We will get the wrong position of buffer in multithreaded environment, so the
> method makes the invalid cell in MSLAB.
> {noformat}
> public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int
> sourceOffset,
> int destinationOffset, int length) {
> if (in.hasArray() && out.hasArray()) {
> // ...
> } else if (UNSAFE_AVAIL) {
> // ...
> } else {
> int outOldPos = out.position();
> out.position(destinationOffset);
> ByteBuffer inDup = in.duplicate();
> inDup.position(sourceOffset).limit(sourceOffset + length);
> out.put(inDup);
> out.position(outOldPos);
> }
> return destinationOffset + length;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-14141) HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits from backed up tables
[
https://issues.apache.org/jira/browse/HBASE-14141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-14141:
---
Summary: HBase Backup/Restore Phase 3: Filter WALs on backup to include
only edits from backed up tables (was: HBase Backup/Restore Phase 3: Filter
WALs on backup to include only edits from backup tables)
> HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits
> from backed up tables
> ---
>
> Key: HBASE-14141
> URL: https://issues.apache.org/jira/browse/HBASE-14141
> Project: HBase
> Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Blocker
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-14141.HBASE-14123.v1.patch, HBASE-14141.v1.patch,
> HBASE-14141.v2.patch, HBASE-14141.v4.patch, HBASE-14141.v5.patch,
> HBASE-14141.v6.patch
>
>
> h2. High level design overview
> * When incremental backup request comes for tables {t} we select all the
> tables already registered in a backup system - {T} and union them with {t},
> which results in a new table set - U(t, T)
> * For every table K from U(t,T) we perform the following:
> ** Convert new WAL files into HFile applying table filter K (only edits for
> table T will pass the filter)
> ** Move these HFile(s) to backup destination
> During restore (incremental):
> * We run full restore first
> * Then collect all HFiles from intermediate incremental images and run them
> through HFileSplitterJob, which splits files into a current tables region
> boundaries
> * Load these files using LoadIncrementalHFiles tool
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-14141) HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits from backup tables
[
https://issues.apache.org/jira/browse/HBASE-14141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956143#comment-15956143
]
Hadoop QA commented on HBASE-14141:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 32s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 113m 26s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 155m 41s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.client.TestAvoidCellReferencesIntoShippedBlocks |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12861970/HBASE-14141.v6.patch |
| JIRA Issue | HBASE-14141 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 682c7319251d 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / e916b79 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/6315/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/6315/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6315/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6315/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically
[jira] [Updated] (HBASE-17766) Generate Javadoc for hbase-spark module
[ https://issues.apache.org/jira/browse/HBASE-17766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liang updated HBASE-17766: - Status: Patch Available (was: Open) > Generate Javadoc for hbase-spark module > > > Key: HBASE-17766 > URL: https://issues.apache.org/jira/browse/HBASE-17766 > Project: HBase > Issue Type: Bug >Reporter: Yi Liang >Assignee: Yi Liang > Fix For: 2.0.0 > > Attachments: HBASE-17766-V1.patch, spark-api.jpg, user-api.jpg > > > Scala classes in hbase-spark module aren't showing up in our API docs nor > our internal API docs. see https://hbase.apache.org/apidocs/ -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17766) Generate Javadoc for hbase-spark module
[ https://issues.apache.org/jira/browse/HBASE-17766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liang updated HBASE-17766: - Attachment: HBASE-17766-V1.patch > Generate Javadoc for hbase-spark module > > > Key: HBASE-17766 > URL: https://issues.apache.org/jira/browse/HBASE-17766 > Project: HBase > Issue Type: Bug >Reporter: Yi Liang >Assignee: Yi Liang > Fix For: 2.0.0 > > Attachments: HBASE-17766-V1.patch, spark-api.jpg, user-api.jpg > > > Scala classes in hbase-spark module aren't showing up in our API docs nor > our internal API docs. see https://hbase.apache.org/apidocs/ -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17766) Generate Javadoc for hbase-spark module
[ https://issues.apache.org/jira/browse/HBASE-17766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956123#comment-15956123 ] Yi Liang commented on HBASE-17766: -- I also investigate the project Apache Flink, which also have java mixed code, and they also use this com.typesafe.genjavadoc plugin, and use mvn javadoc:aggregate to generate javadoc, however HBase use mvn site to generate javadoc. The reason why mvn site can not generate javadoc successfully when using genjavadoc plugin in hbase for the following reasons: (1) Flink does not have annotations, but we have annotation like stable, privatelimited (2) The fake java code generated by com.typesafe.genjavadoc plugin can not be used for mvn site, and can only be used by mvn javadoc. Because the generated fake code only have some definition and have no implementation, and the fake code can not pass the check by mvn site. I will just provide a patch about my proposal, so you guys can have a better understanding. > Generate Javadoc for hbase-spark module > > > Key: HBASE-17766 > URL: https://issues.apache.org/jira/browse/HBASE-17766 > Project: HBase > Issue Type: Bug >Reporter: Yi Liang >Assignee: Yi Liang > Fix For: 2.0.0 > > Attachments: HBASE-17766-V1.patch, spark-api.jpg, user-api.jpg > > > Scala classes in hbase-spark module aren't showing up in our API docs nor > our internal API docs. see https://hbase.apache.org/apidocs/ -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17863) Procedure V2: Some cleanup around Procedure.isFinished() and procedure executor
[ https://issues.apache.org/jira/browse/HBASE-17863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Umesh Agashe updated HBASE-17863: - Attachment: HBASE-17863.v4.patch minor fix > Procedure V2: Some cleanup around Procedure.isFinished() and procedure > executor > --- > > Key: HBASE-17863 > URL: https://issues.apache.org/jira/browse/HBASE-17863 > Project: HBase > Issue Type: Bug > Components: proc-v2 >Reporter: Umesh Agashe >Assignee: Umesh Agashe > Attachments: HBASE-17863.v1.patch, HBASE-17863.v2.patch, > HBASE-17863.v3.patch, HBASE-17863.v3.patch, HBASE-17863.v4.patch > > > Clean up around isFinished() and procedure executor -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-14141) HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits from backup tables
[
https://issues.apache.org/jira/browse/HBASE-14141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956102#comment-15956102
]
Vladimir Rodionov commented on HBASE-14141:
---
OK, go ahead
> HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits
> from backup tables
>
>
> Key: HBASE-14141
> URL: https://issues.apache.org/jira/browse/HBASE-14141
> Project: HBase
> Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Blocker
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-14141.HBASE-14123.v1.patch, HBASE-14141.v1.patch,
> HBASE-14141.v2.patch, HBASE-14141.v4.patch, HBASE-14141.v5.patch,
> HBASE-14141.v6.patch
>
>
> h2. High level design overview
> * When incremental backup request comes for tables {t} we select all the
> tables already registered in a backup system - {T} and union them with {t},
> which results in a new table set - U(t, T)
> * For every table K from U(t,T) we perform the following:
> ** Convert new WAL files into HFile applying table filter K (only edits for
> table T will pass the filter)
> ** Move these HFile(s) to backup destination
> During restore (incremental):
> * We run full restore first
> * Then collect all HFiles from intermediate incremental images and run them
> through HFileSplitterJob, which splits files into a current tables region
> boundaries
> * Load these files using LoadIncrementalHFiles tool
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-14141) HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits from backup tables
[
https://issues.apache.org/jira/browse/HBASE-14141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956092#comment-15956092
]
Ted Yu commented on HBASE-14141:
Vlad:
How about making the interval 2 seconds instead of 1 second ?
If you agree, I can make the change at time of commit.
> HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits
> from backup tables
>
>
> Key: HBASE-14141
> URL: https://issues.apache.org/jira/browse/HBASE-14141
> Project: HBase
> Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Blocker
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-14141.HBASE-14123.v1.patch, HBASE-14141.v1.patch,
> HBASE-14141.v2.patch, HBASE-14141.v4.patch, HBASE-14141.v5.patch,
> HBASE-14141.v6.patch
>
>
> h2. High level design overview
> * When incremental backup request comes for tables {t} we select all the
> tables already registered in a backup system - {T} and union them with {t},
> which results in a new table set - U(t, T)
> * For every table K from U(t,T) we perform the following:
> ** Convert new WAL files into HFile applying table filter K (only edits for
> table T will pass the filter)
> ** Move these HFile(s) to backup destination
> During restore (incremental):
> * We run full restore first
> * Then collect all HFiles from intermediate incremental images and run them
> through HFileSplitterJob, which splits files into a current tables region
> boundaries
> * Load these files using LoadIncrementalHFiles tool
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17800) [C++] handle exceptions in client RPC
[ https://issues.apache.org/jira/browse/HBASE-17800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956020#comment-15956020 ] Xiaobing Zhou edited comment on HBASE-17800 at 4/4/17 11:45 PM: v2 addressed the two points, thanks. was (Author: xiaobingo): v2 address the two points, thanks. > [C++] handle exceptions in client RPC > - > > Key: HBASE-17800 > URL: https://issues.apache.org/jira/browse/HBASE-17800 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HBASE-17800-HBASE-14850.000.patch, > HBASE-17800-HBASE-14850.001.patch, HBASE-17800-HBASE-14850.002.patch > > > Exceptions are ignored in current client RPC. They should be handled properly > to be consumed by RPC retry or propagated up to APIs. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956065#comment-15956065 ] Lars Hofhansl commented on HBASE-17877: --- [~vik.karma], can you attach your performance findings? (micro benchmarks and end-to-end) In my own test I use to vet perf changes I found a 20-30% performance improvement - but I am using short key. The effect will be much less pronounced for longer keys (where memory pure bandwidth becomes the dominating factor) > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956060#comment-15956060 ] Lars Hofhansl edited comment on HBASE-17877 at 4/4/17 11:41 PM: Here's a patch I made that avoids any additional Guava dependencies. All the credit goes to [~vik.karma]. He did the testing. Found the better Hadoop implementation, etc. I assigned the issue to Vikas. was (Author: lhofhansl): Here's a patch I made that avoid any additional Guava dependencies. All the credit goes to [~vik.karma]. He did the testing. Found the better Hadoop implementation, etc. > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17869) UnsafeAvailChecker wrongly returns false on ppc
[ https://issues.apache.org/jira/browse/HBASE-17869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956062#comment-15956062 ] Jerry He commented on HBASE-17869: -- On a x86_64 box, if I hard-code UnsafeAvailChecker.unaligned() to false, the same tests fail. > UnsafeAvailChecker wrongly returns false on ppc > --- > > Key: HBASE-17869 > URL: https://issues.apache.org/jira/browse/HBASE-17869 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.4 >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17869.patch > > > On ppc64 arch, java.nio.Bits.unaligned() wrongly returns false due to a JDK > bug. > https://bugs.openjdk.java.net/browse/JDK-8165231 > This causes some problem for HBase. i.e. FuzzyRowFilter test fails. > Fix it by providing a hard-code workaround for the JDK bug. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-17877: -- Attachment: 17877-1.2.patch Here's a patch I made that avoid any additional Guava dependencies. All the credit goes to [~vik.karma]. He did the testing. Found the better Hadoop implementation, etc. > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl > Attachments: 17877-1.2.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl reassigned HBASE-17877: - Assignee: Lars Hofhansl > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Attachments: 17877-1.2.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HBASE-17877) Replace/improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl reassigned HBASE-17877: - Assignee: Vikas Vishwakarma (was: Lars Hofhansl) > Replace/improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma > Attachments: 17877-1.2.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17877) Replace/improve HBase's byte[] comparator
Lars Hofhansl created HBASE-17877: - Summary: Replace/improve HBase's byte[] comparator Key: HBASE-17877 URL: https://issues.apache.org/jira/browse/HBASE-17877 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl [~vik.karma] did some extensive tests and found that Hadoop's version is faster - dramatically faster in some cases. Patch forthcoming. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17800) [C++] handle exceptions in client RPC
[ https://issues.apache.org/jira/browse/HBASE-17800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956020#comment-15956020 ] Xiaobing Zhou commented on HBASE-17800: --- v2 address the two points, thanks. > [C++] handle exceptions in client RPC > - > > Key: HBASE-17800 > URL: https://issues.apache.org/jira/browse/HBASE-17800 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HBASE-17800-HBASE-14850.000.patch, > HBASE-17800-HBASE-14850.001.patch, HBASE-17800-HBASE-14850.002.patch > > > Exceptions are ignored in current client RPC. They should be handled properly > to be consumed by RPC retry or propagated up to APIs. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17800) [C++] handle exceptions in client RPC
[ https://issues.apache.org/jira/browse/HBASE-17800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HBASE-17800: -- Attachment: HBASE-17800-HBASE-14850.002.patch > [C++] handle exceptions in client RPC > - > > Key: HBASE-17800 > URL: https://issues.apache.org/jira/browse/HBASE-17800 > Project: HBase > Issue Type: Sub-task >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HBASE-17800-HBASE-14850.000.patch, > HBASE-17800-HBASE-14850.001.patch, HBASE-17800-HBASE-14850.002.patch > > > Exceptions are ignored in current client RPC. They should be handled properly > to be consumed by RPC retry or propagated up to APIs. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-14925) Develop HBase shell command/tool to list table's region info through command line
[
https://issues.apache.org/jira/browse/HBASE-14925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956003#comment-15956003
]
Hadoop QA commented on HBASE-14925:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} rubocop {color} | {color:blue} 0m 0s
{color} | {color:blue} rubocop was not available. {color} |
| {color:blue}0{color} | {color:blue} ruby-lint {color} | {color:blue} 0m 0s
{color} | {color:blue} Ruby-lint was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 8s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 3s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 10s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 44s
{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 46s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12861962/HBASE-14925.002.patch
|
| JIRA Issue | HBASE-14925 |
| Optional Tests | asflicense javac javadoc unit rubocop ruby_lint |
| uname | Linux 2dafc8f9070b 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / e916b79 |
| Default Java | 1.8.0_121 |
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/6314/artifact/patchprocess/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6314/testReport/ |
| modules | C: hbase-shell U: hbase-shell |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6314/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Develop HBase shell command/tool to list table's region info through command
> line
> -
>
> Key: HBASE-14925
> URL: https://issues.apache.org/jira/browse/HBASE-14925
> Project: HBase
> Issue Type: Improvement
> Components: shell
>Reporter: Romil Choksi
>Assignee: Karan Mehta
> Attachments: HBASE-14925.002.patch, HBASE-14925.patch
>
>
> I am going through the hbase shell commands to see if there is anything I can
> use to get all the regions info just for a particular table. I don’t see any
> such command that provides me that information.
> It would be better to have a command that provides region info, start key,
> end key etc taking a table name as the input parameter. This is available
> through HBase UI on clicking on a particular table's link
> A tool/shell command to get a list of regions for a table or all tables in a
> tabular structured output (that is machine readable)
[jira] [Commented] (HBASE-17860) Implement secure native client connection
[
https://issues.apache.org/jira/browse/HBASE-17860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955999#comment-15955999
]
Ted Yu commented on HBASE-17860:
[~eclark] was asking why not casting the callback methods.
I answered on RB but the compilation errors are not properly formatted.
However, the assignment of callback functions triggers the following
compilation errors:
{code}
connection/connection-factory.cc:164:22: error: reinterpret_cast from 'int
(SaslHandler::*)(void *, int, const char *)' to 'sasl_callback_ft' (aka 'int
(*)()') is not allowed
callback->proc = reinterpret_cast
(::SaslLogFn);
^~~~
connection/connection-factory.cc:169:22: error: cannot cast from type 'int
(SaslHandler::*)(void *, const char **)' to pointer type 'sasl_callback_ft'
(aka 'int (*)()')
callback->proc = (sasl_callback_ft) ::GetPluginPath;
^~
connection/connection-factory.cc:175:22: error: cannot cast from type 'int
(SaslHandler::*)(void *, int, const char **, unsigned int *)' to pointer type
'sasl_callback_ft' (aka 'int (*)()')
callback->proc = (sasl_callback_ft) ::Simple;
{code}
> Implement secure native client connection
> -
>
> Key: HBASE-17860
> URL: https://issues.apache.org/jira/browse/HBASE-17860
> Project: HBase
> Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Attachments: 17860.v2.txt, 17860.v3.txt, 17860.v4.txt
>
>
> So far, the native client communicates with insecure cluster.
> This JIRA is to add secure connection support for native client using Cyrus
> library.
> The work is based on earlier implementation and is redone via wangle and
> folly frameworks.
> Thanks to [~devaraj] who started the initiative.
> Here is high level description of the design:
> * SaslHandler is declared as:
> {code}
> class SaslHandler
> : public wangle::HandlerAdapter std::unique_ptr>{
> {code}
> It would be inserted between EventBaseHandler and
> LengthFieldBasedFrameDecoder in the pipeline (via
> ConnectionFactory::Connect())
> * SaslHandler would intercept writes to server by buffering the IOBuf's and
> start the handshake process (via sasl_client_XX calls provided by Cyrus)
> * after handshake is complete, SaslHandler would send the buffered IOBuf's to
> server and act as pass-thru from then on
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-14141) HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits from backup tables
[
https://issues.apache.org/jira/browse/HBASE-14141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladimir Rodionov updated HBASE-14141:
--
Attachment: (was: HBASE-14141.v6.patch)
> HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits
> from backup tables
>
>
> Key: HBASE-14141
> URL: https://issues.apache.org/jira/browse/HBASE-14141
> Project: HBase
> Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Blocker
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-14141.HBASE-14123.v1.patch, HBASE-14141.v1.patch,
> HBASE-14141.v2.patch, HBASE-14141.v4.patch, HBASE-14141.v5.patch,
> HBASE-14141.v6.patch
>
>
> h2. High level design overview
> * When incremental backup request comes for tables {t} we select all the
> tables already registered in a backup system - {T} and union them with {t},
> which results in a new table set - U(t, T)
> * For every table K from U(t,T) we perform the following:
> ** Convert new WAL files into HFile applying table filter K (only edits for
> table T will pass the filter)
> ** Move these HFile(s) to backup destination
> During restore (incremental):
> * We run full restore first
> * Then collect all HFiles from intermediate incremental images and run them
> through HFileSplitterJob, which splits files into a current tables region
> boundaries
> * Load these files using LoadIncrementalHFiles tool
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-14141) HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits from backup tables
[
https://issues.apache.org/jira/browse/HBASE-14141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladimir Rodionov updated HBASE-14141:
--
Attachment: HBASE-14141.v6.patch
> HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits
> from backup tables
>
>
> Key: HBASE-14141
> URL: https://issues.apache.org/jira/browse/HBASE-14141
> Project: HBase
> Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Blocker
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-14141.HBASE-14123.v1.patch, HBASE-14141.v1.patch,
> HBASE-14141.v2.patch, HBASE-14141.v4.patch, HBASE-14141.v5.patch,
> HBASE-14141.v6.patch
>
>
> h2. High level design overview
> * When incremental backup request comes for tables {t} we select all the
> tables already registered in a backup system - {T} and union them with {t},
> which results in a new table set - U(t, T)
> * For every table K from U(t,T) we perform the following:
> ** Convert new WAL files into HFile applying table filter K (only edits for
> table T will pass the filter)
> ** Move these HFile(s) to backup destination
> During restore (incremental):
> * We run full restore first
> * Then collect all HFiles from intermediate incremental images and run them
> through HFileSplitterJob, which splits files into a current tables region
> boundaries
> * Load these files using LoadIncrementalHFiles tool
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-14141) HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits from backup tables
[
https://issues.apache.org/jira/browse/HBASE-14141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955988#comment-15955988
]
Vladimir Rodionov commented on HBASE-14141:
---
Checked 4 failed tests locally. They passed.
Submitted patch again.
> HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits
> from backup tables
>
>
> Key: HBASE-14141
> URL: https://issues.apache.org/jira/browse/HBASE-14141
> Project: HBase
> Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Blocker
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-14141.HBASE-14123.v1.patch, HBASE-14141.v1.patch,
> HBASE-14141.v2.patch, HBASE-14141.v4.patch, HBASE-14141.v5.patch,
> HBASE-14141.v6.patch
>
>
> h2. High level design overview
> * When incremental backup request comes for tables {t} we select all the
> tables already registered in a backup system - {T} and union them with {t},
> which results in a new table set - U(t, T)
> * For every table K from U(t,T) we perform the following:
> ** Convert new WAL files into HFile applying table filter K (only edits for
> table T will pass the filter)
> ** Move these HFile(s) to backup destination
> During restore (incremental):
> * We run full restore first
> * Then collect all HFiles from intermediate incremental images and run them
> through HFileSplitterJob, which splits files into a current tables region
> boundaries
> * Load these files using LoadIncrementalHFiles tool
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17869) UnsafeAvailChecker wrongly returns false on ppc
[
https://issues.apache.org/jira/browse/HBASE-17869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955986#comment-15955986
]
Jerry He commented on HBASE-17869:
--
Yeah, I had the same question. Two code paths (true or false) should yield the
same results.
Since the 'true' route is more commonly used, can we assume it has the correct
results?
Then there could be a bug somewhere in the 'false' route. In the FuzzyRowFilter?
{code}
java.lang.AssertionError: expected: but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at
org.apache.hadoop.hbase.filter.TestFuzzyRowFilter.testSatisfiesForward(TestFuzzyRowFilter.java:80)
{code}
{code}
java.lang.AssertionError: expected: but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at
org.apache.hadoop.hbase.filter.TestFuzzyRowFilter.testSatisfiesReverse(TestFuzzyRowFilter.java:120)
{code}
> UnsafeAvailChecker wrongly returns false on ppc
> ---
>
> Key: HBASE-17869
> URL: https://issues.apache.org/jira/browse/HBASE-17869
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.4
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17869.patch
>
>
> On ppc64 arch, java.nio.Bits.unaligned() wrongly returns false due to a JDK
> bug.
> https://bugs.openjdk.java.net/browse/JDK-8165231
> This causes some problem for HBase. i.e. FuzzyRowFilter test fails.
> Fix it by providing a hard-code workaround for the JDK bug.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-14141) HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits from backup tables
[
https://issues.apache.org/jira/browse/HBASE-14141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladimir Rodionov updated HBASE-14141:
--
Status: Patch Available (was: Open)
> HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits
> from backup tables
>
>
> Key: HBASE-14141
> URL: https://issues.apache.org/jira/browse/HBASE-14141
> Project: HBase
> Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Blocker
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-14141.HBASE-14123.v1.patch, HBASE-14141.v1.patch,
> HBASE-14141.v2.patch, HBASE-14141.v4.patch, HBASE-14141.v5.patch,
> HBASE-14141.v6.patch
>
>
> h2. High level design overview
> * When incremental backup request comes for tables {t} we select all the
> tables already registered in a backup system - {T} and union them with {t},
> which results in a new table set - U(t, T)
> * For every table K from U(t,T) we perform the following:
> ** Convert new WAL files into HFile applying table filter K (only edits for
> table T will pass the filter)
> ** Move these HFile(s) to backup destination
> During restore (incremental):
> * We run full restore first
> * Then collect all HFiles from intermediate incremental images and run them
> through HFileSplitterJob, which splits files into a current tables region
> boundaries
> * Load these files using LoadIncrementalHFiles tool
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-14141) HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits from backup tables
[
https://issues.apache.org/jira/browse/HBASE-14141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladimir Rodionov updated HBASE-14141:
--
Status: Open (was: Patch Available)
> HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits
> from backup tables
>
>
> Key: HBASE-14141
> URL: https://issues.apache.org/jira/browse/HBASE-14141
> Project: HBase
> Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Blocker
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-14141.HBASE-14123.v1.patch, HBASE-14141.v1.patch,
> HBASE-14141.v2.patch, HBASE-14141.v4.patch, HBASE-14141.v5.patch,
> HBASE-14141.v6.patch
>
>
> h2. High level design overview
> * When incremental backup request comes for tables {t} we select all the
> tables already registered in a backup system - {T} and union them with {t},
> which results in a new table set - U(t, T)
> * For every table K from U(t,T) we perform the following:
> ** Convert new WAL files into HFile applying table filter K (only edits for
> table T will pass the filter)
> ** Move these HFile(s) to backup destination
> During restore (incremental):
> * We run full restore first
> * Then collect all HFiles from intermediate incremental images and run them
> through HFileSplitterJob, which splits files into a current tables region
> boundaries
> * Load these files using LoadIncrementalHFiles tool
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-14141) HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits from backup tables
[
https://issues.apache.org/jira/browse/HBASE-14141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955979#comment-15955979
]
Ted Yu commented on HBASE-14141:
See if failed tests pass locally.
If they do, submit for another QA run.
> HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits
> from backup tables
>
>
> Key: HBASE-14141
> URL: https://issues.apache.org/jira/browse/HBASE-14141
> Project: HBase
> Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Blocker
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-14141.HBASE-14123.v1.patch, HBASE-14141.v1.patch,
> HBASE-14141.v2.patch, HBASE-14141.v4.patch, HBASE-14141.v5.patch,
> HBASE-14141.v6.patch
>
>
> h2. High level design overview
> * When incremental backup request comes for tables {t} we select all the
> tables already registered in a backup system - {T} and union them with {t},
> which results in a new table set - U(t, T)
> * For every table K from U(t,T) we perform the following:
> ** Convert new WAL files into HFile applying table filter K (only edits for
> table T will pass the filter)
> ** Move these HFile(s) to backup destination
> During restore (incremental):
> * We run full restore first
> * Then collect all HFiles from intermediate incremental images and run them
> through HFileSplitterJob, which splits files into a current tables region
> boundaries
> * Load these files using LoadIncrementalHFiles tool
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17860) Implement secure native client connection
[
https://issues.apache.org/jira/browse/HBASE-17860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955969#comment-15955969
]
Ted Yu commented on HBASE-17860:
https://reviews.apache.org/r/58197/
> Implement secure native client connection
> -
>
> Key: HBASE-17860
> URL: https://issues.apache.org/jira/browse/HBASE-17860
> Project: HBase
> Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Attachments: 17860.v2.txt, 17860.v3.txt, 17860.v4.txt
>
>
> So far, the native client communicates with insecure cluster.
> This JIRA is to add secure connection support for native client using Cyrus
> library.
> The work is based on earlier implementation and is redone via wangle and
> folly frameworks.
> Thanks to [~devaraj] who started the initiative.
> Here is high level description of the design:
> * SaslHandler is declared as:
> {code}
> class SaslHandler
> : public wangle::HandlerAdapter std::unique_ptr>{
> {code}
> It would be inserted between EventBaseHandler and
> LengthFieldBasedFrameDecoder in the pipeline (via
> ConnectionFactory::Connect())
> * SaslHandler would intercept writes to server by buffering the IOBuf's and
> start the handshake process (via sasl_client_XX calls provided by Cyrus)
> * after handshake is complete, SaslHandler would send the buffered IOBuf's to
> server and act as pass-thru from then on
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17860) Implement secure native client connection
[
https://issues.apache.org/jira/browse/HBASE-17860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-17860:
---
Attachment: 17860.v4.txt
> Implement secure native client connection
> -
>
> Key: HBASE-17860
> URL: https://issues.apache.org/jira/browse/HBASE-17860
> Project: HBase
> Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Attachments: 17860.v2.txt, 17860.v3.txt, 17860.v4.txt
>
>
> So far, the native client communicates with insecure cluster.
> This JIRA is to add secure connection support for native client using Cyrus
> library.
> The work is based on earlier implementation and is redone via wangle and
> folly frameworks.
> Thanks to [~devaraj] who started the initiative.
> Here is high level description of the design:
> * SaslHandler is declared as:
> {code}
> class SaslHandler
> : public wangle::HandlerAdapter std::unique_ptr>{
> {code}
> It would be inserted between EventBaseHandler and
> LengthFieldBasedFrameDecoder in the pipeline (via
> ConnectionFactory::Connect())
> * SaslHandler would intercept writes to server by buffering the IOBuf's and
> start the handshake process (via sasl_client_XX calls provided by Cyrus)
> * after handshake is complete, SaslHandler would send the buffered IOBuf's to
> server and act as pass-thru from then on
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-14925) Develop HBase shell command/tool to list table's region info through command line
[ https://issues.apache.org/jira/browse/HBASE-14925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955953#comment-15955953 ] Karan Mehta commented on HBASE-14925: - Added a new patch with required changes. Can you take a look [~rstokes] and [~ashish singhi]? > Develop HBase shell command/tool to list table's region info through command > line > - > > Key: HBASE-14925 > URL: https://issues.apache.org/jira/browse/HBASE-14925 > Project: HBase > Issue Type: Improvement > Components: shell >Reporter: Romil Choksi >Assignee: Karan Mehta > Attachments: HBASE-14925.002.patch, HBASE-14925.patch > > > I am going through the hbase shell commands to see if there is anything I can > use to get all the regions info just for a particular table. I don’t see any > such command that provides me that information. > It would be better to have a command that provides region info, start key, > end key etc taking a table name as the input parameter. This is available > through HBase UI on clicking on a particular table's link > A tool/shell command to get a list of regions for a table or all tables in a > tabular structured output (that is machine readable) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-14925) Develop HBase shell command/tool to list table's region info through command line
[ https://issues.apache.org/jira/browse/HBASE-14925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karan Mehta updated HBASE-14925: Attachment: HBASE-14925.002.patch > Develop HBase shell command/tool to list table's region info through command > line > - > > Key: HBASE-14925 > URL: https://issues.apache.org/jira/browse/HBASE-14925 > Project: HBase > Issue Type: Improvement > Components: shell >Reporter: Romil Choksi >Assignee: Karan Mehta > Attachments: HBASE-14925.002.patch, HBASE-14925.patch > > > I am going through the hbase shell commands to see if there is anything I can > use to get all the regions info just for a particular table. I don’t see any > such command that provides me that information. > It would be better to have a command that provides region info, start key, > end key etc taking a table name as the input parameter. This is available > through HBase UI on clicking on a particular table's link > A tool/shell command to get a list of regions for a table or all tables in a > tabular structured output (that is machine readable) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-14141) HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits from backup tables
[
https://issues.apache.org/jira/browse/HBASE-14141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955921#comment-15955921
]
Hadoop QA commented on HBASE-14141:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 36s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 3s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 142m 40s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
5s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 185m 29s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestBlockEvictionFromClient |
| | hadoop.hbase.regionserver.wal.TestAsyncLogRollPeriod |
| | hadoop.hbase.filter.TestColumnRangeFilter |
| | hadoop.hbase.quotas.TestQuotaThrottle |
| Timed out junit tests | org.apache.hadoop.hbase.client.TestAsyncTableScanAll |
| | org.apache.hadoop.hbase.client.TestMultiRespectsLimits |
| | org.apache.hadoop.hbase.util.TestMiniClusterLoadEncoded |
| | org.apache.hadoop.hbase.regionserver.wal.TestLogRolling |
| | org.apache.hadoop.hbase.TestAcidGuarantees |
| | org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter |
| |
org.apache.hadoop.hbase.client.TestAsyncTableGetMultiThreadedWithEagerCompaction
|
| | org.apache.hadoop.hbase.client.TestAsyncTable |
| | org.apache.hadoop.hbase.filter.TestMultiRowRangeFilter |
| | org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel |
| | org.apache.hadoop.hbase.client.TestHCM |
| | org.apache.hadoop.hbase.client.TestSizeFailures |
| |
org.apache.hadoop.hbase.client.TestCloneSnapshotFromClientWithRegionReplicas |
| | org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClient |
| | org.apache.hadoop.hbase.regionserver.TestPriorityRpc |
| | org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover |
| | org.apache.hadoop.hbase.util.TestIdReadWriteLock |
\\
\\
|| Subsystem || Report/Notes ||
|
[jira] [Commented] (HBASE-17721) Provide streaming APIs with SSL/TLS
[ https://issues.apache.org/jira/browse/HBASE-17721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955875#comment-15955875 ] stack commented on HBASE-17721: --- Thanks for update [~alexaraujo]. > Provide streaming APIs with SSL/TLS > --- > > Key: HBASE-17721 > URL: https://issues.apache.org/jira/browse/HBASE-17721 > Project: HBase > Issue Type: Umbrella > Components: rpc, security >Reporter: Alex Araujo >Assignee: Alex Araujo > Fix For: 2.0.0 > > > Umbrella to add optional client/server streaming capabilities to HBase. > This would allow bandwidth to be used more efficiently for certain > operations, and allow clients to use SSL/TLS for authentication and > encryption. > Desired client/server scaffolding: > - HTTP/2 support > - Protocol negotiation (blocking vs streaming, auth, encryption, etc.) > - TLS/SSL support > - Streaming RPC support > Possibilities (and their tradeoffs): > - gRPC: Some initial work and discussion on HBASE-13467 (Prototype using GRPC > as IPC mechanism) > -- Has most or all of the desired scaffolding > -- Adds additional g* dependencies. Compat story for g* dependencies not > always ideal > - Custom HTTP/2 based client/server APIs > -- More control over compat story > -- Non-trivial to build scaffolding; might reinvent wheels along the way > - Others? > Related Jiras that might be rolled in as sub-tasks (or closed/replaced with > new ones): > HBASE-17708 (Expose config to set two-way auth over TLS in HttpServer and add > a test) > HBASE-8691 (High-Throughput Streaming Scan API) > HBASE-14899 (Create custom Streaming ReplicationEndpoint) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17876) Release 1.2.6
Sean Busbey created HBASE-17876: --- Summary: Release 1.2.6 Key: HBASE-17876 URL: https://issues.apache.org/jira/browse/HBASE-17876 Project: HBase Issue Type: Task Components: community Reporter: Sean Busbey Assignee: Sean Busbey Fix For: 1.2.6 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17721) Provide streaming APIs with SSL/TLS
[ https://issues.apache.org/jira/browse/HBASE-17721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955797#comment-15955797 ] Alex Araujo commented on HBASE-17721: - I was able to compile existing protos and grpc protos within hbase-protocol-shaded using grpc 1.1.2 and protobuf 3.1.0. I got hung up on shading grpc and its dependencies (grpc-core depends on com.google.code.findbugs:jsr305, which HBase does not allow). [~apurtell] suggested building a grpc-shaded module outside of HBase and pulling it in as a dependency rather than building it. I was planning on trying that or punting on shading grpc for the time being to get some benchmarks for HBASE-13467. Given other priorities, I'll likely get to that in a couple of weeks. > Provide streaming APIs with SSL/TLS > --- > > Key: HBASE-17721 > URL: https://issues.apache.org/jira/browse/HBASE-17721 > Project: HBase > Issue Type: Umbrella > Components: rpc, security >Reporter: Alex Araujo >Assignee: Alex Araujo > Fix For: 2.0.0 > > > Umbrella to add optional client/server streaming capabilities to HBase. > This would allow bandwidth to be used more efficiently for certain > operations, and allow clients to use SSL/TLS for authentication and > encryption. > Desired client/server scaffolding: > - HTTP/2 support > - Protocol negotiation (blocking vs streaming, auth, encryption, etc.) > - TLS/SSL support > - Streaming RPC support > Possibilities (and their tradeoffs): > - gRPC: Some initial work and discussion on HBASE-13467 (Prototype using GRPC > as IPC mechanism) > -- Has most or all of the desired scaffolding > -- Adds additional g* dependencies. Compat story for g* dependencies not > always ideal > - Custom HTTP/2 based client/server APIs > -- More control over compat story > -- Non-trivial to build scaffolding; might reinvent wheels along the way > - Others? > Related Jiras that might be rolled in as sub-tasks (or closed/replaced with > new ones): > HBASE-17708 (Expose config to set two-way auth over TLS in HttpServer and add > a test) > HBASE-8691 (High-Throughput Streaming Scan API) > HBASE-14899 (Create custom Streaming ReplicationEndpoint) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-14925) Develop HBase shell command/tool to list table's region info through command line
[
https://issues.apache.org/jira/browse/HBASE-14925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955753#comment-15955753
]
Karan Mehta commented on HBASE-14925:
-
Thanks [~rstokes] for pointing out the use cases. The use cases typically
suggest that the command will be not be used really often. We typically want
the data to be fast, but no specific hard deadlines.
This code does consider the fact that if region server is not supplied, then
information should be returned for all regions. {{start_with}} would match all
the results if provided with an empty string.
This patch is actually optimized for queries that just want all the regions of
a particular table. For the other part, if a typical table has many regions,
then we will have to iterate through each of them and find the relevant ones,
which may be expensive if a single table has plenty of regions. The approach is
similar to the code that {{table.jsp}} uses to display the information in
WEB-UI, the same API's are used. If WEB-UI seems appropriate in terms of
latency, this one also should not be, I feel. Please suggest your comments on
this one.
Moreover, if the following approach is followed as previous patch
{code}
for server in cluster_status.getServers()
for name,region in cluster_status.getLoad(server).getRegionsLoad()
region_name = region.getNameAsString()
regionStoreFileSize = region.getStorefileSizeMB()
regionRequests = region.getRequestsCount()
if region_name.start_with? tgtTable
results << { "server" => server, "name" => region_name, "size" =>
regionStoreFileSize, "requests" => regionRequests }
end
end
{code}
It is difficult to retrieve the {{EndKey}} of a particular region since
{{HRegionInfo}} is not present anywhere. {{StartKey}} is available if we call
{{getLoad(ServerName).getRegionsLoad()}}, retrieve the region name and parse
it. I believe {{EndKey}} is something that we are required to display as well.
This patch can be used if we want to optimize the queries that filter by both
tableName and regionServerName since that condition check can be done before
the start of second loop.
> Develop HBase shell command/tool to list table's region info through command
> line
> -
>
> Key: HBASE-14925
> URL: https://issues.apache.org/jira/browse/HBASE-14925
> Project: HBase
> Issue Type: Improvement
> Components: shell
>Reporter: Romil Choksi
>Assignee: Karan Mehta
> Attachments: HBASE-14925.patch
>
>
> I am going through the hbase shell commands to see if there is anything I can
> use to get all the regions info just for a particular table. I don’t see any
> such command that provides me that information.
> It would be better to have a command that provides region info, start key,
> end key etc taking a table name as the input parameter. This is available
> through HBase UI on clicking on a particular table's link
> A tool/shell command to get a list of regions for a table or all tables in a
> tabular structured output (that is machine readable)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955691#comment-15955691
]
Hadoop QA commented on HBASE-16438:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 37s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 24 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
31s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
34m 51s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 0s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 108m 40s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 162m 45s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12861912/HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch
|
| JIRA Issue | HBASE-16438 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 31f1d2b2b4cf 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / e916b79 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6312/testReport/ |
| modules | C: hbase-common hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6312/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
>
[jira] [Work started] (HBASE-17848) update documentation to use lists.apache.org
[ https://issues.apache.org/jira/browse/HBASE-17848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-17848 started by Jan Hentschel. - > update documentation to use lists.apache.org > > > Key: HBASE-17848 > URL: https://issues.apache.org/jira/browse/HBASE-17848 > Project: HBase > Issue Type: Improvement > Components: documentation >Reporter: Sean Busbey >Assignee: Jan Hentschel >Priority: Minor > > right now our documentation uses search-hadoop.com to link to conversations > on the mailing list where we decided things. Now that lists.apache.org exists > and has proper threading support, we should link there instead. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HBASE-17848) update documentation to use lists.apache.org
[ https://issues.apache.org/jira/browse/HBASE-17848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jan Hentschel reassigned HBASE-17848: - Assignee: Jan Hentschel > update documentation to use lists.apache.org > > > Key: HBASE-17848 > URL: https://issues.apache.org/jira/browse/HBASE-17848 > Project: HBase > Issue Type: Improvement > Components: documentation >Reporter: Sean Busbey >Assignee: Jan Hentschel >Priority: Minor > > right now our documentation uses search-hadoop.com to link to conversations > on the mailing list where we decided things. Now that lists.apache.org exists > and has proper threading support, we should link there instead. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-14141) HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits from backup tables
[
https://issues.apache.org/jira/browse/HBASE-14141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladimir Rodionov updated HBASE-14141:
--
Description:
h2. High level design overview
* When incremental backup request comes for tables {t} we select all the tables
already registered in a backup system - {T} and union them with {t}, which
results in a new table set - U(t, T)
* For every table K from U(t,T) we perform the following:
** Convert new WAL files into HFile applying table filter K (only edits for
table T will pass the filter)
** Move these HFile(s) to backup destination
During restore (incremental):
* We run full restore first
* Then collect all HFiles from intermediate incremental images and run them
through HFileSplitterJob, which splits files into a current tables region
boundaries
* Load these files using LoadIncrementalHFiles tool
> HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits
> from backup tables
>
>
> Key: HBASE-14141
> URL: https://issues.apache.org/jira/browse/HBASE-14141
> Project: HBase
> Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Blocker
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-14141.HBASE-14123.v1.patch, HBASE-14141.v1.patch,
> HBASE-14141.v2.patch, HBASE-14141.v4.patch, HBASE-14141.v5.patch,
> HBASE-14141.v6.patch
>
>
> h2. High level design overview
> * When incremental backup request comes for tables {t} we select all the
> tables already registered in a backup system - {T} and union them with {t},
> which results in a new table set - U(t, T)
> * For every table K from U(t,T) we perform the following:
> ** Convert new WAL files into HFile applying table filter K (only edits for
> table T will pass the filter)
> ** Move these HFile(s) to backup destination
> During restore (incremental):
> * We run full restore first
> * Then collect all HFiles from intermediate incremental images and run them
> through HFileSplitterJob, which splits files into a current tables region
> boundaries
> * Load these files using LoadIncrementalHFiles tool
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-14141) HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits from backup tables
[ https://issues.apache.org/jira/browse/HBASE-14141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-14141: -- Attachment: HBASE-14141.v6.patch v6 addresses last RB comments, javadoc and findbugs warnings. cc: [~tedyu], [~elserj] > HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits > from backup tables > > > Key: HBASE-14141 > URL: https://issues.apache.org/jira/browse/HBASE-14141 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Blocker > Labels: backup > Fix For: 2.0.0 > > Attachments: HBASE-14141.HBASE-14123.v1.patch, HBASE-14141.v1.patch, > HBASE-14141.v2.patch, HBASE-14141.v4.patch, HBASE-14141.v5.patch, > HBASE-14141.v6.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17721) Provide streaming APIs with SSL/TLS
[ https://issues.apache.org/jira/browse/HBASE-17721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955547#comment-15955547 ] stack commented on HBASE-17721: --- Any progress her per chance [~alexaraujo]? > Provide streaming APIs with SSL/TLS > --- > > Key: HBASE-17721 > URL: https://issues.apache.org/jira/browse/HBASE-17721 > Project: HBase > Issue Type: Umbrella > Components: rpc, security >Reporter: Alex Araujo >Assignee: Alex Araujo > Fix For: 2.0.0 > > > Umbrella to add optional client/server streaming capabilities to HBase. > This would allow bandwidth to be used more efficiently for certain > operations, and allow clients to use SSL/TLS for authentication and > encryption. > Desired client/server scaffolding: > - HTTP/2 support > - Protocol negotiation (blocking vs streaming, auth, encryption, etc.) > - TLS/SSL support > - Streaming RPC support > Possibilities (and their tradeoffs): > - gRPC: Some initial work and discussion on HBASE-13467 (Prototype using GRPC > as IPC mechanism) > -- Has most or all of the desired scaffolding > -- Adds additional g* dependencies. Compat story for g* dependencies not > always ideal > - Custom HTTP/2 based client/server APIs > -- More control over compat story > -- Non-trivial to build scaffolding; might reinvent wheels along the way > - Others? > Related Jiras that might be rolled in as sub-tasks (or closed/replaced with > new ones): > HBASE-17708 (Expose config to set two-way auth over TLS in HttpServer and add > a test) > HBASE-8691 (High-Throughput Streaming Scan API) > HBASE-14899 (Create custom Streaming ReplicationEndpoint) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17875) Document why objects over 10MB are not well-suited for hbase
stack created HBASE-17875: - Summary: Document why objects over 10MB are not well-suited for hbase Key: HBASE-17875 URL: https://issues.apache.org/jira/browse/HBASE-17875 Project: HBase Issue Type: Bug Components: documentation Reporter: stack A new-user who has patently done their homework is unable to find in doc why objects above 10MB are not recommended for hbase. Lets add explanation that has a link from MOB on what happens when objects this size are requested form HBase. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955492#comment-15955492
]
Hudson commented on HBASE-17861:
FAILURE: Integrated in Jenkins build HBase-1.4 #687 (See
[https://builds.apache.org/job/HBase-1.4/687/])
HBASE-17861: Regionserver down when checking the permission of staging (tedyu:
rev 4057a6c89c74c9f595cb51ac3bdc288396a0b257)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/SecureBulkLoadEndpoint.java
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861.branch-1.V1.patch,
> HBASE-17861.branch-1.V2.patch, HBASE-17861.branch-1.V3.patch,
> HBASE-17861.branch-1.V4.patch, HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = s3a://xx//xx
> hbase.wal.dir = hdfs://xx/xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-17861:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks for the patch, Li.
Thanks for the reviews, Zach and Jerry.
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861.branch-1.V1.patch,
> HBASE-17861.branch-1.V2.patch, HBASE-17861.branch-1.V3.patch,
> HBASE-17861.branch-1.V4.patch, HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = s3a://xx//xx
> hbase.wal.dir = hdfs://xx/xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-14925) Develop HBase shell command/tool to list table's region info through command line
[ https://issues.apache.org/jira/browse/HBASE-14925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955478#comment-15955478 ] ronan stokes commented on HBASE-14925: -- Just to clarify further - the regionServerName parameter should be optional and the command should still return the correct information if the region server is not supplied , the information should be returned for all regions > Develop HBase shell command/tool to list table's region info through command > line > - > > Key: HBASE-14925 > URL: https://issues.apache.org/jira/browse/HBASE-14925 > Project: HBase > Issue Type: Improvement > Components: shell >Reporter: Romil Choksi >Assignee: Karan Mehta > Attachments: HBASE-14925.patch > > > I am going through the hbase shell commands to see if there is anything I can > use to get all the regions info just for a particular table. I don’t see any > such command that provides me that information. > It would be better to have a command that provides region info, start key, > end key etc taking a table name as the input parameter. This is available > through HBase UI on clicking on a particular table's link > A tool/shell command to get a list of regions for a table or all tables in a > tabular structured output (that is machine readable) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-14925) Develop HBase shell command/tool to list table's region info through command line
[ https://issues.apache.org/jira/browse/HBASE-14925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955474#comment-15955474 ] ronan stokes commented on HBASE-14925: -- Its useful to consider why some one would use this command when looking at the variations and what parameters might be required Some examples: - planning to take a node of out of service => in this case its useful to see what regions are on a specific server in order to determine what you need to move - diagnosing skew or region imbalance => in this case, what you need is a breakdown of the regions by server - but you are interested in the counts,sizes by server along with read and write rates - preparing to manually balance a table => need to filter by table name - informational purposes only => just need to ensure that data is up to date > Develop HBase shell command/tool to list table's region info through command > line > - > > Key: HBASE-14925 > URL: https://issues.apache.org/jira/browse/HBASE-14925 > Project: HBase > Issue Type: Improvement > Components: shell >Reporter: Romil Choksi >Assignee: Karan Mehta > Attachments: HBASE-14925.patch > > > I am going through the hbase shell commands to see if there is anything I can > use to get all the regions info just for a particular table. I don’t see any > such command that provides me that information. > It would be better to have a command that provides region info, start key, > end key etc taking a table name as the input parameter. This is available > through HBase UI on clicking on a particular table's link > A tool/shell command to get a list of regions for a table or all tables in a > tabular structured output (that is machine readable) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17863) Procedure V2: Some cleanup around Procedure.isFinished() and procedure executor
[ https://issues.apache.org/jira/browse/HBASE-17863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955460#comment-15955460 ] Umesh Agashe commented on HBASE-17863: -- Failed again! Looking into it. > Procedure V2: Some cleanup around Procedure.isFinished() and procedure > executor > --- > > Key: HBASE-17863 > URL: https://issues.apache.org/jira/browse/HBASE-17863 > Project: HBase > Issue Type: Bug > Components: proc-v2 >Reporter: Umesh Agashe >Assignee: Umesh Agashe > Attachments: HBASE-17863.v1.patch, HBASE-17863.v2.patch, > HBASE-17863.v3.patch, HBASE-17863.v3.patch > > > Clean up around isFinished() and procedure executor -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955436#comment-15955436
]
Jerry He commented on HBASE-17861:
--
+1
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861.branch-1.V1.patch,
> HBASE-17861.branch-1.V2.patch, HBASE-17861.branch-1.V3.patch,
> HBASE-17861.branch-1.V4.patch, HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = s3a://xx//xx
> hbase.wal.dir = hdfs://xx/xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16438: --- Status: Patch Available (was: Open) > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Attachments: > HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, > HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, > HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, > MemstoreChunkCell_trunk.patch > > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. > Why we need to embed the chunkid in the Cell is described by [~anastas] in > this remark over in parent issue > https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16438: --- Attachment: HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch For QA. > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Attachments: > HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, > HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, > HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, > MemstoreChunkCell_trunk.patch > > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. > Why we need to embed the chunkid in the Cell is described by [~anastas] in > this remark over in parent issue > https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16438: --- Status: Open (was: Patch Available) > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Attachments: > HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_11_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, > HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, > HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, > MemstoreChunkCell_trunk.patch > > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. > Why we need to embed the chunkid in the Cell is described by [~anastas] in > this remark over in parent issue > https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17849) PE tool randomness is not totally random
[ https://issues.apache.org/jira/browse/HBASE-17849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955414#comment-15955414 ] Ted Yu commented on HBASE-17849: +1, pending QA. > PE tool randomness is not totally random > > > Key: HBASE-17849 > URL: https://issues.apache.org/jira/browse/HBASE-17849 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17849.patch > > > Recently we were using the PE tool for doing some bucket cache related > performance tests. One thing that we noted was that the way the random read > works is not totally random. > Suppose we load 200G of data using --size param and then we use --rows=50 > to do the randomRead. The assumption was among the 200G of data it could > generate randomly 50 row keys to do the reads. > But it so happens that the PE tool generates random rows only on those set of > row keys which falls under the first 50 rows. > This was quite evident when we tried to use HBASE-15314 in our testing. > Suppose we split the bucket cache of size 200G into 2 files each 100G the > randomReads with --rows=50 always lands in the first file and not in the > 2nd file. Better to make PE purely random. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955403#comment-15955403 ] ramkrishna.s.vasudevan commented on HBASE-16438: [~anastas] Some how I missed your updates. The latest patch here has the change from long to int for the chunkid. And adds the constant of how many bytes we need for a cell that will be added to CellChunkMap. > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Attachments: > HBASE-16438_10_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_1.patch, HBASE-16438_3_ChunkCreatorwrappingChunkPool.patch, > HBASE-16438_4_ChunkCreatorwrappingChunkPool.patch, > HBASE-16438_8_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438_9_ChunkCreatorwrappingChunkPool_withchunkRef.patch, > HBASE-16438.patch, MemstoreChunkCell_memstoreChunkCreator_oldversion.patch, > MemstoreChunkCell_trunk.patch > > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. > Why we need to embed the chunkid in the Cell is described by [~anastas] in > this remark over in parent issue > https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17849) PE tool randomness is not totally random
[ https://issues.apache.org/jira/browse/HBASE-17849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955399#comment-15955399 ] ramkrishna.s.vasudevan commented on HBASE-17849: Ping !!! > PE tool randomness is not totally random > > > Key: HBASE-17849 > URL: https://issues.apache.org/jira/browse/HBASE-17849 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17849.patch > > > Recently we were using the PE tool for doing some bucket cache related > performance tests. One thing that we noted was that the way the random read > works is not totally random. > Suppose we load 200G of data using --size param and then we use --rows=50 > to do the randomRead. The assumption was among the 200G of data it could > generate randomly 50 row keys to do the reads. > But it so happens that the PE tool generates random rows only on those set of > row keys which falls under the first 50 rows. > This was quite evident when we tried to use HBASE-15314 in our testing. > Suppose we split the bucket cache of size 200G into 2 files each 100G the > randomReads with --rows=50 always lands in the first file and not in the > 2nd file. Better to make PE purely random. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-14978) Don't allow Multi to retain too many blocks
[ https://issues.apache.org/jira/browse/HBASE-14978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955361#comment-15955361 ] Anoop Sam John commented on HBASE-14978: Created HBASE-17874 and made as blocker for 2.0. Will look at it soon. Ya might be we should not account for Cell coming from BucketCache (memory mode) at all. > Don't allow Multi to retain too many blocks > --- > > Key: HBASE-14978 > URL: https://issues.apache.org/jira/browse/HBASE-14978 > Project: HBase > Issue Type: Improvement > Components: io, IPC/RPC, regionserver >Affects Versions: 2.0.0, 1.2.0, 1.3.0 >Reporter: Elliott Clark >Assignee: Elliott Clark >Priority: Blocker > Fix For: 2.0.0, 1.2.0, 1.3.0 > > Attachments: HBASE-14978-branch-1.2.patch, > HBASE-14978-branch-1.2.patch, HBASE-14978-branch-1.2.patch, > HBASE-14978-branch-1.2.patch, HBASE-14978.patch, HBASE-14978-v1.patch, > HBASE-14978-v2.patch, HBASE-14978-v3.patch, HBASE-14978-v4.patch > > > Scans and Multi's have limits on the total size of cells that can be > returned. However if those requests are not all pointing at the same blocks > then the KeyValues can keep alive a lot more data than their size. > Take the following example: > A multi with a list of 1 gets to a fat row. Each column being returned in > in a different block. Each column is small 32 bytes or so. > So the total cell size will be 32 * 1 = ~320kb. However if each block is > 128k then total retained heap size will be almost 2gigs. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17874) Limiting of read request response size based on block size may go wrong when blocks are read from onheap or off heap bucket cache
Anoop Sam John created HBASE-17874:
--
Summary: Limiting of read request response size based on block
size may go wrong when blocks are read from onheap or off heap bucket cache
Key: HBASE-17874
URL: https://issues.apache.org/jira/browse/HBASE-17874
Project: HBase
Issue Type: Bug
Affects Versions: 2.0.0
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Priority: Blocker
Fix For: 2.0.0
HBASE-14978 added this size limiting so as to make sure the multi read requests
do not retain two many blocks. This works well when the blocks are obtained
from any where other than memory mode BucketCache. In case of on heap or off
heap Bucket Cache, the entire cache area is split into N ByteBuffers each of
size 4 MB. When we hit a block in this cache, we no longer do copy data into
temp array. We use the same shared memory (BB). Its capacity is 4 MB.
The block size accounting logic is RSRpcServices is like below
{code}
if (c instanceof ByteBufferCell) {
ByteBufferCell bbCell = (ByteBufferCell) c;
ByteBuffer bb = bbCell.getValueByteBuffer();
if (bb != lastBlock) {
context.incrementResponseBlockSize(bb.capacity());
lastBlock = bb;
}
} else {
// We're using the last block being the same as the current block as
// a proxy for pointing to a new block. This won't be exact.
// If there are multiple gets that bounce back and forth
// Then it's possible that this will over count the size of
// referenced blocks. However it's better to over count and
// use two rpcs than to OOME the regionserver.
byte[] valueArray = c.getValueArray();
if (valueArray != lastBlock) {
context.incrementResponseBlockSize(valueArray.length);
lastBlock = valueArray;
}
}
{code}
We take the BBCell's value buffer and takes its capacity. The cell is backed by
the same BB that backs the HFileBlock. When the HFileBlock is created from the
BC, we do as below duplicating and proper positioning and limiting the BB
{code}
ByteBuffer bb = buffers[i].duplicate();
if (i == startBuffer) {
cnt = bufferSize - startBufferOffset;
if (cnt > len) cnt = len;
bb.limit(startBufferOffset + cnt).position(startBufferOffset);
{code}
Still this BB's capacity is 4 MB.
This will make the size limit breach to happen too soon. What we expect is
block size defaults to 64 KB and so we here by allow cells from different
blocks to appear in response. We have a way to check whether we move from one
block to next.
{code}
if (bb != lastBlock) {
...
lastBlock = bb;
}
{code}
But already just by considering the 1st cell, we added 4 MB size!
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liang updated HBASE-17861:
-
Release Note: Some object store does not support unix style permission,
this jira fix the permission check issue when specify staging dir in different
file system. Currently it covers s3, wasb, swift.
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861.branch-1.V1.patch,
> HBASE-17861.branch-1.V2.patch, HBASE-17861.branch-1.V3.patch,
> HBASE-17861.branch-1.V4.patch, HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = s3a://xx//xx
> hbase.wal.dir = hdfs://xx/xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17872) Make BBU#copyFromBufferToBuffer thread-safe
[
https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955272#comment-15955272
]
Ted Yu commented on HBASE-17872:
Nice finding, Chia-Ping
Please include your test in the patch.
> Make BBU#copyFromBufferToBuffer thread-safe
> ---
>
> Key: HBASE-17872
> URL: https://issues.apache.org/jira/browse/HBASE-17872
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0
>
>
> A case is shown below.
> We will get the wrong position of buffer in multithreaded environment, so the
> method makes the invalid cell in MSLAB.
> {noformat}
> public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int
> sourceOffset,
> int destinationOffset, int length) {
> if (in.hasArray() && out.hasArray()) {
> // ...
> } else if (UNSAFE_AVAIL) {
> // ...
> } else {
> int outOldPos = out.position();
> out.position(destinationOffset);
> ByteBuffer inDup = in.duplicate();
> inDup.position(sourceOffset).limit(sourceOffset + length);
> out.put(inDup);
> out.position(outOldPos);
> }
> return destinationOffset + length;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17857) Remove IS annotations from IA.Public classes
[
https://issues.apache.org/jira/browse/HBASE-17857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955270#comment-15955270
]
Hadoop QA commented on HBASE-17857:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
1s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 37s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 38s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 1m
14s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} scalac {color} | {color:green} 2m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 23s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 1m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 8s
{color} | {color:green} hbase-annotations in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 48s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 16s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 105m 11s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 29s
{color} | {color:green} hbase-endpoint in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 35s
{color} | {color:green} hbase-rest in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 52s
{color} | {color:green} hbase-spark in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
36s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 182m 26s
[jira] [Updated] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-17861:
---
Hadoop Flags: Reviewed
+1
Please fill out release note.
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861.branch-1.V1.patch,
> HBASE-17861.branch-1.V2.patch, HBASE-17861.branch-1.V3.patch,
> HBASE-17861.branch-1.V4.patch, HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = s3a://xx//xx
> hbase.wal.dir = hdfs://xx/xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17872) Make BBU#copyFromBufferToBuffer thread-safe
[
https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955097#comment-15955097
]
Chia-Ping Tsai commented on HBASE-17872:
The unsafe doesn't effect the position in BBU, but the others may effect the
position. We should unify the logic of all methods in BBU.
> Make BBU#copyFromBufferToBuffer thread-safe
> ---
>
> Key: HBASE-17872
> URL: https://issues.apache.org/jira/browse/HBASE-17872
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0
>
>
> A case is shown below.
> We will get the wrong position of buffer in multithreaded environment, so the
> method makes the invalid cell in MSLAB.
> {noformat}
> public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int
> sourceOffset,
> int destinationOffset, int length) {
> if (in.hasArray() && out.hasArray()) {
> // ...
> } else if (UNSAFE_AVAIL) {
> // ...
> } else {
> int outOldPos = out.position();
> out.position(destinationOffset);
> ByteBuffer inDup = in.duplicate();
> inDup.position(sourceOffset).limit(sourceOffset + length);
> out.put(inDup);
> out.position(outOldPos);
> }
> return destinationOffset + length;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17873) Change the IA.Public annotation to IA.Private for unstable API
[ https://issues.apache.org/jira/browse/HBASE-17873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17873: -- Affects Version/s: 2.0.0 Priority: Blocker (was: Major) > Change the IA.Public annotation to IA.Private for unstable API > -- > > Key: HBASE-17873 > URL: https://issues.apache.org/jira/browse/HBASE-17873 > Project: HBase > Issue Type: Sub-task > Components: API >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Priority: Blocker > Fix For: 2.0.0 > > > As discussed in mailing list andHBASE-17857. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17873) Change the IA.Public annotation to IA.Private for unstable API
Duo Zhang created HBASE-17873: - Summary: Change the IA.Public annotation to IA.Private for unstable API Key: HBASE-17873 URL: https://issues.apache.org/jira/browse/HBASE-17873 Project: HBase Issue Type: Sub-task Reporter: Duo Zhang As discussed in mailing list andHBASE-17857. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17872) Make BBU#copyFromBufferToBuffer thread-safe
[
https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955084#comment-15955084
]
Chia-Ping Tsai commented on HBASE-17872:
bq. Any other instances of such cases?
BBU#copyFromArrayToBuffer(ByteBuffer out, int outOffset, byte[] in, int
inOffset, int length). It is used to set the timestamp of BBKV.
> Make BBU#copyFromBufferToBuffer thread-safe
> ---
>
> Key: HBASE-17872
> URL: https://issues.apache.org/jira/browse/HBASE-17872
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0
>
>
> A case is shown below.
> We will get the wrong position of buffer in multithreaded environment, so the
> method makes the invalid cell in MSLAB.
> {noformat}
> public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int
> sourceOffset,
> int destinationOffset, int length) {
> if (in.hasArray() && out.hasArray()) {
> // ...
> } else if (UNSAFE_AVAIL) {
> // ...
> } else {
> int outOldPos = out.position();
> out.position(destinationOffset);
> ByteBuffer inDup = in.duplicate();
> inDup.position(sourceOffset).limit(sourceOffset + length);
> out.put(inDup);
> out.position(outOldPos);
> }
> return destinationOffset + length;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17857) Remove IS annotations from IA.Public classes
[ https://issues.apache.org/jira/browse/HBASE-17857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17857: -- Attachment: HBASE-17857-v3.patch Change the IA annotation back to public for AsyncAdmin and backoff related classes. Will open new issue to address it. > Remove IS annotations from IA.Public classes > > > Key: HBASE-17857 > URL: https://issues.apache.org/jira/browse/HBASE-17857 > Project: HBase > Issue Type: Sub-task > Components: API >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17857.patch, HBASE-17857-v1.patch, > HBASE-17857-v2.patch, HBASE-17857-v3.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17872) Make BBU#copyFromBufferToBuffer thread-safe
[
https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955073#comment-15955073
]
Anoop Sam John commented on HBASE-17872:
Any other instances of such cases?
> Make BBU#copyFromBufferToBuffer thread-safe
> ---
>
> Key: HBASE-17872
> URL: https://issues.apache.org/jira/browse/HBASE-17872
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0
>
>
> A case is shown below.
> We will get the wrong position of buffer in multithreaded environment, so the
> method makes the invalid cell in MSLAB.
> {noformat}
> public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int
> sourceOffset,
> int destinationOffset, int length) {
> if (in.hasArray() && out.hasArray()) {
> // ...
> } else if (UNSAFE_AVAIL) {
> // ...
> } else {
> int outOldPos = out.position();
> out.position(destinationOffset);
> ByteBuffer inDup = in.duplicate();
> inDup.position(sourceOffset).limit(sourceOffset + length);
> out.put(inDup);
> out.position(outOldPos);
> }
> return destinationOffset + length;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17872) Make BBU#copyFromBufferToBuffer thread-safe
[
https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955066#comment-15955066
]
Chia-Ping Tsai commented on HBASE-17872:
bq. +1 to fix it via BB.duplicate() call.
Okay, copy that. I will attach the patch after my local tests.
> Make BBU#copyFromBufferToBuffer thread-safe
> ---
>
> Key: HBASE-17872
> URL: https://issues.apache.org/jira/browse/HBASE-17872
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0
>
>
> A case is shown below.
> We will get the wrong position of buffer in multithreaded environment, so the
> method makes the invalid cell in MSLAB.
> {noformat}
> public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int
> sourceOffset,
> int destinationOffset, int length) {
> if (in.hasArray() && out.hasArray()) {
> // ...
> } else if (UNSAFE_AVAIL) {
> // ...
> } else {
> int outOldPos = out.position();
> out.position(destinationOffset);
> ByteBuffer inDup = in.duplicate();
> inDup.position(sourceOffset).limit(sourceOffset + length);
> out.put(inDup);
> out.position(outOldPos);
> }
> return destinationOffset + length;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17872) Make BBU#copyFromBufferToBuffer thread-safe
[
https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955052#comment-15955052
]
Anoop Sam John edited comment on HBASE-17872 at 4/4/17 12:11 PM:
-
+1 to fix it via BB.duplicate() call. The tests were covering the Unsafe path
and so not revealed the issue may be. Ya it is a mistake at using this method
at MSLABImpl level.
was (Author: anoop.hbase):
+1 to fix it via BB.duplicate() call.
> Make BBU#copyFromBufferToBuffer thread-safe
> ---
>
> Key: HBASE-17872
> URL: https://issues.apache.org/jira/browse/HBASE-17872
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0
>
>
> A case is shown below.
> We will get the wrong position of buffer in multithreaded environment, so the
> method makes the invalid cell in MSLAB.
> {noformat}
> public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int
> sourceOffset,
> int destinationOffset, int length) {
> if (in.hasArray() && out.hasArray()) {
> // ...
> } else if (UNSAFE_AVAIL) {
> // ...
> } else {
> int outOldPos = out.position();
> out.position(destinationOffset);
> ByteBuffer inDup = in.duplicate();
> inDup.position(sourceOffset).limit(sourceOffset + length);
> out.put(inDup);
> out.position(outOldPos);
> }
> return destinationOffset + length;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17872) Make BBU#copyFromBufferToBuffer thread-safe
[
https://issues.apache.org/jira/browse/HBASE-17872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955052#comment-15955052
]
Anoop Sam John commented on HBASE-17872:
+1 to fix it via BB.duplicate() call.
> Make BBU#copyFromBufferToBuffer thread-safe
> ---
>
> Key: HBASE-17872
> URL: https://issues.apache.org/jira/browse/HBASE-17872
> Project: HBase
> Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Critical
> Fix For: 2.0.0
>
>
> A case is shown below.
> We will get the wrong position of buffer in multithreaded environment, so the
> method makes the invalid cell in MSLAB.
> {noformat}
> public static int copyFromBufferToBuffer(ByteBuffer in, ByteBuffer out, int
> sourceOffset,
> int destinationOffset, int length) {
> if (in.hasArray() && out.hasArray()) {
> // ...
> } else if (UNSAFE_AVAIL) {
> // ...
> } else {
> int outOldPos = out.position();
> out.position(destinationOffset);
> ByteBuffer inDup = in.duplicate();
> inDup.position(sourceOffset).limit(sourceOffset + length);
> out.put(inDup);
> out.position(outOldPos);
> }
> return destinationOffset + length;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
