[jira] [Commented] (HBASE-17361) Make HTable thread safe
[
https://issues.apache.org/jira/browse/HBASE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15772232#comment-15772232
]
Hadoop QA commented on HBASE-17361:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
35s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 12s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 1s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 94m 40s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 138m 29s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12844521/HBASE-17361.patch |
| JIRA Issue | HBASE-17361 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux e7fcd2fc9ca4 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 8fb9a91 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5033/testReport/ |
| modules | C: hbase-client hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5033/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Make HTable thread safe
> ---
[jira] [Commented] (HBASE-17174) Refactor the AsyncProcess, BufferedMutatorImpl, and HTable
[ https://issues.apache.org/jira/browse/HBASE-17174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15772187#comment-15772187 ] Duo Zhang commented on HBASE-17174: --- Let's wait for a while to see if there are objections. Will commit tomorrow if no. Thanks. > Refactor the AsyncProcess, BufferedMutatorImpl, and HTable > -- > > Key: HBASE-17174 > URL: https://issues.apache.org/jira/browse/HBASE-17174 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17174.v0.patch, HBASE-17174.v1.patch, > HBASE-17174.v10.patch, HBASE-17174.v11.patch, HBASE-17174.v11.patch, > HBASE-17174.v11.patch, HBASE-17174.v12.patch, HBASE-17174.v2.patch, > HBASE-17174.v3.patch, HBASE-17174.v4.patch, HBASE-17174.v5.patch, > HBASE-17174.v6.patch, HBASE-17174.v7.patch, HBASE-17174.v8.patch, > HBASE-17174.v9.patch > > > The following are reasons of refactoring. > # A update-heavy application, for example, loader, creates many > BufferedMutator for batch updates. But these BufferedMutators can’t share a > large threadpool because the shutdown() method will be called when closing > any BufferedMutator. This patch adds a flag into BufferedMutatorParams for > preventing calling the shutdown() method in BufferedMutatorImpl#close > # The AsyncProcess has the powerful traffic control, but the control is > against the single table currently. We should allow alternate traffic control > implementation for advanced user who want more control. > All suggestions are welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17361) Make HTable thread safe
[
https://issues.apache.org/jira/browse/HBASE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-17361:
--
Affects Version/s: (was: 1.2.4)
(was: 1.1.7)
Priority: Major (was: Critical)
Description:
Currently HTable is marked as NOT thread safe, and this JIRA target at
improving this to take better usage of the thread-safe BufferedMutator.
Some findings/work done:
If we try to do put to the same HTable instance in parallel, there'll be
problem, since now we have {{HTable#getBufferedMutator}} like
{code}
BufferedMutator getBufferedMutator() throws IOException {
if (mutator == null) {
this.mutator = (BufferedMutatorImpl) connection.getBufferedMutator(
new BufferedMutatorParams(tableName)
.pool(pool)
.writeBufferSize(connConfiguration.getWriteBufferSize())
.maxKeyValueSize(connConfiguration.getMaxKeyValueSize())
);
}
mutator.setRpcTimeout(writeRpcTimeout);
mutator.setOperationTimeout(operationTimeout);
return mutator;
}
{code}
And {{HTable#flushCommits}}:
{code}
void flushCommits() throws IOException {
if (mutator == null) {
// nothing to flush if there's no mutator; don't bother creating one.
return;
}
getBufferedMutator().flush();
}
{code}
For {{HTable#put}}
{code}
public void put(final Put put) throws IOException {
getBufferedMutator().mutate(put);
flushCommits();
}
{code}
If we launch multiple threads to put in parallel, below sequence might happen
because {{HTable#getBufferedMutator}} is not thread safe:
{noformat}
1. ThreadA runs to getBufferedMutator and finds mutator==null
2. ThreadB runs to getBufferedMutator and finds mutator==null
3. ThreadA initialize mutator to instanceA, then calls mutator#mutate,
adding one put (putA) into {{writeAsyncBuffer}}
4. ThreadB initialize mutator to instanceB
5. ThreadA runs to flushCommits, now mutator is instanceB, it calls
instanceB's flush method, putA is lost
{noformat}
After fixing this, we will find quite some contention on
{{BufferedMutatorImpl#flush}}, so more efforts required to make HTable thread
safe but with good performance meanwhile.
was:
Now we have {{HTable#getBufferedMutator}} like
{code}
BufferedMutator getBufferedMutator() throws IOException {
if (mutator == null) {
this.mutator = (BufferedMutatorImpl) connection.getBufferedMutator(
new BufferedMutatorParams(tableName)
.pool(pool)
.writeBufferSize(connConfiguration.getWriteBufferSize())
.maxKeyValueSize(connConfiguration.getMaxKeyValueSize())
);
}
mutator.setRpcTimeout(writeRpcTimeout);
mutator.setOperationTimeout(operationTimeout);
return mutator;
}
{code}
And {{HTable#flushCommits}}:
{code}
void flushCommits() throws IOException {
if (mutator == null) {
// nothing to flush if there's no mutator; don't bother creating one.
return;
}
getBufferedMutator().flush();
}
{code}
For {{HTable#put}}
{code}
public void put(final Put put) throws IOException {
getBufferedMutator().mutate(put);
flushCommits();
}
{code}
If we launch multiple threads to put in parallel, below sequence might happen
because {{HTable#getBufferedMutator}} is not thread safe:
{noformat}
1. ThreadA runs to getBufferedMutator and finds mutator==null
2. ThreadB runs to getBufferedMutator and finds mutator==null
3. ThreadA initialize mutator to instanceA, then calls mutator#mutate,
adding one put (putA) into {{writeAsyncBuffer}}
4. ThreadB initialize mutator to instanceB
5. ThreadA runs to flushCommits, now mutator is instanceB, it calls
instanceB's flush method, putA is lost
{noformat}
Will add a UT to cover this case, and fix it in this JIRA.
Issue Type: Improvement (was: Bug)
Summary: Make HTable thread safe (was: HTable#getBufferedMutator
is not thread safe and could cause data loss)
> Make HTable thread safe
> ---
>
> Key: HBASE-17361
> URL: https://issues.apache.org/jira/browse/HBASE-17361
> Project: HBase
> Issue Type: Improvement
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17361.patch, HBASE-17361.patch
>
>
> Currently HTable is marked as NOT thread safe, and this JIRA target at
> improving this to take better usage of the thread-safe BufferedMutator.
> Some findings/work done:
> If we try to do put to the same HTable instance in parallel, there'll be
> problem, since now we have {{HTable#getBufferedMutator}} like
> {code}
>BufferedMutator getBufferedMutator() throws IOException {
> if (mutator == null) {
> this.mutator = (BufferedMutatorImpl) connection.getBufferedMutator(
> new BufferedMutatorParams(tableName)
> .pool(pool)
> .write
[jira] [Commented] (HBASE-17361) HTable#getBufferedMutator is not thread safe and could cause data loss
[
https://issues.apache.org/jira/browse/HBASE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15772138#comment-15772138
]
Duo Zhang commented on HBASE-17361:
---
Maybe add an ErrorHandlingConfig for each method? A TableBuilder still means
you need to construct a new Table every time...
> HTable#getBufferedMutator is not thread safe and could cause data loss
> --
>
> Key: HBASE-17361
> URL: https://issues.apache.org/jira/browse/HBASE-17361
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.7, 1.2.4
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Attachments: HBASE-17361.patch, HBASE-17361.patch
>
>
> Now we have {{HTable#getBufferedMutator}} like
> {code}
>BufferedMutator getBufferedMutator() throws IOException {
> if (mutator == null) {
> this.mutator = (BufferedMutatorImpl) connection.getBufferedMutator(
> new BufferedMutatorParams(tableName)
> .pool(pool)
> .writeBufferSize(connConfiguration.getWriteBufferSize())
> .maxKeyValueSize(connConfiguration.getMaxKeyValueSize())
> );
> }
> mutator.setRpcTimeout(writeRpcTimeout);
> mutator.setOperationTimeout(operationTimeout);
> return mutator;
> }
> {code}
> And {{HTable#flushCommits}}:
> {code}
> void flushCommits() throws IOException {
> if (mutator == null) {
> // nothing to flush if there's no mutator; don't bother creating one.
> return;
> }
> getBufferedMutator().flush();
> }
> {code}
> For {{HTable#put}}
> {code}
> public void put(final Put put) throws IOException {
> getBufferedMutator().mutate(put);
> flushCommits();
> }
> {code}
> If we launch multiple threads to put in parallel, below sequence might happen
> because {{HTable#getBufferedMutator}} is not thread safe:
> {noformat}
> 1. ThreadA runs to getBufferedMutator and finds mutator==null
> 2. ThreadB runs to getBufferedMutator and finds mutator==null
> 3. ThreadA initialize mutator to instanceA, then calls mutator#mutate,
> adding one put (putA) into {{writeAsyncBuffer}}
> 4. ThreadB initialize mutator to instanceB
> 5. ThreadA runs to flushCommits, now mutator is instanceB, it calls
> instanceB's flush method, putA is lost
> {noformat}
> Will add a UT to cover this case, and fix it in this JIRA.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17361) HTable#getBufferedMutator is not thread safe and could cause data loss
[
https://issues.apache.org/jira/browse/HBASE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15772124#comment-15772124
]
Yu Li commented on HBASE-17361:
---
Sorry but I never tried to get any BufferedMutator directly, actually never
tried to get a BufferedMutator instance but all through the HTable API. Please
refer to the UT case, although it's not that proper considering HTable is
marked as not thread safe for now.
> HTable#getBufferedMutator is not thread safe and could cause data loss
> --
>
> Key: HBASE-17361
> URL: https://issues.apache.org/jira/browse/HBASE-17361
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.7, 1.2.4
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Attachments: HBASE-17361.patch, HBASE-17361.patch
>
>
> Now we have {{HTable#getBufferedMutator}} like
> {code}
>BufferedMutator getBufferedMutator() throws IOException {
> if (mutator == null) {
> this.mutator = (BufferedMutatorImpl) connection.getBufferedMutator(
> new BufferedMutatorParams(tableName)
> .pool(pool)
> .writeBufferSize(connConfiguration.getWriteBufferSize())
> .maxKeyValueSize(connConfiguration.getMaxKeyValueSize())
> );
> }
> mutator.setRpcTimeout(writeRpcTimeout);
> mutator.setOperationTimeout(operationTimeout);
> return mutator;
> }
> {code}
> And {{HTable#flushCommits}}:
> {code}
> void flushCommits() throws IOException {
> if (mutator == null) {
> // nothing to flush if there's no mutator; don't bother creating one.
> return;
> }
> getBufferedMutator().flush();
> }
> {code}
> For {{HTable#put}}
> {code}
> public void put(final Put put) throws IOException {
> getBufferedMutator().mutate(put);
> flushCommits();
> }
> {code}
> If we launch multiple threads to put in parallel, below sequence might happen
> because {{HTable#getBufferedMutator}} is not thread safe:
> {noformat}
> 1. ThreadA runs to getBufferedMutator and finds mutator==null
> 2. ThreadB runs to getBufferedMutator and finds mutator==null
> 3. ThreadA initialize mutator to instanceA, then calls mutator#mutate,
> adding one put (putA) into {{writeAsyncBuffer}}
> 4. ThreadB initialize mutator to instanceB
> 5. ThreadA runs to flushCommits, now mutator is instanceB, it calls
> instanceB's flush method, putA is lost
> {noformat}
> Will add a UT to cover this case, and fix it in this JIRA.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17361) HTable#getBufferedMutator is not thread safe and could cause data loss
[
https://issues.apache.org/jira/browse/HBASE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15772117#comment-15772117
]
Duo Zhang commented on HBASE-17361:
---
Then you'd better change the title to 'HTable should be thread safe for better
performance' rather than the current one? It will make people confusing...
> HTable#getBufferedMutator is not thread safe and could cause data loss
> --
>
> Key: HBASE-17361
> URL: https://issues.apache.org/jira/browse/HBASE-17361
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.7, 1.2.4
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Attachments: HBASE-17361.patch, HBASE-17361.patch
>
>
> Now we have {{HTable#getBufferedMutator}} like
> {code}
>BufferedMutator getBufferedMutator() throws IOException {
> if (mutator == null) {
> this.mutator = (BufferedMutatorImpl) connection.getBufferedMutator(
> new BufferedMutatorParams(tableName)
> .pool(pool)
> .writeBufferSize(connConfiguration.getWriteBufferSize())
> .maxKeyValueSize(connConfiguration.getMaxKeyValueSize())
> );
> }
> mutator.setRpcTimeout(writeRpcTimeout);
> mutator.setOperationTimeout(operationTimeout);
> return mutator;
> }
> {code}
> And {{HTable#flushCommits}}:
> {code}
> void flushCommits() throws IOException {
> if (mutator == null) {
> // nothing to flush if there's no mutator; don't bother creating one.
> return;
> }
> getBufferedMutator().flush();
> }
> {code}
> For {{HTable#put}}
> {code}
> public void put(final Put put) throws IOException {
> getBufferedMutator().mutate(put);
> flushCommits();
> }
> {code}
> If we launch multiple threads to put in parallel, below sequence might happen
> because {{HTable#getBufferedMutator}} is not thread safe:
> {noformat}
> 1. ThreadA runs to getBufferedMutator and finds mutator==null
> 2. ThreadB runs to getBufferedMutator and finds mutator==null
> 3. ThreadA initialize mutator to instanceA, then calls mutator#mutate,
> adding one put (putA) into {{writeAsyncBuffer}}
> 4. ThreadB initialize mutator to instanceB
> 5. ThreadA runs to flushCommits, now mutator is instanceB, it calls
> instanceB's flush method, putA is lost
> {noformat}
> Will add a UT to cover this case, and fix it in this JIRA.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17174) Refactor the AsyncProcess, BufferedMutatorImpl, and HTable
[ https://issues.apache.org/jira/browse/HBASE-17174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15772118#comment-15772118 ] ChiaPing Tsai commented on HBASE-17174: --- failed test isn't related to v12 patch, and it passes locally. > Refactor the AsyncProcess, BufferedMutatorImpl, and HTable > -- > > Key: HBASE-17174 > URL: https://issues.apache.org/jira/browse/HBASE-17174 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17174.v0.patch, HBASE-17174.v1.patch, > HBASE-17174.v10.patch, HBASE-17174.v11.patch, HBASE-17174.v11.patch, > HBASE-17174.v11.patch, HBASE-17174.v12.patch, HBASE-17174.v2.patch, > HBASE-17174.v3.patch, HBASE-17174.v4.patch, HBASE-17174.v5.patch, > HBASE-17174.v6.patch, HBASE-17174.v7.patch, HBASE-17174.v8.patch, > HBASE-17174.v9.patch > > > The following are reasons of refactoring. > # A update-heavy application, for example, loader, creates many > BufferedMutator for batch updates. But these BufferedMutators can’t share a > large threadpool because the shutdown() method will be called when closing > any BufferedMutator. This patch adds a flag into BufferedMutatorParams for > preventing calling the shutdown() method in BufferedMutatorImpl#close > # The AsyncProcess has the powerful traffic control, but the control is > against the single table currently. We should allow alternate traffic control > implementation for advanced user who want more control. > All suggestions are welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17361) HTable#getBufferedMutator is not thread safe and could cause data loss
[
https://issues.apache.org/jira/browse/HBASE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15772110#comment-15772110
]
Phil Yang commented on HBASE-17361:
---
Besides methods with timeout/retries... params, I think we can remove
setXXXTimeout and use a TableBuilder to set several default configurations as a
constructor?
> HTable#getBufferedMutator is not thread safe and could cause data loss
> --
>
> Key: HBASE-17361
> URL: https://issues.apache.org/jira/browse/HBASE-17361
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.7, 1.2.4
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Attachments: HBASE-17361.patch, HBASE-17361.patch
>
>
> Now we have {{HTable#getBufferedMutator}} like
> {code}
>BufferedMutator getBufferedMutator() throws IOException {
> if (mutator == null) {
> this.mutator = (BufferedMutatorImpl) connection.getBufferedMutator(
> new BufferedMutatorParams(tableName)
> .pool(pool)
> .writeBufferSize(connConfiguration.getWriteBufferSize())
> .maxKeyValueSize(connConfiguration.getMaxKeyValueSize())
> );
> }
> mutator.setRpcTimeout(writeRpcTimeout);
> mutator.setOperationTimeout(operationTimeout);
> return mutator;
> }
> {code}
> And {{HTable#flushCommits}}:
> {code}
> void flushCommits() throws IOException {
> if (mutator == null) {
> // nothing to flush if there's no mutator; don't bother creating one.
> return;
> }
> getBufferedMutator().flush();
> }
> {code}
> For {{HTable#put}}
> {code}
> public void put(final Put put) throws IOException {
> getBufferedMutator().mutate(put);
> flushCommits();
> }
> {code}
> If we launch multiple threads to put in parallel, below sequence might happen
> because {{HTable#getBufferedMutator}} is not thread safe:
> {noformat}
> 1. ThreadA runs to getBufferedMutator and finds mutator==null
> 2. ThreadB runs to getBufferedMutator and finds mutator==null
> 3. ThreadA initialize mutator to instanceA, then calls mutator#mutate,
> adding one put (putA) into {{writeAsyncBuffer}}
> 4. ThreadB initialize mutator to instanceB
> 5. ThreadA runs to flushCommits, now mutator is instanceB, it calls
> instanceB's flush method, putA is lost
> {noformat}
> Will add a UT to cover this case, and fix it in this JIRA.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17361) HTable#getBufferedMutator is not thread safe and could cause data loss
[
https://issues.apache.org/jira/browse/HBASE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15772109#comment-15772109
]
Yu Li commented on HBASE-17361:
---
Thanks for chiming in guys [~yangzhe1991] and [~Apache9], as I mentioned in
last comment, I have a pretty good reason for making HTable thread safe but
still not ready to share. This JIRA opened here not because of some experiment
w/o necessary knowledge but towards something, will be back for the discussion
later (smile).
> HTable#getBufferedMutator is not thread safe and could cause data loss
> --
>
> Key: HBASE-17361
> URL: https://issues.apache.org/jira/browse/HBASE-17361
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.7, 1.2.4
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Attachments: HBASE-17361.patch, HBASE-17361.patch
>
>
> Now we have {{HTable#getBufferedMutator}} like
> {code}
>BufferedMutator getBufferedMutator() throws IOException {
> if (mutator == null) {
> this.mutator = (BufferedMutatorImpl) connection.getBufferedMutator(
> new BufferedMutatorParams(tableName)
> .pool(pool)
> .writeBufferSize(connConfiguration.getWriteBufferSize())
> .maxKeyValueSize(connConfiguration.getMaxKeyValueSize())
> );
> }
> mutator.setRpcTimeout(writeRpcTimeout);
> mutator.setOperationTimeout(operationTimeout);
> return mutator;
> }
> {code}
> And {{HTable#flushCommits}}:
> {code}
> void flushCommits() throws IOException {
> if (mutator == null) {
> // nothing to flush if there's no mutator; don't bother creating one.
> return;
> }
> getBufferedMutator().flush();
> }
> {code}
> For {{HTable#put}}
> {code}
> public void put(final Put put) throws IOException {
> getBufferedMutator().mutate(put);
> flushCommits();
> }
> {code}
> If we launch multiple threads to put in parallel, below sequence might happen
> because {{HTable#getBufferedMutator}} is not thread safe:
> {noformat}
> 1. ThreadA runs to getBufferedMutator and finds mutator==null
> 2. ThreadB runs to getBufferedMutator and finds mutator==null
> 3. ThreadA initialize mutator to instanceA, then calls mutator#mutate,
> adding one put (putA) into {{writeAsyncBuffer}}
> 4. ThreadB initialize mutator to instanceB
> 5. ThreadA runs to flushCommits, now mutator is instanceB, it calls
> instanceB's flush method, putA is lost
> {noformat}
> Will add a UT to cover this case, and fix it in this JIRA.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17361) HTable#getBufferedMutator is not thread safe and could cause data loss
[
https://issues.apache.org/jira/browse/HBASE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15772097#comment-15772097
]
Duo Zhang commented on HBASE-17361:
---
See my comments above, the way you get a BufferedMutator is wrong. You should
use methods declared in Connection to get a BufferedMutator, not the one in
HTable.
> HTable#getBufferedMutator is not thread safe and could cause data loss
> --
>
> Key: HBASE-17361
> URL: https://issues.apache.org/jira/browse/HBASE-17361
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.7, 1.2.4
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Attachments: HBASE-17361.patch, HBASE-17361.patch
>
>
> Now we have {{HTable#getBufferedMutator}} like
> {code}
>BufferedMutator getBufferedMutator() throws IOException {
> if (mutator == null) {
> this.mutator = (BufferedMutatorImpl) connection.getBufferedMutator(
> new BufferedMutatorParams(tableName)
> .pool(pool)
> .writeBufferSize(connConfiguration.getWriteBufferSize())
> .maxKeyValueSize(connConfiguration.getMaxKeyValueSize())
> );
> }
> mutator.setRpcTimeout(writeRpcTimeout);
> mutator.setOperationTimeout(operationTimeout);
> return mutator;
> }
> {code}
> And {{HTable#flushCommits}}:
> {code}
> void flushCommits() throws IOException {
> if (mutator == null) {
> // nothing to flush if there's no mutator; don't bother creating one.
> return;
> }
> getBufferedMutator().flush();
> }
> {code}
> For {{HTable#put}}
> {code}
> public void put(final Put put) throws IOException {
> getBufferedMutator().mutate(put);
> flushCommits();
> }
> {code}
> If we launch multiple threads to put in parallel, below sequence might happen
> because {{HTable#getBufferedMutator}} is not thread safe:
> {noformat}
> 1. ThreadA runs to getBufferedMutator and finds mutator==null
> 2. ThreadB runs to getBufferedMutator and finds mutator==null
> 3. ThreadA initialize mutator to instanceA, then calls mutator#mutate,
> adding one put (putA) into {{writeAsyncBuffer}}
> 4. ThreadB initialize mutator to instanceB
> 5. ThreadA runs to flushCommits, now mutator is instanceB, it calls
> instanceB's flush method, putA is lost
> {noformat}
> Will add a UT to cover this case, and fix it in this JIRA.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17366) Run TestHFile#testReaderWithoutBlockCache failes
[ https://issues.apache.org/jira/browse/HBASE-17366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15772096#comment-15772096 ] Ted Yu commented on HBASE-17366: lgtm > Run TestHFile#testReaderWithoutBlockCache failes > > > Key: HBASE-17366 > URL: https://issues.apache.org/jira/browse/HBASE-17366 > Project: HBase > Issue Type: Bug >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-17366-master-001.patch > > > Running TestHFile#testReaderWithoutBlockCache failes because cacheConf is not > initialized. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17174) Refactor the AsyncProcess, BufferedMutatorImpl, and HTable
[
https://issues.apache.org/jira/browse/HBASE-17174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15772091#comment-15772091
]
Hadoop QA commented on HBASE-17174:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 35s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 57s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 14s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 133m 49s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestHRegion |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12844515/HBASE-17174.v12.patch
|
| JIRA Issue | HBASE-17174 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 2f7c6a7d5b12 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 8fb9a91 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/5032/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/5032/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.a
[jira] [Updated] (HBASE-15806) An endpoint-based export tool
[ https://issues.apache.org/jira/browse/HBASE-15806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai updated HBASE-15806: -- Status: Patch Available (was: Open) > An endpoint-based export tool > - > > Key: HBASE-15806 > URL: https://issues.apache.org/jira/browse/HBASE-15806 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: Experiment.png, HBASE-15806-v1.patch, > HBASE-15806-v2.patch, HBASE-15806-v3.patch, HBASE-15806.patch, > HBASE-15806.v4.patch, HBASE-15806.v5.patch, HBASE-15806.v6.patch, > HBASE-15806.v7.patch > > > The time for exporting table can be reduced, if we use the endpoint technique > to export the hdfs files by the region server rather than by hbase client. > In my experiments, the elapsed time of endpoint-based export can be less than > half of current export tool (enable the hdfs compression) > But the shortcomings is we need to alter table for deploying the endpoint > any comments about this? thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15806) An endpoint-based export tool
[ https://issues.apache.org/jira/browse/HBASE-15806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai updated HBASE-15806: -- Attachment: HBASE-15806.v7.patch v7 includes some trivial change due to [HBASE-15432|https://issues.apache.org/jira/browse/HBASE-15432] > An endpoint-based export tool > - > > Key: HBASE-15806 > URL: https://issues.apache.org/jira/browse/HBASE-15806 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: Experiment.png, HBASE-15806-v1.patch, > HBASE-15806-v2.patch, HBASE-15806-v3.patch, HBASE-15806.patch, > HBASE-15806.v4.patch, HBASE-15806.v5.patch, HBASE-15806.v6.patch, > HBASE-15806.v7.patch > > > The time for exporting table can be reduced, if we use the endpoint technique > to export the hdfs files by the region server rather than by hbase client. > In my experiments, the elapsed time of endpoint-based export can be less than > half of current export tool (enable the hdfs compression) > But the shortcomings is we need to alter table for deploying the endpoint > any comments about this? thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17361) HTable#getBufferedMutator is not thread safe and could cause data loss
[
https://issues.apache.org/jira/browse/HBASE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15772046#comment-15772046
]
Duo Zhang commented on HBASE-17361:
---
The problem is the timeout config in Table and AsyncTable. If we allow set
timeout and then make a call to the HBase cluster, then it is useless to just
make the calls to HBase cluster thread safe...
Maybe we should find another way to let use specific a timeout for each call?
> HTable#getBufferedMutator is not thread safe and could cause data loss
> --
>
> Key: HBASE-17361
> URL: https://issues.apache.org/jira/browse/HBASE-17361
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.7, 1.2.4
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Attachments: HBASE-17361.patch, HBASE-17361.patch
>
>
> Now we have {{HTable#getBufferedMutator}} like
> {code}
>BufferedMutator getBufferedMutator() throws IOException {
> if (mutator == null) {
> this.mutator = (BufferedMutatorImpl) connection.getBufferedMutator(
> new BufferedMutatorParams(tableName)
> .pool(pool)
> .writeBufferSize(connConfiguration.getWriteBufferSize())
> .maxKeyValueSize(connConfiguration.getMaxKeyValueSize())
> );
> }
> mutator.setRpcTimeout(writeRpcTimeout);
> mutator.setOperationTimeout(operationTimeout);
> return mutator;
> }
> {code}
> And {{HTable#flushCommits}}:
> {code}
> void flushCommits() throws IOException {
> if (mutator == null) {
> // nothing to flush if there's no mutator; don't bother creating one.
> return;
> }
> getBufferedMutator().flush();
> }
> {code}
> For {{HTable#put}}
> {code}
> public void put(final Put put) throws IOException {
> getBufferedMutator().mutate(put);
> flushCommits();
> }
> {code}
> If we launch multiple threads to put in parallel, below sequence might happen
> because {{HTable#getBufferedMutator}} is not thread safe:
> {noformat}
> 1. ThreadA runs to getBufferedMutator and finds mutator==null
> 2. ThreadB runs to getBufferedMutator and finds mutator==null
> 3. ThreadA initialize mutator to instanceA, then calls mutator#mutate,
> adding one put (putA) into {{writeAsyncBuffer}}
> 4. ThreadB initialize mutator to instanceB
> 5. ThreadA runs to flushCommits, now mutator is instanceB, it calls
> instanceB's flush method, putA is lost
> {noformat}
> Will add a UT to cover this case, and fix it in this JIRA.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15806) An endpoint-based export tool
[ https://issues.apache.org/jira/browse/HBASE-15806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai updated HBASE-15806: -- Status: Open (was: Patch Available) > An endpoint-based export tool > - > > Key: HBASE-15806 > URL: https://issues.apache.org/jira/browse/HBASE-15806 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: Experiment.png, HBASE-15806-v1.patch, > HBASE-15806-v2.patch, HBASE-15806-v3.patch, HBASE-15806.patch, > HBASE-15806.v4.patch, HBASE-15806.v5.patch, HBASE-15806.v6.patch > > > The time for exporting table can be reduced, if we use the endpoint technique > to export the hdfs files by the region server rather than by hbase client. > In my experiments, the elapsed time of endpoint-based export can be less than > half of current export tool (enable the hdfs compression) > But the shortcomings is we need to alter table for deploying the endpoint > any comments about this? thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17361) HTable#getBufferedMutator is not thread safe and could cause data loss
[
https://issues.apache.org/jira/browse/HBASE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15772032#comment-15772032
]
Duo Zhang commented on HBASE-17361:
---
We do not expose the BufferedMutator in HTable to upper layer I think? The
getBufferedMutator method is marked as @VisibleForTesting so I think it is a
private method, which means you should not use it outside HTable.
The right way is to get a BufferedMutator with the method in Connection.
{code:title=Connection.java}
BufferedMutator getBufferedMutator(TableName tableName) throws IOException;
BufferedMutator getBufferedMutator(BufferedMutatorParams params) throws
IOException;
{code}
You can call the methods of this BufferedMutator instance concurrently with
multiple threads.
> HTable#getBufferedMutator is not thread safe and could cause data loss
> --
>
> Key: HBASE-17361
> URL: https://issues.apache.org/jira/browse/HBASE-17361
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.7, 1.2.4
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Attachments: HBASE-17361.patch, HBASE-17361.patch
>
>
> Now we have {{HTable#getBufferedMutator}} like
> {code}
>BufferedMutator getBufferedMutator() throws IOException {
> if (mutator == null) {
> this.mutator = (BufferedMutatorImpl) connection.getBufferedMutator(
> new BufferedMutatorParams(tableName)
> .pool(pool)
> .writeBufferSize(connConfiguration.getWriteBufferSize())
> .maxKeyValueSize(connConfiguration.getMaxKeyValueSize())
> );
> }
> mutator.setRpcTimeout(writeRpcTimeout);
> mutator.setOperationTimeout(operationTimeout);
> return mutator;
> }
> {code}
> And {{HTable#flushCommits}}:
> {code}
> void flushCommits() throws IOException {
> if (mutator == null) {
> // nothing to flush if there's no mutator; don't bother creating one.
> return;
> }
> getBufferedMutator().flush();
> }
> {code}
> For {{HTable#put}}
> {code}
> public void put(final Put put) throws IOException {
> getBufferedMutator().mutate(put);
> flushCommits();
> }
> {code}
> If we launch multiple threads to put in parallel, below sequence might happen
> because {{HTable#getBufferedMutator}} is not thread safe:
> {noformat}
> 1. ThreadA runs to getBufferedMutator and finds mutator==null
> 2. ThreadB runs to getBufferedMutator and finds mutator==null
> 3. ThreadA initialize mutator to instanceA, then calls mutator#mutate,
> adding one put (putA) into {{writeAsyncBuffer}}
> 4. ThreadB initialize mutator to instanceB
> 5. ThreadA runs to flushCommits, now mutator is instanceB, it calls
> instanceB's flush method, putA is lost
> {noformat}
> Will add a UT to cover this case, and fix it in this JIRA.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17361) HTable#getBufferedMutator is not thread safe and could cause data loss
[
https://issues.apache.org/jira/browse/HBASE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15772023#comment-15772023
]
Phil Yang commented on HBASE-17361:
---
Same concern with [~jinghe]. Table/HTable is not thread safe but
BufferedMutator is. However it seems that there will not be any trouble if we
call most of methods in HTable in different threads... And I really suspect
that many users don't know Table is not thread safe :)
Maybe we can make Table thread-safe since 2.0? Especially it will not be a hard
work. After we having AsyncTable I think we can implement a thread-safe Table
by get a CompletableFuture and blocking on get(). Any ideas?
> HTable#getBufferedMutator is not thread safe and could cause data loss
> --
>
> Key: HBASE-17361
> URL: https://issues.apache.org/jira/browse/HBASE-17361
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.7, 1.2.4
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Attachments: HBASE-17361.patch, HBASE-17361.patch
>
>
> Now we have {{HTable#getBufferedMutator}} like
> {code}
>BufferedMutator getBufferedMutator() throws IOException {
> if (mutator == null) {
> this.mutator = (BufferedMutatorImpl) connection.getBufferedMutator(
> new BufferedMutatorParams(tableName)
> .pool(pool)
> .writeBufferSize(connConfiguration.getWriteBufferSize())
> .maxKeyValueSize(connConfiguration.getMaxKeyValueSize())
> );
> }
> mutator.setRpcTimeout(writeRpcTimeout);
> mutator.setOperationTimeout(operationTimeout);
> return mutator;
> }
> {code}
> And {{HTable#flushCommits}}:
> {code}
> void flushCommits() throws IOException {
> if (mutator == null) {
> // nothing to flush if there's no mutator; don't bother creating one.
> return;
> }
> getBufferedMutator().flush();
> }
> {code}
> For {{HTable#put}}
> {code}
> public void put(final Put put) throws IOException {
> getBufferedMutator().mutate(put);
> flushCommits();
> }
> {code}
> If we launch multiple threads to put in parallel, below sequence might happen
> because {{HTable#getBufferedMutator}} is not thread safe:
> {noformat}
> 1. ThreadA runs to getBufferedMutator and finds mutator==null
> 2. ThreadB runs to getBufferedMutator and finds mutator==null
> 3. ThreadA initialize mutator to instanceA, then calls mutator#mutate,
> adding one put (putA) into {{writeAsyncBuffer}}
> 4. ThreadB initialize mutator to instanceB
> 5. ThreadA runs to flushCommits, now mutator is instanceB, it calls
> instanceB's flush method, putA is lost
> {noformat}
> Will add a UT to cover this case, and fix it in this JIRA.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17361) HTable#getBufferedMutator is not thread safe and could cause data loss
[
https://issues.apache.org/jira/browse/HBASE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15772018#comment-15772018
]
Duo Zhang commented on HBASE-17361:
---
I think HTable is known to be not thread safe from long long ago?
> HTable#getBufferedMutator is not thread safe and could cause data loss
> --
>
> Key: HBASE-17361
> URL: https://issues.apache.org/jira/browse/HBASE-17361
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.7, 1.2.4
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Attachments: HBASE-17361.patch, HBASE-17361.patch
>
>
> Now we have {{HTable#getBufferedMutator}} like
> {code}
>BufferedMutator getBufferedMutator() throws IOException {
> if (mutator == null) {
> this.mutator = (BufferedMutatorImpl) connection.getBufferedMutator(
> new BufferedMutatorParams(tableName)
> .pool(pool)
> .writeBufferSize(connConfiguration.getWriteBufferSize())
> .maxKeyValueSize(connConfiguration.getMaxKeyValueSize())
> );
> }
> mutator.setRpcTimeout(writeRpcTimeout);
> mutator.setOperationTimeout(operationTimeout);
> return mutator;
> }
> {code}
> And {{HTable#flushCommits}}:
> {code}
> void flushCommits() throws IOException {
> if (mutator == null) {
> // nothing to flush if there's no mutator; don't bother creating one.
> return;
> }
> getBufferedMutator().flush();
> }
> {code}
> For {{HTable#put}}
> {code}
> public void put(final Put put) throws IOException {
> getBufferedMutator().mutate(put);
> flushCommits();
> }
> {code}
> If we launch multiple threads to put in parallel, below sequence might happen
> because {{HTable#getBufferedMutator}} is not thread safe:
> {noformat}
> 1. ThreadA runs to getBufferedMutator and finds mutator==null
> 2. ThreadB runs to getBufferedMutator and finds mutator==null
> 3. ThreadA initialize mutator to instanceA, then calls mutator#mutate,
> adding one put (putA) into {{writeAsyncBuffer}}
> 4. ThreadB initialize mutator to instanceB
> 5. ThreadA runs to flushCommits, now mutator is instanceB, it calls
> instanceB's flush method, putA is lost
> {noformat}
> Will add a UT to cover this case, and fix it in this JIRA.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17361) HTable#getBufferedMutator is not thread safe and could cause data loss
[
https://issues.apache.org/jira/browse/HBASE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15772011#comment-15772011
]
Yu Li commented on HBASE-17361:
---
Thanks for the reference to HTable javadoc [~jerryhe].
It's awkward that BufferedMutator is thread safe but never got chance to be
accessed in parallel
It's also true that if we share the same BufferedMutator per connection the
locking will be extreme as [~eclark] mentioned in HBASE-14687, see
{{BufferedMutatorImpl#flush}} which is synchronized...
Personally I think we should try to resolve the contention inside
BufferedMutator and then make HTable thread safe. We should resolve the issue
in a way or another rather than leaving it there after all...
I'm working on something interesting and have a pretty good reason of making
HTable thread safe, will share later when it's ready (smile). Before that,
let's just leave the JIRA open here for later check.
> HTable#getBufferedMutator is not thread safe and could cause data loss
> --
>
> Key: HBASE-17361
> URL: https://issues.apache.org/jira/browse/HBASE-17361
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.7, 1.2.4
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Attachments: HBASE-17361.patch, HBASE-17361.patch
>
>
> Now we have {{HTable#getBufferedMutator}} like
> {code}
>BufferedMutator getBufferedMutator() throws IOException {
> if (mutator == null) {
> this.mutator = (BufferedMutatorImpl) connection.getBufferedMutator(
> new BufferedMutatorParams(tableName)
> .pool(pool)
> .writeBufferSize(connConfiguration.getWriteBufferSize())
> .maxKeyValueSize(connConfiguration.getMaxKeyValueSize())
> );
> }
> mutator.setRpcTimeout(writeRpcTimeout);
> mutator.setOperationTimeout(operationTimeout);
> return mutator;
> }
> {code}
> And {{HTable#flushCommits}}:
> {code}
> void flushCommits() throws IOException {
> if (mutator == null) {
> // nothing to flush if there's no mutator; don't bother creating one.
> return;
> }
> getBufferedMutator().flush();
> }
> {code}
> For {{HTable#put}}
> {code}
> public void put(final Put put) throws IOException {
> getBufferedMutator().mutate(put);
> flushCommits();
> }
> {code}
> If we launch multiple threads to put in parallel, below sequence might happen
> because {{HTable#getBufferedMutator}} is not thread safe:
> {noformat}
> 1. ThreadA runs to getBufferedMutator and finds mutator==null
> 2. ThreadB runs to getBufferedMutator and finds mutator==null
> 3. ThreadA initialize mutator to instanceA, then calls mutator#mutate,
> adding one put (putA) into {{writeAsyncBuffer}}
> 4. ThreadB initialize mutator to instanceB
> 5. ThreadA runs to flushCommits, now mutator is instanceB, it calls
> instanceB's flush method, putA is lost
> {noformat}
> Will add a UT to cover this case, and fix it in this JIRA.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17361) HTable#getBufferedMutator is not thread safe and could cause data loss
[
https://issues.apache.org/jira/browse/HBASE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-17361:
--
Attachment: HBASE-17361.patch
> HTable#getBufferedMutator is not thread safe and could cause data loss
> --
>
> Key: HBASE-17361
> URL: https://issues.apache.org/jira/browse/HBASE-17361
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.7, 1.2.4
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Attachments: HBASE-17361.patch, HBASE-17361.patch
>
>
> Now we have {{HTable#getBufferedMutator}} like
> {code}
>BufferedMutator getBufferedMutator() throws IOException {
> if (mutator == null) {
> this.mutator = (BufferedMutatorImpl) connection.getBufferedMutator(
> new BufferedMutatorParams(tableName)
> .pool(pool)
> .writeBufferSize(connConfiguration.getWriteBufferSize())
> .maxKeyValueSize(connConfiguration.getMaxKeyValueSize())
> );
> }
> mutator.setRpcTimeout(writeRpcTimeout);
> mutator.setOperationTimeout(operationTimeout);
> return mutator;
> }
> {code}
> And {{HTable#flushCommits}}:
> {code}
> void flushCommits() throws IOException {
> if (mutator == null) {
> // nothing to flush if there's no mutator; don't bother creating one.
> return;
> }
> getBufferedMutator().flush();
> }
> {code}
> For {{HTable#put}}
> {code}
> public void put(final Put put) throws IOException {
> getBufferedMutator().mutate(put);
> flushCommits();
> }
> {code}
> If we launch multiple threads to put in parallel, below sequence might happen
> because {{HTable#getBufferedMutator}} is not thread safe:
> {noformat}
> 1. ThreadA runs to getBufferedMutator and finds mutator==null
> 2. ThreadB runs to getBufferedMutator and finds mutator==null
> 3. ThreadA initialize mutator to instanceA, then calls mutator#mutate,
> adding one put (putA) into {{writeAsyncBuffer}}
> 4. ThreadB initialize mutator to instanceB
> 5. ThreadA runs to flushCommits, now mutator is instanceB, it calls
> instanceB's flush method, putA is lost
> {noformat}
> Will add a UT to cover this case, and fix it in this JIRA.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11392) add/remove peer requests should be routed through master
[ https://issues.apache.org/jira/browse/HBASE-11392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15771948#comment-15771948 ] Guanghao Zhang commented on HBASE-11392: It is compatible. The old version client can operate zk directly. > add/remove peer requests should be routed through master > > > Key: HBASE-11392 > URL: https://issues.apache.org/jira/browse/HBASE-11392 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Guanghao Zhang >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-11392-v1.patch, HBASE-11392-v2.patch, > HBASE-11392-v3.patch, HBASE-11392-v4.patch, HBASE-11392-v5.patch, > HBASE-11392-v6.patch > > > ReplicationAdmin directly operates over the zookeeper data for replication > setup. We should move these operations to be routed through master for two > reasons: > - Replication implementation details are exposed to client. We should move > most of the replication related classes to hbase-server package. > - Routing the requests through master is the standard practice for all other > operations. It allows for decoupling implementation details from the client > and code. > Review board: https://reviews.apache.org/r/54730/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16981) Expand Mob Compaction Partition policy from daily to weekly, monthly and beyond
[ https://issues.apache.org/jira/browse/HBASE-16981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15771938#comment-15771938 ] Jingcheng Du edited comment on HBASE-16981 at 12/23/16 5:03 AM: Hi [~huaxiang], would you mind sharing the RB link for this patch? Thanks. And have you test the hbase shell already? was (Author: jingcheng.du): Hi [~huaxiang], would you mind sharing the RB link for this patch? Thanks. > Expand Mob Compaction Partition policy from daily to weekly, monthly and > beyond > --- > > Key: HBASE-16981 > URL: https://issues.apache.org/jira/browse/HBASE-16981 > Project: HBase > Issue Type: New Feature > Components: mob >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-16981.master.001.patch, > HBASE-16981.master.002.patch, HBASE-16981.master.003.patch, > HBASE-16981.master.004.patch, > Supportingweeklyandmonthlymobcompactionpartitionpolicyinhbase.pdf > > > Today the mob region holds all mob files for all regions. With daily > partition mob compaction policy, after major mob compaction, there is still > one file per region daily. Given there is 365 days in one year, at least 365 > files per region. Since HDFS has limitation for number of files under one > folder, this is not going to scale if there are lots of regions. To reduce > mob file number, we want to introduce other partition policies such as > weekly, monthly to compact mob files within one week or month into one file. > This jira is create to track this effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (HBASE-16981) Expand Mob Compaction Partition policy from daily to weekly, monthly and beyond
[ https://issues.apache.org/jira/browse/HBASE-16981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-16981: - Comment: was deleted (was: And have you tested the hbase shell already? Thanks.) > Expand Mob Compaction Partition policy from daily to weekly, monthly and > beyond > --- > > Key: HBASE-16981 > URL: https://issues.apache.org/jira/browse/HBASE-16981 > Project: HBase > Issue Type: New Feature > Components: mob >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-16981.master.001.patch, > HBASE-16981.master.002.patch, HBASE-16981.master.003.patch, > HBASE-16981.master.004.patch, > Supportingweeklyandmonthlymobcompactionpartitionpolicyinhbase.pdf > > > Today the mob region holds all mob files for all regions. With daily > partition mob compaction policy, after major mob compaction, there is still > one file per region daily. Given there is 365 days in one year, at least 365 > files per region. Since HDFS has limitation for number of files under one > folder, this is not going to scale if there are lots of regions. To reduce > mob file number, we want to introduce other partition policies such as > weekly, monthly to compact mob files within one week or month into one file. > This jira is create to track this effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16981) Expand Mob Compaction Partition policy from daily to weekly, monthly and beyond
[ https://issues.apache.org/jira/browse/HBASE-16981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15771941#comment-15771941 ] Jingcheng Du commented on HBASE-16981: -- And have you tested the hbase shell already? Thanks. > Expand Mob Compaction Partition policy from daily to weekly, monthly and > beyond > --- > > Key: HBASE-16981 > URL: https://issues.apache.org/jira/browse/HBASE-16981 > Project: HBase > Issue Type: New Feature > Components: mob >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-16981.master.001.patch, > HBASE-16981.master.002.patch, HBASE-16981.master.003.patch, > HBASE-16981.master.004.patch, > Supportingweeklyandmonthlymobcompactionpartitionpolicyinhbase.pdf > > > Today the mob region holds all mob files for all regions. With daily > partition mob compaction policy, after major mob compaction, there is still > one file per region daily. Given there is 365 days in one year, at least 365 > files per region. Since HDFS has limitation for number of files under one > folder, this is not going to scale if there are lots of regions. To reduce > mob file number, we want to introduce other partition policies such as > weekly, monthly to compact mob files within one week or month into one file. > This jira is create to track this effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16981) Expand Mob Compaction Partition policy from daily to weekly, monthly and beyond
[ https://issues.apache.org/jira/browse/HBASE-16981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15771938#comment-15771938 ] Jingcheng Du commented on HBASE-16981: -- Hi [~huaxiang], would you mind sharing the RB link for this patch? Thanks. > Expand Mob Compaction Partition policy from daily to weekly, monthly and > beyond > --- > > Key: HBASE-16981 > URL: https://issues.apache.org/jira/browse/HBASE-16981 > Project: HBase > Issue Type: New Feature > Components: mob >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-16981.master.001.patch, > HBASE-16981.master.002.patch, HBASE-16981.master.003.patch, > HBASE-16981.master.004.patch, > Supportingweeklyandmonthlymobcompactionpartitionpolicyinhbase.pdf > > > Today the mob region holds all mob files for all regions. With daily > partition mob compaction policy, after major mob compaction, there is still > one file per region daily. Given there is 365 days in one year, at least 365 > files per region. Since HDFS has limitation for number of files under one > folder, this is not going to scale if there are lots of regions. To reduce > mob file number, we want to introduce other partition policies such as > weekly, monthly to compact mob files within one week or month into one file. > This jira is create to track this effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17345) Implement batch
[
https://issues.apache.org/jira/browse/HBASE-17345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15771909#comment-15771909
]
Hadoop QA commented on HBASE-17345:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
35s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
23m 39s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 58s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 89m 20s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 128m 46s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12844508/HBASE-17345-v4.patch |
| JIRA Issue | HBASE-17345 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 6be9b661c8d7 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / b3f2bec |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5031/testReport/ |
| modules | C: hbase-client hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5031/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Implement batch
> ---
>
[jira] [Updated] (HBASE-17174) Refactor the AsyncProcess, BufferedMutatorImpl, and HTable
[ https://issues.apache.org/jira/browse/HBASE-17174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai updated HBASE-17174: -- Status: Patch Available (was: Open) > Refactor the AsyncProcess, BufferedMutatorImpl, and HTable > -- > > Key: HBASE-17174 > URL: https://issues.apache.org/jira/browse/HBASE-17174 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17174.v0.patch, HBASE-17174.v1.patch, > HBASE-17174.v10.patch, HBASE-17174.v11.patch, HBASE-17174.v11.patch, > HBASE-17174.v11.patch, HBASE-17174.v12.patch, HBASE-17174.v2.patch, > HBASE-17174.v3.patch, HBASE-17174.v4.patch, HBASE-17174.v5.patch, > HBASE-17174.v6.patch, HBASE-17174.v7.patch, HBASE-17174.v8.patch, > HBASE-17174.v9.patch > > > The following are reasons of refactoring. > # A update-heavy application, for example, loader, creates many > BufferedMutator for batch updates. But these BufferedMutators can’t share a > large threadpool because the shutdown() method will be called when closing > any BufferedMutator. This patch adds a flag into BufferedMutatorParams for > preventing calling the shutdown() method in BufferedMutatorImpl#close > # The AsyncProcess has the powerful traffic control, but the control is > against the single table currently. We should allow alternate traffic control > implementation for advanced user who want more control. > All suggestions are welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17174) Refactor the AsyncProcess, BufferedMutatorImpl, and HTable
[ https://issues.apache.org/jira/browse/HBASE-17174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai updated HBASE-17174: -- Attachment: HBASE-17174.v12.patch v12 addresses [~Apache9] suggestion > Refactor the AsyncProcess, BufferedMutatorImpl, and HTable > -- > > Key: HBASE-17174 > URL: https://issues.apache.org/jira/browse/HBASE-17174 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17174.v0.patch, HBASE-17174.v1.patch, > HBASE-17174.v10.patch, HBASE-17174.v11.patch, HBASE-17174.v11.patch, > HBASE-17174.v11.patch, HBASE-17174.v12.patch, HBASE-17174.v2.patch, > HBASE-17174.v3.patch, HBASE-17174.v4.patch, HBASE-17174.v5.patch, > HBASE-17174.v6.patch, HBASE-17174.v7.patch, HBASE-17174.v8.patch, > HBASE-17174.v9.patch > > > The following are reasons of refactoring. > # A update-heavy application, for example, loader, creates many > BufferedMutator for batch updates. But these BufferedMutators can’t share a > large threadpool because the shutdown() method will be called when closing > any BufferedMutator. This patch adds a flag into BufferedMutatorParams for > preventing calling the shutdown() method in BufferedMutatorImpl#close > # The AsyncProcess has the powerful traffic control, but the control is > against the single table currently. We should allow alternate traffic control > implementation for advanced user who want more control. > All suggestions are welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11392) add/remove peer requests should be routed through master
[ https://issues.apache.org/jira/browse/HBASE-11392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15771828#comment-15771828 ] Phil Yang commented on HBASE-11392: --- Hi [~zghaobac] Are your recently works incompatible? Can an old version client add/remove peer to a new version cluster? If not we can mark these issues as "Incompatible change"? Thanks. > add/remove peer requests should be routed through master > > > Key: HBASE-11392 > URL: https://issues.apache.org/jira/browse/HBASE-11392 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Guanghao Zhang >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-11392-v1.patch, HBASE-11392-v2.patch, > HBASE-11392-v3.patch, HBASE-11392-v4.patch, HBASE-11392-v5.patch, > HBASE-11392-v6.patch > > > ReplicationAdmin directly operates over the zookeeper data for replication > setup. We should move these operations to be routed through master for two > reasons: > - Replication implementation details are exposed to client. We should move > most of the replication related classes to hbase-server package. > - Routing the requests through master is the standard practice for all other > operations. It allows for decoupling implementation details from the client > and code. > Review board: https://reviews.apache.org/r/54730/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17174) Refactor the AsyncProcess, BufferedMutatorImpl, and HTable
[ https://issues.apache.org/jira/browse/HBASE-17174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai updated HBASE-17174: -- Status: Open (was: Patch Available) > Refactor the AsyncProcess, BufferedMutatorImpl, and HTable > -- > > Key: HBASE-17174 > URL: https://issues.apache.org/jira/browse/HBASE-17174 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17174.v0.patch, HBASE-17174.v1.patch, > HBASE-17174.v10.patch, HBASE-17174.v11.patch, HBASE-17174.v11.patch, > HBASE-17174.v11.patch, HBASE-17174.v2.patch, HBASE-17174.v3.patch, > HBASE-17174.v4.patch, HBASE-17174.v5.patch, HBASE-17174.v6.patch, > HBASE-17174.v7.patch, HBASE-17174.v8.patch, HBASE-17174.v9.patch > > > The following are reasons of refactoring. > # A update-heavy application, for example, loader, creates many > BufferedMutator for batch updates. But these BufferedMutators can’t share a > large threadpool because the shutdown() method will be called when closing > any BufferedMutator. This patch adds a flag into BufferedMutatorParams for > preventing calling the shutdown() method in BufferedMutatorImpl#close > # The AsyncProcess has the powerful traffic control, but the control is > against the single table currently. We should allow alternate traffic control > implementation for advanced user who want more control. > All suggestions are welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17345) Implement batch
[
https://issues.apache.org/jira/browse/HBASE-17345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15771805#comment-15771805
]
Hadoop QA commented on HBASE-17345:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
31s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 40s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 16s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 94m 41s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 140m 14s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12844497/HBASE-17345-v3.patch |
| JIRA Issue | HBASE-17345 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 5b083e2a4bab 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / b3f2bec |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5030/testReport/ |
| modules | C: hbase-client hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5030/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Implement batch
> ---
[jira] [Commented] (HBASE-17314) Limit total buffered size for all replication sources
[ https://issues.apache.org/jira/browse/HBASE-17314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15771793#comment-15771793 ] Phil Yang commented on HBASE-17314: --- Mock is better, will change the code when commit. > Limit total buffered size for all replication sources > - > > Key: HBASE-17314 > URL: https://issues.apache.org/jira/browse/HBASE-17314 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Phil Yang >Assignee: Phil Yang > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17314.branch-1.v01.patch, > HBASE-17314.branch-1.v02.patch, HBASE-17314.v01.patch, HBASE-17314.v02.patch, > HBASE-17314.v03.patch, HBASE-17314.v04.patch, HBASE-17314.v05.patch, > HBASE-17314.v06.patch > > > If we have many peers or some servers have many recovered queues, we will > hold many entries in memory which will increase the pressure of GC, even > maybe OOM because we will read entries for 64MB to buffer in default for one > source. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17359) Implement async admin
[ https://issues.apache.org/jira/browse/HBASE-17359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-17359: --- Description: And as we will return a CompletableFuture, I think we can just remove the XXXAsync methods, and make all the methods blocking which means we will only finish the CompletableFuture when the operation is done. User can choose whether to wait on the returned CompletableFuture. Convert this to a umbrella task. There maybe some sub-tasks. 1. Table admin operations. 2. Region admin operations. 3. Namespace admin operations. 4. Snapshot admin operations. 5. Replication admin operations. 6. Other operations, like quota, balance.. was:And as we will return a CompletableFuture, I think we can just remove the XXXAsync methods, and make all the methods blocking which means we will only finish the CompletableFuture when the operation is done. User can choose whether to wait on the returned CompletableFuture. > Implement async admin > - > > Key: HBASE-17359 > URL: https://issues.apache.org/jira/browse/HBASE-17359 > Project: HBase > Issue Type: Umbrella > Components: Client >Reporter: Duo Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0 > > > And as we will return a CompletableFuture, I think we can just remove the > XXXAsync methods, and make all the methods blocking which means we will only > finish the CompletableFuture when the operation is done. User can choose > whether to wait on the returned CompletableFuture. > Convert this to a umbrella task. There maybe some sub-tasks. > 1. Table admin operations. > 2. Region admin operations. > 3. Namespace admin operations. > 4. Snapshot admin operations. > 5. Replication admin operations. > 6. Other operations, like quota, balance.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17359) Implement async admin
[ https://issues.apache.org/jira/browse/HBASE-17359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-17359: --- Issue Type: Umbrella (was: Sub-task) Parent: (was: HBASE-16833) > Implement async admin > - > > Key: HBASE-17359 > URL: https://issues.apache.org/jira/browse/HBASE-17359 > Project: HBase > Issue Type: Umbrella > Components: Client >Reporter: Duo Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0 > > > And as we will return a CompletableFuture, I think we can just remove the > XXXAsync methods, and make all the methods blocking which means we will only > finish the CompletableFuture when the operation is done. User can choose > whether to wait on the returned CompletableFuture. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-17359) Implement async admin
[ https://issues.apache.org/jira/browse/HBASE-17359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang reassigned HBASE-17359: -- Assignee: Guanghao Zhang > Implement async admin > - > > Key: HBASE-17359 > URL: https://issues.apache.org/jira/browse/HBASE-17359 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Duo Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0 > > > And as we will return a CompletableFuture, I think we can just remove the > XXXAsync methods, and make all the methods blocking which means we will only > finish the CompletableFuture when the operation is done. User can choose > whether to wait on the returned CompletableFuture. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17330) SnapshotFileCache will always refresh the file cache
[
https://issues.apache.org/jira/browse/HBASE-17330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15771727#comment-15771727
]
Jianwei Cui commented on HBASE-17330:
-
Thanks for pointing out the mod time problem, [~stack]. I tried the patch
locally as:
1. start a client to take snapshot periodically;
2. make {{SnapshotFileCache#refreshCache}} log the loading hfile names each
time it scheduled.
The log shows {{SnapshotFileCache}} could load the hfiles referenced by
snapshots taken before {{refreshCache}} starting. However, as you mentioned,
relying on the mod time is risky, the accuracy of mod time depends on the
implementation of underlying file system, and the mod time could also be
updated(such as by {{FSNamesystem#setTimes}}). To be more safe, we can make
{{SnapshotFileCache#getUnreferencedFiles}} load hfile names through on-disk
snapshots if the passed file is not in memory cache? as:
{code}
public synchronized Iterable
getUnreferencedFiles(Iterable files,
final SnapshotManager snapshotManager)
throws IOException {
...
for (FileStatus file : files) {
String fileName = file.getPath().getName();
if (!refreshed && !cache.contains(fileName)) {
refreshCache(); // ==> Always load hfile names through on-disk
snapshots(not consider the mod time).
refreshed = true;
}
if (cache.contains(fileName)) {
continue;
}
{code}
> SnapshotFileCache will always refresh the file cache
>
>
> Key: HBASE-17330
> URL: https://issues.apache.org/jira/browse/HBASE-17330
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 2.0.0, 1.3.1, 0.98.23
>Reporter: Jianwei Cui
>Assignee: Jianwei Cui
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17330-v1.patch, HBASE-17330-v2.patch
>
>
> In {{SnapshotFileCache#refreshCache}}, the {{hasChanges}} will be judged as:
> {code}
> try {
> FileStatus dirStatus = fs.getFileStatus(snapshotDir);
> lastTimestamp = dirStatus.getModificationTime();
> hasChanges |= (lastTimestamp >= lastModifiedTime); // >= will make
> hasChanges always be true
> {code}
> The {{(lastTimestamp >= lastModifiedTime)}} will make {{hasChanges}} always
> be true because {{lastTimestamp}} will be updated as:
> {code}
> this.lastModifiedTime = lastTimestamp;
> {code}
> So, SnapshotFileCache will always refresh the file cache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17314) Limit total buffered size for all replication sources
[
https://issues.apache.org/jira/browse/HBASE-17314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15771682#comment-15771682
]
Guanghao Zhang commented on HBASE-17314:
{code}
207 if (manager == null) {
208 // It must in a unit test, it is ok to use a local one.
209 this.totalBufferUsed = new AtomicLong();
210 } else {
211 this.totalBufferUsed = manager.getTotalBufferUsed();
212 }
{code}
How about mock a ReplicationSourceManager in the unit test?
> Limit total buffered size for all replication sources
> -
>
> Key: HBASE-17314
> URL: https://issues.apache.org/jira/browse/HBASE-17314
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17314.branch-1.v01.patch,
> HBASE-17314.branch-1.v02.patch, HBASE-17314.v01.patch, HBASE-17314.v02.patch,
> HBASE-17314.v03.patch, HBASE-17314.v04.patch, HBASE-17314.v05.patch,
> HBASE-17314.v06.patch
>
>
> If we have many peers or some servers have many recovered queues, we will
> hold many entries in memory which will increase the pressure of GC, even
> maybe OOM because we will read entries for 64MB to buffer in default for one
> source.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17345) Implement batch
[ https://issues.apache.org/jira/browse/HBASE-17345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15771663#comment-15771663 ] Duo Zhang commented on HBASE-17345: --- Any other concerns? [~stack] [~carp84]. Thanks. > Implement batch > --- > > Key: HBASE-17345 > URL: https://issues.apache.org/jira/browse/HBASE-17345 > Project: HBase > Issue Type: Sub-task > Components: asyncclient, Client >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17345-v1.patch, HBASE-17345-v2.patch, > HBASE-17345-v3.patch, HBASE-17345-v4.patch, HBASE-17345.patch > > > Add the support for general batch based on the code introduced in HBASE-17142. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17345) Implement batch
[ https://issues.apache.org/jira/browse/HBASE-17345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17345: -- Attachment: HBASE-17345-v4.patch Fix rpc timeout. > Implement batch > --- > > Key: HBASE-17345 > URL: https://issues.apache.org/jira/browse/HBASE-17345 > Project: HBase > Issue Type: Sub-task > Components: asyncclient, Client >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17345-v1.patch, HBASE-17345-v2.patch, > HBASE-17345-v3.patch, HBASE-17345-v4.patch, HBASE-17345.patch > > > Add the support for general batch based on the code introduced in HBASE-17142. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17345) Implement batch
[
https://issues.apache.org/jira/browse/HBASE-17345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15771587#comment-15771587
]
Duo Zhang commented on HBASE-17345:
---
{quote}
About ConnectionUtils:
In voidBatch and voidBatchAll, mind explain why table. batch(actions)
rather than table. batch(actions)?
{quote}
We do not care about the return value of put or delete, but I'm sure that we do
have a return value for these operations and it is not Void... So a safe way is
to keep it as Object and map it to ''null'. 'null' can be any type so it is
safe to cast it to Void.
{quote}
About AsyncTableBase:
1. Add javadoc for newly added methods: exists(List)/existsAll,
put(List)/putAll, delete(List)/deleteAll, batch(List)/batchAll?
2. Add more UT cases to cover them?
{quote}
Done.
{quote}
About TestAsyncGetMultiThread:
1. Now it's making chaos for each split key, including split-and-compact,
balance and move, and sleep 5 seconds in between each, which will make the test
run for over 2 min. Maybe simplify it a little bit to make the test finish
faster?
2. Also feel the name confusing, maybe TestAsyncGetWithMultiThread is better?
{quote}
I've renamed it to TestAsyncTableGetMultiThreaded. And it is designed to test
the failover of our async client implementation stack, so I think it is
reasonable to keep it running for a while?
> Implement batch
> ---
>
> Key: HBASE-17345
> URL: https://issues.apache.org/jira/browse/HBASE-17345
> Project: HBase
> Issue Type: Sub-task
> Components: asyncclient, Client
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17345-v1.patch, HBASE-17345-v2.patch,
> HBASE-17345-v3.patch, HBASE-17345.patch
>
>
> Add the support for general batch based on the code introduced in HBASE-17142.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17335) enable/disable replication peer requests should be routed through master
[ https://issues.apache.org/jira/browse/HBASE-17335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-17335: --- Resolution: Fixed Status: Resolved (was: Patch Available) > enable/disable replication peer requests should be routed through master > > > Key: HBASE-17335 > URL: https://issues.apache.org/jira/browse/HBASE-17335 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17335-v1.patch > > > As HBASE-11392 description says, we should move replication operations to be > routed through master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17335) enable/disable replication peer requests should be routed through master
[ https://issues.apache.org/jira/browse/HBASE-17335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15771580#comment-15771580 ] Guanghao Zhang commented on HBASE-17335: Pushed to master. Thanks [~enis] for reviewing. > enable/disable replication peer requests should be routed through master > > > Key: HBASE-17335 > URL: https://issues.apache.org/jira/browse/HBASE-17335 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17335-v1.patch > > > As HBASE-11392 description says, we should move replication operations to be > routed through master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17359) Implement async admin
[ https://issues.apache.org/jira/browse/HBASE-17359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15771569#comment-15771569 ] Duo Zhang commented on HBASE-17359: --- Sure. Go ahead. > Implement async admin > - > > Key: HBASE-17359 > URL: https://issues.apache.org/jira/browse/HBASE-17359 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Duo Zhang > Fix For: 2.0.0 > > > And as we will return a CompletableFuture, I think we can just remove the > XXXAsync methods, and make all the methods blocking which means we will only > finish the CompletableFuture when the operation is done. User can choose > whether to wait on the returned CompletableFuture. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17359) Implement async admin
[ https://issues.apache.org/jira/browse/HBASE-17359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15771565#comment-15771565 ] Guanghao Zhang commented on HBASE-17359: [~Apache9] I am interested in this. Can I assign this issue to me? > Implement async admin > - > > Key: HBASE-17359 > URL: https://issues.apache.org/jira/browse/HBASE-17359 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Duo Zhang > Fix For: 2.0.0 > > > And as we will return a CompletableFuture, I think we can just remove the > XXXAsync methods, and make all the methods blocking which means we will only > finish the CompletableFuture when the operation is done. User can choose > whether to wait on the returned CompletableFuture. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17345) Implement batch
[ https://issues.apache.org/jira/browse/HBASE-17345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17345: -- Attachment: HBASE-17345-v3.patch Remove debug code. > Implement batch > --- > > Key: HBASE-17345 > URL: https://issues.apache.org/jira/browse/HBASE-17345 > Project: HBase > Issue Type: Sub-task > Components: asyncclient, Client >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17345-v1.patch, HBASE-17345-v2.patch, > HBASE-17345-v3.patch, HBASE-17345.patch > > > Add the support for general batch based on the code introduced in HBASE-17142. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17345) Implement batch
[
https://issues.apache.org/jira/browse/HBASE-17345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15771555#comment-15771555
]
Duo Zhang commented on HBASE-17345:
---
{quote}
It has to be an Impl in the below, it can't be Interface?
private final AsyncConnectionImpl conn;
{quote}
We need to use some internal field of AsyncConnectionImpl. For example, the
rpcControllerFactory. We can declare a interface like ClusterConnection, but I
do not think it is necessary as now we only have one implementation, can do it
later.
> Implement batch
> ---
>
> Key: HBASE-17345
> URL: https://issues.apache.org/jira/browse/HBASE-17345
> Project: HBase
> Issue Type: Sub-task
> Components: asyncclient, Client
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17345-v1.patch, HBASE-17345-v2.patch,
> HBASE-17345.patch
>
>
> Add the support for general batch based on the code introduced in HBASE-17142.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17366) Run TestHFile#testReaderWithoutBlockCache failes
[
https://issues.apache.org/jira/browse/HBASE-17366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15771544#comment-15771544
]
Hadoop QA commented on HBASE-17366:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 9s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
0s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 10s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 90m 21s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 128m 55s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12844485/HBASE-17366-master-001.patch
|
| JIRA Issue | HBASE-17366 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 64f04f4dbaf2 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 45da294 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5029/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5029/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Run TestHFile#testReaderWithoutBlockCache failes
>
>
> Key: HBASE-17366
> URL: https://issues.apache.org/jira/browse/HBASE-17366
> Project: HBase
> Issue Type: Bug
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Trivial
> Fix For: 2.0.0
>
> Attachments: HBASE-17366-master-001.patch
>
>
> Running
[jira] [Commented] (HBASE-15905) Makefile build env incorrectly links in tests
[ https://issues.apache.org/jira/browse/HBASE-15905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15771503#comment-15771503 ] Enis Soztutar commented on HBASE-15905: --- Can you also rebase the patch please. > Makefile build env incorrectly links in tests > - > > Key: HBASE-15905 > URL: https://issues.apache.org/jira/browse/HBASE-15905 > Project: HBase > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Priyadharshini karthikeyan > Attachments: HBASE-15905.HBASE-14850.v1.patch, > HBASE-15905.HBASE-14850.v2.patch, HBASE-15905.HBASE-14850.v3.patch > > > Right now the makefile build system doesn't seem to do so well. > * Tests are included in the lib > * Documentation includes the protobuf dir > * just running make on a clean check out fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15905) Makefile build env incorrectly links in tests
[
https://issues.apache.org/jira/browse/HBASE-15905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15771501#comment-15771501
]
Enis Soztutar commented on HBASE-15905:
---
[~sudeeps] the patch looks good.
Why are we not removing the lib artifacts in {{make clean}}
{code}
- @rm -rf $(DEBUG_BUILD_DIR) $(RELEASE_BUILD_DIR) $(LIB_RELEASE)
$(LIB_DEBUG) $(ARC_RELEASE) $(ARC_DEBUG) docs buck-out build
+ @rm -rf docs buck-out build
{code}
> Makefile build env incorrectly links in tests
> -
>
> Key: HBASE-15905
> URL: https://issues.apache.org/jira/browse/HBASE-15905
> Project: HBase
> Issue Type: Sub-task
>Reporter: Elliott Clark
>Assignee: Priyadharshini karthikeyan
> Attachments: HBASE-15905.HBASE-14850.v1.patch,
> HBASE-15905.HBASE-14850.v2.patch, HBASE-15905.HBASE-14850.v3.patch
>
>
> Right now the makefile build system doesn't seem to do so well.
> * Tests are included in the lib
> * Documentation includes the protobuf dir
> * just running make on a clean check out fails.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16010) Put draining function through Admin API
[
https://issues.apache.org/jira/browse/HBASE-16010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15771486#comment-15771486
]
Enis Soztutar commented on HBASE-16010:
---
bq. Let me open a subtask right away and make sure it will be in.
Ok, as long as there is a critical issue marked against the versions, and a
commitment to fix it, I'll be happy.
bq. You think we should combine the two steps into one?
I think so, having the draining functionality in a ruby script is not
acceptable (for the long term). That is why I was asking for the plans around
that. We do not have to address them now, but at least track those in a jira.
> Put draining function through Admin API
> ---
>
> Key: HBASE-16010
> URL: https://issues.apache.org/jira/browse/HBASE-16010
> Project: HBase
> Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Matt Warhaftig
>Priority: Minor
> Attachments: hbase-16010-v1.patch, hbase-16010-v2.patch
>
>
> Currently, there is no Amdin API for draining function. Client has to
> interact directly with Zookeeper draining node to add and remove draining
> servers.
> For example, in draining_servers.rb:
> {code}
> zkw = org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.new(config,
> "draining_servers", nil)
> parentZnode = zkw.drainingZNode
> begin
> for server in servers
> node = ZKUtil.joinZNode(parentZnode, server)
> ZKUtil.createAndFailSilent(zkw, node)
> end
> ensure
> zkw.close()
> end
> {code}
> This is not good in cases like secure clusters with protected Zookeeper nodes.
> Let's put draining function through Admin API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17366) Run TestHFile#testReaderWithoutBlockCache failes
[ https://issues.apache.org/jira/browse/HBASE-17366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15771381#comment-15771381 ] huaxiang sun commented on HBASE-17366: -- Thanks [~vrodionov] for more info, will update accordingly. > Run TestHFile#testReaderWithoutBlockCache failes > > > Key: HBASE-17366 > URL: https://issues.apache.org/jira/browse/HBASE-17366 > Project: HBase > Issue Type: Bug >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-17366-master-001.patch > > > Running TestHFile#testReaderWithoutBlockCache failes because cacheConf is not > initialized. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-17366) Run TestHFile#testReaderWithoutBlockCache failes
[ https://issues.apache.org/jira/browse/HBASE-17366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15771360#comment-15771360 ] Vladimir Rodionov edited comment on HBASE-17366 at 12/22/16 11:14 PM: -- Seems some recent patch introduced this issue. In this case it is not clear how this patch got approval. Interesting. We need cache-less HFile.Writer implementation for applications which writes HFile(s) directly. Please refer to CacheConf.DISABLED and to HFile.Reader CacheConf - less constructor. It is in HBASE-17152. was (Author: vrodionov): Seems some recent patch introduced this issue. In this case it is not clear how this patch got approval. Interesting. We need cache-less HFile.Writer implementation for applications which writes HFile(s) directly. Please refer to CacheConf.DISABLED and to HFile.Reader CacheConf - less constructor. Please refer to HBASE-17152. > Run TestHFile#testReaderWithoutBlockCache failes > > > Key: HBASE-17366 > URL: https://issues.apache.org/jira/browse/HBASE-17366 > Project: HBase > Issue Type: Bug >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-17366-master-001.patch > > > Running TestHFile#testReaderWithoutBlockCache failes because cacheConf is not > initialized. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-17366) Run TestHFile#testReaderWithoutBlockCache failes
[ https://issues.apache.org/jira/browse/HBASE-17366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15771360#comment-15771360 ] Vladimir Rodionov edited comment on HBASE-17366 at 12/22/16 11:13 PM: -- Seems some recent patch introduced this issue. In this case it is not clear how this patch got approval. Interesting. We need cache-less HFile.Writer implementation for applications which writes HFile(s) directly. Please refer to CacheConf.DISABLED and to HFile.Reader CacheConf - less constructor. Please refer to HBASE-17152. was (Author: vrodionov): Seems some recent patch introduced this issue. In this case it is not clear how this patch got approval. Interesting. We need cache-less HFile.Writer implementation for applications which writes HFile(s) directly. Please refer to CacheConf.DISABLED and to HFile.Reader CacheConf - less constructor. > Run TestHFile#testReaderWithoutBlockCache failes > > > Key: HBASE-17366 > URL: https://issues.apache.org/jira/browse/HBASE-17366 > Project: HBase > Issue Type: Bug >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-17366-master-001.patch > > > Running TestHFile#testReaderWithoutBlockCache failes because cacheConf is not > initialized. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17366) Run TestHFile#testReaderWithoutBlockCache failes
[ https://issues.apache.org/jira/browse/HBASE-17366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15771360#comment-15771360 ] Vladimir Rodionov commented on HBASE-17366: --- Seems some recent patch introduced this issue. In this case it is not clear how this patch got approval. Interesting. We need cache-less HFile.Writer implementation for applications which writes HFile(s) directly. Please refer to CacheConf.DISABLED and to HFile.Reader CacheConf - less constructor. > Run TestHFile#testReaderWithoutBlockCache failes > > > Key: HBASE-17366 > URL: https://issues.apache.org/jira/browse/HBASE-17366 > Project: HBase > Issue Type: Bug >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-17366-master-001.patch > > > Running TestHFile#testReaderWithoutBlockCache failes because cacheConf is not > initialized. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17366) Run TestHFile#testReaderWithoutBlockCache failes
[ https://issues.apache.org/jira/browse/HBASE-17366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun updated HBASE-17366: - Attachment: HBASE-17366-master-001.patch Minor fix. > Run TestHFile#testReaderWithoutBlockCache failes > > > Key: HBASE-17366 > URL: https://issues.apache.org/jira/browse/HBASE-17366 > Project: HBase > Issue Type: Bug >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-17366-master-001.patch > > > Running TestHFile#testReaderWithoutBlockCache failes because cacheConf is not > initialized. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17366) Run TestHFile#testReaderWithoutBlockCache failes
[ https://issues.apache.org/jira/browse/HBASE-17366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun updated HBASE-17366: - Status: Patch Available (was: Open) > Run TestHFile#testReaderWithoutBlockCache failes > > > Key: HBASE-17366 > URL: https://issues.apache.org/jira/browse/HBASE-17366 > Project: HBase > Issue Type: Bug >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-17366-master-001.patch > > > Running TestHFile#testReaderWithoutBlockCache failes because cacheConf is not > initialized. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17366) Run TestHFile#testReaderWithoutBlockCache failes
huaxiang sun created HBASE-17366: Summary: Run TestHFile#testReaderWithoutBlockCache failes Key: HBASE-17366 URL: https://issues.apache.org/jira/browse/HBASE-17366 Project: HBase Issue Type: Bug Reporter: huaxiang sun Assignee: huaxiang sun Priority: Trivial Fix For: 2.0.0 Running TestHFile#testReaderWithoutBlockCache failes because cacheConf is not initialized. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16981) Expand Mob Compaction Partition policy from daily to weekly, monthly and beyond
[ https://issues.apache.org/jira/browse/HBASE-16981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15771325#comment-15771325 ] huaxiang sun commented on HBASE-16981: -- Thanks Jingcheng. I updated the diff a bit to clean out the doc warning. This is a minor change in the doc part so I did not upload the new diff in the RB, just upload it here. > Expand Mob Compaction Partition policy from daily to weekly, monthly and > beyond > --- > > Key: HBASE-16981 > URL: https://issues.apache.org/jira/browse/HBASE-16981 > Project: HBase > Issue Type: New Feature > Components: mob >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-16981.master.001.patch, > HBASE-16981.master.002.patch, HBASE-16981.master.003.patch, > HBASE-16981.master.004.patch, > Supportingweeklyandmonthlymobcompactionpartitionpolicyinhbase.pdf > > > Today the mob region holds all mob files for all regions. With daily > partition mob compaction policy, after major mob compaction, there is still > one file per region daily. Given there is 365 days in one year, at least 365 > files per region. Since HDFS has limitation for number of files under one > folder, this is not going to scale if there are lots of regions. To reduce > mob file number, we want to introduce other partition policies such as > weekly, monthly to compact mob files within one week or month into one file. > This jira is create to track this effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17362) HBase Backup/Restore Phase 4
[ https://issues.apache.org/jira/browse/HBASE-17362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-17362: -- Description: The umbrella JIRA for next features of the backup/restore module (was: The umbrella JIRA for next features of backup/restore module) > HBase Backup/Restore Phase 4 > > > Key: HBASE-17362 > URL: https://issues.apache.org/jira/browse/HBASE-17362 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > The umbrella JIRA for next features of the backup/restore module -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17362) HBase Backup/Restore Phase 4
[ https://issues.apache.org/jira/browse/HBASE-17362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15771313#comment-15771313 ] Vladimir Rodionov commented on HBASE-17362: --- cc: [~te...@apache.org], [~devaraj], [~enis] > HBase Backup/Restore Phase 4 > > > Key: HBASE-17362 > URL: https://issues.apache.org/jira/browse/HBASE-17362 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > The umbrella JIRA for next features of backup/restore module -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17365) Data at rest encryption support (cloud backups)
Vladimir Rodionov created HBASE-17365: - Summary: Data at rest encryption support (cloud backups) Key: HBASE-17365 URL: https://issues.apache.org/jira/browse/HBASE-17365 Project: HBase Issue Type: New Feature Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Support encryption of data at rest in cloud backup destinations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16981) Expand Mob Compaction Partition policy from daily to weekly, monthly and beyond
[
https://issues.apache.org/jira/browse/HBASE-16981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15771309#comment-15771309
]
Hadoop QA commented on HBASE-16981:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} rubocop {color} | {color:blue} 0m 0s
{color} | {color:blue} rubocop was not available. {color} |
| {color:blue}0{color} | {color:blue} ruby-lint {color} | {color:blue} 0m 0s
{color} | {color:blue} Ruby-lint was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 5s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 10s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 8s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 100m 52s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 31s
{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 156m 34s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12844454/HBASE-16981.master.004.patch
|
| JIRA Issue | HBASE-16981 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile rubocop ruby_lint |
| uname | Linux 1d9c12e18c9b 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revis
[jira] [Created] (HBASE-17364) Point in time (PIT) restore
Vladimir Rodionov created HBASE-17364: - Summary: Point in time (PIT) restore Key: HBASE-17364 URL: https://issues.apache.org/jira/browse/HBASE-17364 Project: HBase Issue Type: New Feature Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov For applications where HBase version is a real timestamp we can implement PIT, including *restore current state* option. *Restore current state* command will restore table from the most recent backup and then continue restore operation by re-playing edits from WAL files which are still around. Another option : restore time range of a table -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17283) [C++] Result class
[
https://issues.apache.org/jira/browse/HBASE-17283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15771295#comment-15771295
]
Enis Soztutar commented on HBASE-17283:
---
bq. Using a return by value will have an overhead of temporaries. Returning a
reference of local variable is not safe. Since we need the Cell instance to
live beyond the scope of the method we can create it on heap and we are using a
smart pointer i.e. unique_ptr in our case.
Thanks for the explanations above. I was asking about why are we returning
unique_ptr with copying the Cell, versus returning Cells uncopied with
shared_ptr's. I think Result should keep cells as vector> and
return vectors of shared_ptrs, so that there is no copying of the data when the
application reads back the cells from Result objects. Another alternative is
having everything return by reference to Cells or values, and tie the returned
stuff's lifetimes to that of the Result object. Not sure which one would be
better for consumers.
The first if is not needed here:
{code}
+ if (!cells.empty()) {
+for (const auto &cell : cells) {
{code}
here as well:
{code}
+ if (!result_map_.empty()) {
+if (!IsEmpty()) {
{code}
The iterators will not iterate anyway, no?
Result is a construct-once, and read-only object. Why are you checking the
row_.size here in the ctor? You can simply do the second line, no?
{code}
+ if (0 == row_.size()) {
+row_ = (0 == cells_.size() ? "" : cells_[0]->Row());
+ }
{code}
This is a nit, but in java we use (x == 0) rather than (0 == x). We should use
that form unless it is more common in C++. See
https://en.wikipedia.org/wiki/Yoda_conditions.
The maps should not be using {{unsigned long}}, because the Timestamp is not a
{{long}} anymore. We are using int64_t, did you see the changes I've pushed in
HBASE-17220?
{code}
std::map>
{code}
Can you please run {{bin/format-code}} and also {{make lint}} for the incoming
patches. The cpplint is unfortunately will report existing issues as well
(which we do not need to address in this patch), but if there are issues that
are due to the new patch, we should address those.
This should be fine, you can remove the TODO (but keep the comment). We only
build the Result object once when reading back from an RPC. Partial results
might change that, but we can address those issues when we have that support
down the line.
{code}
+ //TODO Feedback needed.
+ // We create the map when cells are added. unlike java where map is
created when result.getMap() is called
{code}
The following is by-design. In most of the use cases, the client is only
interested in the latest version, and by default there is only 1 version
anyway. So this is a convenience method. Maybe add a javadoc in FamilyMap()
method saying something like {{ Returns map of qualifiers to values, only
includes latest values}}.
{code}
+ //TODO Feedback needed.
+ //We break after inserting the first value. Result.java takes
only the first value
{code}
> [C++] Result class
> --
>
> Key: HBASE-17283
> URL: https://issues.apache.org/jira/browse/HBASE-17283
> Project: HBase
> Issue Type: Sub-task
>Reporter: Sudeep Sunthankar
>Assignee: Sudeep Sunthankar
> Attachments: HBASE-17283.HBASE-14850.v1.patch,
> HBASE-17283.HBASE-14850.v3.patch, hbase-17283_v2.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-17363) Snapshot-less backup
Vladimir Rodionov created HBASE-17363: - Summary: Snapshot-less backup Key: HBASE-17363 URL: https://issues.apache.org/jira/browse/HBASE-17363 Project: HBase Issue Type: New Feature Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Currently, during full table backup we perform table snapshot. Too long and not very robust. This is brainstorming JIRA, how can we improve snapshots or avoid them completely. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17362) HBase Backup/Restore Phase 4
[ https://issues.apache.org/jira/browse/HBASE-17362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-17362: -- Summary: HBase Backup/Restore Phase 4 (was: HBase BAckup/Restore Phase 4) > HBase Backup/Restore Phase 4 > > > Key: HBASE-17362 > URL: https://issues.apache.org/jira/browse/HBASE-17362 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > The umbrella JIRA for next features of backup/restore module -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-17362) HBase BAckup/Restore Phase 4
Vladimir Rodionov created HBASE-17362: - Summary: HBase BAckup/Restore Phase 4 Key: HBASE-17362 URL: https://issues.apache.org/jira/browse/HBASE-17362 Project: HBase Issue Type: Umbrella Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov The umbrella JIRA for next features of backup/restore module -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-17283) [C++] Result class
[ https://issues.apache.org/jira/browse/HBASE-17283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-17283: -- Summary: [C++] Result class (was: [C++] Result and ResultScanner classes) > [C++] Result class > -- > > Key: HBASE-17283 > URL: https://issues.apache.org/jira/browse/HBASE-17283 > Project: HBase > Issue Type: Sub-task >Reporter: Sudeep Sunthankar >Assignee: Sudeep Sunthankar > Attachments: HBASE-17283.HBASE-14850.v1.patch, > HBASE-17283.HBASE-14850.v3.patch, hbase-17283_v2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17335) enable/disable replication peer requests should be routed through master
[ https://issues.apache.org/jira/browse/HBASE-17335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15771069#comment-15771069 ] Enis Soztutar commented on HBASE-17335: --- +1. > enable/disable replication peer requests should be routed through master > > > Key: HBASE-17335 > URL: https://issues.apache.org/jira/browse/HBASE-17335 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17335-v1.patch > > > As HBASE-11392 description says, we should move replication operations to be > routed through master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17314) Limit total buffered size for all replication sources
[
https://issues.apache.org/jira/browse/HBASE-17314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15771029#comment-15771029
]
Ted Yu commented on HBASE-17314:
Patch v6 looks good.
{code}
106 public static void setDownAfterClass() throws Exception {
{code}
Fix the typo above: set -> tear
> Limit total buffered size for all replication sources
> -
>
> Key: HBASE-17314
> URL: https://issues.apache.org/jira/browse/HBASE-17314
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17314.branch-1.v01.patch,
> HBASE-17314.branch-1.v02.patch, HBASE-17314.v01.patch, HBASE-17314.v02.patch,
> HBASE-17314.v03.patch, HBASE-17314.v04.patch, HBASE-17314.v05.patch,
> HBASE-17314.v06.patch
>
>
> If we have many peers or some servers have many recovered queues, we will
> hold many entries in memory which will increase the pressure of GC, even
> maybe OOM because we will read entries for 64MB to buffer in default for one
> source.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16981) Expand Mob Compaction Partition policy from daily to weekly, monthly and beyond
[ https://issues.apache.org/jira/browse/HBASE-16981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun updated HBASE-16981: - Attachment: HBASE-16981.master.004.patch > Expand Mob Compaction Partition policy from daily to weekly, monthly and > beyond > --- > > Key: HBASE-16981 > URL: https://issues.apache.org/jira/browse/HBASE-16981 > Project: HBase > Issue Type: New Feature > Components: mob >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-16981.master.001.patch, > HBASE-16981.master.002.patch, HBASE-16981.master.003.patch, > HBASE-16981.master.004.patch, > Supportingweeklyandmonthlymobcompactionpartitionpolicyinhbase.pdf > > > Today the mob region holds all mob files for all regions. With daily > partition mob compaction policy, after major mob compaction, there is still > one file per region daily. Given there is 365 days in one year, at least 365 > files per region. Since HDFS has limitation for number of files under one > folder, this is not going to scale if there are lots of regions. To reduce > mob file number, we want to introduce other partition policies such as > weekly, monthly to compact mob files within one week or month into one file. > This jira is create to track this effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17160) Undo unnecessary inter-module dependency; spark to hbase-it and hbase-it to shell
[ https://issues.apache.org/jira/browse/HBASE-17160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15770941#comment-15770941 ] Hudson commented on HBASE-17160: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2179 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2179/]) HBASE-17160 Undo unnecessary inter-module dependency; spark to hbase-it (stack: rev 45da294a171faea10b34b022c9d3990bdb72d0e8) * (edit) hbase-endpoint/pom.xml > Undo unnecessary inter-module dependency; spark to hbase-it and hbase-it to > shell > - > > Key: HBASE-17160 > URL: https://issues.apache.org/jira/browse/HBASE-17160 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17160.addendum.txt, HBASE-17160.master.001.patch, > HBASE-17160.master.002.patch, HBASE-17160.master.002.patch, hbase.png, > minor_hbase.png, untangled_hbase.png > > > Very minor untangling. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17306) IntegrationTestRSGroup#testRegionMove may fail due to region server not online
[
https://issues.apache.org/jira/browse/HBASE-17306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15770869#comment-15770869
]
Francis Liu commented on HBASE-17306:
-
[~tedyu] I can't recall that I have.
> IntegrationTestRSGroup#testRegionMove may fail due to region server not online
> --
>
> Key: HBASE-17306
> URL: https://issues.apache.org/jira/browse/HBASE-17306
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Priority: Minor
> Attachments: 17306.v1.txt
>
>
> {code}
> 2016-12-13 05:26:57,965|INFO|MainThread|machine.py:145 - run()|2)
> testRegionMove(org.apache.hadoop.hbase.rsgroup.IntegrationTestRSGroup)
> 2016-12-13 05:26:57,965|INFO|MainThread|machine.py:145 -
> run()|org.apache.hadoop.hbase.constraint.ConstraintException:
> org.apache.hadoop.hbase.constraint.ConstraintException:
> Server ctr-e77-1481596162056-0240-01-05.a.com:16020 is not an online
> server in default group.
> 2016-12-13 05:26:57,966|INFO|MainThread|machine.py:145 - run()|at
> org.apache.hadoop.hbase.rsgroup.RSGroupAdminServer.moveServers(RSGroupAdminServer.java:135)
> 2016-12-13 05:26:57,966|INFO|MainThread|machine.py:145 - run()|at
> org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint.moveServers(RSGroupAdminEndpoint.java:169)
> 2016-12-13 05:26:57,966|INFO|MainThread|machine.py:145 - run()|at
> org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos$RSGroupAdminService.
> callMethod(RSGroupAdminProtos.java:11136)
> 2016-12-13 05:26:57,966|INFO|MainThread|machine.py:145 - run()|at
> org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:679)
> 2016-12-13 05:26:57,966|INFO|MainThread|machine.py:145 - run()|at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2
> {code}
> Shortly before the test failure, the server was shutdown:
> {code}
> 2016-12-13 05:21:25,428 INFO
> [MASTER_SERVER_OPERATIONS-ctr-e77-1481596162056-0240-01-08:2-4]
> handler.ServerShutdownHandler: Finished processing of shutdown of ctr-
> e77-1481596162056-0240-01-05.a.com,16020,1481606309159
> ...
> 2016-12-13 05:26:57,935 INFO
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=2]
> master.ServerManager: Registering
> server=ctr-e77-1481596162056-0240-01-05.hwx. site,16020,1481606803303
> 2016-12-13 05:27:06,219 DEBUG [main-EventThread]
> zookeeper.RegionServerTracker: Added tracking of RS
> /hbase-secure/rs/ctr-e77-1481596162056-0240-01-05.a.com,16020,
> 1481606803303
> {code}
> The registration of the new server (start code1481606803303) happened shortly
> after the test failure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17069) RegionServer writes invalid META entries for split daughters in some circumstances
[
https://issues.apache.org/jira/browse/HBASE-17069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15770860#comment-15770860
]
stack commented on HBASE-17069:
---
Thanks [~apurtell]
> RegionServer writes invalid META entries for split daughters in some
> circumstances
> --
>
> Key: HBASE-17069
> URL: https://issues.apache.org/jira/browse/HBASE-17069
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.4
>Reporter: Andrew Purtell
>Priority: Critical
> Attachments: daughter_1_d55ef81c2f8299abbddfce0445067830.log,
> daughter_2_08629d59564726da2497f70451aafcdb.log, logs.tar.gz,
> parent-393d2bfd8b1c52ce08540306659624f2.log
>
>
> I have been seeing frequent ITBLL failures testing various versions of 1.2.x.
> Over the lifetime of 1.2.x the following issues have been fixed:
> - HBASE-15315 (Remove always set super user call as high priority)
> - HBASE-16093 (Fix splits failed before creating daughter regions leave meta
> inconsistent)
> And this one is pending:
> - HBASE-17044 (Fix merge failed before creating merged region leaves meta
> inconsistent)
> I can apply all of the above to branch-1.2 and still see this failure:
> *The life of stillborn region d55ef81c2f8299abbddfce0445067830*
> *Master sees SPLITTING_NEW*
> {noformat}
> 2016-11-08 04:23:21,186 INFO [AM.ZK.Worker-pool2-t82] master.RegionStates:
> Transition null to {d55ef81c2f8299abbddfce0445067830 state=SPLITTING_NEW,
> ts=1478579001186, server=node-3.cluster,16020,1478578389506}
> {noformat}
> *The RegionServer creates it*
> {noformat}
> 2016-11-08 04:23:26,035 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for GomnU: blockCache=LruBlockCache{blockCount=34,
> currentSize=14996112, freeSize=12823716208, maxSize=12838712320,
> heapSize=14996112, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,038 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for big: blockCache=LruBlockCache{blockCount=34,
> currentSize=14996112, freeSize=12823716208, maxSize=12838712320,
> heapSize=14996112, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,442 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for meta: blockCache=LruBlockCache{blockCount=63,
> currentSize=17187656, freeSize=12821524664, maxSize=12838712320,
> heapSize=17187656, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,713 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for nwmrW: blockCache=LruBlockCache{blockCount=96,
> currentSize=19178440, freeSize=12819533880, maxSize=12838712320,
> heapSize=19178440, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,715 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for piwbr: blockCache=LruBlockCache{blockCount=96,
> currentSize=19178440, freeSize=12819533880, maxSize=12838712320,
> heapSize=19178440, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,717 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for tiny: blockCache=LruBlockCache{blockCount=96,
> currentSize=19178440, freeSize=12819533880, maxSize=12838712320,
> heapSize=19178440, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, sing
[jira] [Commented] (HBASE-17361) HTable#getBufferedMutator is not thread safe and could cause data loss
[
https://issues.apache.org/jira/browse/HBASE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15770838#comment-15770838
]
Jerry He commented on HBASE-17361:
--
HBASE-17174 also tries to share AsyncProcess, BufferedMutator and their
relations with HTable.
> HTable#getBufferedMutator is not thread safe and could cause data loss
> --
>
> Key: HBASE-17361
> URL: https://issues.apache.org/jira/browse/HBASE-17361
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.7, 1.2.4
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Attachments: HBASE-17361.patch
>
>
> Now we have {{HTable#getBufferedMutator}} like
> {code}
>BufferedMutator getBufferedMutator() throws IOException {
> if (mutator == null) {
> this.mutator = (BufferedMutatorImpl) connection.getBufferedMutator(
> new BufferedMutatorParams(tableName)
> .pool(pool)
> .writeBufferSize(connConfiguration.getWriteBufferSize())
> .maxKeyValueSize(connConfiguration.getMaxKeyValueSize())
> );
> }
> mutator.setRpcTimeout(writeRpcTimeout);
> mutator.setOperationTimeout(operationTimeout);
> return mutator;
> }
> {code}
> And {{HTable#flushCommits}}:
> {code}
> void flushCommits() throws IOException {
> if (mutator == null) {
> // nothing to flush if there's no mutator; don't bother creating one.
> return;
> }
> getBufferedMutator().flush();
> }
> {code}
> For {{HTable#put}}
> {code}
> public void put(final Put put) throws IOException {
> getBufferedMutator().mutate(put);
> flushCommits();
> }
> {code}
> If we launch multiple threads to put in parallel, below sequence might happen
> because {{HTable#getBufferedMutator}} is not thread safe:
> {noformat}
> 1. ThreadA runs to getBufferedMutator and finds mutator==null
> 2. ThreadB runs to getBufferedMutator and finds mutator==null
> 3. ThreadA initialize mutator to instanceA, then calls mutator#mutate,
> adding one put (putA) into {{writeAsyncBuffer}}
> 4. ThreadB initialize mutator to instanceB
> 5. ThreadA runs to flushCommits, now mutator is instanceB, it calls
> instanceB's flush method, putA is lost
> {noformat}
> Will add a UT to cover this case, and fix it in this JIRA.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17069) RegionServer writes invalid META entries for split daughters in some circumstances
[
https://issues.apache.org/jira/browse/HBASE-17069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15770836#comment-15770836
]
Andrew Purtell commented on HBASE-17069:
TRACE logging really slows everything down. The test has been running for ~24
hours, is not finished yet, has not reproduced the problem. The timing is off
and we are always winning the race instead of occasionally losing. Will redo
with DEBUG level logging next week after the holiday.
[~stack]
> RegionServer writes invalid META entries for split daughters in some
> circumstances
> --
>
> Key: HBASE-17069
> URL: https://issues.apache.org/jira/browse/HBASE-17069
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.4
>Reporter: Andrew Purtell
>Priority: Critical
> Attachments: daughter_1_d55ef81c2f8299abbddfce0445067830.log,
> daughter_2_08629d59564726da2497f70451aafcdb.log, logs.tar.gz,
> parent-393d2bfd8b1c52ce08540306659624f2.log
>
>
> I have been seeing frequent ITBLL failures testing various versions of 1.2.x.
> Over the lifetime of 1.2.x the following issues have been fixed:
> - HBASE-15315 (Remove always set super user call as high priority)
> - HBASE-16093 (Fix splits failed before creating daughter regions leave meta
> inconsistent)
> And this one is pending:
> - HBASE-17044 (Fix merge failed before creating merged region leaves meta
> inconsistent)
> I can apply all of the above to branch-1.2 and still see this failure:
> *The life of stillborn region d55ef81c2f8299abbddfce0445067830*
> *Master sees SPLITTING_NEW*
> {noformat}
> 2016-11-08 04:23:21,186 INFO [AM.ZK.Worker-pool2-t82] master.RegionStates:
> Transition null to {d55ef81c2f8299abbddfce0445067830 state=SPLITTING_NEW,
> ts=1478579001186, server=node-3.cluster,16020,1478578389506}
> {noformat}
> *The RegionServer creates it*
> {noformat}
> 2016-11-08 04:23:26,035 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for GomnU: blockCache=LruBlockCache{blockCount=34,
> currentSize=14996112, freeSize=12823716208, maxSize=12838712320,
> heapSize=14996112, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,038 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for big: blockCache=LruBlockCache{blockCount=34,
> currentSize=14996112, freeSize=12823716208, maxSize=12838712320,
> heapSize=14996112, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,442 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for meta: blockCache=LruBlockCache{blockCount=63,
> currentSize=17187656, freeSize=12821524664, maxSize=12838712320,
> heapSize=17187656, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,713 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for nwmrW: blockCache=LruBlockCache{blockCount=96,
> currentSize=19178440, freeSize=12819533880, maxSize=12838712320,
> heapSize=19178440, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,715 INFO
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created
> cacheConfig for piwbr: blockCache=LruBlockCache{blockCount=96,
> currentSize=19178440, freeSize=12819533880, maxSize=12838712320,
> heapSize=19178440, minSize=12196776960, minFactor=0.95, multiSize=6098388480,
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false
> 2016-11-08 04:23:26,717 INFO
> [StoreOpener-d55ef81c2f8299abbddfce04
[jira] [Commented] (HBASE-17361) HTable#getBufferedMutator is not thread safe and could cause data loss
[
https://issues.apache.org/jira/browse/HBASE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15770792#comment-15770792
]
stack commented on HBASE-17361:
---
Yes.
" * This class is NOT thread safe for reads nor writes.
* In the case of writes (Put, Delete), the underlying write buffer can
* be corrupted if multiple threads contend over a single HTable instance.
* In the case of reads, some fields used by a Scan are shared among all
threads."
Then HBASE-14687 talks of BM being thread-safe.
"... the javadoc for BufferdMutator makes it sound like HTable where there
should be one per thread. That ends up being really really bad as there's one
AsyncProcess per buffered mutator. This can lead to a single regionserver being
pounded. However if you don't use more than one buffered mutator then the
locking is extreme and contention eats up your cpu."
I do not know how to reconcile the two contending concerns above.
> HTable#getBufferedMutator is not thread safe and could cause data loss
> --
>
> Key: HBASE-17361
> URL: https://issues.apache.org/jira/browse/HBASE-17361
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.7, 1.2.4
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Attachments: HBASE-17361.patch
>
>
> Now we have {{HTable#getBufferedMutator}} like
> {code}
>BufferedMutator getBufferedMutator() throws IOException {
> if (mutator == null) {
> this.mutator = (BufferedMutatorImpl) connection.getBufferedMutator(
> new BufferedMutatorParams(tableName)
> .pool(pool)
> .writeBufferSize(connConfiguration.getWriteBufferSize())
> .maxKeyValueSize(connConfiguration.getMaxKeyValueSize())
> );
> }
> mutator.setRpcTimeout(writeRpcTimeout);
> mutator.setOperationTimeout(operationTimeout);
> return mutator;
> }
> {code}
> And {{HTable#flushCommits}}:
> {code}
> void flushCommits() throws IOException {
> if (mutator == null) {
> // nothing to flush if there's no mutator; don't bother creating one.
> return;
> }
> getBufferedMutator().flush();
> }
> {code}
> For {{HTable#put}}
> {code}
> public void put(final Put put) throws IOException {
> getBufferedMutator().mutate(put);
> flushCommits();
> }
> {code}
> If we launch multiple threads to put in parallel, below sequence might happen
> because {{HTable#getBufferedMutator}} is not thread safe:
> {noformat}
> 1. ThreadA runs to getBufferedMutator and finds mutator==null
> 2. ThreadB runs to getBufferedMutator and finds mutator==null
> 3. ThreadA initialize mutator to instanceA, then calls mutator#mutate,
> adding one put (putA) into {{writeAsyncBuffer}}
> 4. ThreadB initialize mutator to instanceB
> 5. ThreadA runs to flushCommits, now mutator is instanceB, it calls
> instanceB's flush method, putA is lost
> {noformat}
> Will add a UT to cover this case, and fix it in this JIRA.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17361) HTable#getBufferedMutator is not thread safe and could cause data loss
[
https://issues.apache.org/jira/browse/HBASE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15770793#comment-15770793
]
stack commented on HBASE-17361:
---
But see below...
> HTable#getBufferedMutator is not thread safe and could cause data loss
> --
>
> Key: HBASE-17361
> URL: https://issues.apache.org/jira/browse/HBASE-17361
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.7, 1.2.4
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Attachments: HBASE-17361.patch
>
>
> Now we have {{HTable#getBufferedMutator}} like
> {code}
>BufferedMutator getBufferedMutator() throws IOException {
> if (mutator == null) {
> this.mutator = (BufferedMutatorImpl) connection.getBufferedMutator(
> new BufferedMutatorParams(tableName)
> .pool(pool)
> .writeBufferSize(connConfiguration.getWriteBufferSize())
> .maxKeyValueSize(connConfiguration.getMaxKeyValueSize())
> );
> }
> mutator.setRpcTimeout(writeRpcTimeout);
> mutator.setOperationTimeout(operationTimeout);
> return mutator;
> }
> {code}
> And {{HTable#flushCommits}}:
> {code}
> void flushCommits() throws IOException {
> if (mutator == null) {
> // nothing to flush if there's no mutator; don't bother creating one.
> return;
> }
> getBufferedMutator().flush();
> }
> {code}
> For {{HTable#put}}
> {code}
> public void put(final Put put) throws IOException {
> getBufferedMutator().mutate(put);
> flushCommits();
> }
> {code}
> If we launch multiple threads to put in parallel, below sequence might happen
> because {{HTable#getBufferedMutator}} is not thread safe:
> {noformat}
> 1. ThreadA runs to getBufferedMutator and finds mutator==null
> 2. ThreadB runs to getBufferedMutator and finds mutator==null
> 3. ThreadA initialize mutator to instanceA, then calls mutator#mutate,
> adding one put (putA) into {{writeAsyncBuffer}}
> 4. ThreadB initialize mutator to instanceB
> 5. ThreadA runs to flushCommits, now mutator is instanceB, it calls
> instanceB's flush method, putA is lost
> {noformat}
> Will add a UT to cover this case, and fix it in this JIRA.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17361) HTable#getBufferedMutator is not thread safe and could cause data loss
[
https://issues.apache.org/jira/browse/HBASE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15770777#comment-15770777
]
stack commented on HBASE-17361:
---
Thanks for the reference. Ok on the patch then.
> HTable#getBufferedMutator is not thread safe and could cause data loss
> --
>
> Key: HBASE-17361
> URL: https://issues.apache.org/jira/browse/HBASE-17361
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.7, 1.2.4
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Attachments: HBASE-17361.patch
>
>
> Now we have {{HTable#getBufferedMutator}} like
> {code}
>BufferedMutator getBufferedMutator() throws IOException {
> if (mutator == null) {
> this.mutator = (BufferedMutatorImpl) connection.getBufferedMutator(
> new BufferedMutatorParams(tableName)
> .pool(pool)
> .writeBufferSize(connConfiguration.getWriteBufferSize())
> .maxKeyValueSize(connConfiguration.getMaxKeyValueSize())
> );
> }
> mutator.setRpcTimeout(writeRpcTimeout);
> mutator.setOperationTimeout(operationTimeout);
> return mutator;
> }
> {code}
> And {{HTable#flushCommits}}:
> {code}
> void flushCommits() throws IOException {
> if (mutator == null) {
> // nothing to flush if there's no mutator; don't bother creating one.
> return;
> }
> getBufferedMutator().flush();
> }
> {code}
> For {{HTable#put}}
> {code}
> public void put(final Put put) throws IOException {
> getBufferedMutator().mutate(put);
> flushCommits();
> }
> {code}
> If we launch multiple threads to put in parallel, below sequence might happen
> because {{HTable#getBufferedMutator}} is not thread safe:
> {noformat}
> 1. ThreadA runs to getBufferedMutator and finds mutator==null
> 2. ThreadB runs to getBufferedMutator and finds mutator==null
> 3. ThreadA initialize mutator to instanceA, then calls mutator#mutate,
> adding one put (putA) into {{writeAsyncBuffer}}
> 4. ThreadB initialize mutator to instanceB
> 5. ThreadA runs to flushCommits, now mutator is instanceB, it calls
> instanceB's flush method, putA is lost
> {noformat}
> Will add a UT to cover this case, and fix it in this JIRA.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17361) HTable#getBufferedMutator is not thread safe and could cause data loss
[
https://issues.apache.org/jira/browse/HBASE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15770759#comment-15770759
]
Jerry He commented on HBASE-17361:
--
HTable level is not guaranteed to be thread safe, and it is a private internal
class now? Or I miss something?
> HTable#getBufferedMutator is not thread safe and could cause data loss
> --
>
> Key: HBASE-17361
> URL: https://issues.apache.org/jira/browse/HBASE-17361
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.7, 1.2.4
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Attachments: HBASE-17361.patch
>
>
> Now we have {{HTable#getBufferedMutator}} like
> {code}
>BufferedMutator getBufferedMutator() throws IOException {
> if (mutator == null) {
> this.mutator = (BufferedMutatorImpl) connection.getBufferedMutator(
> new BufferedMutatorParams(tableName)
> .pool(pool)
> .writeBufferSize(connConfiguration.getWriteBufferSize())
> .maxKeyValueSize(connConfiguration.getMaxKeyValueSize())
> );
> }
> mutator.setRpcTimeout(writeRpcTimeout);
> mutator.setOperationTimeout(operationTimeout);
> return mutator;
> }
> {code}
> And {{HTable#flushCommits}}:
> {code}
> void flushCommits() throws IOException {
> if (mutator == null) {
> // nothing to flush if there's no mutator; don't bother creating one.
> return;
> }
> getBufferedMutator().flush();
> }
> {code}
> For {{HTable#put}}
> {code}
> public void put(final Put put) throws IOException {
> getBufferedMutator().mutate(put);
> flushCommits();
> }
> {code}
> If we launch multiple threads to put in parallel, below sequence might happen
> because {{HTable#getBufferedMutator}} is not thread safe:
> {noformat}
> 1. ThreadA runs to getBufferedMutator and finds mutator==null
> 2. ThreadB runs to getBufferedMutator and finds mutator==null
> 3. ThreadA initialize mutator to instanceA, then calls mutator#mutate,
> adding one put (putA) into {{writeAsyncBuffer}}
> 4. ThreadB initialize mutator to instanceB
> 5. ThreadA runs to flushCommits, now mutator is instanceB, it calls
> instanceB's flush method, putA is lost
> {noformat}
> Will add a UT to cover this case, and fix it in this JIRA.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17361) HTable#getBufferedMutator is not thread safe and could cause data loss
[
https://issues.apache.org/jira/browse/HBASE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15770731#comment-15770731
]
Ted Yu commented on HBASE-17361:
+1 on patch.
> HTable#getBufferedMutator is not thread safe and could cause data loss
> --
>
> Key: HBASE-17361
> URL: https://issues.apache.org/jira/browse/HBASE-17361
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.7, 1.2.4
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Attachments: HBASE-17361.patch
>
>
> Now we have {{HTable#getBufferedMutator}} like
> {code}
>BufferedMutator getBufferedMutator() throws IOException {
> if (mutator == null) {
> this.mutator = (BufferedMutatorImpl) connection.getBufferedMutator(
> new BufferedMutatorParams(tableName)
> .pool(pool)
> .writeBufferSize(connConfiguration.getWriteBufferSize())
> .maxKeyValueSize(connConfiguration.getMaxKeyValueSize())
> );
> }
> mutator.setRpcTimeout(writeRpcTimeout);
> mutator.setOperationTimeout(operationTimeout);
> return mutator;
> }
> {code}
> And {{HTable#flushCommits}}:
> {code}
> void flushCommits() throws IOException {
> if (mutator == null) {
> // nothing to flush if there's no mutator; don't bother creating one.
> return;
> }
> getBufferedMutator().flush();
> }
> {code}
> For {{HTable#put}}
> {code}
> public void put(final Put put) throws IOException {
> getBufferedMutator().mutate(put);
> flushCommits();
> }
> {code}
> If we launch multiple threads to put in parallel, below sequence might happen
> because {{HTable#getBufferedMutator}} is not thread safe:
> {noformat}
> 1. ThreadA runs to getBufferedMutator and finds mutator==null
> 2. ThreadB runs to getBufferedMutator and finds mutator==null
> 3. ThreadA initialize mutator to instanceA, then calls mutator#mutate,
> adding one put (putA) into {{writeAsyncBuffer}}
> 4. ThreadB initialize mutator to instanceB
> 5. ThreadA runs to flushCommits, now mutator is instanceB, it calls
> instanceB's flush method, putA is lost
> {noformat}
> Will add a UT to cover this case, and fix it in this JIRA.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17361) HTable#getBufferedMutator is not thread safe and could cause data loss
[
https://issues.apache.org/jira/browse/HBASE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15770703#comment-15770703
]
Yu Li commented on HBASE-17361:
---
btw, making all mutator access synchronized will downgrade performance, please
refer to HBASE-14687 for more details.
> HTable#getBufferedMutator is not thread safe and could cause data loss
> --
>
> Key: HBASE-17361
> URL: https://issues.apache.org/jira/browse/HBASE-17361
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.7, 1.2.4
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Attachments: HBASE-17361.patch
>
>
> Now we have {{HTable#getBufferedMutator}} like
> {code}
>BufferedMutator getBufferedMutator() throws IOException {
> if (mutator == null) {
> this.mutator = (BufferedMutatorImpl) connection.getBufferedMutator(
> new BufferedMutatorParams(tableName)
> .pool(pool)
> .writeBufferSize(connConfiguration.getWriteBufferSize())
> .maxKeyValueSize(connConfiguration.getMaxKeyValueSize())
> );
> }
> mutator.setRpcTimeout(writeRpcTimeout);
> mutator.setOperationTimeout(operationTimeout);
> return mutator;
> }
> {code}
> And {{HTable#flushCommits}}:
> {code}
> void flushCommits() throws IOException {
> if (mutator == null) {
> // nothing to flush if there's no mutator; don't bother creating one.
> return;
> }
> getBufferedMutator().flush();
> }
> {code}
> For {{HTable#put}}
> {code}
> public void put(final Put put) throws IOException {
> getBufferedMutator().mutate(put);
> flushCommits();
> }
> {code}
> If we launch multiple threads to put in parallel, below sequence might happen
> because {{HTable#getBufferedMutator}} is not thread safe:
> {noformat}
> 1. ThreadA runs to getBufferedMutator and finds mutator==null
> 2. ThreadB runs to getBufferedMutator and finds mutator==null
> 3. ThreadA initialize mutator to instanceA, then calls mutator#mutate,
> adding one put (putA) into {{writeAsyncBuffer}}
> 4. ThreadB initialize mutator to instanceB
> 5. ThreadA runs to flushCommits, now mutator is instanceB, it calls
> instanceB's flush method, putA is lost
> {noformat}
> Will add a UT to cover this case, and fix it in this JIRA.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17361) HTable#getBufferedMutator is not thread safe and could cause data loss
[
https://issues.apache.org/jira/browse/HBASE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15770655#comment-15770655
]
Yu Li commented on HBASE-17361:
---
Not necessarily IMO, only need to make sure of thread safe during
initialization, then volatile should be enough. It's a similar case as
{{RegionGroupingProvider#getWAL}} except for we only have a single instance
instead of a map here (so a simple lock is enough instead of using
{{IdReadWriteLock}} in RegionGroupingProvider). Or please let me know if any
other concerns sir, thanks.
> HTable#getBufferedMutator is not thread safe and could cause data loss
> --
>
> Key: HBASE-17361
> URL: https://issues.apache.org/jira/browse/HBASE-17361
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.7, 1.2.4
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Attachments: HBASE-17361.patch
>
>
> Now we have {{HTable#getBufferedMutator}} like
> {code}
>BufferedMutator getBufferedMutator() throws IOException {
> if (mutator == null) {
> this.mutator = (BufferedMutatorImpl) connection.getBufferedMutator(
> new BufferedMutatorParams(tableName)
> .pool(pool)
> .writeBufferSize(connConfiguration.getWriteBufferSize())
> .maxKeyValueSize(connConfiguration.getMaxKeyValueSize())
> );
> }
> mutator.setRpcTimeout(writeRpcTimeout);
> mutator.setOperationTimeout(operationTimeout);
> return mutator;
> }
> {code}
> And {{HTable#flushCommits}}:
> {code}
> void flushCommits() throws IOException {
> if (mutator == null) {
> // nothing to flush if there's no mutator; don't bother creating one.
> return;
> }
> getBufferedMutator().flush();
> }
> {code}
> For {{HTable#put}}
> {code}
> public void put(final Put put) throws IOException {
> getBufferedMutator().mutate(put);
> flushCommits();
> }
> {code}
> If we launch multiple threads to put in parallel, below sequence might happen
> because {{HTable#getBufferedMutator}} is not thread safe:
> {noformat}
> 1. ThreadA runs to getBufferedMutator and finds mutator==null
> 2. ThreadB runs to getBufferedMutator and finds mutator==null
> 3. ThreadA initialize mutator to instanceA, then calls mutator#mutate,
> adding one put (putA) into {{writeAsyncBuffer}}
> 4. ThreadB initialize mutator to instanceB
> 5. ThreadA runs to flushCommits, now mutator is instanceB, it calls
> instanceB's flush method, putA is lost
> {noformat}
> Will add a UT to cover this case, and fix it in this JIRA.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17335) enable/disable replication peer requests should be routed through master
[
https://issues.apache.org/jira/browse/HBASE-17335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15770644#comment-15770644
]
Hadoop QA commented on HBASE-17335:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 33s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 11m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
35s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 50s
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 29s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 3m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 11m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
33s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 14 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
23m 39s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 2m
31s {color} | {color:green} the patch passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 1s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 88m 30s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 110m 54s
{color} | {color:green} root in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
2s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 283m 46s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
[jira] [Commented] (HBASE-17330) SnapshotFileCache will always refresh the file cache
[
https://issues.apache.org/jira/browse/HBASE-17330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15770633#comment-15770633
]
stack commented on HBASE-17330:
---
[~cuijianwei] Have you tried your fix? Is mod time reliable?
> SnapshotFileCache will always refresh the file cache
>
>
> Key: HBASE-17330
> URL: https://issues.apache.org/jira/browse/HBASE-17330
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 2.0.0, 1.3.1, 0.98.23
>Reporter: Jianwei Cui
>Assignee: Jianwei Cui
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17330-v1.patch, HBASE-17330-v2.patch
>
>
> In {{SnapshotFileCache#refreshCache}}, the {{hasChanges}} will be judged as:
> {code}
> try {
> FileStatus dirStatus = fs.getFileStatus(snapshotDir);
> lastTimestamp = dirStatus.getModificationTime();
> hasChanges |= (lastTimestamp >= lastModifiedTime); // >= will make
> hasChanges always be true
> {code}
> The {{(lastTimestamp >= lastModifiedTime)}} will make {{hasChanges}} always
> be true because {{lastTimestamp}} will be updated as:
> {code}
> this.lastModifiedTime = lastTimestamp;
> {code}
> So, SnapshotFileCache will always refresh the file cache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17361) HTable#getBufferedMutator is not thread safe and could cause data loss
[
https://issues.apache.org/jira/browse/HBASE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15770626#comment-15770626
]
stack commented on HBASE-17361:
---
Thanks. Should we just synchronize all buffered mutator access? Would be more
straight-forward than volatile with null and a dedicated lock? Thanks [~carp84]
> HTable#getBufferedMutator is not thread safe and could cause data loss
> --
>
> Key: HBASE-17361
> URL: https://issues.apache.org/jira/browse/HBASE-17361
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.7, 1.2.4
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Attachments: HBASE-17361.patch
>
>
> Now we have {{HTable#getBufferedMutator}} like
> {code}
>BufferedMutator getBufferedMutator() throws IOException {
> if (mutator == null) {
> this.mutator = (BufferedMutatorImpl) connection.getBufferedMutator(
> new BufferedMutatorParams(tableName)
> .pool(pool)
> .writeBufferSize(connConfiguration.getWriteBufferSize())
> .maxKeyValueSize(connConfiguration.getMaxKeyValueSize())
> );
> }
> mutator.setRpcTimeout(writeRpcTimeout);
> mutator.setOperationTimeout(operationTimeout);
> return mutator;
> }
> {code}
> And {{HTable#flushCommits}}:
> {code}
> void flushCommits() throws IOException {
> if (mutator == null) {
> // nothing to flush if there's no mutator; don't bother creating one.
> return;
> }
> getBufferedMutator().flush();
> }
> {code}
> For {{HTable#put}}
> {code}
> public void put(final Put put) throws IOException {
> getBufferedMutator().mutate(put);
> flushCommits();
> }
> {code}
> If we launch multiple threads to put in parallel, below sequence might happen
> because {{HTable#getBufferedMutator}} is not thread safe:
> {noformat}
> 1. ThreadA runs to getBufferedMutator and finds mutator==null
> 2. ThreadB runs to getBufferedMutator and finds mutator==null
> 3. ThreadA initialize mutator to instanceA, then calls mutator#mutate,
> adding one put (putA) into {{writeAsyncBuffer}}
> 4. ThreadB initialize mutator to instanceB
> 5. ThreadA runs to flushCommits, now mutator is instanceB, it calls
> instanceB's flush method, putA is lost
> {noformat}
> Will add a UT to cover this case, and fix it in this JIRA.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17341) Add a timeout during replication endpoint termination
[ https://issues.apache.org/jira/browse/HBASE-17341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15770585#comment-15770585 ] Hudson commented on HBASE-17341: FAILURE: Integrated in Jenkins build HBase-0.98-on-Hadoop-1.1 #1300 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1300/]) HBASE-17341 Add a timeout during replication endpoint termination (apurtell: rev 0eff7dffb9acc16b8d8f6b073a13c3f8a5fb6b5c) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSource.java > Add a timeout during replication endpoint termination > - > > Key: HBASE-17341 > URL: https://issues.apache.org/jira/browse/HBASE-17341 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 0.98.23, 1.2.4 >Reporter: Vincent Poon >Assignee: Vincent Poon >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 0.98.24, 1.1.8 > > Attachments: HBASE-17341.branch-1.1.v1.patch, > HBASE-17341.branch-1.1.v2.patch, HBASE-17341.master.v1.patch, > HBASE-17341.master.v2.patch > > > In ReplicationSource#terminate(), a Future is obtained from > ReplicationEndpoint#stop(). Future.get() is then called, but can potentially > hang there if something went wrong in the endpoint stop(). > Hanging there has serious implications, because the thread could potentially > be the ZK event thread (e.g. watcher calls > ReplicationSourceManager#removePeer() -> ReplicationSource#terminate() -> > blocked). This means no other events in the ZK event queue will get > processed, which for HBase means other ZK watches such as replication watch > notifications, snapshot watch notifications, even RegionServer shutdown will > all get blocked. > The short term fix addressed here is to simply add a timeout for > Future.get(). But the severe consequences seen here perhaps suggest a > broader refactoring of the ZKWatcher usage in HBase is in order, to protect > against situations like this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17361) HTable#getBufferedMutator is not thread safe and could cause data loss
[
https://issues.apache.org/jira/browse/HBASE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15770482#comment-15770482
]
Jean-Marc Spaggiari commented on HBASE-17361:
-
But is this going to impact the performances?
> HTable#getBufferedMutator is not thread safe and could cause data loss
> --
>
> Key: HBASE-17361
> URL: https://issues.apache.org/jira/browse/HBASE-17361
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Attachments: HBASE-17361.patch
>
>
> Now we have {{HTable#getBufferedMutator}} like
> {code}
>BufferedMutator getBufferedMutator() throws IOException {
> if (mutator == null) {
> this.mutator = (BufferedMutatorImpl) connection.getBufferedMutator(
> new BufferedMutatorParams(tableName)
> .pool(pool)
> .writeBufferSize(connConfiguration.getWriteBufferSize())
> .maxKeyValueSize(connConfiguration.getMaxKeyValueSize())
> );
> }
> mutator.setRpcTimeout(writeRpcTimeout);
> mutator.setOperationTimeout(operationTimeout);
> return mutator;
> }
> {code}
> And {{HTable#flushCommits}}:
> {code}
> void flushCommits() throws IOException {
> if (mutator == null) {
> // nothing to flush if there's no mutator; don't bother creating one.
> return;
> }
> getBufferedMutator().flush();
> }
> {code}
> For {{HTable#put}}
> {code}
> public void put(final Put put) throws IOException {
> getBufferedMutator().mutate(put);
> flushCommits();
> }
> {code}
> If we launch multiple threads to put in parallel, below sequence might happen
> because {{HTable#getBufferedMutator}} is not thread safe:
> {noformat}
> 1. ThreadA runs to getBufferedMutator and finds mutator==null
> 2. ThreadB runs to getBufferedMutator and finds mutator==null
> 3. ThreadA initialize mutator to instanceA, then calls mutator#mutate,
> adding one put (putA) into {{writeAsyncBuffer}}
> 4. ThreadB initialize mutator to instanceB
> 5. ThreadA runs to flushCommits, now mutator is instanceB, it calls
> instanceB's flush method, putA is lost
> {noformat}
> Will add a UT to cover this case, and fix it in this JIRA.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17361) HTable#getBufferedMutator is not thread safe and could cause data loss
[
https://issues.apache.org/jira/browse/HBASE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-17361:
--
Affects Version/s: 1.1.7
1.2.4
This issue exists at least since 1.1.2, didn't get chance to check older
versions.
> HTable#getBufferedMutator is not thread safe and could cause data loss
> --
>
> Key: HBASE-17361
> URL: https://issues.apache.org/jira/browse/HBASE-17361
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.7, 1.2.4
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Attachments: HBASE-17361.patch
>
>
> Now we have {{HTable#getBufferedMutator}} like
> {code}
>BufferedMutator getBufferedMutator() throws IOException {
> if (mutator == null) {
> this.mutator = (BufferedMutatorImpl) connection.getBufferedMutator(
> new BufferedMutatorParams(tableName)
> .pool(pool)
> .writeBufferSize(connConfiguration.getWriteBufferSize())
> .maxKeyValueSize(connConfiguration.getMaxKeyValueSize())
> );
> }
> mutator.setRpcTimeout(writeRpcTimeout);
> mutator.setOperationTimeout(operationTimeout);
> return mutator;
> }
> {code}
> And {{HTable#flushCommits}}:
> {code}
> void flushCommits() throws IOException {
> if (mutator == null) {
> // nothing to flush if there's no mutator; don't bother creating one.
> return;
> }
> getBufferedMutator().flush();
> }
> {code}
> For {{HTable#put}}
> {code}
> public void put(final Put put) throws IOException {
> getBufferedMutator().mutate(put);
> flushCommits();
> }
> {code}
> If we launch multiple threads to put in parallel, below sequence might happen
> because {{HTable#getBufferedMutator}} is not thread safe:
> {noformat}
> 1. ThreadA runs to getBufferedMutator and finds mutator==null
> 2. ThreadB runs to getBufferedMutator and finds mutator==null
> 3. ThreadA initialize mutator to instanceA, then calls mutator#mutate,
> adding one put (putA) into {{writeAsyncBuffer}}
> 4. ThreadB initialize mutator to instanceB
> 5. ThreadA runs to flushCommits, now mutator is instanceB, it calls
> instanceB's flush method, putA is lost
> {noformat}
> Will add a UT to cover this case, and fix it in this JIRA.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17361) HTable#getBufferedMutator is not thread safe and could cause data loss
[
https://issues.apache.org/jira/browse/HBASE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15770495#comment-15770495
]
Yu Li commented on HBASE-17361:
---
Should be quite little since the lock only happens before the mutator got
initialized.
And IMHO, w/o correctness performance means nothing (smile)
> HTable#getBufferedMutator is not thread safe and could cause data loss
> --
>
> Key: HBASE-17361
> URL: https://issues.apache.org/jira/browse/HBASE-17361
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.7, 1.2.4
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Attachments: HBASE-17361.patch
>
>
> Now we have {{HTable#getBufferedMutator}} like
> {code}
>BufferedMutator getBufferedMutator() throws IOException {
> if (mutator == null) {
> this.mutator = (BufferedMutatorImpl) connection.getBufferedMutator(
> new BufferedMutatorParams(tableName)
> .pool(pool)
> .writeBufferSize(connConfiguration.getWriteBufferSize())
> .maxKeyValueSize(connConfiguration.getMaxKeyValueSize())
> );
> }
> mutator.setRpcTimeout(writeRpcTimeout);
> mutator.setOperationTimeout(operationTimeout);
> return mutator;
> }
> {code}
> And {{HTable#flushCommits}}:
> {code}
> void flushCommits() throws IOException {
> if (mutator == null) {
> // nothing to flush if there's no mutator; don't bother creating one.
> return;
> }
> getBufferedMutator().flush();
> }
> {code}
> For {{HTable#put}}
> {code}
> public void put(final Put put) throws IOException {
> getBufferedMutator().mutate(put);
> flushCommits();
> }
> {code}
> If we launch multiple threads to put in parallel, below sequence might happen
> because {{HTable#getBufferedMutator}} is not thread safe:
> {noformat}
> 1. ThreadA runs to getBufferedMutator and finds mutator==null
> 2. ThreadB runs to getBufferedMutator and finds mutator==null
> 3. ThreadA initialize mutator to instanceA, then calls mutator#mutate,
> adding one put (putA) into {{writeAsyncBuffer}}
> 4. ThreadB initialize mutator to instanceB
> 5. ThreadA runs to flushCommits, now mutator is instanceB, it calls
> instanceB's flush method, putA is lost
> {noformat}
> Will add a UT to cover this case, and fix it in this JIRA.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17361) HTable#getBufferedMutator is not thread safe and could cause data loss
[
https://issues.apache.org/jira/browse/HBASE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-17361:
--
Status: Patch Available (was: Open)
Submit patch for HadoopQA
> HTable#getBufferedMutator is not thread safe and could cause data loss
> --
>
> Key: HBASE-17361
> URL: https://issues.apache.org/jira/browse/HBASE-17361
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Attachments: HBASE-17361.patch
>
>
> Now we have {{HTable#getBufferedMutator}} like
> {code}
>BufferedMutator getBufferedMutator() throws IOException {
> if (mutator == null) {
> this.mutator = (BufferedMutatorImpl) connection.getBufferedMutator(
> new BufferedMutatorParams(tableName)
> .pool(pool)
> .writeBufferSize(connConfiguration.getWriteBufferSize())
> .maxKeyValueSize(connConfiguration.getMaxKeyValueSize())
> );
> }
> mutator.setRpcTimeout(writeRpcTimeout);
> mutator.setOperationTimeout(operationTimeout);
> return mutator;
> }
> {code}
> And {{HTable#flushCommits}}:
> {code}
> void flushCommits() throws IOException {
> if (mutator == null) {
> // nothing to flush if there's no mutator; don't bother creating one.
> return;
> }
> getBufferedMutator().flush();
> }
> {code}
> For {{HTable#put}}
> {code}
> public void put(final Put put) throws IOException {
> getBufferedMutator().mutate(put);
> flushCommits();
> }
> {code}
> If we launch multiple threads to put in parallel, below sequence might happen
> because {{HTable#getBufferedMutator}} is not thread safe:
> {noformat}
> 1. ThreadA runs to getBufferedMutator and finds mutator==null
> 2. ThreadB runs to getBufferedMutator and finds mutator==null
> 3. ThreadA initialize mutator to instanceA, then calls mutator#mutate,
> adding one put (putA) into {{writeAsyncBuffer}}
> 4. ThreadB initialize mutator to instanceB
> 5. ThreadA runs to flushCommits, now mutator is instanceB, it calls
> instanceB's flush method, putA is lost
> {noformat}
> Will add a UT to cover this case, and fix it in this JIRA.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17361) HTable#getBufferedMutator is not thread safe and could cause data loss
[
https://issues.apache.org/jira/browse/HBASE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15770476#comment-15770476
]
Yu Li commented on HBASE-17361:
---
W/o fix here, the newly added UT in patch will always fail.
> HTable#getBufferedMutator is not thread safe and could cause data loss
> --
>
> Key: HBASE-17361
> URL: https://issues.apache.org/jira/browse/HBASE-17361
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Attachments: HBASE-17361.patch
>
>
> Now we have {{HTable#getBufferedMutator}} like
> {code}
>BufferedMutator getBufferedMutator() throws IOException {
> if (mutator == null) {
> this.mutator = (BufferedMutatorImpl) connection.getBufferedMutator(
> new BufferedMutatorParams(tableName)
> .pool(pool)
> .writeBufferSize(connConfiguration.getWriteBufferSize())
> .maxKeyValueSize(connConfiguration.getMaxKeyValueSize())
> );
> }
> mutator.setRpcTimeout(writeRpcTimeout);
> mutator.setOperationTimeout(operationTimeout);
> return mutator;
> }
> {code}
> And {{HTable#flushCommits}}:
> {code}
> void flushCommits() throws IOException {
> if (mutator == null) {
> // nothing to flush if there's no mutator; don't bother creating one.
> return;
> }
> getBufferedMutator().flush();
> }
> {code}
> For {{HTable#put}}
> {code}
> public void put(final Put put) throws IOException {
> getBufferedMutator().mutate(put);
> flushCommits();
> }
> {code}
> If we launch multiple threads to put in parallel, below sequence might happen
> because {{HTable#getBufferedMutator}} is not thread safe:
> {noformat}
> 1. ThreadA runs to getBufferedMutator and finds mutator==null
> 2. ThreadB runs to getBufferedMutator and finds mutator==null
> 3. ThreadA initialize mutator to instanceA, then calls mutator#mutate,
> adding one put (putA) into {{writeAsyncBuffer}}
> 4. ThreadB initialize mutator to instanceB
> 5. ThreadA runs to flushCommits, now mutator is instanceB, it calls
> instanceB's flush method, putA is lost
> {noformat}
> Will add a UT to cover this case, and fix it in this JIRA.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17361) HTable#getBufferedMutator is not thread safe and could cause data loss
[
https://issues.apache.org/jira/browse/HBASE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-17361:
--
Attachment: HBASE-17361.patch
> HTable#getBufferedMutator is not thread safe and could cause data loss
> --
>
> Key: HBASE-17361
> URL: https://issues.apache.org/jira/browse/HBASE-17361
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
> Attachments: HBASE-17361.patch
>
>
> Now we have {{HTable#getBufferedMutator}} like
> {code}
>BufferedMutator getBufferedMutator() throws IOException {
> if (mutator == null) {
> this.mutator = (BufferedMutatorImpl) connection.getBufferedMutator(
> new BufferedMutatorParams(tableName)
> .pool(pool)
> .writeBufferSize(connConfiguration.getWriteBufferSize())
> .maxKeyValueSize(connConfiguration.getMaxKeyValueSize())
> );
> }
> mutator.setRpcTimeout(writeRpcTimeout);
> mutator.setOperationTimeout(operationTimeout);
> return mutator;
> }
> {code}
> And {{HTable#flushCommits}}:
> {code}
> void flushCommits() throws IOException {
> if (mutator == null) {
> // nothing to flush if there's no mutator; don't bother creating one.
> return;
> }
> getBufferedMutator().flush();
> }
> {code}
> For {{HTable#put}}
> {code}
> public void put(final Put put) throws IOException {
> getBufferedMutator().mutate(put);
> flushCommits();
> }
> {code}
> If we launch multiple threads to put in parallel, below sequence might happen
> because {{HTable#getBufferedMutator}} is not thread safe:
> {noformat}
> 1. ThreadA runs to getBufferedMutator and finds mutator==null
> 2. ThreadB runs to getBufferedMutator and finds mutator==null
> 3. ThreadA initialize mutator to instanceA, then calls mutator#mutate,
> adding one put (putA) into {{writeAsyncBuffer}}
> 4. ThreadB initialize mutator to instanceB
> 5. ThreadA runs to flushCommits, now mutator is instanceB, it calls
> instanceB's flush method, putA is lost
> {noformat}
> Will add a UT to cover this case, and fix it in this JIRA.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-17361) HTable#getBufferedMutator is not thread safe and could get data loss
Yu Li created HBASE-17361:
-
Summary: HTable#getBufferedMutator is not thread safe and could
get data loss
Key: HBASE-17361
URL: https://issues.apache.org/jira/browse/HBASE-17361
Project: HBase
Issue Type: Bug
Reporter: Yu Li
Assignee: Yu Li
Priority: Critical
Now we have {{HTable#getBufferedMutator}} like
{code}
BufferedMutator getBufferedMutator() throws IOException {
if (mutator == null) {
this.mutator = (BufferedMutatorImpl) connection.getBufferedMutator(
new BufferedMutatorParams(tableName)
.pool(pool)
.writeBufferSize(connConfiguration.getWriteBufferSize())
.maxKeyValueSize(connConfiguration.getMaxKeyValueSize())
);
}
mutator.setRpcTimeout(writeRpcTimeout);
mutator.setOperationTimeout(operationTimeout);
return mutator;
}
{code}
And {{HTable#flushCommits}}:
{code}
void flushCommits() throws IOException {
if (mutator == null) {
// nothing to flush if there's no mutator; don't bother creating one.
return;
}
getBufferedMutator().flush();
}
{code}
For {{HTable#put}}
{code}
public void put(final Put put) throws IOException {
getBufferedMutator().mutate(put);
flushCommits();
}
{code}
If we launch multiple threads to put in parallel, below sequence might happen
because {{HTable#getBufferedMutator}} is not thread safe:
{noformat}
1. ThreadA runs to getBufferedMutator and finds mutator==null
2. ThreadB runs to getBufferedMutator and finds mutator==null
3. ThreadA initialize mutator to instanceA, then calls mutator#mutate,
adding one put (putA) into {{writeAsyncBuffer}}
4. ThreadB initialize mutator to instanceB
5. ThreadA runs to flushCommits, now mutator is instanceB, it calls
instanceB's flush method, putA is lost
{noformat}
Will add a UT to cover this case, and fix it in this JIRA.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-17361) HTable#getBufferedMutator is not thread safe and could cause data loss
[
https://issues.apache.org/jira/browse/HBASE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-17361:
--
Summary: HTable#getBufferedMutator is not thread safe and could cause data
loss (was: HTable#getBufferedMutator is not thread safe and could get data
loss)
> HTable#getBufferedMutator is not thread safe and could cause data loss
> --
>
> Key: HBASE-17361
> URL: https://issues.apache.org/jira/browse/HBASE-17361
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Critical
>
> Now we have {{HTable#getBufferedMutator}} like
> {code}
>BufferedMutator getBufferedMutator() throws IOException {
> if (mutator == null) {
> this.mutator = (BufferedMutatorImpl) connection.getBufferedMutator(
> new BufferedMutatorParams(tableName)
> .pool(pool)
> .writeBufferSize(connConfiguration.getWriteBufferSize())
> .maxKeyValueSize(connConfiguration.getMaxKeyValueSize())
> );
> }
> mutator.setRpcTimeout(writeRpcTimeout);
> mutator.setOperationTimeout(operationTimeout);
> return mutator;
> }
> {code}
> And {{HTable#flushCommits}}:
> {code}
> void flushCommits() throws IOException {
> if (mutator == null) {
> // nothing to flush if there's no mutator; don't bother creating one.
> return;
> }
> getBufferedMutator().flush();
> }
> {code}
> For {{HTable#put}}
> {code}
> public void put(final Put put) throws IOException {
> getBufferedMutator().mutate(put);
> flushCommits();
> }
> {code}
> If we launch multiple threads to put in parallel, below sequence might happen
> because {{HTable#getBufferedMutator}} is not thread safe:
> {noformat}
> 1. ThreadA runs to getBufferedMutator and finds mutator==null
> 2. ThreadB runs to getBufferedMutator and finds mutator==null
> 3. ThreadA initialize mutator to instanceA, then calls mutator#mutate,
> adding one put (putA) into {{writeAsyncBuffer}}
> 4. ThreadB initialize mutator to instanceB
> 5. ThreadA runs to flushCommits, now mutator is instanceB, it calls
> instanceB's flush method, putA is lost
> {noformat}
> Will add a UT to cover this case, and fix it in this JIRA.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17160) Undo unnecessary inter-module dependency; spark to hbase-it and hbase-it to shell
[ https://issues.apache.org/jira/browse/HBASE-17160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15770419#comment-15770419 ] stack commented on HBASE-17160: --- Pushed the addendum. Thanks for finding and trying [~chia7712] > Undo unnecessary inter-module dependency; spark to hbase-it and hbase-it to > shell > - > > Key: HBASE-17160 > URL: https://issues.apache.org/jira/browse/HBASE-17160 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-17160.addendum.txt, HBASE-17160.master.001.patch, > HBASE-17160.master.002.patch, HBASE-17160.master.002.patch, hbase.png, > minor_hbase.png, untangled_hbase.png > > > Very minor untangling. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17334) Add locate row before/after support for AsyncRegionLocator
[ https://issues.apache.org/jira/browse/HBASE-17334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15770331#comment-15770331 ] Hudson commented on HBASE-17334: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2178 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2178/]) HBASE-17334 Add locate row before/after support for AsyncRegionLocator (zhangduo: rev 09bb4287631563df934cfe88b16fa6dc03e490e6) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncNonMetaRegionLocatorConcurrenyLimit.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncClientScanner.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncNonMetaRegionLocator.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncScanSingleRegionRpcRetryingCaller.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncSmallScanRpcRetryingCaller.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncRegionLocatorTimeout.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncTableRegionLocatorImpl.java * (add) hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionLocateType.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncMultiGetRpcRetryingCaller.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRegionLocator.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncNonMetaRegionLocator.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncSingleRequestRpcRetryingCaller.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncSingleRequestRpcRetryingCaller.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRpcRetryingCallerFactory.java > Add locate row before/after support for AsyncRegionLocator > -- > > Key: HBASE-17334 > URL: https://issues.apache.org/jira/browse/HBASE-17334 > Project: HBase > Issue Type: Sub-task > Components: Client >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17334-v1.patch, HBASE-17334-v2.patch, > HBASE-17334.patch > > > Now we only have a getPreviousRegionLocation method which is only used for > reverse scan, and it is not perfect as it can not deal with region merge. As > we want to add inclusive/exclusive support for start row and end row of a > scan, we need to implement general locate to row before/after method for > AsyncRegionLocator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-17330) SnapshotFileCache will always refresh the file cache
[
https://issues.apache.org/jira/browse/HBASE-17330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15770330#comment-15770330
]
Hudson commented on HBASE-17330:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2178 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/2178/])
HBASE-17330 SnapshotFileCache will always refresh the file cache (tedyu: rev
66781864aaf78e8c8afb0978a7f68b6773d69649)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotFileCache.java
> SnapshotFileCache will always refresh the file cache
>
>
> Key: HBASE-17330
> URL: https://issues.apache.org/jira/browse/HBASE-17330
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Affects Versions: 2.0.0, 1.3.1, 0.98.23
>Reporter: Jianwei Cui
>Assignee: Jianwei Cui
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17330-v1.patch, HBASE-17330-v2.patch
>
>
> In {{SnapshotFileCache#refreshCache}}, the {{hasChanges}} will be judged as:
> {code}
> try {
> FileStatus dirStatus = fs.getFileStatus(snapshotDir);
> lastTimestamp = dirStatus.getModificationTime();
> hasChanges |= (lastTimestamp >= lastModifiedTime); // >= will make
> hasChanges always be true
> {code}
> The {{(lastTimestamp >= lastModifiedTime)}} will make {{hasChanges}} always
> be true because {{lastTimestamp}} will be updated as:
> {code}
> this.lastModifiedTime = lastTimestamp;
> {code}
> So, SnapshotFileCache will always refresh the file cache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-17345) Implement batch
[
https://issues.apache.org/jira/browse/HBASE-17345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15770290#comment-15770290
]
Hadoop QA commented on HBASE-17345:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 50s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 57s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 92m 13s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 135m 21s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12844399/HBASE-17345-v2.patch |
| JIRA Issue | HBASE-17345 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 717b1a160ed3 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 09bb428 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/5026/testReport/ |
| modules | C: hbase-client hbase-server U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/5026/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Implement batch
> ---
