[jira] [Commented] (HBASE-17292) Add observer notification before bulk loaded hfile is moved to region directory

2017-10-11 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199884#comment-16199884
 ] 

Hudson commented on HBASE-17292:


SUCCESS: Integrated in Jenkins build HBase-1.3-JDK8 #317 (See 
[https://builds.apache.org/job/HBase-1.3-JDK8/317/])
HBASE-18910 Backport HBASE-17292 - revert due to incompatibility (tedyu: rev 
eb475f00af711455b4c31d47f06317529db62232)
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java


> Add observer notification before bulk loaded hfile is moved to region 
> directory
> ---
>
> Key: HBASE-17292
> URL: https://issues.apache.org/jira/browse/HBASE-17292
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 17292.addendum, 17292.branch-1.v1.txt, 17292.v1.txt, 
> 17292.v2.txt, 17292.v3.txt
>
>
> Currently the postBulkLoadHFile() hook notifies the locations of bulk loaded 
> hfiles.
> However, if bulk load fails after hfile is moved to region directory but 
> before postBulkLoadHFile() hook is called, there is no way for pluggable 
> components (replication - see HBASE-17290, backup / restore) to know which 
> hfile(s) have been moved to region directory.
> Even if postBulkLoadHFile() is called in finally block, the write (to backup 
> table or zookeeper) issued from postBulkLoadHFile() may fail, ending up with 
> same situation.
> This issue adds a preCommitStoreFile() hook which notifies path of to be 
> committed hfile before bulk loaded hfile is moved to region directory.
> With preCommitStoreFile() hook, write (to backup table or zookeeper) can be 
> issued before the movement of hfile.
> If write fails, IOException would make bulk load fail, not leaving hfile in 
> region directory.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-17292) Add observer notification before bulk loaded hfile is moved to region directory

2017-10-10 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199635#comment-16199635
 ] 

Hudson commented on HBASE-17292:


SUCCESS: Integrated in Jenkins build HBase-1.3-IT #231 (See 
[https://builds.apache.org/job/HBase-1.3-IT/231/])
HBASE-18910 Backport HBASE-17292 - revert due to incompatibility (tedyu: rev 
eb475f00af711455b4c31d47f06317529db62232)
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java


> Add observer notification before bulk loaded hfile is moved to region 
> directory
> ---
>
> Key: HBASE-17292
> URL: https://issues.apache.org/jira/browse/HBASE-17292
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 17292.addendum, 17292.branch-1.v1.txt, 17292.v1.txt, 
> 17292.v2.txt, 17292.v3.txt
>
>
> Currently the postBulkLoadHFile() hook notifies the locations of bulk loaded 
> hfiles.
> However, if bulk load fails after hfile is moved to region directory but 
> before postBulkLoadHFile() hook is called, there is no way for pluggable 
> components (replication - see HBASE-17290, backup / restore) to know which 
> hfile(s) have been moved to region directory.
> Even if postBulkLoadHFile() is called in finally block, the write (to backup 
> table or zookeeper) issued from postBulkLoadHFile() may fail, ending up with 
> same situation.
> This issue adds a preCommitStoreFile() hook which notifies path of to be 
> committed hfile before bulk loaded hfile is moved to region directory.
> With preCommitStoreFile() hook, write (to backup table or zookeeper) can be 
> issued before the movement of hfile.
> If write fails, IOException would make bulk load fail, not leaving hfile in 
> region directory.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-17292) Add observer notification before bulk loaded hfile is moved to region directory

2017-10-10 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199626#comment-16199626
 ] 

Hudson commented on HBASE-17292:


FAILURE: Integrated in Jenkins build HBase-1.3-JDK7 #302 (See 
[https://builds.apache.org/job/HBase-1.3-JDK7/302/])
HBASE-18910 Backport HBASE-17292 - revert due to incompatibility (tedyu: rev 
eb475f00af711455b4c31d47f06317529db62232)
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java


> Add observer notification before bulk loaded hfile is moved to region 
> directory
> ---
>
> Key: HBASE-17292
> URL: https://issues.apache.org/jira/browse/HBASE-17292
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 17292.addendum, 17292.branch-1.v1.txt, 17292.v1.txt, 
> 17292.v2.txt, 17292.v3.txt
>
>
> Currently the postBulkLoadHFile() hook notifies the locations of bulk loaded 
> hfiles.
> However, if bulk load fails after hfile is moved to region directory but 
> before postBulkLoadHFile() hook is called, there is no way for pluggable 
> components (replication - see HBASE-17290, backup / restore) to know which 
> hfile(s) have been moved to region directory.
> Even if postBulkLoadHFile() is called in finally block, the write (to backup 
> table or zookeeper) issued from postBulkLoadHFile() may fail, ending up with 
> same situation.
> This issue adds a preCommitStoreFile() hook which notifies path of to be 
> committed hfile before bulk loaded hfile is moved to region directory.
> With preCommitStoreFile() hook, write (to backup table or zookeeper) can be 
> issued before the movement of hfile.
> If write fails, IOException would make bulk load fail, not leaving hfile in 
> region directory.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-17292) Add observer notification before bulk loaded hfile is moved to region directory

2016-12-20 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765856#comment-15765856
 ] 

Hudson commented on HBASE-17292:


FAILURE: Integrated in Jenkins build HBase-1.4 #573 (See 
[https://builds.apache.org/job/HBase-1.4/573/])
HBASE-17292 Add observer notification before bulk loaded hfile is moved (tedyu: 
rev 0b69f59133f46a1aee6294aa40b54b763fe4222d)
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java


> Add observer notification before bulk loaded hfile is moved to region 
> directory
> ---
>
> Key: HBASE-17292
> URL: https://issues.apache.org/jira/browse/HBASE-17292
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 17292.addendum, 17292.branch-1.v1.txt, 17292.v1.txt, 
> 17292.v2.txt, 17292.v3.txt
>
>
> Currently the postBulkLoadHFile() hook notifies the locations of bulk loaded 
> hfiles.
> However, if bulk load fails after hfile is moved to region directory but 
> before postBulkLoadHFile() hook is called, there is no way for pluggable 
> components (replication - see HBASE-17290, backup / restore) to know which 
> hfile(s) have been moved to region directory.
> Even if postBulkLoadHFile() is called in finally block, the write (to backup 
> table or zookeeper) issued from postBulkLoadHFile() may fail, ending up with 
> same situation.
> This issue adds a preCommitStoreFile() hook which notifies path of to be 
> committed hfile before bulk loaded hfile is moved to region directory.
> With preCommitStoreFile() hook, write (to backup table or zookeeper) can be 
> issued before the movement of hfile.
> If write fails, IOException would make bulk load fail, not leaving hfile in 
> region directory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17292) Add observer notification before bulk loaded hfile is moved to region directory

2016-12-20 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765279#comment-15765279
 ] 

Hadoop QA commented on HBASE-17292:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 1s 
{color} | {color:blue} The patch file was not named according to hbase's naming 
conventions. Please see 
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for 
instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
46s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s 
{color} | {color:green} branch-1 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s 
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
55s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
17s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 55s 
{color} | {color:red} hbase-server in branch-1 has 2 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s 
{color} | {color:green} branch-1 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s 
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s 
{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s 
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
14m 49s {color} | {color:green} The patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 7s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s 
{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s 
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 84m 3s 
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
18s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 113m 5s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:e01ee2f |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12844112/17292.branch-1.v1.txt 
|
| JIRA Issue | HBASE-17292 |
| Optional Tests |  asflicense  javac  javadoc  unit

[jira] [Commented] (HBASE-17292) Add observer notification before bulk loaded hfile is moved to region directory

2016-12-16 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15756250#comment-15756250
 ] 

Hudson commented on HBASE-17292:


SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2145 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/2145/])
HBASE-17292 Add observer notification before bulk loaded hfile is moved (tedyu: 
rev 49b0bab50436433feefd842a223e0656e86df247)
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java


> Add observer notification before bulk loaded hfile is moved to region 
> directory
> ---
>
> Key: HBASE-17292
> URL: https://issues.apache.org/jira/browse/HBASE-17292
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 17292.addendum, 17292.v1.txt, 17292.v2.txt, 17292.v3.txt
>
>
> Currently the postBulkLoadHFile() hook notifies the locations of bulk loaded 
> hfiles.
> However, if bulk load fails after hfile is moved to region directory but 
> before postBulkLoadHFile() hook is called, there is no way for pluggable 
> components (replication - see HBASE-17290, backup / restore) to know which 
> hfile(s) have been moved to region directory.
> Even if postBulkLoadHFile() is called in finally block, the write (to backup 
> table or zookeeper) issued from postBulkLoadHFile() may fail, ending up with 
> same situation.
> This issue adds a preCommitStoreFile() hook which notifies path of to be 
> committed hfile before bulk loaded hfile is moved to region directory.
> With preCommitStoreFile() hook, write (to backup table or zookeeper) can be 
> issued before the movement of hfile.
> If write fails, IOException would make bulk load fail, not leaving hfile in 
> region directory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17292) Add observer notification before bulk loaded hfile is moved to region directory

2016-12-15 Thread Hudson (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752912#comment-15752912
 ] 

Hudson commented on HBASE-17292:


SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2138 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/2138/])
HBASE-17292 Add observer notification before bulk loaded hfile is moved (tedyu: 
rev 35f0718a418f1aaf4d86da5bf88d3b2db3d6be67)
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SecureBulkLoadManager.java


> Add observer notification before bulk loaded hfile is moved to region 
> directory
> ---
>
> Key: HBASE-17292
> URL: https://issues.apache.org/jira/browse/HBASE-17292
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 17292.v1.txt, 17292.v2.txt, 17292.v3.txt
>
>
> Currently the postBulkLoadHFile() hook notifies the locations of bulk loaded 
> hfiles.
> However, if bulk load fails after hfile is moved to region directory but 
> before postBulkLoadHFile() hook is called, there is no way for pluggable 
> components (replication - see HBASE-17290, backup / restore) to know which 
> hfile(s) have been moved to region directory.
> Even if postBulkLoadHFile() is called in finally block, the write (to backup 
> table or zookeeper) issued from postBulkLoadHFile() may fail, ending up with 
> same situation.
> This issue adds a preCommitStoreFile() hook which notifies path of to be 
> committed hfile before bulk loaded hfile is moved to region directory.
> With preCommitStoreFile() hook, write (to backup table or zookeeper) can be 
> issued before the movement of hfile.
> If write fails, IOException would make bulk load fail, not leaving hfile in 
> region directory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17292) Add observer notification before bulk loaded hfile is moved to region directory

2016-12-15 Thread Ted Yu (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751973#comment-15751973
 ] 

Ted Yu commented on HBASE-17292:


Thanks for review, Jerry.

I updated description, explaining why the current postBulkLoadHFile() hook 
doesn't suffice for fault tolerance purposes.

> Add observer notification before bulk loaded hfile is moved to region 
> directory
> ---
>
> Key: HBASE-17292
> URL: https://issues.apache.org/jira/browse/HBASE-17292
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 17292.v1.txt, 17292.v2.txt, 17292.v3.txt
>
>
> Currently the postBulkLoadHFile() hook notifies the locations of bulk loaded 
> hfiles.
> However, if bulk load fails after hfile is moved to region directory but 
> before postBulkLoadHFile() hook is called, there is no way for pluggable 
> components (replication - see HBASE-17290, backup / restore) to know which 
> hfile(s) have been moved to region directory.
> Even if postBulkLoadHFile() is called in finally block, the write (to backup 
> table or zookeeper) issued from postBulkLoadHFile() may fail, ending up with 
> same situation.
> This issue adds a preCommitStoreFile() hook which notifies path of to be 
> committed hfile before bulk loaded hfile is moved to region directory.
> With preCommitStoreFile() hook, write (to backup table or zookeeper) can be 
> issued before the movement of hfile.
> If write fails, IOException would make bulk load fail, not leaving hfile in 
> region directory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17292) Add observer notification before bulk loaded hfile is moved to region directory

2016-12-15 Thread Jerry He (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751945#comment-15751945
 ] 

Jerry He commented on HBASE-17292:
--

bq. For your specific problem you extend public API (RegionObserver). 
I guess I have the same question. You have been adding quite a few things in 
the bulk load recently. Hope you have good use cases for them.

Patch lgtm.

> Add observer notification before bulk loaded hfile is moved to region 
> directory
> ---
>
> Key: HBASE-17292
> URL: https://issues.apache.org/jira/browse/HBASE-17292
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 17292.v1.txt, 17292.v2.txt, 17292.v3.txt
>
>
> Currently the postBulkLoadHFile() hook notifies the locations of bulk loaded 
> hfiles.
> However, if bulk load fails after hfile is moved to region directory but 
> before postBulkLoadHFile() hook is called, there is no way for pluggable 
> components (replication - see HBASE-17290, backup / restore) to know which 
> hfile(s) have been moved to region directory.
> This issue adds a preCommitStoreFile() hook which notifies path of to be 
> committed hfile before bulk loaded hfile is moved to region directory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17292) Add observer notification before bulk loaded hfile is moved to region directory

2016-12-14 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15750612#comment-15750612
 ] 

Hadoop QA commented on HBASE-17292:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 2s 
{color} | {color:blue} The patch file was not named according to hbase's naming 
conventions. Please see 
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for 
instructions. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
55s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
13s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
24m 55s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 89m 42s 
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
12s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 126m 14s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12843351/17292.v3.txt |
| JIRA Issue | HBASE-17292 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 6fb704a4ddb4 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / a5a6036 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4925/artifact/patchprocess/whitespace-eol.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4925/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4925/console |
| Powered by | Apache Yetus 0.3.0

[jira] [Commented] (HBASE-17292) Add observer notification before bulk loaded hfile is moved to region directory

2016-12-14 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15749848#comment-15749848
 ] 

Hadoop QA commented on HBASE-17292:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 2s 
{color} | {color:blue} The patch file was not named according to hbase's naming 
conventions. Please see 
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for 
instructions. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
34s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
35s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
23m 11s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 87m 1s 
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
16s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 121m 1s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12843297/17292.v2.txt |
| JIRA Issue | HBASE-17292 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 1809d5831c98 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 691f266 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4919/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4919/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> Add observer notification before bulk loaded hfile is moved to region 
> directory
>

[jira] [Commented] (HBASE-17292) Add observer notification before bulk loaded hfile is moved to region directory

2016-12-14 Thread Vladimir Rodionov (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15749297#comment-15749297
 ] 

Vladimir Rodionov commented on HBASE-17292:
---

Yes, add postCommit hook and fix . lgtm after that. 

> Add observer notification before bulk loaded hfile is moved to region 
> directory
> ---
>
> Key: HBASE-17292
> URL: https://issues.apache.org/jira/browse/HBASE-17292
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 17292.v1.txt
>
>
> Currently the postBulkLoadHFile() hook notifies the locations of bulk loaded 
> hfiles.
> However, if bulk load fails after hfile is moved to region directory but 
> before postBulkLoadHFile() hook is called, there is no way for pluggable 
> components (replication - see HBASE-17290, backup / restore) to know which 
> hfile(s) have been moved to region directory.
> This issue adds a preCommitStoreFile() hook which notifies path of to be 
> committed hfile before bulk loaded hfile is moved to region directory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17292) Add observer notification before bulk loaded hfile is moved to region directory

2016-12-14 Thread Ted Yu (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15749244#comment-15749244
 ] 

Ted Yu commented on HBASE-17292:


bq. Hooks in RegionObserver usually comes in pairs: pre- and post-
Just want to confirm: you're suggesting adding postCommitStoreFile() in 
HStore#bulkLoadHFile() after fs.commitStoreFile() call:
{code}
  public Path bulkLoadHFile(String srcPathStr, Path dstPath) throws IOException 
{
Path srcPath = new Path(srcPathStr);
fs.commitStoreFile(srcPath, dstPath);
{code}

bq. Who else is going to use this hook

Once this hook is added, bulk load support for replication would be able to use 
it as well.

> Add observer notification before bulk loaded hfile is moved to region 
> directory
> ---
>
> Key: HBASE-17292
> URL: https://issues.apache.org/jira/browse/HBASE-17292
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 17292.v1.txt
>
>
> Currently the postBulkLoadHFile() hook notifies the locations of bulk loaded 
> hfiles.
> However, if bulk load fails after hfile is moved to region directory but 
> before postBulkLoadHFile() hook is called, there is no way for pluggable 
> components (replication, backup / restore) to know which hfile(s) have been 
> moved to region directory.
> This issue adds a preCommitStoreFile() hook which notifies path of to be 
> committed hfile before bulk loaded hfile is moved to region directory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17292) Add observer notification before bulk loaded hfile is moved to region directory

2016-12-14 Thread Vladimir Rodionov (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15749233#comment-15749233
 ] 

Vladimir Rodionov commented on HBASE-17292:
---

Another note:

For your specific problem you extend public API (RegionObserver). Make sure you 
have other use cases for new API and you can convince community in necessity of 
this decision. 

> Add observer notification before bulk loaded hfile is moved to region 
> directory
> ---
>
> Key: HBASE-17292
> URL: https://issues.apache.org/jira/browse/HBASE-17292
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 17292.v1.txt
>
>
> Currently the postBulkLoadHFile() hook notifies the locations of bulk loaded 
> hfiles.
> However, if bulk load fails after hfile is moved to region directory but 
> before postBulkLoadHFile() hook is called, there is no way for pluggable 
> components (replication, backup / restore) to know which hfile(s) have been 
> moved to region directory.
> This issue adds a preCommitStoreFile() hook which notifies path of to be 
> committed hfile before bulk loaded hfile is moved to region directory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17292) Add observer notification before bulk loaded hfile is moved to region directory

2016-12-14 Thread Vladimir Rodionov (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15749199#comment-15749199
 ] 

Vladimir Rodionov commented on HBASE-17292:
---

Ted, couple notes:

# Hooks in RegionObserver usually comes in pairs: pre- and post-. You introduce 
only one.
# Who else is going to use this hook in your opinion?  

{code}
   /**
   * Called before moving bulk loaded hfile to region directory.
   *
   * @param ctx
   * @param family column family
   * @param pairs List of pairs of { HFile location in staging dir, HFile path 
in region dir }
   * Each pair are for the same hfile.
   * @throws IOException
   */
  void preCommitStoreFile(final ObserverContext 
ctx,
  final byte[] family, final List> pairs) throws 
IOException;
 
{code}

You have String and Path for file locations. Make them both Path.

> Add observer notification before bulk loaded hfile is moved to region 
> directory
> ---
>
> Key: HBASE-17292
> URL: https://issues.apache.org/jira/browse/HBASE-17292
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 17292.v1.txt
>
>
> Currently the postBulkLoadHFile() hook notifies the locations of bulk loaded 
> hfiles.
> However, if bulk load fails after hfile is moved to region directory but 
> before postBulkLoadHFile() hook is called, there is no way for pluggable 
> components (replication, backup / restore) to know which hfile(s) have been 
> moved to region directory.
> This issue adds a preCommitStoreFile() hook which notifies path of to be 
> committed hfile before bulk loaded hfile is moved to region directory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17292) Add observer notification before bulk loaded hfile is moved to region directory

2016-12-12 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15741429#comment-15741429
 ] 

Hadoop QA commented on HBASE-17292:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 0s 
{color} | {color:blue} The patch file was not named according to hbase's naming 
conventions. Please see 
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for 
instructions. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
2s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
1s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
25m 32s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 89m 33s 
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
14s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 127m 9s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12842761/17292.v1.txt |
| JIRA Issue | HBASE-17292 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 2fecb9d06598 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 1615f45 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4878/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4878/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> Add observer notification before bulk loaded hfile is moved to region 
> directory
>