[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517586#comment-15517586
]
Hudson commented on HBASE-16257:
SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #1660 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1660/])
HBASE-16257 Move staging dir to be under hbase root dir (jerryjch: rev
50b051ade1d38bba0b936db1c751e5045bc103cb)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/SecureBulkLoadEndpointClient.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/util/hbck/HFileCorruptionChecker.java
* (edit) hbase-common/src/main/resources/hbase-default.xml
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/SecureBulkLoadClient.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplit.java
* (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterWalManager.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java
* (delete)
hbase-client/src/main/java/org/apache/hadoop/hbase/security/SecureBulkLoadUtil.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SecureBulkLoadManager.java
* (edit)
hbase-common/src/test/java/org/apache/hadoop/hbase/HBaseCommonTestingUtility.java
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch,
> HBASE-16257-v3.patch, HBASE-16257-v4.patch, HBASE-16257-v5.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517005#comment-15517005
]
Jerry He commented on HBASE-16257:
--
Thanks for the review.
bq. Is this between 1.x -> 2.0 replication?
i didn't test this case. But this JIRA would not affect it, as we had
discussed.
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch,
> HBASE-16257-v3.patch, HBASE-16257-v4.patch, HBASE-16257-v5.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514127#comment-15514127
]
Enis Soztutar commented on HBASE-16257:
---
bq. I tested with a local standalone instance and on a Keberos cluster. All
looks good.
Cool. Thanks Jerry for the test. Is this between 1.x -> 2.0 replication?
+1 for v5.
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch,
> HBASE-16257-v3.patch, HBASE-16257-v4.patch, HBASE-16257-v5.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505478#comment-15505478
]
Hadoop QA commented on HBASE-16257:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 13s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
34s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 38s
{color} | {color:red} hbase-common in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 38s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 48s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 2s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 85m 53s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 138m 14s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.client.TestReplicasClient |
| | org.apache.hadoop.hbase.client.TestFromClientSide |
| | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient |
| | org.apache.hadoop.hbase.client.TestMobSnapshotCloneIndependence |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829314/HBASE-16257-v5.patch |
| JIRA Issue | HBASE-16257 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile xml |
| uname | Linux 2912f63a6ac9 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505232#comment-15505232
]
Jerry He commented on HBASE-16257:
--
bq. If it is not the case, then we can commit this patch as it is.
Is this +1 [~enis]?
Thanks for the comment, [~ashish singhi]
I tested with a local standalone instance and on a Keberos cluster. All looks
good.
The directories are created or modified with the correct permissions:
{noformat}
hadoop fs -ls /apps/hbase/
drwx--x--x - hbase hdfs 0 2016-09-19 13:48 /apps/hbase/data
hadoop fs -ls /apps/hbase/data
drwx-- - hbase hdfs 0 2016-09-19 11:25 /apps/hbase/data/.hbck
drwx-- - hbase hdfs 0 2016-09-19 13:48 /apps/hbase/data/.tmp
drwx-- - hbase hdfs 0 2016-09-19 18:01
/apps/hbase/data/MasterProcWALs
drwx-- - hbase hdfs 0 2016-09-19 13:48 /apps/hbase/data/WALs
drwx-- - hbase hdfs 0 2016-09-19 14:01 /apps/hbase/data/archive
drwx-- - hbase hdfs 0 2016-06-17 15:25 /apps/hbase/data/corrupt
drwx-- - hbase hdfs 0 2016-06-09 12:36 /apps/hbase/data/data
-rw--- 3 hbase hdfs 42 2016-06-09 12:35 /apps/hbase/data/hbase.id
-rw--- 3 hbase hdfs 7 2016-06-09 12:35
/apps/hbase/data/hbase.version
drwx-- - hbase hdfs 0 2016-09-19 11:25 /apps/hbase/data/mobdir
drwx-- - hbase hdfs 0 2016-09-19 17:59 /apps/hbase/data/oldWALs
drwx--x--x - hbase hdfs 0 2016-09-19 17:34 /apps/hbase/data/staging
{noformat}
Ran a few bulk loads as well.
v5 adds back creating staging dir in SecureBulkLoadManager in the region server
side if it does not exist, so that it is ok to upgrade region server before
master.
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch,
> HBASE-16257-v3.patch, HBASE-16257-v4.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504211#comment-15504211
]
Enis Soztutar commented on HBASE-16257:
---
If source is 2.0 with this patch, but sink is 1.x without this patch, then the
sink will not create the staging dir under root. However, on the source side,
we will still be accessing the staging dir under root on the sink fs.
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch,
> HBASE-16257-v3.patch, HBASE-16257-v4.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504158#comment-15504158
]
Ashish Singhi commented on HBASE-16257:
---
It need not. It's HFileReplicator which will create and access the directory in
the sink cluster and it will run in sink cluster so there should not be any
problem.
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch,
> HBASE-16257-v3.patch, HBASE-16257-v4.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504140#comment-15504140
]
Enis Soztutar commented on HBASE-16257:
---
In secure 1.x clusters, root dir is protected with 700. So unless the user is
the same, source cluster cannot create or access a directory under the sink
cluster.
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch,
> HBASE-16257-v3.patch, HBASE-16257-v4.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504142#comment-15504142
]
Enis Soztutar commented on HBASE-16257:
---
In secure 1.x clusters, root dir is protected with 700. So unless the user is
the same, source cluster cannot create or access a directory under the sink
cluster.
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch,
> HBASE-16257-v3.patch, HBASE-16257-v4.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503034#comment-15503034
]
Ashish Singhi commented on HBASE-16257:
---
Your thinking is correct, HFileReplicator is using the staging dir configured
on the sink cluster. It is getting the staging dir path from the sink cluster
conf. If the staging dir doesn't exist it will create it and copy the HFiles
from the source cluster to the sink cluster staging dir and then run the
complete bulkload tool.
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch,
> HBASE-16257-v3.patch, HBASE-16257-v4.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15502243#comment-15502243
]
Enis Soztutar commented on HBASE-16257:
---
Sorry, I was thinking that HFileReplicator is using the staging dir on the
remote end. If it is not the case, then we can commit this patch as it is.
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch,
> HBASE-16257-v3.patch, HBASE-16257-v4.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498233#comment-15498233
]
Ashish Singhi commented on HBASE-16257:
---
You got it right [~jerryhe].
bq. the staging dir will not be there under the 1.x root, which will then case
the HFileReplicator to fail.
Sorry I did not get this why HFileReplicator will fail here ?
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch,
> HBASE-16257-v3.patch, HBASE-16257-v4.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498070#comment-15498070
]
Jerry He commented on HBASE-16257:
--
Hi, [~enis]
Thinking a bit more. Correct me if I am wrong.
HFileReplicator operates on the sink side. It tries to copy from the sourceFs
to local sink staging dir. If we have 2.x -> 1.x, there is no rootDir/staging
on the sink. But it is ok. Sink knows how to handle its own staging dir, new
way or old way. Source will not try to access sink staging dir and sink will
not try to access source staging dir.
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch,
> HBASE-16257-v3.patch, HBASE-16257-v4.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497701#comment-15497701
]
Jerry He commented on HBASE-16257:
--
Good point.Let me think thru this case.
FYI Ashish Singhi
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch,
> HBASE-16257-v3.patch, HBASE-16257-v4.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497422#comment-15497422
]
Enis Soztutar commented on HBASE-16257:
---
Thanks Jerry for the latest patch. Sorry, but one last item may need to be
addressed. Otherwise, everything looks good. The HFileReplicator now uses the
bulk load staging dir as a path under root. This will become a problem if the
source cluster is upgraded first before the sink cluster. In case of cyclic
replication, it will always be the case that you can have 2.x -> 1.x
replication. If that is the case, the staging dir will not be there under the
1.x root, which will then case the HFileReplicator to fail. Maybe we can do a
simple exists check on the remote FS to see whether we can use the new
hbase.root.dir/staging or we should fall back to using the value from
configuration.
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch,
> HBASE-16257-v3.patch, HBASE-16257-v4.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15488906#comment-15488906
]
Hadoop QA commented on HBASE-16257:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
56s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 31s
{color} | {color:red} hbase-common in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
24m 18s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 40s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 50s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 86m 38s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 132m 42s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.client.TestReplicasClient |
| | org.apache.hadoop.hbase.client.TestFromClientSide |
| | org.apache.hadoop.hbase.client.TestMetaWithReplicas |
| | org.apache.hadoop.hbase.client.TestFromClientSide3 |
| | org.apache.hadoop.hbase.client.TestTableSnapshotScanner |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12828340/HBASE-16257-v4.patch |
| JIRA Issue | HBASE-16257 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile xml |
| uname | Linux 021e35ca14df 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15486284#comment-15486284
]
Jerry He commented on HBASE-16257:
--
bq. Maybe extract the subdirs away, and add getters so that rest of the code
does not have to deal with parsing the conf again:
Do you mean to do something similar to the temp dir?
{code}
this.tempdir = new Path(this.rootdir, HConstants.HBASE_TEMP_DIRECTORY);
...
/**
* @return HBase temp dir.
*/
public Path getTempDir() {
return this.tempdir;
}
{code}
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch,
> HBASE-16257-v3.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15486278#comment-15486278
]
Jerry He commented on HBASE-16257:
--
bq. We create this dir with default perms. If there is a way to get the default
perms, and after ensuring that we have x, setting it to root should be fine.
[~enis] This would work for you? As the last step:
{code}
FsPermission currentRootPerms =
fs.getFileStatus(this.rootdir).getPermission();
if (!currentRootPerms.getUserAction().implies(FsAction.EXECUTE)
|| !currentRootPerms.getGroupAction().implies(FsAction.EXECUTE)
|| !currentRootPerms.getOtherAction().implies(FsAction.EXECUTE)) {
LOG.warn("rootdir permissions do not contain 'excute' for user, group or
other. "
+ "Automatically adding 'excute' permission for all");
fs.setPermission(
this.rootdir,
new FsPermission(currentRootPerms.getUserAction().or(FsAction.EXECUTE),
currentRootPerms
.getGroupAction().or(FsAction.EXECUTE),
currentRootPerms.getOtherAction().or(
FsAction.EXECUTE)));
}
{code}
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch,
> HBASE-16257-v3.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15485879#comment-15485879
]
Hadoop QA commented on HBASE-16257:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
37s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 15s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 36s
{color} | {color:red} hbase-common in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 13s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 42s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 51s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 9s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 130m 54s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.wal.TestWALSplit |
| | hadoop.hbase.regionserver.TestScannerHeartbeatMessages |
| | hadoop.hbase.wal.TestWALSplitCompressed |
| Timed out junit tests |
org.apache.hadoop.hbase.client.TestMultiRespectsLimits |
| | org.apache.hadoop.hbase.client.TestSnapshotMetadata |
| | org.apache.hadoop.hbase.client.TestTableSnapshotScanner |
| | org.apache.hadoop.hbase.client.TestCloneSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12828131/HBASE-16257-v3.patch |
| JIRA Issue | HBASE-16257 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15485860#comment-15485860
]
Hadoop QA commented on HBASE-16257:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 46s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
30s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 10s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 39s
{color} | {color:red} hbase-common in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 2s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 47s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 53s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 96m 30s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
42s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 152m 9s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.snapshot.TestSnapshotClientRetries |
| | org.apache.hadoop.hbase.snapshot.TestMobSecureExportSnapshot |
| | org.apache.hadoop.hbase.filter.TestFilterListOrOperatorWithBlkCnt |
| | org.apache.hadoop.hbase.snapshot.TestRestoreFlushSnapshotFromClient |
| | org.apache.hadoop.hbase.client.TestHCM |
| | org.apache.hadoop.hbase.snapshot.TestMobFlushSnapshotFromClient |
| | org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient |
| | org.apache.hadoop.hbase.TestMetaTableAccessor |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12828125/HBASE-16257-v3.patch |
| JIRA Issue |
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15485678#comment-15485678
]
Enis Soztutar commented on HBASE-16257:
---
bq. Hmm where is the other place?
Here:
{code}
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SecureBulkLoadManager.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SecureBulkLoadManager.java
+ private final static String BULKLOAD_STAGING_DIR =
"hbase.bulkload.staging.dir";
{code}
bq. Is the following ok?
We create this dir with default perms. If there is a way to get the default
perms, and after ensuring that we have {{x}}, setting it to root should be
fine.
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch,
> HBASE-16257-v3.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15485653#comment-15485653
]
Jerry He commented on HBASE-16257:
--
bq. Can we either use all upper case or all lower case for the names of these
bq. Let's add a comment here talking about the layout of rootdir and the
permissions of its subdirs.
Will do.
bq. This is now defined in two places
Hmm where is the other place?
bq. I was mentioning above that, for upgrades, we have to "open-up" the
permissions for the root directory
Is the following ok?
{noformat}
+// Handle the last few special files and set the final rootdir permissions
+// rootdir needs 'x' for all for bulk load staging dir
+if (isSecurityEnabled) {
+ fs.setPermission(new Path(rootdir, HConstants.VERSION_FILE_NAME),
secureRootFilePerms);
+ fs.setPermission(new Path(rootdir, HConstants.CLUSTER_ID_FILE_NAME),
secureRootFilePerms);
+}
+fs.setPermission(this.rootdir, this.rootPerms);
{noformat}
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch,
> HBASE-16257-v3.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15485581#comment-15485581
]
Enis Soztutar commented on HBASE-16257:
---
Can we either use all upper case or all lower case for the names of these:
{code}
+ private final FsPermission rootPerms = FsPermission.valueOf("-rwxr-xr-x");
+ // permissions for the files under rootDir that need secure protection
+ private final FsPermission secureRootFilePerms =
FsPermission.valueOf("-rw---");
+ // permissions for the directories under rootDir that need secure protection
+ private final FsPermission secureRootSubDirPerms;
+ // permissions for bulk load staging directory under rootDir
+ private final FsPermission PERM_HIDDEN = FsPermission.valueOf("-rwx--x--x");
{code}
Let's add a comment here talking about the layout of rootdir and the
permissions of its subdirs.
This is now defined in two places:
{code}
+ public static final String BULKLOAD_STAGING_DIR_NAME = "staging";
{code}
I was mentioning above that, for upgrades, we have to "open-up" the permissions
for the root directory because if a branch-1 cluster is run in secure mode, the
root dir is already wiht perms 700. After upgrade, the staging dir will be
inside this root dir, and all BL operation will fail after the upgrade.
{code}
// Filesystem is good. Go ahead and check for hbase.rootdir.
try {
if (!fs.exists(rd)) {
-if (isSecurityEnabled) {
- fs.mkdirs(rd, rootDirPerms);
-} else {
- fs.mkdirs(rd);
-}
+fs.mkdirs(rd);
{code}
Maybe extract the subdirs away, and add getters so that rest of the code does
not have to deal with parsing the conf again:
{code}
+checkSubDir(new Path(this.rootdir, HConstants.BASE_NAMESPACE_DIR));
+checkSubDir(new Path(this.rootdir, HConstants.HFILE_ARCHIVE_DIRECTORY));
+checkSubDir(new Path(this.rootdir, HConstants.HREGION_LOGDIR_NAME));
{code}
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch,
> HBASE-16257-v3.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15475899#comment-15475899
]
Jerry He commented on HBASE-16257:
--
bq. How do we handle upgrades?
This would be compatible with rolling upgrade, as indicated in the parent JIRA.
Changing the staging dir in the middle won't break the compatibility. Old and
new staging dirs can co-exists. Bulk load token is full path.
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15475863#comment-15475863
]
Jerry He commented on HBASE-16257:
--
Hi, Enis
Thanks for the comments. I will update the patch as commented.
bq. Are these changes related?
Not directly related. The intention was to put the CORRUPT_DIR_NAME always
under hbase.rootdir and get rid of the property "hbase.hfile.quarantine.dir".
It is not referenced or documented anywhere else.
Ok to remove? I can put a deprecate warning in the log.
bq. Is this conf coming from the remote(sink) cluster for the HFileReplicator?
Yes
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15474713#comment-15474713
]
Enis Soztutar commented on HBASE-16257:
---
bq. Could you give more details on the whitelist/blacklist suggestion, Enis
Soztutar? Thanks.
I was trying to suggest that we do a black list of sub-directories inside the
root directory for "special permissions" other than 700, instead of a white
list of subdirectories with permission "700". The approach we have in the patch
seems fine.
What happens if we create the staging dir and on the next restart we do run
checkSubDir() methods? Will this fail because it does not match 700?
How do we handle upgrades? In secure clusters, the root dir is already set to
700, so the master has to open it up, but only after changing the perms for the
subdirectories.
I think the MasterFS should own and manage creating the staging directory as
well, instead of the SBLM since MFS is responsible for owning the directory
layout.
Are these changes related?
{code}
-Path corruptBaseDir = new Path(FSUtils.getRootDir(conf), conf.get(
-"hbase.hfile.quarantine.dir", HConstants.CORRUPT_DIR_NAME));
+Path corruptBaseDir = new Path(FSUtils.getRootDir(conf),
HConstants.CORRUPT_DIR_NAME);
{code}
Is this conf coming from the remote(sink) cluster for the HFileReplicator?
Sorry I did not check the code.
{code}
+this.hbaseStagingDir = new Path(FSUtils.getRootDir(conf),
HConstants.BULKLOAD_STAGING_DIR_NAME);
{code}
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15423778#comment-15423778
]
Jerry He commented on HBASE-16257:
--
Ping for comments. [~enis], [~mbertozzi]?
Thanks.
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403314#comment-15403314
]
Jerry He commented on HBASE-16257:
--
Could you give more details on the whitelist/blacklist suggestion, [~enis]?
Thanks.
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402539#comment-15402539
]
Enis Soztutar commented on HBASE-16257:
---
bq. Instead, we will set all the subdirs that need protection to 700
individually, .e.g. /hbase/data, /hbase/WALs, /hbase/oldWALs.
We have to be careful if we add a new subdirectory, we should make sure that it
also gets correct permissions. So do this as a black-list, rather than
whitelist maybe.
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402531#comment-15402531
]
Enis Soztutar commented on HBASE-16257:
---
Indeed.
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15401540#comment-15401540
]
Hadoop QA commented on HBASE-16257:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
6s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 36s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 19s
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 21s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 32s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 29s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
35m 14s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 21s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 15s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 59s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 3s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 17m 13s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 87m 55s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821259/HBASE-16257-v2.patch |
| JIRA Issue | HBASE-16257 |
|
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400818#comment-15400818
]
Hadoop QA commented on HBASE-16257:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
27s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 21s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 12s
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 25s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 12s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 19m 57s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 22m 52s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 25m 45s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 28m 36s
{color} | {color:red} Patch causes 11 errors with Hadoop v2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 46s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 58s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 8s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green}
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388083#comment-15388083
]
Jerry He commented on HBASE-16257:
--
Yes, we have to deal with the permission on the hbase root dir and its subdirs.
/hbase/staging will be 711.
Currently under Kerberos, /hbase is 700. We need to relax it.
Instead, we will set all the subdirs that need protection to 700 individually,
.e.g. /hbase/data, /hbase/WALs, /hbase/oldWALs.
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387933#comment-15387933
]
Dima Spivak commented on HBASE-16257:
-
Wouldn't this also require making /hbase permissions 711 by default?
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387222#comment-15387222
]
Jerry He commented on HBASE-16257:
--
bq. You mean you want to just remove this config and go with a fixed path?
Yes. It will be like:
/hbase/data
/hbase/WALs
/hbase/oldWALs
/hbase/staging
This change looks backward compatible, since it is more like a tmp dir and we
pass the full path as bulk token between server and client.
But ignoring the property is a change from the user's perspective Any
suggestion?
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387209#comment-15387209
]
Anoop Sam John commented on HBASE-16257:
You mean you want to just remove this config and go with a fixed path? After
this change, we will append this config value under the hbase root path and
make the full path. So that is infact a behave change no?
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387188#comment-15387188
]
Jerry He commented on HBASE-16257:
--
We will deprecate 'hbase.bulkload.staging.dir'. I was thinking just ignoring
it in the new server, maybe print a message.
This will not cause any BC problem on its own because the directory is only
relevant internally to the region server. Right?
But we still need a deprecation cycle?
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387119#comment-15387119
]
Anoop Sam John commented on HBASE-16257:
So now this config 'hbase.bulkload.staging.dir' expects a full path and you
want to change the meaning of that also? When we say move it under hbase root
dir, it has to be always under that. So BC issue?
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387103#comment-15387103
]
Anoop Sam John commented on HBASE-16257:
+1 for doing this.
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15385281#comment-15385281
]
Dima Spivak commented on HBASE-16257:
-
Not having it under /hbase also causes issues with bulk load when HDFS has
encryption zones set up. Good idea.
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
