[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15517586#comment-15517586 ] Hudson commented on HBASE-16257: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #1660 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1660/]) HBASE-16257 Move staging dir to be under hbase root dir (jerryjch: rev 50b051ade1d38bba0b936db1c751e5045bc103cb) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/SecureBulkLoadEndpointClient.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/hbck/HFileCorruptionChecker.java * (edit) hbase-common/src/main/resources/hbase-default.xml * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/SecureBulkLoadClient.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplit.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterWalManager.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java * (delete) hbase-client/src/main/java/org/apache/hadoop/hbase/security/SecureBulkLoadUtil.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SecureBulkLoadManager.java * (edit) hbase-common/src/test/java/org/apache/hadoop/hbase/HBaseCommonTestingUtility.java > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch, > HBASE-16257-v3.patch, HBASE-16257-v4.patch, HBASE-16257-v5.patch > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15517005#comment-15517005 ] Jerry He commented on HBASE-16257: -- Thanks for the review. bq. Is this between 1.x -> 2.0 replication? i didn't test this case. But this JIRA would not affect it, as we had discussed. > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch, > HBASE-16257-v3.patch, HBASE-16257-v4.patch, HBASE-16257-v5.patch > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514127#comment-15514127 ] Enis Soztutar commented on HBASE-16257: --- bq. I tested with a local standalone instance and on a Keberos cluster. All looks good. Cool. Thanks Jerry for the test. Is this between 1.x -> 2.0 replication? +1 for v5. > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch, > HBASE-16257-v3.patch, HBASE-16257-v4.patch, HBASE-16257-v5.patch > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15505478#comment-15505478 ] Hadoop QA commented on HBASE-16257: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 5 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 36s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 13s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 4s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 34s {color} | {color:green} master passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 38s {color} | {color:red} hbase-common in master has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s {color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 22s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 4s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 4s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 56s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 30s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s {color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 27m 38s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} | | {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 29s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 23s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 48s {color} | {color:green} hbase-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 2s {color} | {color:green} hbase-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 85m 53s {color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 38s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 138m 14s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Timed out junit tests | org.apache.hadoop.hbase.client.TestReplicasClient | | | org.apache.hadoop.hbase.client.TestFromClientSide | | | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient | | | org.apache.hadoop.hbase.client.TestMobSnapshotCloneIndependence | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12829314/HBASE-16257-v5.patch | | JIRA Issue | HBASE-16257 | | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile xml | | uname | Linux 2912f63a6ac9 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15505232#comment-15505232 ] Jerry He commented on HBASE-16257: -- bq. If it is not the case, then we can commit this patch as it is. Is this +1 [~enis]? Thanks for the comment, [~ashish singhi] I tested with a local standalone instance and on a Keberos cluster. All looks good. The directories are created or modified with the correct permissions: {noformat} hadoop fs -ls /apps/hbase/ drwx--x--x - hbase hdfs 0 2016-09-19 13:48 /apps/hbase/data hadoop fs -ls /apps/hbase/data drwx-- - hbase hdfs 0 2016-09-19 11:25 /apps/hbase/data/.hbck drwx-- - hbase hdfs 0 2016-09-19 13:48 /apps/hbase/data/.tmp drwx-- - hbase hdfs 0 2016-09-19 18:01 /apps/hbase/data/MasterProcWALs drwx-- - hbase hdfs 0 2016-09-19 13:48 /apps/hbase/data/WALs drwx-- - hbase hdfs 0 2016-09-19 14:01 /apps/hbase/data/archive drwx-- - hbase hdfs 0 2016-06-17 15:25 /apps/hbase/data/corrupt drwx-- - hbase hdfs 0 2016-06-09 12:36 /apps/hbase/data/data -rw--- 3 hbase hdfs 42 2016-06-09 12:35 /apps/hbase/data/hbase.id -rw--- 3 hbase hdfs 7 2016-06-09 12:35 /apps/hbase/data/hbase.version drwx-- - hbase hdfs 0 2016-09-19 11:25 /apps/hbase/data/mobdir drwx-- - hbase hdfs 0 2016-09-19 17:59 /apps/hbase/data/oldWALs drwx--x--x - hbase hdfs 0 2016-09-19 17:34 /apps/hbase/data/staging {noformat} Ran a few bulk loads as well. v5 adds back creating staging dir in SecureBulkLoadManager in the region server side if it does not exist, so that it is ok to upgrade region server before master. > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch, > HBASE-16257-v3.patch, HBASE-16257-v4.patch > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15504211#comment-15504211 ] Enis Soztutar commented on HBASE-16257: --- If source is 2.0 with this patch, but sink is 1.x without this patch, then the sink will not create the staging dir under root. However, on the source side, we will still be accessing the staging dir under root on the sink fs. > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch, > HBASE-16257-v3.patch, HBASE-16257-v4.patch > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15504158#comment-15504158 ] Ashish Singhi commented on HBASE-16257: --- It need not. It's HFileReplicator which will create and access the directory in the sink cluster and it will run in sink cluster so there should not be any problem. > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch, > HBASE-16257-v3.patch, HBASE-16257-v4.patch > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15504140#comment-15504140 ] Enis Soztutar commented on HBASE-16257: --- In secure 1.x clusters, root dir is protected with 700. So unless the user is the same, source cluster cannot create or access a directory under the sink cluster. > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch, > HBASE-16257-v3.patch, HBASE-16257-v4.patch > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15504142#comment-15504142 ] Enis Soztutar commented on HBASE-16257: --- In secure 1.x clusters, root dir is protected with 700. So unless the user is the same, source cluster cannot create or access a directory under the sink cluster. > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch, > HBASE-16257-v3.patch, HBASE-16257-v4.patch > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15503034#comment-15503034 ] Ashish Singhi commented on HBASE-16257: --- Your thinking is correct, HFileReplicator is using the staging dir configured on the sink cluster. It is getting the staging dir path from the sink cluster conf. If the staging dir doesn't exist it will create it and copy the HFiles from the source cluster to the sink cluster staging dir and then run the complete bulkload tool. > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch, > HBASE-16257-v3.patch, HBASE-16257-v4.patch > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15502243#comment-15502243 ] Enis Soztutar commented on HBASE-16257: --- Sorry, I was thinking that HFileReplicator is using the staging dir on the remote end. If it is not the case, then we can commit this patch as it is. > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch, > HBASE-16257-v3.patch, HBASE-16257-v4.patch > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15498233#comment-15498233 ] Ashish Singhi commented on HBASE-16257: --- You got it right [~jerryhe]. bq. the staging dir will not be there under the 1.x root, which will then case the HFileReplicator to fail. Sorry I did not get this why HFileReplicator will fail here ? > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch, > HBASE-16257-v3.patch, HBASE-16257-v4.patch > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15498070#comment-15498070 ] Jerry He commented on HBASE-16257: -- Hi, [~enis] Thinking a bit more. Correct me if I am wrong. HFileReplicator operates on the sink side. It tries to copy from the sourceFs to local sink staging dir. If we have 2.x -> 1.x, there is no rootDir/staging on the sink. But it is ok. Sink knows how to handle its own staging dir, new way or old way. Source will not try to access sink staging dir and sink will not try to access source staging dir. > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch, > HBASE-16257-v3.patch, HBASE-16257-v4.patch > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15497701#comment-15497701 ] Jerry He commented on HBASE-16257: -- Good point.Let me think thru this case. FYI Ashish Singhi > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch, > HBASE-16257-v3.patch, HBASE-16257-v4.patch > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15497422#comment-15497422 ] Enis Soztutar commented on HBASE-16257: --- Thanks Jerry for the latest patch. Sorry, but one last item may need to be addressed. Otherwise, everything looks good. The HFileReplicator now uses the bulk load staging dir as a path under root. This will become a problem if the source cluster is upgraded first before the sink cluster. In case of cyclic replication, it will always be the case that you can have 2.x -> 1.x replication. If that is the case, the staging dir will not be there under the 1.x root, which will then case the HFileReplicator to fail. Maybe we can do a simple exists check on the remote FS to see whether we can use the new hbase.root.dir/staging or we should fall back to using the value from configuration. > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch, > HBASE-16257-v3.patch, HBASE-16257-v4.patch > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15488906#comment-15488906 ] Hadoop QA commented on HBASE-16257: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 5 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 56s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 55s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 30s {color} | {color:green} master passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 31s {color} | {color:red} hbase-common in master has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s {color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 12s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 1s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 56s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 29s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s {color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 24m 18s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} | | {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 28s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 13s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 40s {color} | {color:green} hbase-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 50s {color} | {color:green} hbase-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 86m 38s {color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 39s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 132m 42s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Timed out junit tests | org.apache.hadoop.hbase.client.TestReplicasClient | | | org.apache.hadoop.hbase.client.TestFromClientSide | | | org.apache.hadoop.hbase.client.TestMetaWithReplicas | | | org.apache.hadoop.hbase.client.TestFromClientSide3 | | | org.apache.hadoop.hbase.client.TestTableSnapshotScanner | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12828340/HBASE-16257-v4.patch | | JIRA Issue | HBASE-16257 | | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile xml | | uname | Linux 021e35ca14df 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 201
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486284#comment-15486284 ] Jerry He commented on HBASE-16257: -- bq. Maybe extract the subdirs away, and add getters so that rest of the code does not have to deal with parsing the conf again: Do you mean to do something similar to the temp dir? {code} this.tempdir = new Path(this.rootdir, HConstants.HBASE_TEMP_DIRECTORY); ... /** * @return HBase temp dir. */ public Path getTempDir() { return this.tempdir; } {code} > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch, > HBASE-16257-v3.patch > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486278#comment-15486278 ] Jerry He commented on HBASE-16257: -- bq. We create this dir with default perms. If there is a way to get the default perms, and after ensuring that we have x, setting it to root should be fine. [~enis] This would work for you? As the last step: {code} FsPermission currentRootPerms = fs.getFileStatus(this.rootdir).getPermission(); if (!currentRootPerms.getUserAction().implies(FsAction.EXECUTE) || !currentRootPerms.getGroupAction().implies(FsAction.EXECUTE) || !currentRootPerms.getOtherAction().implies(FsAction.EXECUTE)) { LOG.warn("rootdir permissions do not contain 'excute' for user, group or other. " + "Automatically adding 'excute' permission for all"); fs.setPermission( this.rootdir, new FsPermission(currentRootPerms.getUserAction().or(FsAction.EXECUTE), currentRootPerms .getGroupAction().or(FsAction.EXECUTE), currentRootPerms.getOtherAction().or( FsAction.EXECUTE))); } {code} > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch, > HBASE-16257-v3.patch > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15485879#comment-15485879 ] Hadoop QA commented on HBASE-16257: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 5 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 37s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 15s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 5s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 35s {color} | {color:green} master passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 36s {color} | {color:red} hbase-common in master has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s {color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 26s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 4s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 4s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 57s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 31s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s {color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 25m 13s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} | | {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 30s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 29s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 42s {color} | {color:green} hbase-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 51s {color} | {color:green} hbase-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 9s {color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 41s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 130m 54s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hbase.wal.TestWALSplit | | | hadoop.hbase.regionserver.TestScannerHeartbeatMessages | | | hadoop.hbase.wal.TestWALSplitCompressed | | Timed out junit tests | org.apache.hadoop.hbase.client.TestMultiRespectsLimits | | | org.apache.hadoop.hbase.client.TestSnapshotMetadata | | | org.apache.hadoop.hbase.client.TestTableSnapshotScanner | | | org.apache.hadoop.hbase.client.TestCloneSnapshotFromClient | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12828131/HBASE-16257-v3.patch | | JIRA Issue | HBASE-16257 | | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15485860#comment-15485860 ] Hadoop QA commented on HBASE-16257: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 46s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 5 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 30s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 10s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 1s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 33s {color} | {color:green} master passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 39s {color} | {color:red} hbase-common in master has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s {color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 23s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 11s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 11s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 0s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 32s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s {color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 29m 2s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} | | {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 29s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 23s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 47s {color} | {color:green} hbase-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 53s {color} | {color:green} hbase-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 96m 30s {color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 42s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 152m 9s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Timed out junit tests | org.apache.hadoop.hbase.snapshot.TestSnapshotClientRetries | | | org.apache.hadoop.hbase.snapshot.TestMobSecureExportSnapshot | | | org.apache.hadoop.hbase.filter.TestFilterListOrOperatorWithBlkCnt | | | org.apache.hadoop.hbase.snapshot.TestRestoreFlushSnapshotFromClient | | | org.apache.hadoop.hbase.client.TestHCM | | | org.apache.hadoop.hbase.snapshot.TestMobFlushSnapshotFromClient | | | org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient | | | org.apache.hadoop.hbase.TestMetaTableAccessor | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12828125/HBASE-16257-v3.patch | | JIRA
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15485678#comment-15485678 ] Enis Soztutar commented on HBASE-16257: --- bq. Hmm where is the other place? Here: {code} --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SecureBulkLoadManager.java +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SecureBulkLoadManager.java + private final static String BULKLOAD_STAGING_DIR = "hbase.bulkload.staging.dir"; {code} bq. Is the following ok? We create this dir with default perms. If there is a way to get the default perms, and after ensuring that we have {{x}}, setting it to root should be fine. > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch, > HBASE-16257-v3.patch > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15485653#comment-15485653 ] Jerry He commented on HBASE-16257: -- bq. Can we either use all upper case or all lower case for the names of these bq. Let's add a comment here talking about the layout of rootdir and the permissions of its subdirs. Will do. bq. This is now defined in two places Hmm where is the other place? bq. I was mentioning above that, for upgrades, we have to "open-up" the permissions for the root directory Is the following ok? {noformat} +// Handle the last few special files and set the final rootdir permissions +// rootdir needs 'x' for all for bulk load staging dir +if (isSecurityEnabled) { + fs.setPermission(new Path(rootdir, HConstants.VERSION_FILE_NAME), secureRootFilePerms); + fs.setPermission(new Path(rootdir, HConstants.CLUSTER_ID_FILE_NAME), secureRootFilePerms); +} +fs.setPermission(this.rootdir, this.rootPerms); {noformat} > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch, > HBASE-16257-v3.patch > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15485581#comment-15485581 ] Enis Soztutar commented on HBASE-16257: --- Can we either use all upper case or all lower case for the names of these: {code} + private final FsPermission rootPerms = FsPermission.valueOf("-rwxr-xr-x"); + // permissions for the files under rootDir that need secure protection + private final FsPermission secureRootFilePerms = FsPermission.valueOf("-rw---"); + // permissions for the directories under rootDir that need secure protection + private final FsPermission secureRootSubDirPerms; + // permissions for bulk load staging directory under rootDir + private final FsPermission PERM_HIDDEN = FsPermission.valueOf("-rwx--x--x"); {code} Let's add a comment here talking about the layout of rootdir and the permissions of its subdirs. This is now defined in two places: {code} + public static final String BULKLOAD_STAGING_DIR_NAME = "staging"; {code} I was mentioning above that, for upgrades, we have to "open-up" the permissions for the root directory because if a branch-1 cluster is run in secure mode, the root dir is already wiht perms 700. After upgrade, the staging dir will be inside this root dir, and all BL operation will fail after the upgrade. {code} // Filesystem is good. Go ahead and check for hbase.rootdir. try { if (!fs.exists(rd)) { -if (isSecurityEnabled) { - fs.mkdirs(rd, rootDirPerms); -} else { - fs.mkdirs(rd); -} +fs.mkdirs(rd); {code} Maybe extract the subdirs away, and add getters so that rest of the code does not have to deal with parsing the conf again: {code} +checkSubDir(new Path(this.rootdir, HConstants.BASE_NAMESPACE_DIR)); +checkSubDir(new Path(this.rootdir, HConstants.HFILE_ARCHIVE_DIRECTORY)); +checkSubDir(new Path(this.rootdir, HConstants.HREGION_LOGDIR_NAME)); {code} > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch, > HBASE-16257-v3.patch > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15475899#comment-15475899 ] Jerry He commented on HBASE-16257: -- bq. How do we handle upgrades? This would be compatible with rolling upgrade, as indicated in the parent JIRA. Changing the staging dir in the middle won't break the compatibility. Old and new staging dirs can co-exists. Bulk load token is full path. > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15475863#comment-15475863 ] Jerry He commented on HBASE-16257: -- Hi, Enis Thanks for the comments. I will update the patch as commented. bq. Are these changes related? Not directly related. The intention was to put the CORRUPT_DIR_NAME always under hbase.rootdir and get rid of the property "hbase.hfile.quarantine.dir". It is not referenced or documented anywhere else. Ok to remove? I can put a deprecate warning in the log. bq. Is this conf coming from the remote(sink) cluster for the HFileReplicator? Yes > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15474713#comment-15474713 ] Enis Soztutar commented on HBASE-16257: --- bq. Could you give more details on the whitelist/blacklist suggestion, Enis Soztutar? Thanks. I was trying to suggest that we do a black list of sub-directories inside the root directory for "special permissions" other than 700, instead of a white list of subdirectories with permission "700". The approach we have in the patch seems fine. What happens if we create the staging dir and on the next restart we do run checkSubDir() methods? Will this fail because it does not match 700? How do we handle upgrades? In secure clusters, the root dir is already set to 700, so the master has to open it up, but only after changing the perms for the subdirectories. I think the MasterFS should own and manage creating the staging directory as well, instead of the SBLM since MFS is responsible for owning the directory layout. Are these changes related? {code} -Path corruptBaseDir = new Path(FSUtils.getRootDir(conf), conf.get( -"hbase.hfile.quarantine.dir", HConstants.CORRUPT_DIR_NAME)); +Path corruptBaseDir = new Path(FSUtils.getRootDir(conf), HConstants.CORRUPT_DIR_NAME); {code} Is this conf coming from the remote(sink) cluster for the HFileReplicator? Sorry I did not check the code. {code} +this.hbaseStagingDir = new Path(FSUtils.getRootDir(conf), HConstants.BULKLOAD_STAGING_DIR_NAME); {code} > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15423778#comment-15423778 ] Jerry He commented on HBASE-16257: -- Ping for comments. [~enis], [~mbertozzi]? Thanks. > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15403314#comment-15403314 ] Jerry He commented on HBASE-16257: -- Could you give more details on the whitelist/blacklist suggestion, [~enis]? Thanks. > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15402539#comment-15402539 ] Enis Soztutar commented on HBASE-16257: --- bq. Instead, we will set all the subdirs that need protection to 700 individually, .e.g. /hbase/data, /hbase/WALs, /hbase/oldWALs. We have to be careful if we add a new subdirectory, we should make sure that it also gets correct permissions. So do this as a black-list, rather than whitelist maybe. > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15402531#comment-15402531 ] Enis Soztutar commented on HBASE-16257: --- Indeed. > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15401540#comment-15401540 ] Hadoop QA commented on HBASE-16257: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s {color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 5 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 6s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 36s {color} | {color:green} master passed with JDK v1.8.0 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 19s {color} | {color:green} master passed with JDK v1.7.0_80 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 15s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 50s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 20s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 21s {color} | {color:green} master passed with JDK v1.8.0 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s {color} | {color:green} master passed with JDK v1.7.0_80 {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 38s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 32s {color} | {color:green} the patch passed with JDK v1.8.0 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 32s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 29s {color} | {color:green} the patch passed with JDK v1.7.0_80 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 29s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 54s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s {color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 35m 14s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 6s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 21s {color} | {color:green} the patch passed with JDK v1.8.0 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 15s {color} | {color:green} the patch passed with JDK v1.7.0_80 {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 59s {color} | {color:green} hbase-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 3s {color} | {color:green} hbase-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 17m 13s {color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 27s {color} | {color:green} Patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 87m 55s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12821259/HBASE-16257-v2.patch | | JIRA Issue |
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15400818#comment-15400818 ] Hadoop QA commented on HBASE-16257: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s {color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 5 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 27s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 21s {color} | {color:green} master passed with JDK v1.8.0 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 12s {color} | {color:green} master passed with JDK v1.7.0_80 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 10s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 43s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 43s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s {color} | {color:green} master passed with JDK v1.8.0 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s {color} | {color:green} master passed with JDK v1.7.0_80 {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 26s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 25s {color} | {color:green} the patch passed with JDK v1.8.0 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 25s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 12s {color} | {color:green} the patch passed with JDK v1.7.0_80 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 12s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 10s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 44s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s {color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 19m 57s {color} | {color:red} Patch causes 11 errors with Hadoop v2.6.1. {color} | | {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 22m 52s {color} | {color:red} Patch causes 11 errors with Hadoop v2.6.2. {color} | | {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 25m 45s {color} | {color:red} Patch causes 11 errors with Hadoop v2.6.3. {color} | | {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 28m 36s {color} | {color:red} Patch causes 11 errors with Hadoop v2.7.1. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 23s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s {color} | {color:green} the patch passed with JDK v1.8.0 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s {color} | {color:green} the patch passed with JDK v1.7.0_80 {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 46s {color} | {color:green} hbase-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 58s {color} | {color:green} hbase-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 8s {color} | {color:red} hbase-server in the patch failed. {color} | | {color:green}+1{color}
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388083#comment-15388083 ] Jerry He commented on HBASE-16257: -- Yes, we have to deal with the permission on the hbase root dir and its subdirs. /hbase/staging will be 711. Currently under Kerberos, /hbase is 700. We need to relax it. Instead, we will set all the subdirs that need protection to 700 individually, .e.g. /hbase/data, /hbase/WALs, /hbase/oldWALs. > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 2.0.0 > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15387933#comment-15387933 ] Dima Spivak commented on HBASE-16257: - Wouldn't this also require making /hbase permissions 711 by default? > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 2.0.0 > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15387222#comment-15387222 ] Jerry He commented on HBASE-16257: -- bq. You mean you want to just remove this config and go with a fixed path? Yes. It will be like: /hbase/data /hbase/WALs /hbase/oldWALs /hbase/staging This change looks backward compatible, since it is more like a tmp dir and we pass the full path as bulk token between server and client. But ignoring the property is a change from the user's perspective Any suggestion? > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 2.0.0 > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15387209#comment-15387209 ] Anoop Sam John commented on HBASE-16257: You mean you want to just remove this config and go with a fixed path? After this change, we will append this config value under the hbase root path and make the full path. So that is infact a behave change no? > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 2.0.0 > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15387188#comment-15387188 ] Jerry He commented on HBASE-16257: -- We will deprecate 'hbase.bulkload.staging.dir'. I was thinking just ignoring it in the new server, maybe print a message. This will not cause any BC problem on its own because the directory is only relevant internally to the region server. Right? But we still need a deprecation cycle? > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 2.0.0 > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15387119#comment-15387119 ] Anoop Sam John commented on HBASE-16257: So now this config 'hbase.bulkload.staging.dir' expects a full path and you want to change the meaning of that also? When we say move it under hbase root dir, it has to be always under that. So BC issue? > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 2.0.0 > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15387103#comment-15387103 ] Anoop Sam John commented on HBASE-16257: +1 for doing this. > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 2.0.0 > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[ https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15385281#comment-15385281 ] Dima Spivak commented on HBASE-16257: - Not having it under /hbase also causes issues with bulk load when HDFS has encryption zones set up. Good idea. > Move staging dir to be under hbase root dir > --- > > Key: HBASE-16257 > URL: https://issues.apache.org/jira/browse/HBASE-16257 > Project: HBase > Issue Type: Sub-task >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 2.0.0 > > > The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then > defaults to > {code} > public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/" > + System.getProperty("user.name") + "/hbase-staging"; > {code} > This default would have problem on local file system standalone case. > We can move the staging dir to be under hbase.rootdir. We are bringing > secure bulkload to the core. It makes sense to bring it under core control as > well, instead of an optional property. -- This message was sent by Atlassian JIRA (v6.3.4#6332)