[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16585729#comment-16585729
]
Hudson commented on HBASE-17861:
Results for branch branch-1.3
[build #433 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/433/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/433//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/433//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/433//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
>Priority: Major
> Labels: filesystem, s3, wal
> Fix For: 1.4.0, 1.2.7, 1.3.3
>
> Attachments: HBASE-17861-V1.patch, HBASE-17861.branch-1.V1.patch,
> HBASE-17861.branch-1.V2.patch, HBASE-17861.branch-1.V3.patch,
> HBASE-17861.branch-1.V4.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = s3a://xx//xx
> hbase.wal.dir = hdfs://xx/xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16585715#comment-16585715
]
Hudson commented on HBASE-17861:
Results for branch branch-1.2
[build #437 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/437/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/437//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/437//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/437//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
>Priority: Major
> Labels: filesystem, s3, wal
> Fix For: 1.4.0, 1.2.7, 1.3.3
>
> Attachments: HBASE-17861-V1.patch, HBASE-17861.branch-1.V1.patch,
> HBASE-17861.branch-1.V2.patch, HBASE-17861.branch-1.V3.patch,
> HBASE-17861.branch-1.V4.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = s3a://xx//xx
> hbase.wal.dir = hdfs://xx/xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16585356#comment-16585356
]
Hudson commented on HBASE-17861:
FAILURE: Integrated in Jenkins build HBase-1.3-IT #455 (See
[https://builds.apache.org/job/HBase-1.3-IT/455/])
HBASE-17861: Regionserver down when checking the permission of staging (busbey:
rev 7d7279d19d534f660eaa8dd9dec63fe8b547fb89)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/SecureBulkLoadEndpoint.java
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
>Priority: Major
> Labels: filesystem, s3, wal
> Fix For: 1.4.0, 1.2.7, 1.3.3
>
> Attachments: HBASE-17861-V1.patch, HBASE-17861.branch-1.V1.patch,
> HBASE-17861.branch-1.V2.patch, HBASE-17861.branch-1.V3.patch,
> HBASE-17861.branch-1.V4.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = s3a://xx//xx
> hbase.wal.dir = hdfs://xx/xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16585345#comment-16585345
]
Hudson commented on HBASE-17861:
SUCCESS: Integrated in Jenkins build HBase-1.2-IT #1148 (See
[https://builds.apache.org/job/HBase-1.2-IT/1148/])
HBASE-17861: Regionserver down when checking the permission of staging (busbey:
rev 0466474d845413b4145e4602028ca49bf20130a4)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/SecureBulkLoadEndpoint.java
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
>Priority: Major
> Labels: filesystem, s3, wal
> Fix For: 1.4.0, 1.2.7, 1.3.3
>
> Attachments: HBASE-17861-V1.patch, HBASE-17861.branch-1.V1.patch,
> HBASE-17861.branch-1.V2.patch, HBASE-17861.branch-1.V3.patch,
> HBASE-17861.branch-1.V4.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = s3a://xx//xx
> hbase.wal.dir = hdfs://xx/xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969342#comment-15969342
]
Umesh Agashe commented on HBASE-17861:
--
Thanks to [~waterlx] for pointing this JIRA to me. Sometime back when I tried
running HBase on S3 I got the same issue. I discussed it with my colleague. All
these filesystems will differ in what typical FS features they support. I think
this needs to be handled at broader level than hard-coded list. We can have a
configuration toggle for checking permissions and when the exception is thrown
due to underlying filesystem limitations, we can point to the configuration
that needs to be toggled. This can be done for all similar FS issues that we
may come across in future.
Eventually we can document all configurations options in a reference doc to
support certain filesystem. Let me know your thoughts. Thanks!
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861.branch-1.V1.patch,
> HBASE-17861.branch-1.V2.patch, HBASE-17861.branch-1.V3.patch,
> HBASE-17861.branch-1.V4.patch, HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = s3a://xx//xx
> hbase.wal.dir = hdfs://xx/xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955492#comment-15955492
]
Hudson commented on HBASE-17861:
FAILURE: Integrated in Jenkins build HBase-1.4 #687 (See
[https://builds.apache.org/job/HBase-1.4/687/])
HBASE-17861: Regionserver down when checking the permission of staging (tedyu:
rev 4057a6c89c74c9f595cb51ac3bdc288396a0b257)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/SecureBulkLoadEndpoint.java
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861.branch-1.V1.patch,
> HBASE-17861.branch-1.V2.patch, HBASE-17861.branch-1.V3.patch,
> HBASE-17861.branch-1.V4.patch, HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = s3a://xx//xx
> hbase.wal.dir = hdfs://xx/xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15955436#comment-15955436
]
Jerry He commented on HBASE-17861:
--
+1
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861.branch-1.V1.patch,
> HBASE-17861.branch-1.V2.patch, HBASE-17861.branch-1.V3.patch,
> HBASE-17861.branch-1.V4.patch, HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = s3a://xx//xx
> hbase.wal.dir = hdfs://xx/xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954585#comment-15954585
]
Hadoop QA commented on HBASE-17861:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m 52s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
49s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} branch-1 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 55s
{color} | {color:red} hbase-server in branch-1 has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} branch-1 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
14m 52s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 82m 58s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 122m 45s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:e01ee2f |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12861809/HBASE-17861.branch-1.V4.patch
|
| JIRA Issue | HBASE-17861 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 069d176cb5e4 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24
10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954524#comment-15954524
]
Yi Liang commented on HBASE-17861:
--
new patch carry Ted's comments
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861.branch-1.V1.patch,
> HBASE-17861.branch-1.V2.patch, HBASE-17861.branch-1.V3.patch,
> HBASE-17861.branch-1.V4.patch, HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = s3a://xx//xx
> hbase.wal.dir = hdfs://xx/xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954519#comment-15954519
]
Ted Yu commented on HBASE-17861:
I don't think the failure of TestReplicasClient had anything to do with bulk
load.
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861.branch-1.V1.patch,
> HBASE-17861.branch-1.V2.patch, HBASE-17861.branch-1.V3.patch,
> HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = s3a://xx//xx
> hbase.wal.dir = hdfs://xx/xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954512#comment-15954512
]
Yi Liang commented on HBASE-17861:
--
ok, let me check if there are too many schemes
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861.branch-1.V1.patch,
> HBASE-17861.branch-1.V2.patch, HBASE-17861.branch-1.V3.patch,
> HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = s3a://xx//xx
> hbase.wal.dir = hdfs://xx/xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954506#comment-15954506
]
Ted Yu commented on HBASE-17861:
How about:
{code}
Set h = new HashSet(Arrays.asList("s3a", "s3", "s3n", "wasb",
"wasbs", ...));
{code}
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861.branch-1.V1.patch,
> HBASE-17861.branch-1.V2.patch, HBASE-17861.branch-1.V3.patch,
> HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = s3a://xx//xx
> hbase.wal.dir = hdfs://xx/xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954502#comment-15954502
]
Yi Liang commented on HBASE-17861:
--
Hi Ted, thanks for review, the reason why I did not use set is
(1) In java, When we initialize set, we still need to initialize as below,
transform a array to set; or we use add() method for multiple times to add
values. It seems not very efficient
Set h = new HashSet(Arrays.asList("a", "b")); or h.add(a);
h.add(b);
(2) In the code, in the check
condition(scheme.startsWith(FsNotSupportPermission[i])), I use startwith()
method, since there are schemes like: s3a, s3, s3n, wasb, wasbs and so on, the
Set.contains(FsNotSupportPermission[i]) method seems does not work quite well
in our case.
Let me know if you have any good idea to implement it, thanks, :)
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861.branch-1.V1.patch,
> HBASE-17861.branch-1.V2.patch, HBASE-17861.branch-1.V3.patch,
> HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = s3a://xx//xx
> hbase.wal.dir = hdfs://xx/xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954497#comment-15954497
]
Hadoop QA commented on HBASE-17861:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
48s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s
{color} | {color:green} branch-1 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 56s
{color} | {color:red} hbase-server in branch-1 has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} branch-1 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
14m 44s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 25s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 112m 30s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestReplicasClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:e01ee2f |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12861789/HBASE-17861.branch-1.V3.patch
|
| JIRA Issue | HBASE-17861 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux a858e26b4706 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
|
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954443#comment-15954443
]
Ted Yu commented on HBASE-17861:
FsNotSupportPermission -> FsWithoutPermissionSupport
Why do you use array instead of Set ?
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861.branch-1.V1.patch,
> HBASE-17861.branch-1.V2.patch, HBASE-17861.branch-1.V3.patch,
> HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = s3a://xx//xx
> hbase.wal.dir = hdfs://xx/xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954424#comment-15954424
]
Yi Liang commented on HBASE-17861:
--
new patch attaced
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861.branch-1.V1.patch,
> HBASE-17861.branch-1.V2.patch, HBASE-17861.branch-1.V3.patch,
> HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = s3a://xx//xx
> hbase.wal.dir = hdfs://xx/xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954405#comment-15954405
]
Hadoop QA commented on HBASE-17861:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
48s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} branch-1 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 56s
{color} | {color:red} hbase-server in branch-1 has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} branch-1 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
14m 53s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 17s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 112m 43s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestScannerHeartbeatMessages |
| | hadoop.hbase.client.TestReplicasClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:e01ee2f |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12861769/HBASE-17861.branch-1.V2.patch
|
| JIRA Issue | HBASE-17861 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux d024ff5a0daf 3.13.0-107-generic #154-Ubuntu SMP
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954327#comment-15954327
]
Zach York commented on HBASE-17861:
---
bq. There may be more file system(s) added in the future, how about putting the
3 known FS in a private Set so that the above check is more readable ?
+1 I meant to say this earlier, but forgot before you uploaded the patch.
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861.branch-1.V1.patch,
> HBASE-17861.branch-1.V2.patch, HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = s3a://xx//xx
> hbase.wal.dir = hdfs://xx/xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954318#comment-15954318
]
Ted Yu commented on HBASE-17861:
{code}
157 if (!scheme.startsWith("s3") && !scheme.startsWith("wasb") &&
!scheme.startsWith("swift") && !status.getPermission().equals(PERM_HIDDEN)) {
{code}
There may be more file system(s) added in the future, how about putting the 3
known FS in a private Set so that the above check is more readable ?
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861.branch-1.V1.patch,
> HBASE-17861.branch-1.V2.patch, HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = s3a://xx//xx
> hbase.wal.dir = hdfs://xx/xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954246#comment-15954246
]
Yi Liang commented on HBASE-17861:
--
New patch carry Jerry and Zach's comments
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861.branch-1.V1.patch,
> HBASE-17861.branch-1.V2.patch, HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = s3a://xx//xx
> hbase.wal.dir = hdfs://xx/xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952565#comment-15952565
]
Yi Liang commented on HBASE-17861:
--
oh, my bad, will change the description
I mean when set root dir on s3
hbase.rootdir = s3a://xx//xx
hbase.wal.dir = hdfs://xx/xx
After checking the code, I found that on master branch, the staging dir can
only on the same filesystem as root dir, but on branch-1, it seems not.
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861.branch-1.V1.patch, HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = hdfs://xx/xx
> hbase.wal.dir = s3a://xx//xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952106#comment-15952106
]
Anoop Sam John commented on HBASE-17861:
{quote}
hbase.rootdir = hdfs://xx/xx
hbase.wal.dir = s3a://xx//xx
{quote}
U mean root dir in S3 while WAL in HDFS? When root is in HDFS am not sure why
bulk load staging dir is getting created inside S3.
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861.branch-1.V1.patch, HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = hdfs://xx/xx
> hbase.wal.dir = s3a://xx//xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952042#comment-15952042
]
Jerry He commented on HBASE-17861:
--
We can just these three: s3, wasb and swift, as [~zyork] suggested, and ignore
case.
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861.branch-1.V1.patch, HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = hdfs://xx/xx
> hbase.wal.dir = s3a://xx//xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951884#comment-15951884
]
Hadoop QA commented on HBASE-17861:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m 57s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
53s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s
{color} | {color:green} branch-1 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 49s
{color} | {color:red} hbase-server in branch-1 has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s
{color} | {color:green} branch-1 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
14m 49s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 85m 27s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 125m 6s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestReplicasClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:e01ee2f |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12861535/HBASE-17861.branch-1.V1.patch
|
| JIRA Issue | HBASE-17861 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 93b9d3eddaec 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
|
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951757#comment-15951757
]
Zach York commented on HBASE-17861:
---
For example, it could be something like this:
instead of:
if(fs.getScheme().startsWith("hdfs")
->
if (fs.getScheme().startsWith("s3").
Also are scheme's always lowercase or do you need to ensure it is lowercase
before the comparison?
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861.branch-1.V1.patch, HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = hdfs://xx/xx
> hbase.wal.dir = s3a://xx//xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951750#comment-15951750
]
Zach York commented on HBASE-17861:
---
I'm not saying that you need to check every conceivable filesystem. I'm just
saying perhaps the patch should exclude if the scheme contains s3 (which would
cover s3, s3a, s3n, etc) rather than removing it for everything but hdfs (since
there are likely other FS implementations other than hdfs that DO support rwx
style permissions).
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861.branch-1.V1.patch, HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = hdfs://xx/xx
> hbase.wal.dir = s3a://xx//xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951739#comment-15951739
]
Yi Liang commented on HBASE-17861:
--
>From Enis comment in HBASE-11045:
[[HBase is supported (to the extend that corresponding vendors support it) on a
couple of file systems other than HDFS: gpfs, maprfs, EMC Isilon, Microsoft
WASB are the ones from the top of my head. You should contact the corresponding
vendor if you want to learn more. ]]
And I just check Microsoft WASB, it seems it does not support permission, see
https://hadoop.apache.org/docs/stable/hadoop-azure/index.html. It says "File
owner and group are persisted, but the permissions model is not enforced.
Authorization occurs at the level of the entire Azure Blob Storage account."
I probably could not cover all the above FS
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = hdfs://xx/xx
> hbase.wal.dir = s3a://xx//xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951728#comment-15951728
]
Jerry He commented on HBASE-17861:
--
bq. It seems like this should be the exception instead of the rule.
Makes sense.
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = hdfs://xx/xx
> hbase.wal.dir = s3a://xx//xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951702#comment-15951702
]
Zach York commented on HBASE-17861:
---
Are there other FileSystems other than S3 that don't have permissions? It seems
like this should be the exception instead of the rule.
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = hdfs://xx/xx
> hbase.wal.dir = s3a://xx//xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951696#comment-15951696
]
Ted Yu commented on HBASE-17861:
Name your patch HBASE-17861.branch-1.V1.patch
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = hdfs://xx/xx
> hbase.wal.dir = s3a://xx//xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951695#comment-15951695
]
Yi Liang commented on HBASE-17861:
--
I mean branch-1
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = hdfs://xx/xx
> hbase.wal.dir = s3a://xx//xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951640#comment-15951640
]
Ted Yu commented on HBASE-17861:
You can name your patch HBASE-17861.branch-1.0.V1.patch
But is there going to be any more release for branch-1.0 ?
Did you mean branch-1.1 ?
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = hdfs://xx/xx
> hbase.wal.dir = s3a://xx//xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951607#comment-15951607
]
Yi Liang commented on HBASE-17861:
--
Hi Ted, Thanks for reviewing, I have test it on S3 with HBase 1.2.4.
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = hdfs://xx/xx
> hbase.wal.dir = s3a://xx//xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See Object stores have differerent authorization
> models in
> https://hadoop.apache.org/docs/current3/hadoop-aws/tools/hadoop-aws/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951606#comment-15951606
]
Hadoop QA commented on HBASE-17861:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s {color}
| {color:red} HBASE-17861 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12861500/HBASE-17861-V1.patch |
| JIRA Issue | HBASE-17861 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6288/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = hdfs://xx/xx
> hbase.wal.dir = s3a://xx//xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See simulated permission section in
> https://docs.hortonworks.com/HDPDocuments/HDCloudAWS/HDCloudAWS-1.8.0/bk_hdcloud-aws/content/s3-s3aclient/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951605#comment-15951605
]
Ted Yu commented on HBASE-17861:
lgtm
Have you tested with s3a filesystem ?
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Labels: filesystem, s3, wal
> Fix For: 1.4.0
>
> Attachments: HBASE-17861-V1.patch
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = hdfs://xx/xx
> hbase.wal.dir = s3a://xx//xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See simulated permission section in
> https://docs.hortonworks.com/HDPDocuments/HDCloudAWS/HDCloudAWS-1.8.0/bk_hdcloud-aws/content/s3-s3aclient/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17861) Regionserver down when checking the permission of staging dir if hbase.rootdir is on S3
[
https://issues.apache.org/jira/browse/HBASE-17861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951589#comment-15951589
]
Yi Liang commented on HBASE-17861:
--
And in our master branch, the code in SecureBulkLoadEndpoint has been
refactored, and the statement check the folder permission is removed. so the
patch is only for branch-1.0
> Regionserver down when checking the permission of staging dir if
> hbase.rootdir is on S3
> ---
>
> Key: HBASE-17861
> URL: https://issues.apache.org/jira/browse/HBASE-17861
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Fix For: 1.4.0
>
>
> Found some issue, when set up HBASE-17437: Support specifying a WAL directory
> outside of the root directory.
> The region server are showdown when I add following config into
> hbase-site.xml
> hbase.rootdir = hdfs://xx/xx
> hbase.wal.dir = s3a://xx//xx
> hbase.coprocessor.region.classes =
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> Error is below
> {noformat}
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint threw
> java.lang.IllegalStateException: Directory already exists but permissions
> aren't set to '-rwx--x--x'
> {noformat}
> The reason is that, when hbase enable securebulkload, hbase will create a
> folder in s3, it can not set above permission, because in s3, all files are
> listed as having full read/write permissions and all directories appear to
> have full rwx permissions. See simulated permission section in
> https://docs.hortonworks.com/HDPDocuments/HDCloudAWS/HDCloudAWS-1.8.0/bk_hdcloud-aws/content/s3-s3aclient/index.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
