[jira] [Updated] (HADOOP-13933) Add haadmin -getAllServiceState option to get the HA state of all the NameNodes/ResourceManagers
[
https://issues.apache.org/jira/browse/HADOOP-13933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HADOOP-13933:
---
Target Version/s: 2.9.0
Issue Type: New Feature (was: Improvement)
Summary: Add haadmin -getAllServiceState option to get the HA
state of all the NameNodes/ResourceManagers (was: Add haadmin command to get
HA state of all the namenodes)
> Add haadmin -getAllServiceState option to get the HA state of all the
> NameNodes/ResourceManagers
>
>
> Key: HADOOP-13933
> URL: https://issues.apache.org/jira/browse/HADOOP-13933
> Project: Hadoop Common
> Issue Type: New Feature
> Components: tools
>Affects Versions: 2.7.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HADOOP-13933.002.patch, HADOOP-13933.003.patch,
> HADOOP-13933.003.patch, HADOOP-13933.004.patch, HDFS-9559.01.patch
>
>
> Currently we have one command to get state of namenode.
> {code}
> ./hdfs haadmin -getServiceState
> {code}
> It will be good to have command which will give state of all the namenodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13961) mvn install fails on trunk
[
https://issues.apache.org/jira/browse/HADOOP-13961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15814029#comment-15814029
]
John Zhuge commented on HADOOP-13961:
-
[~sjlee0] Couldn't find out a better approach than patch 001, probably because
hadoop-hdfs:test depends on hadoop-kms:test which in turn depends on
hadoop-kms:main. Not easy to set up this kind of transitive dependencies. But I
am not Maven expect, someone might have a good solution?
> mvn install fails on trunk
> --
>
> Key: HADOOP-13961
> URL: https://issues.apache.org/jira/browse/HADOOP-13961
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha2
>Reporter: Karthik Kambatla
>Assignee: John Zhuge
>Priority: Blocker
> Attachments: HADOOP-13961.001.patch
>
>
> mvn install fails for me on trunk on a new environment with the following:
> {noformat}
> [ERROR] Failed to execute goal on project hadoop-hdfs: Could not resolve
> dependencies for project
> org.apache.hadoop:hadoop-hdfs:jar:3.0.0-alpha2-SNAPSHOT: Could not find
> artifact
> org.apache.hadoop:hadoop-kms:jar:classes:3.0.0-alpha2-20161228.102554-925 in
> apache.snapshots.https
> (https://repository.apache.org/content/repositories/snapshots) -> [Help 1]
> {noformat}
> This works on an existing dev setup, likely because I have the jar in my m2
> cache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13962) Update ADLS SDK to 2.1.4
[ https://issues.apache.org/jira/browse/HADOOP-13962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813937#comment-15813937 ] John Zhuge commented on HADOOP-13962: - I don't think so. [~ASikaria], could you comment? Should also wait for HADOOP-13927 fix which might require ADLS backend fix and SDK api-version bump. > Update ADLS SDK to 2.1.4 > > > Key: HADOOP-13962 > URL: https://issues.apache.org/jira/browse/HADOOP-13962 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Affects Versions: 3.0.0-alpha2 >Reporter: John Zhuge >Assignee: John Zhuge > > ADLS has multiple upgrades since the version 2.0.11 we are using: 2.1.1, > 2.1.2, and 2.1.4. Change list: > https://github.com/Azure/azure-data-lake-store-java/blob/master/CHANGES.md. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13961) mvn install fails on trunk
[
https://issues.apache.org/jira/browse/HADOOP-13961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HADOOP-13961:
Status: In Progress (was: Patch Available)
> mvn install fails on trunk
> --
>
> Key: HADOOP-13961
> URL: https://issues.apache.org/jira/browse/HADOOP-13961
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha2
>Reporter: Karthik Kambatla
>Assignee: John Zhuge
>Priority: Blocker
> Attachments: HADOOP-13961.001.patch
>
>
> mvn install fails for me on trunk on a new environment with the following:
> {noformat}
> [ERROR] Failed to execute goal on project hadoop-hdfs: Could not resolve
> dependencies for project
> org.apache.hadoop:hadoop-hdfs:jar:3.0.0-alpha2-SNAPSHOT: Could not find
> artifact
> org.apache.hadoop:hadoop-kms:jar:classes:3.0.0-alpha2-20161228.102554-925 in
> apache.snapshots.https
> (https://repository.apache.org/content/repositories/snapshots) -> [Help 1]
> {noformat}
> This works on an existing dev setup, likely because I have the jar in my m2
> cache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13961) mvn install fails on trunk
[
https://issues.apache.org/jira/browse/HADOOP-13961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813682#comment-15813682
]
Hadoop QA commented on HADOOP-13961:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
14s{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13961 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12846475/HADOOP-13961.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml |
| uname | Linux 8d141d8bb620 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 945db55 |
| Default Java | 1.8.0_111 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11405/testReport/ |
| modules | C: hadoop-common-project/hadoop-kms U:
hadoop-common-project/hadoop-kms |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11405/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> mvn install fails on trunk
> --
>
> Key: HADOOP-13961
> URL: https://issues.apache.org/jira/browse/HADOOP-13961
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha2
>Reporter: Karthik Kambatla
>Assignee: John Zhuge
>Priority: Blocker
> Attachments: HADOOP-13961.001.patch
>
>
> mvn install fails for me on trunk on a new environment with the following:
> {noformat}
> [ERROR] Failed to execute goal on project hadoop-hdfs: Could not resolve
> dependencies for project
> org.apache.hadoop:hadoop-hdfs:jar:3.0.0-alpha2-SNAPSHOT: Could not find
> artifact
> org.apache.hadoop:hadoop-kms:jar:classes:3.0.0-alpha2-20161228.102554-925 in
> apache.snapshots.https
> (https://repository.apache.org/content/repositories/snapshots) -> [Help 1]
>
[jira] [Commented] (HADOOP-13433) Race in UGI.reloginFromKeytab
[
https://issues.apache.org/jira/browse/HADOOP-13433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813669#comment-15813669
]
Duo Zhang commented on HADOOP-13433:
{quote}
Have you deployed this fix on your clusters?
{quote}
Yes, we have been running several clusters with this patch in for about half a
year. It works fine.
{quote}
Should the test case be integrated into the patch
{quote}
Sure. Let me prepare a new patch.
> Race in UGI.reloginFromKeytab
> -
>
> Key: HADOOP-13433
> URL: https://issues.apache.org/jira/browse/HADOOP-13433
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Attachments: HADOOP-13433-v1.patch, HADOOP-13433-v2.patch,
> HADOOP-13433.patch, HBASE-13433-testcase-v3.patch
>
>
> This is a problem that has troubled us for several years. For our HBase
> cluster, sometimes the RS will be stuck due to
> {noformat}
> 2016-06-20,03:44:12,936 INFO org.apache.hadoop.ipc.SecureClient: Exception
> encountered while connecting to the server :
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: The ticket
> isn't for us (35) - BAD TGS SERVER NAME)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194)
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:140)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupSaslConnection(SecureClient.java:187)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.access$700(SecureClient.java:95)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:325)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:322)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1781)
> at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
> at org.apache.hadoop.hbase.security.User.call(User.java:607)
> at org.apache.hadoop.hbase.security.User.access$700(User.java:51)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:461)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstreams(SecureClient.java:321)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1164)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1004)
> at
> org.apache.hadoop.hbase.ipc.SecureRpcEngine$Invoker.invoke(SecureRpcEngine.java:107)
> at $Proxy24.replicateLogEntries(Unknown Source)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:962)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.runLoop(ReplicationSource.java:466)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:515)
> Caused by: GSSException: No valid credentials provided (Mechanism level: The
> ticket isn't for us (35) - BAD TGS SERVER NAME)
> at
> sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:663)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:180)
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175)
> ... 23 more
> Caused by: KrbException: The ticket isn't for us (35) - BAD TGS SERVER NAME
> at sun.security.krb5.KrbTgsRep.(KrbTgsRep.java:64)
> at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:185)
> at
> sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:294)
> at
> sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:106)
> at
> sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:557)
> at
> sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:594)
> ... 26 more
> Caused by: KrbException: Identifier doesn't match expected value (906)
> at sun.security.krb5.internal.KDCRep.init(KDCRep.java:133)
> at sun.security.krb5.internal.TGSRep.init(TGSRep.java:58)
> at
[jira] [Commented] (HADOOP-13961) mvn install fails on trunk
[
https://issues.apache.org/jira/browse/HADOOP-13961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813663#comment-15813663
]
John Zhuge commented on HADOOP-13961:
-
That is what I thought initially. Maybe hadoop-hdfs pom should be updated. Let
me give it a try.
> mvn install fails on trunk
> --
>
> Key: HADOOP-13961
> URL: https://issues.apache.org/jira/browse/HADOOP-13961
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha2
>Reporter: Karthik Kambatla
>Assignee: John Zhuge
>Priority: Blocker
> Attachments: HADOOP-13961.001.patch
>
>
> mvn install fails for me on trunk on a new environment with the following:
> {noformat}
> [ERROR] Failed to execute goal on project hadoop-hdfs: Could not resolve
> dependencies for project
> org.apache.hadoop:hadoop-hdfs:jar:3.0.0-alpha2-SNAPSHOT: Could not find
> artifact
> org.apache.hadoop:hadoop-kms:jar:classes:3.0.0-alpha2-20161228.102554-925 in
> apache.snapshots.https
> (https://repository.apache.org/content/repositories/snapshots) -> [Help 1]
> {noformat}
> This works on an existing dev setup, likely because I have the jar in my m2
> cache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13885) Implement getLinkTarget for ViewFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-13885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813606#comment-15813606 ] Hudson commented on HADOOP-13885: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11094 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11094/]) HADOOP-13885. Implement getLinkTarget for ViewFileSystem. Contributed by (wang: rev 511d39e0740f36bf937e7bcf974e1050f0e7c1e0) * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ChRootedFileSystem.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java > Implement getLinkTarget for ViewFileSystem > -- > > Key: HADOOP-13885 > URL: https://issues.apache.org/jira/browse/HADOOP-13885 > Project: Hadoop Common > Issue Type: Task > Components: viewfs >Affects Versions: 3.0.0-alpha1 >Reporter: Manoj Govindassamy >Assignee: Manoj Govindassamy > Fix For: 3.0.0-alpha2 > > Attachments: HADOOP-13885.01.patch, HADOOP-13885.02.patch > > > ViewFileSystem doesn't override FileSystem#getLinkTarget(). So, when view > filesystem is used to resolve the symbolic links, the default FileSystem > implementation throws UnsupportedOperationException. > The proposal is to define getLinkTarget() for ViewFileSystem and invoke the > target FileSystem for resolving the symbolic links. Path thus returned is > preferred to be a viewfs qualified path, so that it can be used again on the > ViewFileSystem handle. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13961) mvn install fails on trunk
[
https://issues.apache.org/jira/browse/HADOOP-13961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813597#comment-15813597
]
Sangjin Lee commented on HADOOP-13961:
--
Thanks for the patch [~jzhuge]!
Quick question: now that we've switched from war to jar for kms, do we need a
separate classes.jar any more? Can hadoop-hdfs simply depend on the main kms
jar? Is the main jar redundant with the classes.jar or are they different so
that hadoop-hdfs should depend on classes.jar still?
> mvn install fails on trunk
> --
>
> Key: HADOOP-13961
> URL: https://issues.apache.org/jira/browse/HADOOP-13961
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha2
>Reporter: Karthik Kambatla
>Assignee: John Zhuge
>Priority: Blocker
> Attachments: HADOOP-13961.001.patch
>
>
> mvn install fails for me on trunk on a new environment with the following:
> {noformat}
> [ERROR] Failed to execute goal on project hadoop-hdfs: Could not resolve
> dependencies for project
> org.apache.hadoop:hadoop-hdfs:jar:3.0.0-alpha2-SNAPSHOT: Could not find
> artifact
> org.apache.hadoop:hadoop-kms:jar:classes:3.0.0-alpha2-20161228.102554-925 in
> apache.snapshots.https
> (https://repository.apache.org/content/repositories/snapshots) -> [Help 1]
> {noformat}
> This works on an existing dev setup, likely because I have the jar in my m2
> cache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13961) mvn install fails on trunk
[
https://issues.apache.org/jira/browse/HADOOP-13961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HADOOP-13961:
Status: Patch Available (was: In Progress)
> mvn install fails on trunk
> --
>
> Key: HADOOP-13961
> URL: https://issues.apache.org/jira/browse/HADOOP-13961
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha2
>Reporter: Karthik Kambatla
>Assignee: John Zhuge
>Priority: Blocker
> Attachments: HADOOP-13961.001.patch
>
>
> mvn install fails for me on trunk on a new environment with the following:
> {noformat}
> [ERROR] Failed to execute goal on project hadoop-hdfs: Could not resolve
> dependencies for project
> org.apache.hadoop:hadoop-hdfs:jar:3.0.0-alpha2-SNAPSHOT: Could not find
> artifact
> org.apache.hadoop:hadoop-kms:jar:classes:3.0.0-alpha2-20161228.102554-925 in
> apache.snapshots.https
> (https://repository.apache.org/content/repositories/snapshots) -> [Help 1]
> {noformat}
> This works on an existing dev setup, likely because I have the jar in my m2
> cache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13961) mvn install fails on trunk
[
https://issues.apache.org/jira/browse/HADOOP-13961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HADOOP-13961:
Attachment: HADOOP-13961.001.patch
Patch 001
* Restore the {{maven-jar-plugin}} sections removed accidentally by
HADOOP-13597.
Testing #1 existing source tree
# rm -fr ~/.m2/repository/
# ( cd hadoop-maven-plugins/ && mvn install )
# mvn clean
# mvn install -DskipTests -Dmaven.javadoc.skip -Pnative
Testing #2 new source tree
# rm -fr ~/.m2/repository/
# mvn install -DskipTests -Dmaven.javadoc.skip -Pnative
Should have run both test cases for any major pom change.
> mvn install fails on trunk
> --
>
> Key: HADOOP-13961
> URL: https://issues.apache.org/jira/browse/HADOOP-13961
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha2
>Reporter: Karthik Kambatla
>Assignee: John Zhuge
>Priority: Blocker
> Attachments: HADOOP-13961.001.patch
>
>
> mvn install fails for me on trunk on a new environment with the following:
> {noformat}
> [ERROR] Failed to execute goal on project hadoop-hdfs: Could not resolve
> dependencies for project
> org.apache.hadoop:hadoop-hdfs:jar:3.0.0-alpha2-SNAPSHOT: Could not find
> artifact
> org.apache.hadoop:hadoop-kms:jar:classes:3.0.0-alpha2-20161228.102554-925 in
> apache.snapshots.https
> (https://repository.apache.org/content/repositories/snapshots) -> [Help 1]
> {noformat}
> This works on an existing dev setup, likely because I have the jar in my m2
> cache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13885) Implement getLinkTarget for ViewFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-13885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HADOOP-13885: - Resolution: Fixed Fix Version/s: 3.0.0-alpha2 Status: Resolved (was: Patch Available) LGTM +1, thanks for the contribution Manoj! I've committed this to trunk. > Implement getLinkTarget for ViewFileSystem > -- > > Key: HADOOP-13885 > URL: https://issues.apache.org/jira/browse/HADOOP-13885 > Project: Hadoop Common > Issue Type: Task > Components: viewfs >Affects Versions: 3.0.0-alpha1 >Reporter: Manoj Govindassamy >Assignee: Manoj Govindassamy > Fix For: 3.0.0-alpha2 > > Attachments: HADOOP-13885.01.patch, HADOOP-13885.02.patch > > > ViewFileSystem doesn't override FileSystem#getLinkTarget(). So, when view > filesystem is used to resolve the symbolic links, the default FileSystem > implementation throws UnsupportedOperationException. > The proposal is to define getLinkTarget() for ViewFileSystem and invoke the > target FileSystem for resolving the symbolic links. Path thus returned is > preferred to be a viewfs qualified path, so that it can be used again on the > ViewFileSystem handle. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-13961) mvn install fails on trunk
[
https://issues.apache.org/jira/browse/HADOOP-13961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HADOOP-13961 started by John Zhuge.
---
> mvn install fails on trunk
> --
>
> Key: HADOOP-13961
> URL: https://issues.apache.org/jira/browse/HADOOP-13961
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha2
>Reporter: Karthik Kambatla
>Assignee: John Zhuge
>Priority: Blocker
>
> mvn install fails for me on trunk on a new environment with the following:
> {noformat}
> [ERROR] Failed to execute goal on project hadoop-hdfs: Could not resolve
> dependencies for project
> org.apache.hadoop:hadoop-hdfs:jar:3.0.0-alpha2-SNAPSHOT: Could not find
> artifact
> org.apache.hadoop:hadoop-kms:jar:classes:3.0.0-alpha2-20161228.102554-925 in
> apache.snapshots.https
> (https://repository.apache.org/content/repositories/snapshots) -> [Help 1]
> {noformat}
> This works on an existing dev setup, likely because I have the jar in my m2
> cache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12946) Ensure one JVMPauseMonitor thread per JVM
[ https://issues.apache.org/jira/browse/HADOOP-12946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-12946: Resolution: Won't Fix Status: Resolved (was: Patch Available) Opt for HADOOP-12908. > Ensure one JVMPauseMonitor thread per JVM > - > > Key: HADOOP-12946 > URL: https://issues.apache.org/jira/browse/HADOOP-12946 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.8.0 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Minor > Attachments: HADOOP-12946-001.patch, HADOOP-12946-002.patch > > > Ensure at most one JVMPauseMonitor thread is running per JVM when there are > multiple JVMPauseMonitor instances, e.g., in mini clusters. This will prevent > redundant GC pause log messages while still maintaining one monitor thread > running. > This is a different way to fix HADOOP-12855. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12855) Add option to disable JVMPauseMonitor across services
[ https://issues.apache.org/jira/browse/HADOOP-12855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-12855: Resolution: Won't Fix Status: Resolved (was: Patch Available) Opt for HADOOP-12908. > Add option to disable JVMPauseMonitor across services > - > > Key: HADOOP-12855 > URL: https://issues.apache.org/jira/browse/HADOOP-12855 > Project: Hadoop Common > Issue Type: Improvement > Components: performance, test >Affects Versions: 2.8.0 > Environment: JVMs with miniHDFS and miniYarn clusters >Reporter: Steve Loughran >Assignee: John Zhuge > Attachments: HADOOP-12855-001.patch, HADOOP-12855-002.patch, > HADOOP-12855-003.patch, HADOOP-12855-004.patch, HADOOP-12855-005.patch, > HADOOP-12855-option-001.patch, HADOOP-12855-option-002.patch > > > Now that the YARN and HDFS services automatically start a JVM pause monitor, > if you start up the mini HDFS and YARN clusters, with history server, you are > spinning off 5 + threads, all looking for JVM pauses, all printing things out > when it happens. > We do not need these monitors in minicluster testing; they merely add load > and noise to tests. > Rather than retrofit new options everywhere, how about having a > "jvm.pause.monitor.enabled" flag (default true), which, when set, starts off > the monitor thread. > That way, the existing code is unchanged, there is always a JVM pause monitor > for the various services —it just isn't spinning up threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13589) S3Guard: Allow execution of all S3A integration tests with S3Guard enabled.
[
https://issues.apache.org/jira/browse/HADOOP-13589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813374#comment-15813374
]
Mingliang Liu commented on HADOOP-13589:
I like this patch.
{code:title=s3guard.md}
282 run with S3Guard by using the `s3guard` profile. When set, this will run
283 all the tests with a local dynamo DB instance set to "authoritative"
mode.
{code}
So currently we are running the S3Guard integration tests against the Amazon
DynamoDB service. Local DynamoDB mode is currently used only on unit test
TestDynamoDBMetadataStore, though we can add more unit tests using this.
+1 other than that.
> S3Guard: Allow execution of all S3A integration tests with S3Guard enabled.
> ---
>
> Key: HADOOP-13589
> URL: https://issues.apache.org/jira/browse/HADOOP-13589
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Chris Nauroth
>Assignee: Steve Loughran
> Attachments: HADOOP-13589-HADOOP-13345-001.patch
>
>
> With S3Guard enabled, S3A must continue to be functionally correct. The best
> way to verify this is to execute our existing S3A integration tests in a mode
> with S3A enabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13908) S3Guard: Existing tables may not be initialized correctly in DynamoDBMetadataStore
[
https://issues.apache.org/jira/browse/HADOOP-13908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813310#comment-15813310
]
Mingliang Liu commented on HADOOP-13908:
Thanks [~cnauroth], [~ste...@apache.org] and [~rajesh.balamohan] for discussion
and review. Thanks Chris for commit!
p.s. [HADOOP-13959] is to replace the {{describe()}} with more lightweight
operation for probing the table status. Let's track effort there.
> S3Guard: Existing tables may not be initialized correctly in
> DynamoDBMetadataStore
> --
>
> Key: HADOOP-13908
> URL: https://issues.apache.org/jira/browse/HADOOP-13908
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: HADOOP-13345
>
> Attachments: HADOOP-13908-HADOOP-13345.000.patch,
> HADOOP-13908-HADOOP-13345.001.patch, HADOOP-13908-HADOOP-13345.002.patch,
> HADOOP-13908-HADOOP-13345.002.patch, HADOOP-13908-HADOOP-13345.003.patch,
> HADOOP-13908-HADOOP-13345.004.patch, HADOOP-13908-HADOOP-13345.005.patch,
> HADOOP-13908-HADOOP-13345.006.patch
>
>
> This was based on discussion in [HADOOP-13455]. Though we should not create
> table unless the config {{fs.s3a.s3guard.ddb.table.create}} is set true, we
> still have to get the existing table in
> {{DynamoDBMetadataStore#initialize()}} and wait for its becoming active,
> before any table/item operations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13589) S3Guard: Allow execution of all S3A integration tests with S3Guard enabled.
[ https://issues.apache.org/jira/browse/HADOOP-13589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813303#comment-15813303 ] Aaron Fabbri commented on HADOOP-13589: --- This looks awesome [~ste...@apache.org]. One minor nit: I'd expect fs.s3a.metadatastore.authoritative to be false by default. That should be the "safer" configuration we encourage for initial production use. Does that make sense? As long as the documentation is clear to this effect, I'm +1 on this. > S3Guard: Allow execution of all S3A integration tests with S3Guard enabled. > --- > > Key: HADOOP-13589 > URL: https://issues.apache.org/jira/browse/HADOOP-13589 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Chris Nauroth >Assignee: Steve Loughran > Attachments: HADOOP-13589-HADOOP-13345-001.patch > > > With S3Guard enabled, S3A must continue to be functionally correct. The best > way to verify this is to execute our existing S3A integration tests in a mode > with S3A enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13908) S3Guard: Existing tables may not be initialized correctly in DynamoDBMetadataStore
[
https://issues.apache.org/jira/browse/HADOOP-13908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-13908:
---
Resolution: Fixed
Fix Version/s: HADOOP-13345
Status: Resolved (was: Patch Available)
I committed this to the HADOOP-13345 feature branch. [~liuml07], thank you for
the patch.
> S3Guard: Existing tables may not be initialized correctly in
> DynamoDBMetadataStore
> --
>
> Key: HADOOP-13908
> URL: https://issues.apache.org/jira/browse/HADOOP-13908
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: HADOOP-13345
>
> Attachments: HADOOP-13908-HADOOP-13345.000.patch,
> HADOOP-13908-HADOOP-13345.001.patch, HADOOP-13908-HADOOP-13345.002.patch,
> HADOOP-13908-HADOOP-13345.002.patch, HADOOP-13908-HADOOP-13345.003.patch,
> HADOOP-13908-HADOOP-13345.004.patch, HADOOP-13908-HADOOP-13345.005.patch,
> HADOOP-13908-HADOOP-13345.006.patch
>
>
> This was based on discussion in [HADOOP-13455]. Though we should not create
> table unless the config {{fs.s3a.s3guard.ddb.table.create}} is set true, we
> still have to get the existing table in
> {{DynamoDBMetadataStore#initialize()}} and wait for its becoming active,
> before any table/item operations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13650) S3Guard: Provide command line tools to manipulate metadata store.
[
https://issues.apache.org/jira/browse/HADOOP-13650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813296#comment-15813296
]
Lei (Eddy) Xu commented on HADOOP-13650:
Hi, [~cnauroth].
I am working on rebase it and cooperating [~liuml07]'s HADOOP-13960 into the
patch. I will post the new patch by EOD.
> S3Guard: Provide command line tools to manipulate metadata store.
> -
>
> Key: HADOOP-13650
> URL: https://issues.apache.org/jira/browse/HADOOP-13650
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HADOOP-13650-HADOOP-13345.000.patch,
> HADOOP-13650-HADOOP-13345.001.patch, HADOOP-13650-HADOOP-13345.002.patch,
> HADOOP-13650-HADOOP-13345.003.patch, HADOOP-13650-HADOOP-13345.004.patch
>
>
> Similar systems like EMRFS has the CLI tools to manipulate the metadata
> store, i.e., create or delete metadata store, or {{import}}, {{sync}} the
> file metadata between metadata store and S3.
> http://docs.aws.amazon.com//ElasticMapReduce/latest/ReleaseGuide/emrfs-cli-reference.html
> S3Guard should offer similar functionality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13914) s3guard: improve S3AFileStatus#isEmptyDirectory handling
[ https://issues.apache.org/jira/browse/HADOOP-13914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813283#comment-15813283 ] Aaron Fabbri commented on HADOOP-13914: --- Ok, thanks for clarifying. Enjoy your break! > s3guard: improve S3AFileStatus#isEmptyDirectory handling > > > Key: HADOOP-13914 > URL: https://issues.apache.org/jira/browse/HADOOP-13914 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: HADOOP-13345 >Reporter: Aaron Fabbri >Assignee: Mingliang Liu > Attachments: HADOOP-13914-HADOOP-13345.000.patch, > s3guard-empty-dirs.md, test-only-HADOOP-13914.patch > > > As discussed in HADOOP-13449, proper support for the isEmptyDirectory() flag > stored in S3AFileStatus is missing from DynamoDBMetadataStore. > The approach taken by LocalMetadataStore is not suitable for the DynamoDB > implementation, and also sacrifices good code separation to minimize > S3AFileSystem changes pre-merge to trunk. > I will attach a design doc that attempts to clearly explain the problem and > preferred solution. I suggest we do this work after merging the HADOOP-13345 > branch to trunk, but am open to suggestions. > I can also attach a patch of a integration test that exercises the missing > case and demonstrates a failure with DynamoDBMetadataStore. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13952) tools dependency hooks are throwing errors
[
https://issues.apache.org/jira/browse/HADOOP-13952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HADOOP-13952:
-
Priority: Critical (was: Blocker)
I'm downgrading this from a Blocker since there's currently no assignee, and
the two errors that I looked at were false positives. If there are functional
issues, then we can re-upgrade the priority.
> tools dependency hooks are throwing errors
> --
>
> Key: HADOOP-13952
> URL: https://issues.apache.org/jira/browse/HADOOP-13952
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha2
>Reporter: Allen Wittenauer
>Priority: Critical
>
> During build, we are throwing these errors:
> {code}
> ERROR: hadoop-aliyun has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-aliyun has missing dependencies: json-lib-jdk15.jar
> ERROR: hadoop-archive-logs has missing dependencies:
> jasper-compiler-5.5.23.jar
> ERROR: hadoop-archives has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-aws has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-azure has missing dependencies:
> jetty-util-ajax-9.3.11.v20160721.jar
> ERROR: hadoop-azure-datalake has missing dependencies: okhttp-2.4.0.jar
> ERROR: hadoop-azure-datalake has missing dependencies: okio-1.4.0.jar
> ERROR: hadoop-extras has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-gridmix has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-kafka has missing dependencies: lz4-1.2.0.jar
> ERROR: hadoop-kafka has missing dependencies: kafka-clients-0.8.2.1.jar
> ERROR: hadoop-openstack has missing dependencies: commons-httpclient-3.1.jar
> ERROR: hadoop-rumen has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-sls has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-sls has missing dependencies: metrics-core-3.0.1.jar
> ERROR: hadoop-streaming has missing dependencies: jasper-compiler-5.5.23.jar
> {code}
> Likely a variety of reasons for the failures. Kafka is HADOOP-12556, but
> others need to be investigated. Probably just need to look at more than just
> common/lib in dist-tools-hooks-maker now that shading has gone in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13914) s3guard: improve S3AFileStatus#isEmptyDirectory handling
[ https://issues.apache.org/jira/browse/HADOOP-13914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813276#comment-15813276 ] Mingliang Liu commented on HADOOP-13914: Thanks. I saw the doc, it's very helpful. Yeah, tristate seems more meaningful. The v0 patch was from offline discussion with Steve last week. I'll rebase later. p.s. [~fabbri] and [~eddyxu] I'll be on PTO for ~1mo. I'll not be responsive during that period, but I'll catch up emails/commits silently. > s3guard: improve S3AFileStatus#isEmptyDirectory handling > > > Key: HADOOP-13914 > URL: https://issues.apache.org/jira/browse/HADOOP-13914 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: HADOOP-13345 >Reporter: Aaron Fabbri >Assignee: Mingliang Liu > Attachments: HADOOP-13914-HADOOP-13345.000.patch, > s3guard-empty-dirs.md, test-only-HADOOP-13914.patch > > > As discussed in HADOOP-13449, proper support for the isEmptyDirectory() flag > stored in S3AFileStatus is missing from DynamoDBMetadataStore. > The approach taken by LocalMetadataStore is not suitable for the DynamoDB > implementation, and also sacrifices good code separation to minimize > S3AFileSystem changes pre-merge to trunk. > I will attach a design doc that attempts to clearly explain the problem and > preferred solution. I suggest we do this work after merging the HADOOP-13345 > branch to trunk, but am open to suggestions. > I can also attach a patch of a integration test that exercises the missing > case and demonstrates a failure with DynamoDBMetadataStore. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13953) Make FTPFileSystem's data connection mode and transfer mode configurable
[ https://issues.apache.org/jira/browse/HADOOP-13953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813267#comment-15813267 ] Hudson commented on HADOOP-13953: - FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #11092 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11092/]) HADOOP-13953. Make FTPFileSystem's data connection mode and transfer (weichiu: rev 0a212a40fcbd12a11294bff7a31e7433111733c9) * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/ftp/TestFTPFileSystem.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java * (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ftp/FTPFileSystem.java > Make FTPFileSystem's data connection mode and transfer mode configurable > > > Key: HADOOP-13953 > URL: https://issues.apache.org/jira/browse/HADOOP-13953 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 0.22.0 >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HADOOP-13953.01.patch, HADOOP-13953.02.patch, > HADOOP-13953.03.patch, HADOOP-13953.04.patch > > > The FTP transfer mode used by FTPFileSystem is BLOCK_TRANSFER_MODE. FTP Data > connection mode used by FTPFileSystem is ACTIVE_LOCAL_DATA_CONNECTION_MODE. > This jira makes them configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13914) s3guard: improve S3AFileStatus#isEmptyDirectory handling
[
https://issues.apache.org/jira/browse/HADOOP-13914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813249#comment-15813249
]
Aaron Fabbri commented on HADOOP-13914:
---
Thanks for uploading an RFC patch [~liuml07].
Did you have a chance to read [the
doc|https://gist.github.com/ajfabbri/db3b2ad5b1f111277681ba4b7f788404] I
attached here? If I'm reading this patch right, the test case I attached to
this JIRA would not pass.
In particular, my doc asserts that MetadataStore#isEmptyDirectory would need to
return a tristate, not a boolean, as it can answer {yes, no, not enough
information}.
> s3guard: improve S3AFileStatus#isEmptyDirectory handling
>
>
> Key: HADOOP-13914
> URL: https://issues.apache.org/jira/browse/HADOOP-13914
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Aaron Fabbri
>Assignee: Mingliang Liu
> Attachments: HADOOP-13914-HADOOP-13345.000.patch,
> s3guard-empty-dirs.md, test-only-HADOOP-13914.patch
>
>
> As discussed in HADOOP-13449, proper support for the isEmptyDirectory() flag
> stored in S3AFileStatus is missing from DynamoDBMetadataStore.
> The approach taken by LocalMetadataStore is not suitable for the DynamoDB
> implementation, and also sacrifices good code separation to minimize
> S3AFileSystem changes pre-merge to trunk.
> I will attach a design doc that attempts to clearly explain the problem and
> preferred solution. I suggest we do this work after merging the HADOOP-13345
> branch to trunk, but am open to suggestions.
> I can also attach a patch of a integration test that exercises the missing
> case and demonstrates a failure with DynamoDBMetadataStore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-10584) ActiveStandbyElector goes down if ZK quorum become unavailable
[ https://issues.apache.org/jira/browse/HADOOP-10584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-10584: -- Assignee: (was: Karthik Kambatla) > ActiveStandbyElector goes down if ZK quorum become unavailable > -- > > Key: HADOOP-10584 > URL: https://issues.apache.org/jira/browse/HADOOP-10584 > Project: Hadoop Common > Issue Type: Bug > Components: ha >Affects Versions: 2.4.0 >Reporter: Karthik Kambatla >Priority: Critical > Attachments: hadoop-10584-prelim.patch, rm.log > > > ActiveStandbyElector retries operations for a few times. If the ZK quorum > itself is down, it goes down and the daemons will have to be brought up > again. > Instead, it should log the fact that it is unable to talk to ZK, call > becomeStandby on its client, and continue to attempt connecting to ZK. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13336) S3A to support per-bucket configuration
[
https://issues.apache.org/jira/browse/HADOOP-13336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813225#comment-15813225
]
Larry McCay commented on HADOOP-13336:
--
[~ste...@apache.org] - Ahhh - I missed the constant names difference.
Eyes are getting tired (old). I thought that I could see some difference but I
couldn't find it.
The patchCredentialProvider approach looks good.
I am still curious whether the following will end up logging something
sensitive - given that this is all about providing separate credentials per
bucket:
{noformat}
+LOG.debug("Setting {} to value of {} : {} (was {})",
+generic, key, value, source.get(generic));
{noformat}
Am I missing what the "value" of these properties actually are?
> S3A to support per-bucket configuration
> ---
>
> Key: HADOOP-13336
> URL: https://issues.apache.org/jira/browse/HADOOP-13336
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13336-006.patch, HADOOP-13336-007.patch,
> HADOOP-13336-HADOOP-13345-001.patch, HADOOP-13336-HADOOP-13345-002.patch,
> HADOOP-13336-HADOOP-13345-003.patch, HADOOP-13336-HADOOP-13345-004.patch,
> HADOOP-13336-HADOOP-13345-005.patch, HADOOP-13336-HADOOP-13345-006.patch,
> HADOOP-13336-HADOOP-13345-008.patch
>
>
> S3a now supports different regions, by way of declaring the endpoint —but you
> can't do things like read in one region, write back in another (e.g. a distcp
> backup), because only one region can be specified in a configuration.
> If s3a supported region declaration in the URL, e.g. s3a://b1.frankfurt
> s3a://b2.seol , then this would be possible.
> Swift does this with a full filesystem binding/config: endpoints, username,
> etc, in the XML file. Would we need to do that much? It'd be simpler
> initially to use a domain suffix of a URL to set the region of a bucket from
> the domain and have the aws library sort the details out itself, maybe with
> some config options for working with non-AWS infra
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13914) s3guard: improve S3AFileStatus#isEmptyDirectory handling
[
https://issues.apache.org/jira/browse/HADOOP-13914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813216#comment-15813216
]
Aaron Fabbri commented on HADOOP-13914:
---
Thanks [~ste...@apache.org]. I'm not sure I'm parsing your comment right so
let me try to paraphrase, and you can correct me:
{quote}
We only want that dir as an optimisation of followon work in s3aFS, so that if
you get a delete(path) you can do a getFileStatus, and, if status=directory,
see if it is empty (so skip the need for recursive=true) without another round
trip.
{quote}
Do you mean that the only purpose of the isEmptyDirectory() predicate in the
future will be for saving a round trip on directory deletes?
{quote}
with s3guard you don't need that caching of state. It can be be done on demand,
only in those few cases where we actually need to know about it...which pushes
for it being something that the metadatastore can work out on demand. We would
need to document that the status field is only valid without an MD store
{quote}
Makes sense. And this "status field only valid w/o MD store" could be
encapsulated in a helper function.
Does the basic solution I proposed still make sense for the case where
determining that flag is still needed?
> s3guard: improve S3AFileStatus#isEmptyDirectory handling
>
>
> Key: HADOOP-13914
> URL: https://issues.apache.org/jira/browse/HADOOP-13914
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Aaron Fabbri
>Assignee: Mingliang Liu
> Attachments: HADOOP-13914-HADOOP-13345.000.patch,
> s3guard-empty-dirs.md, test-only-HADOOP-13914.patch
>
>
> As discussed in HADOOP-13449, proper support for the isEmptyDirectory() flag
> stored in S3AFileStatus is missing from DynamoDBMetadataStore.
> The approach taken by LocalMetadataStore is not suitable for the DynamoDB
> implementation, and also sacrifices good code separation to minimize
> S3AFileSystem changes pre-merge to trunk.
> I will attach a design doc that attempts to clearly explain the problem and
> preferred solution. I suggest we do this work after merging the HADOOP-13345
> branch to trunk, but am open to suggestions.
> I can also attach a patch of a integration test that exercises the missing
> case and demonstrates a failure with DynamoDBMetadataStore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13876) S3Guard: better support for multi-bucket access including read-only
[
https://issues.apache.org/jira/browse/HADOOP-13876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813187#comment-15813187
]
Hadoop QA commented on HADOOP-13876:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
39s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
14s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
35s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13876 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12846434/HADOOP-13876-HADOOP-13345.000.patch
|
| Optional Tests | asflicense mvnsite compile javac javadoc mvninstall
unit findbugs checkstyle |
| uname | Linux 87d8309311ca 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HADOOP-13345 / e3f2002 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11404/testReport/ |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11404/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> S3Guard: better support for multi-bucket access including read-only
> ---
>
> Key: HADOOP-13876
> URL: https://issues.apache.org/jira/browse/HADOOP-13876
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Aaron Fabbri
> Attachments: HADOOP-13876-HADOOP-13345.000.patch
>
>
> HADOOP-13449 adds support for DynamoDBMetadataStore.
> The code currently supports two options
[jira] [Comment Edited] (HADOOP-13114) DistCp should have option to compress data on write
[
https://issues.apache.org/jira/browse/HADOOP-13114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813083#comment-15813083
]
Ravi Prakash edited comment on HADOOP-13114 at 1/9/17 11:20 PM:
Here's a rebase of the patch from Suraj and Yongjun. To try it out, you could
use this command:
{code}
hadoop distcp
-Ddistcp.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec
--compressoutput /input /output
{code}
was (Author: raviprak):
Here's rebase of the patch from Suraj and Yongjun. To try it out, you could use
this command:
{code}
hadoop distcp
-Ddistcp.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec
--compressoutput /input /output
{code}
> DistCp should have option to compress data on write
> ---
>
> Key: HADOOP-13114
> URL: https://issues.apache.org/jira/browse/HADOOP-13114
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
>Affects Versions: 2.8.0, 2.7.3, 3.0.0-alpha1
>Reporter: Suraj Nayak
>Assignee: Suraj Nayak
>Priority: Minor
> Labels: distcp
> Attachments: HADOOP-13114-trunk_2016-05-07-1.patch,
> HADOOP-13114-trunk_2016-05-08-1.patch, HADOOP-13114-trunk_2016-05-10-1.patch,
> HADOOP-13114-trunk_2016-05-12-1.patch, HADOOP-13114.05.patch,
> HADOOP-13114.06.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> DistCp utility should have capability to store data in user specified
> compression format. This avoids one hop of compressing data after transfer.
> Backup strategies to different cluster also get benefit of saving one IO
> operation to and from HDFS, thus saving resources, time and effort.
> * Create an option -compressOutput defaulting to
> {{org.apache.hadoop.io.compress.BZip2Codec}}.
> * Users will be able to change codec with {{-D
> mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.GzipCodec}}
> * If distcp compression is enabled, suffix the filenames with default codec
> extension to indicate the file is compressed. Thus users can be aware of what
> codec was used to compress the data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13114) DistCp should have option to compress data on write
[
https://issues.apache.org/jira/browse/HADOOP-13114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813177#comment-15813177
]
Ravi Prakash commented on HADOOP-13114:
---
Thanks for your comment Koji!
I guess it'd be useful for any files which are compressible, right? And also
the target HDFS can have less free space. Are you thinking there may be
downsides?
> DistCp should have option to compress data on write
> ---
>
> Key: HADOOP-13114
> URL: https://issues.apache.org/jira/browse/HADOOP-13114
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
>Affects Versions: 2.8.0, 2.7.3, 3.0.0-alpha1
>Reporter: Suraj Nayak
>Assignee: Suraj Nayak
>Priority: Minor
> Labels: distcp
> Attachments: HADOOP-13114-trunk_2016-05-07-1.patch,
> HADOOP-13114-trunk_2016-05-08-1.patch, HADOOP-13114-trunk_2016-05-10-1.patch,
> HADOOP-13114-trunk_2016-05-12-1.patch, HADOOP-13114.05.patch,
> HADOOP-13114.06.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> DistCp utility should have capability to store data in user specified
> compression format. This avoids one hop of compressing data after transfer.
> Backup strategies to different cluster also get benefit of saving one IO
> operation to and from HDFS, thus saving resources, time and effort.
> * Create an option -compressOutput defaulting to
> {{org.apache.hadoop.io.compress.BZip2Codec}}.
> * Users will be able to change codec with {{-D
> mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.GzipCodec}}
> * If distcp compression is enabled, suffix the filenames with default codec
> extension to indicate the file is compressed. Thus users can be aware of what
> codec was used to compress the data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13114) DistCp should have option to compress data on write
[
https://issues.apache.org/jira/browse/HADOOP-13114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813173#comment-15813173
]
Hadoop QA commented on HADOOP-13114:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 13s{color} | {color:orange} hadoop-tools/hadoop-distcp: The patch generated
2 new + 178 unchanged - 0 fixed = 180 total (was 178) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 11m 41s{color}
| {color:red} hadoop-distcp in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 9s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.tools.TestDistCpCompression |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13114 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12846433/HADOOP-13114.06.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux b2f024a91ddf 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 91bf504 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11403/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-distcp.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11403/artifact/patchprocess/whitespace-eol.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11403/artifact/patchprocess/whitespace-tabs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11403/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-distcp.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11403/testReport/ |
| modules |
[jira] [Updated] (HADOOP-13914) s3guard: improve S3AFileStatus#isEmptyDirectory handling
[ https://issues.apache.org/jira/browse/HADOOP-13914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu updated HADOOP-13914: --- Attachment: HADOOP-13914-HADOOP-13345.000.patch I upload the v0 patch for discussion. > s3guard: improve S3AFileStatus#isEmptyDirectory handling > > > Key: HADOOP-13914 > URL: https://issues.apache.org/jira/browse/HADOOP-13914 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: HADOOP-13345 >Reporter: Aaron Fabbri >Assignee: Mingliang Liu > Attachments: HADOOP-13914-HADOOP-13345.000.patch, > s3guard-empty-dirs.md, test-only-HADOOP-13914.patch > > > As discussed in HADOOP-13449, proper support for the isEmptyDirectory() flag > stored in S3AFileStatus is missing from DynamoDBMetadataStore. > The approach taken by LocalMetadataStore is not suitable for the DynamoDB > implementation, and also sacrifices good code separation to minimize > S3AFileSystem changes pre-merge to trunk. > I will attach a design doc that attempts to clearly explain the problem and > preferred solution. I suggest we do this work after merging the HADOOP-13345 > branch to trunk, but am open to suggestions. > I can also attach a patch of a integration test that exercises the missing > case and demonstrates a failure with DynamoDBMetadataStore. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13952) tools dependency hooks are throwing errors
[
https://issues.apache.org/jira/browse/HADOOP-13952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813130#comment-15813130
]
Allen Wittenauer commented on HADOOP-13952:
---
bq. jasper-compiler is a test-scoped dependency, but it gets picked up by grep
since it has the word "compile" in it. The dep plugin can print a specific
scope with the "includeScope" option, which seems better.
I had a lot of problems getting includeScope to work correctly for our build,
so gave up on it. We can change the grep to be grep -E compile$ and it should
work as well.
bq. The build-classpath goal prints things out as jar names, which is what we
really want.
It does, but without getting scoping to work, this doesn't work well.
bq. should this be looking for "runtime" scoped dependencies rather than
"compile"?
I seem to recall that there was something that broke with just the runtime
listed. This is likely a pom issue, but I was doing all of them at once sooo...
bq. all the modules except hadoop-pipes have this dependency plugin invocation.
Not quite. They call the file name extensions different things in order to
build different types of shell profiles.
> tools dependency hooks are throwing errors
> --
>
> Key: HADOOP-13952
> URL: https://issues.apache.org/jira/browse/HADOOP-13952
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha2
>Reporter: Allen Wittenauer
>Priority: Blocker
>
> During build, we are throwing these errors:
> {code}
> ERROR: hadoop-aliyun has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-aliyun has missing dependencies: json-lib-jdk15.jar
> ERROR: hadoop-archive-logs has missing dependencies:
> jasper-compiler-5.5.23.jar
> ERROR: hadoop-archives has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-aws has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-azure has missing dependencies:
> jetty-util-ajax-9.3.11.v20160721.jar
> ERROR: hadoop-azure-datalake has missing dependencies: okhttp-2.4.0.jar
> ERROR: hadoop-azure-datalake has missing dependencies: okio-1.4.0.jar
> ERROR: hadoop-extras has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-gridmix has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-kafka has missing dependencies: lz4-1.2.0.jar
> ERROR: hadoop-kafka has missing dependencies: kafka-clients-0.8.2.1.jar
> ERROR: hadoop-openstack has missing dependencies: commons-httpclient-3.1.jar
> ERROR: hadoop-rumen has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-sls has missing dependencies: jasper-compiler-5.5.23.jar
> ERROR: hadoop-sls has missing dependencies: metrics-core-3.0.1.jar
> ERROR: hadoop-streaming has missing dependencies: jasper-compiler-5.5.23.jar
> {code}
> Likely a variety of reasons for the failures. Kafka is HADOOP-12556, but
> others need to be investigated. Probably just need to look at more than just
> common/lib in dist-tools-hooks-maker now that shading has gone in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13876) S3Guard: better support for multi-bucket access including read-only
[
https://issues.apache.org/jira/browse/HADOOP-13876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-13876:
---
Status: Patch Available (was: Open)
> S3Guard: better support for multi-bucket access including read-only
> ---
>
> Key: HADOOP-13876
> URL: https://issues.apache.org/jira/browse/HADOOP-13876
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Aaron Fabbri
> Attachments: HADOOP-13876-HADOOP-13345.000.patch
>
>
> HADOOP-13449 adds support for DynamoDBMetadataStore.
> The code currently supports two options for choosing DynamoDB table names:
> 1. Use name of each s3 bucket and auto-create a DynamoDB table for each.
> 2. Configure a table name in the {{fs.s3a.s3guard.ddb.table}} parameter.
> One of the issues is with accessing read-only buckets. If a user accesses a
> read-only bucket with credentials that do not have DynamoDB write
> permissions, they will get errors when trying to access the read-only bucket.
> This manifests causes test failures for {{ITestS3AAWSCredentialsProvider}}.
> Goals for this JIRA:
> - Fix {{ITestS3AAWSCredentialsProvider}} in a way that makes sense for the
> real use-case.
> - Allow for a "one DynamoDB table per cluster" configuration with a way to
> chose which credentials are used for DynamoDB.
> - Document limitations etc. in the s3guard.md site doc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13876) S3Guard: better support for multi-bucket access including read-only
[
https://issues.apache.org/jira/browse/HADOOP-13876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-13876:
---
Attachment: HADOOP-13876-HADOOP-13345.000.patch
For the goals of this JIRA, the v0 patch is to address the failing tests: it
skips the test by not supporting the credentials in the binding URI as it's
very unsafe and thus deprecated; it also skips the test by not guarding the
read-only S3 buckets. And it documents in s3guard.md the limitations of not
supporting inline credentials in URI.
Tested in US-West-1 region:
{code}
$ mvn -Dit.test='ITestS3A*' -Dscale -q clean verify
Results :
Tests in error:
ITestS3AFileSystemContract>FileSystemContractBaseTest.testRenameToDirWithSamePrefixAllowed:669->FileSystemContractBaseTest.rename:525
» AWSServiceIO
Tests run: 321, Failures: 0, Errors: 1, Skipped: 8
{code}
> S3Guard: better support for multi-bucket access including read-only
> ---
>
> Key: HADOOP-13876
> URL: https://issues.apache.org/jira/browse/HADOOP-13876
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Aaron Fabbri
> Attachments: HADOOP-13876-HADOOP-13345.000.patch
>
>
> HADOOP-13449 adds support for DynamoDBMetadataStore.
> The code currently supports two options for choosing DynamoDB table names:
> 1. Use name of each s3 bucket and auto-create a DynamoDB table for each.
> 2. Configure a table name in the {{fs.s3a.s3guard.ddb.table}} parameter.
> One of the issues is with accessing read-only buckets. If a user accesses a
> read-only bucket with credentials that do not have DynamoDB write
> permissions, they will get errors when trying to access the read-only bucket.
> This manifests causes test failures for {{ITestS3AAWSCredentialsProvider}}.
> Goals for this JIRA:
> - Fix {{ITestS3AAWSCredentialsProvider}} in a way that makes sense for the
> real use-case.
> - Allow for a "one DynamoDB table per cluster" configuration with a way to
> chose which credentials are used for DynamoDB.
> - Document limitations etc. in the s3guard.md site doc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13589) S3Guard: Allow execution of all S3A integration tests with S3Guard enabled.
[
https://issues.apache.org/jira/browse/HADOOP-13589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813087#comment-15813087
]
Chris Nauroth commented on HADOOP-13589:
This looks great. Thanks for picking it up. I tested against us-west-2 with
and without the new S3Guard options enabled.
Steve, do you want to skip the {{StringBuilder}} in
{{NullMetadataStore#toString}}, since it always returns the same string
literal? With that and a pre-commit run, I think this patch will be ready to
go.
> S3Guard: Allow execution of all S3A integration tests with S3Guard enabled.
> ---
>
> Key: HADOOP-13589
> URL: https://issues.apache.org/jira/browse/HADOOP-13589
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Chris Nauroth
>Assignee: Steve Loughran
> Attachments: HADOOP-13589-HADOOP-13345-001.patch
>
>
> With S3Guard enabled, S3A must continue to be functionally correct. The best
> way to verify this is to execute our existing S3A integration tests in a mode
> with S3A enabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13114) DistCp should have option to compress data on write
[
https://issues.apache.org/jira/browse/HADOOP-13114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Prakash updated HADOOP-13114:
--
Attachment: HADOOP-13114.06.patch
Here's rebase of the patch from Suraj and Yongjun. To try it out, you could use
this command:
{code}
hadoop distcp
-Ddistcp.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec
--compressoutput /input /output
{code}
> DistCp should have option to compress data on write
> ---
>
> Key: HADOOP-13114
> URL: https://issues.apache.org/jira/browse/HADOOP-13114
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
>Affects Versions: 2.8.0, 2.7.3, 3.0.0-alpha1
>Reporter: Suraj Nayak
>Assignee: Suraj Nayak
>Priority: Minor
> Labels: distcp
> Attachments: HADOOP-13114-trunk_2016-05-07-1.patch,
> HADOOP-13114-trunk_2016-05-08-1.patch, HADOOP-13114-trunk_2016-05-10-1.patch,
> HADOOP-13114-trunk_2016-05-12-1.patch, HADOOP-13114.05.patch,
> HADOOP-13114.06.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> DistCp utility should have capability to store data in user specified
> compression format. This avoids one hop of compressing data after transfer.
> Backup strategies to different cluster also get benefit of saving one IO
> operation to and from HDFS, thus saving resources, time and effort.
> * Create an option -compressOutput defaulting to
> {{org.apache.hadoop.io.compress.BZip2Codec}}.
> * Users will be able to change codec with {{-D
> mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.GzipCodec}}
> * If distcp compression is enabled, suffix the filenames with default codec
> extension to indicate the file is compressed. Thus users can be aware of what
> codec was used to compress the data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12664) UGI auto-renewer does not verify kinit availability during initialization
[
https://issues.apache.org/jira/browse/HADOOP-12664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ray Chiang updated HADOOP-12664:
Status: Open (was: Patch Available)
> UGI auto-renewer does not verify kinit availability during initialization
> -
>
> Key: HADOOP-12664
> URL: https://issues.apache.org/jira/browse/HADOOP-12664
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Arpit Agarwal
>Assignee: Ray Chiang
>Priority: Minor
> Labels: supportability
> Attachments: HADOOP-12664.001.patch, HADOOP-12664.002.patch,
> HADOOP-12664.003.patch
>
>
> UGI auto-renewer does not verify that {{hadoop.kerberos.kinit.command}} is in
> the path during initialization. If not available, the auto-renewal thread
> will hit an error during TGT renewal. We recently saw a case where it
> manifests as transient errors during client program execution which can be
> hard to track down without UGI logging.
> It seems like {{kinit}} availability should be verified during initialization
> to make the behavior more predictable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13964) Remove vestigal templates directories creation
[
https://issues.apache.org/jira/browse/HADOOP-13964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-13964:
--
Hadoop Flags: Incompatible change
Release Note: This patch removes
share/hadoop/{hadoop,hdfs,mapred,yarn}/templates directories and contents.
> Remove vestigal templates directories creation
> --
>
> Key: HADOOP-13964
> URL: https://issues.apache.org/jira/browse/HADOOP-13964
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha1
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
> Attachments: HADOOP-13964.00.patch
>
>
> the share/hadoop/(component)/template directories are from when RPM and such
> were built as part of the build system. that no longer happens and now those
> files cause more harm than good since they are in the classpath. let's
> remove them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13964) Remove vestigal templates directories creation
[ https://issues.apache.org/jira/browse/HADOOP-13964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-13964: -- Attachment: HADOOP-13964.00.patch -00: * initial patch > Remove vestigal templates directories creation > -- > > Key: HADOOP-13964 > URL: https://issues.apache.org/jira/browse/HADOOP-13964 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: HADOOP-13964.00.patch > > > the share/hadoop/(component)/template directories are from when RPM and such > were built as part of the build system. that no longer happens and now those > files cause more harm than good since they are in the classpath. let's > remove them. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13964) Remove vestigal templates directories creation
[ https://issues.apache.org/jira/browse/HADOOP-13964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-13964: -- Summary: Remove vestigal templates directories creation (was: Remove vestigal template directory creation) > Remove vestigal templates directories creation > -- > > Key: HADOOP-13964 > URL: https://issues.apache.org/jira/browse/HADOOP-13964 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: HADOOP-13964.00.patch > > > the share/hadoop/(component)/template directories are from when RPM and such > were built as part of the build system. that no longer happens and now those > files cause more harm than good since they are in the classpath. let's > remove them. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-13964) Remove vestigal templates directories creation
[ https://issues.apache.org/jira/browse/HADOOP-13964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-13964 started by Allen Wittenauer. - > Remove vestigal templates directories creation > -- > > Key: HADOOP-13964 > URL: https://issues.apache.org/jira/browse/HADOOP-13964 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: HADOOP-13964.00.patch > > > the share/hadoop/(component)/template directories are from when RPM and such > were built as part of the build system. that no longer happens and now those > files cause more harm than good since they are in the classpath. let's > remove them. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13908) S3Guard: Existing tables may not be initialized correctly in DynamoDBMetadataStore
[
https://issues.apache.org/jira/browse/HADOOP-13908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813010#comment-15813010
]
Hadoop QA commented on HADOOP-13908:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
14s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
35s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13908 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12846423/HADOOP-13908-HADOOP-13345.006.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 6f9a076c37f0 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HADOOP-13345 / e3f2002 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11402/testReport/ |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11402/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> S3Guard: Existing tables may not be initialized correctly in
> DynamoDBMetadataStore
> --
>
> Key: HADOOP-13908
> URL: https://issues.apache.org/jira/browse/HADOOP-13908
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-13908-HADOOP-13345.000.patch,
>
[jira] [Commented] (HADOOP-13336) S3A to support per-bucket configuration
[
https://issues.apache.org/jira/browse/HADOOP-13336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812979#comment-15812979
]
Hadoop QA commented on HADOOP-13336:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 10 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
53s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
20s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
31s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
34s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
35s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
40s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
3s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
15s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 35s{color} | {color:orange} root: The patch generated 7 new + 34 unchanged -
1 fixed = 41 total (was 35) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
5s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
48s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 79m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13336 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12846406/HADOOP-13336-HADOOP-13345-008.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml findbugs checkstyle |
| uname | Linux 5e1c74172b14 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24
10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HADOOP-13345 / e3f2002 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11401/artifact/patchprocess/diff-checkstyle-root.txt
|
| whitespace |
[jira] [Commented] (HADOOP-13908) S3Guard: Existing tables may not be initialized correctly in DynamoDBMetadataStore
[
https://issues.apache.org/jira/browse/HADOOP-13908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812974#comment-15812974
]
Mingliang Liu commented on HADOOP-13908:
This is a good test report as the failures are to be addressed elsewhere.
> S3Guard: Existing tables may not be initialized correctly in
> DynamoDBMetadataStore
> --
>
> Key: HADOOP-13908
> URL: https://issues.apache.org/jira/browse/HADOOP-13908
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-13908-HADOOP-13345.000.patch,
> HADOOP-13908-HADOOP-13345.001.patch, HADOOP-13908-HADOOP-13345.002.patch,
> HADOOP-13908-HADOOP-13345.002.patch, HADOOP-13908-HADOOP-13345.003.patch,
> HADOOP-13908-HADOOP-13345.004.patch, HADOOP-13908-HADOOP-13345.005.patch,
> HADOOP-13908-HADOOP-13345.006.patch
>
>
> This was based on discussion in [HADOOP-13455]. Though we should not create
> table unless the config {{fs.s3a.s3guard.ddb.table.create}} is set true, we
> still have to get the existing table in
> {{DynamoDBMetadataStore#initialize()}} and wait for its becoming active,
> before any table/item operations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13908) S3Guard: Existing tables may not be initialized correctly in DynamoDBMetadataStore
[
https://issues.apache.org/jira/browse/HADOOP-13908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HADOOP-13908:
---
Attachment: HADOOP-13908-HADOOP-13345.006.patch
The v6 patch rebases from feature branch and resolves minor conflicts.
Tested against US-West-1 region (both table existent and non-existent before
integration test):
{code}
$ mvn -Dit.test='ITestS3A*' -Dscale -q clean verify
Results :
Tests in error:
ITestS3AAWSCredentialsProvider.testAnonymousProvider:133 » AWSServiceIO
initTa...
ITestS3ACredentialsInURL.testInstantiateFromURL:86 » InterruptedIO initTable:
...
ITestS3ACredentialsInURL.testInvalidCredentialsFail:127 » AccessDenied
s3a://m...
ITestS3AFileSystemContract>FileSystemContractBaseTest.testRenameToDirWithSamePrefixAllowed:669->FileSystemContractBaseTest.rename:525
» AWSServiceIO
Tests run: 321, Failures: 0, Errors: 4, Skipped: 8
{code}
> S3Guard: Existing tables may not be initialized correctly in
> DynamoDBMetadataStore
> --
>
> Key: HADOOP-13908
> URL: https://issues.apache.org/jira/browse/HADOOP-13908
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-13908-HADOOP-13345.000.patch,
> HADOOP-13908-HADOOP-13345.001.patch, HADOOP-13908-HADOOP-13345.002.patch,
> HADOOP-13908-HADOOP-13345.002.patch, HADOOP-13908-HADOOP-13345.003.patch,
> HADOOP-13908-HADOOP-13345.004.patch, HADOOP-13908-HADOOP-13345.005.patch,
> HADOOP-13908-HADOOP-13345.006.patch
>
>
> This was based on discussion in [HADOOP-13455]. Though we should not create
> table unless the config {{fs.s3a.s3guard.ddb.table.create}} is set true, we
> still have to get the existing table in
> {{DynamoDBMetadataStore#initialize()}} and wait for its becoming active,
> before any table/item operations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13964) Remove vestigal template directory creation
Allen Wittenauer created HADOOP-13964: - Summary: Remove vestigal template directory creation Key: HADOOP-13964 URL: https://issues.apache.org/jira/browse/HADOOP-13964 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 3.0.0-alpha1 Reporter: Allen Wittenauer Assignee: Allen Wittenauer the share/hadoop/(component)/template directories are from when RPM and such were built as part of the build system. that no longer happens and now those files cause more harm than good since they are in the classpath. let's remove them. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13336) S3A to support per-bucket configuration
[ https://issues.apache.org/jira/browse/HADOOP-13336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13336: Attachment: HADOOP-13336-HADOOP-13345-008.patch HADOOP-13336 patch 008: permit bucket customisation of hadoop credential providers, by adding an fs.s3a. property for declaring an extra provider path. This means the same property names inside the files are used, you just give a different file for each endpoint. > S3A to support per-bucket configuration > --- > > Key: HADOOP-13336 > URL: https://issues.apache.org/jira/browse/HADOOP-13336 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13336-006.patch, HADOOP-13336-007.patch, > HADOOP-13336-HADOOP-13345-001.patch, HADOOP-13336-HADOOP-13345-002.patch, > HADOOP-13336-HADOOP-13345-003.patch, HADOOP-13336-HADOOP-13345-004.patch, > HADOOP-13336-HADOOP-13345-005.patch, HADOOP-13336-HADOOP-13345-006.patch, > HADOOP-13336-HADOOP-13345-008.patch > > > S3a now supports different regions, by way of declaring the endpoint —but you > can't do things like read in one region, write back in another (e.g. a distcp > backup), because only one region can be specified in a configuration. > If s3a supported region declaration in the URL, e.g. s3a://b1.frankfurt > s3a://b2.seol , then this would be possible. > Swift does this with a full filesystem binding/config: endpoints, username, > etc, in the XML file. Would we need to do that much? It'd be simpler > initially to use a domain suffix of a URL to set the region of a bucket from > the domain and have the aws library sort the details out itself, maybe with > some config options for working with non-AWS infra -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13336) S3A to support per-bucket configuration
[ https://issues.apache.org/jira/browse/HADOOP-13336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13336: Status: Patch Available (was: Open) tested, s3a ireland > S3A to support per-bucket configuration > --- > > Key: HADOOP-13336 > URL: https://issues.apache.org/jira/browse/HADOOP-13336 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13336-006.patch, HADOOP-13336-007.patch, > HADOOP-13336-HADOOP-13345-001.patch, HADOOP-13336-HADOOP-13345-002.patch, > HADOOP-13336-HADOOP-13345-003.patch, HADOOP-13336-HADOOP-13345-004.patch, > HADOOP-13336-HADOOP-13345-005.patch, HADOOP-13336-HADOOP-13345-006.patch, > HADOOP-13336-HADOOP-13345-008.patch > > > S3a now supports different regions, by way of declaring the endpoint —but you > can't do things like read in one region, write back in another (e.g. a distcp > backup), because only one region can be specified in a configuration. > If s3a supported region declaration in the URL, e.g. s3a://b1.frankfurt > s3a://b2.seol , then this would be possible. > Swift does this with a full filesystem binding/config: endpoints, username, > etc, in the XML file. Would we need to do that much? It'd be simpler > initially to use a domain suffix of a URL to set the region of a bucket from > the domain and have the aws library sort the details out itself, maybe with > some config options for working with non-AWS infra -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13963) /bin/bash is hard coded in some of the scripts
[ https://issues.apache.org/jira/browse/HADOOP-13963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812696#comment-15812696 ] Miklos Szegedi commented on HADOOP-13963: - This bug was opened based on the discussion with [~aw] in YARN-6060. > /bin/bash is hard coded in some of the scripts > -- > > Key: HADOOP-13963 > URL: https://issues.apache.org/jira/browse/HADOOP-13963 > Project: Hadoop Common > Issue Type: Bug >Reporter: Miklos Szegedi > > /bin/bash is hard coded in some of the scripts. We should consider using > #!/usr/bin/env bash at the beginning instead. > Candidates: > hadoop-functions_test_helper.bash > start-build-env.sh > findHangingTest.sh > verify-xml.sh > hadoop_env_checks.sh -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13963) /bin/bash is hard coded in some of the scripts
Miklos Szegedi created HADOOP-13963: --- Summary: /bin/bash is hard coded in some of the scripts Key: HADOOP-13963 URL: https://issues.apache.org/jira/browse/HADOOP-13963 Project: Hadoop Common Issue Type: Bug Reporter: Miklos Szegedi /bin/bash is hard coded in some of the scripts. We should consider using #!/usr/bin/env bash at the beginning instead. Candidates: hadoop-functions_test_helper.bash start-build-env.sh findHangingTest.sh verify-xml.sh hadoop_env_checks.sh -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13959) S3guard: replace dynamo.describe() call in init with more efficient query
[
https://issues.apache.org/jira/browse/HADOOP-13959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812667#comment-15812667
]
Mingliang Liu edited comment on HADOOP-13959 at 1/9/17 7:51 PM:
+1 for the proposal.
Thanks [~ste...@apache.org] for filing this! This is definitely something we
have to do for scaling the S3Guard much better. According to our effort in
[HADOOP-13908] v4 patch, we may have to test this patch carefully especially
when waiting a newly created table for active. [AWS
Forum|https://forums.aws.amazon.com/message.jspa?messageID=759966] has an open
discussion thread.
was (Author: liuml07):
+1 for the proposal.
Thanks [~ste...@apache.org] for filing this! This is definitely something we
have to do for scaling the S3Guard much better. According to our effort in
[HADOOP-13908], we may have to test this patch carefully especially when
waiting a newly created table for active. [AWS
Forum|https://forums.aws.amazon.com/message.jspa?messageID=759966] has an open
discussion thread.
> S3guard: replace dynamo.describe() call in init with more efficient query
> -
>
> Key: HADOOP-13959
> URL: https://issues.apache.org/jira/browse/HADOOP-13959
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Steve Loughran
>Priority: Minor
>
> HADOOP-13908 adds initialization when a table isn't created, using the
> {{describe()}} call.
> AWS document this as inefficient, and throttle it. We should be able to get
> away with a simple table lookup as the probe
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13959) S3guard: replace dynamo.describe() call in init with more efficient query
[
https://issues.apache.org/jira/browse/HADOOP-13959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812667#comment-15812667
]
Mingliang Liu commented on HADOOP-13959:
+1 for the proposal.
Thanks [~ste...@apache.org] for filing this! This is definitely something we
have to do for scaling the S3Guard much better. According to our effort in
[HADOOP-13908], we may have to test this patch carefully especially when
waiting a newly created table for active. [AWS
Forum|https://forums.aws.amazon.com/message.jspa?messageID=759966] has an open
discussion thread.
> S3guard: replace dynamo.describe() call in init with more efficient query
> -
>
> Key: HADOOP-13959
> URL: https://issues.apache.org/jira/browse/HADOOP-13959
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Steve Loughran
>Priority: Minor
>
> HADOOP-13908 adds initialization when a table isn't created, using the
> {{describe()}} call.
> AWS document this as inefficient, and throttle it. We should be able to get
> away with a simple table lookup as the probe
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12664) UGI auto-renewer does not verify kinit availability during initialization
[
https://issues.apache.org/jira/browse/HADOOP-12664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812650#comment-15812650
]
Ray Chiang commented on HADOOP-12664:
-
I'm pretty agnostic with respect to whatever solution we end up choosing. At
the very least, I see the following errors:
1) kinit not found (not installed, configuration problem)
2) kinit temporarily not accessible (e.g. flaky filesystem)
3) kinit fails to renew intermittently or has slow response/timeouts (e.g.
network)
Currently, there are two possible checks:
A) Do the kinit path search
B) Simply check kinit exit code 127 for "command not found"
G) Do check outside the kinit retry thread
H) Do check within the kinit retry thread
Given potentially 1) and 2) getting conflated, if we choose option B) and H),
we might want to set some kind of consecutive "command not found" retry error
threshold and throwing an exception if we exit based on that situation.
Thoughts?
> UGI auto-renewer does not verify kinit availability during initialization
> -
>
> Key: HADOOP-12664
> URL: https://issues.apache.org/jira/browse/HADOOP-12664
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Arpit Agarwal
>Assignee: Ray Chiang
>Priority: Minor
> Labels: supportability
> Attachments: HADOOP-12664.001.patch, HADOOP-12664.002.patch,
> HADOOP-12664.003.patch
>
>
> UGI auto-renewer does not verify that {{hadoop.kerberos.kinit.command}} is in
> the path during initialization. If not available, the auto-renewal thread
> will hit an error during TGT renewal. We recently saw a case where it
> manifests as transient errors during client program execution which can be
> hard to track down without UGI logging.
> It seems like {{kinit}} availability should be verified during initialization
> to make the behavior more predictable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13960) Initialize DynamoDBMetadataStore without associated S3AFileSystem
[
https://issues.apache.org/jira/browse/HADOOP-13960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812647#comment-15812647
]
Mingliang Liu commented on HADOOP-13960:
Thanks [~eddyxu] for discussion, review and commit!
Aside from the unit test against DynamoDBLocal simulator, I should also have
posted the integration test results as following (region US-West-1):
{code}
$ mvn -Dit.test='ITestS3A*' -Dscale -q clean verify
Results :
Tests in error:
ITestS3AAWSCredentialsProvider.testAnonymousProvider:133 » IO Failed to
instan...
ITestS3ACredentialsInURL.testInstantiateFromURL:86 » IO Failed to instantiate
...
ITestS3ACredentialsInURL.testInvalidCredentialsFail:127 » AccessDenied
s3a://m...
ITestS3AFileSystemContract>FileSystemContractBaseTest.testRenameToDirWithSamePrefixAllowed:669->FileSystemContractBaseTest.rename:525
» AWSServiceIO
Tests run: 321, Failures: 0, Errors: 4, Skipped: 8
{code}
According to our recent discussion, this is a good run (failures are tracked
elsewhere).
> Initialize DynamoDBMetadataStore without associated S3AFileSystem
> -
>
> Key: HADOOP-13960
> URL: https://issues.apache.org/jira/browse/HADOOP-13960
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: HADOOP-13345
>
> Attachments: HADOOP-13960-HADOOP-13345.000.patch
>
>
> Per the discussion in [HADOOP-13650], it's helpful to initialize a
> DynamoDBMetadataStore object without S3AFileSystem. In the current code, we
> can achieve this via {{DynamoDBMetadataStore#initialize(Configuration)}}.
> However, users still have to provide the associated S3AFileSystem URI in the
> configuration, by means of either setting the {{fs.defaultFS}} in
> configuration file or {{-fs s3://bucket}} command line parameter. Setting the
> default FileSystem in configuration seems not necessary as the command line
> is to manipulate metadata store, e.g. command line tools on an existing HDFS
> cluster.
> This JIRA is to track the effort of initializing a DynamoDBMetadataStore
> without associating any S3 buckets, so that S3AFileSystem is not needed.
> Users have to specify in configuration the DynamoDB endpoints (for region)
> and table name along with credentials, AWS client settings.
> See [~eddyxu] and [~liuml07]'s comments in [HADOOP-13650] for more details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13336) S3A to support per-bucket configuration
[ https://issues.apache.org/jira/browse/HADOOP-13336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13336: Status: Open (was: Patch Available) > S3A to support per-bucket configuration > --- > > Key: HADOOP-13336 > URL: https://issues.apache.org/jira/browse/HADOOP-13336 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13336-006.patch, HADOOP-13336-007.patch, > HADOOP-13336-HADOOP-13345-001.patch, HADOOP-13336-HADOOP-13345-002.patch, > HADOOP-13336-HADOOP-13345-003.patch, HADOOP-13336-HADOOP-13345-004.patch, > HADOOP-13336-HADOOP-13345-005.patch, HADOOP-13336-HADOOP-13345-006.patch > > > S3a now supports different regions, by way of declaring the endpoint —but you > can't do things like read in one region, write back in another (e.g. a distcp > backup), because only one region can be specified in a configuration. > If s3a supported region declaration in the URL, e.g. s3a://b1.frankfurt > s3a://b2.seol , then this would be possible. > Swift does this with a full filesystem binding/config: endpoints, username, > etc, in the XML file. Would we need to do that much? It'd be simpler > initially to use a domain suffix of a URL to set the region of a bucket from > the domain and have the aws library sort the details out itself, maybe with > some config options for working with non-AWS infra -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13929) ADLS should not check in contract-test-options.xml
[
https://issues.apache.org/jira/browse/HADOOP-13929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812531#comment-15812531
]
Vishwajeet Dusane commented on HADOOP-13929:
[~jzhuge] and [~cnauroth] - Thanks a lot for the clean up, i have not got a
chance to go through the patch yet. I am on vacation and would be back on 16th
Jan.
CC: Engaging folks, who can also help [~srevanka], [~shrikant], [~ASikaria] and
[~chris.douglas]
> ADLS should not check in contract-test-options.xml
> --
>
> Key: HADOOP-13929
> URL: https://issues.apache.org/jira/browse/HADOOP-13929
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/adl, test
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: John Zhuge
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13929.001.patch, HADOOP-13929.002.patch,
> HADOOP-13929.003.patch, HADOOP-13929.004.patch, HADOOP-13929.005.patch,
> HADOOP-13929.006.patch, HADOOP-13929.007.patch, HADOOP-13929.008.patch
>
>
> Should not check in the file {{contract-test-options.xml}}. Make sure the
> file is excluded by {{.gitignore}}. Make sure ADLS {{index.md}} provides a
> complete example of this file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13336) S3A to support per-bucket configuration
[
https://issues.apache.org/jira/browse/HADOOP-13336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812523#comment-15812523
]
Steve Loughran commented on HADOOP-13336:
-
Larrrt there's a scan for a forbidden prefix (currently bucket.") and then
actual unmodifiable value. I actually think I could fix them to just the simple
checks, and not-overengineer this.
w.r.t Configuration.getPassword(), I see the problem you are alluding to. Even
though we are migrating fs.s3a.bucket.* to fs.s3a.*, that does nothing to the
credential providers, as they have hard-coded keys in their key:value mappings;
this isn't changing anything.
hmmm.
Would it be possible for us to update the
{{"hadoop.security.credential.provider.path"}} at the same time, that is add a
special property to s3a, say {{fs.s3a.security.credential.provider.path}}. All
the contents of that property would be _prepanded_ to that of the base one.
You'd then need to specify different providers for the different endpoints. By
prepending the values, we can ensure that properties stated in a bucket will
take priority over any in the default provider path.
We'd need to document this, especially how it's likely that once there's a
secret in a JCEKS file, then you must overrride those secrets with new files:
you can't move back to a password from a credentials file: you can't downgrade
security.
Would that work? If so, I can include that in this patch as it's related to the
per-bucket config, isn't it?
> S3A to support per-bucket configuration
> ---
>
> Key: HADOOP-13336
> URL: https://issues.apache.org/jira/browse/HADOOP-13336
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13336-006.patch, HADOOP-13336-007.patch,
> HADOOP-13336-HADOOP-13345-001.patch, HADOOP-13336-HADOOP-13345-002.patch,
> HADOOP-13336-HADOOP-13345-003.patch, HADOOP-13336-HADOOP-13345-004.patch,
> HADOOP-13336-HADOOP-13345-005.patch, HADOOP-13336-HADOOP-13345-006.patch
>
>
> S3a now supports different regions, by way of declaring the endpoint —but you
> can't do things like read in one region, write back in another (e.g. a distcp
> backup), because only one region can be specified in a configuration.
> If s3a supported region declaration in the URL, e.g. s3a://b1.frankfurt
> s3a://b2.seol , then this would be possible.
> Swift does this with a full filesystem binding/config: endpoints, username,
> etc, in the XML file. Would we need to do that much? It'd be simpler
> initially to use a domain suffix of a URL to set the region of a bucket from
> the domain and have the aws library sort the details out itself, maybe with
> some config options for working with non-AWS infra
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13650) S3Guard: Provide command line tools to manipulate metadata store.
[
https://issues.apache.org/jira/browse/HADOOP-13650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812476#comment-15812476
]
Chris Nauroth commented on HADOOP-13650:
[~eddyxu], it looks like this patch needs a rebase. Would you mind doing that
before I take another look? I'd prefer to be able to do a full distro build
and test the sub-command in action while reviewing the code.
> S3Guard: Provide command line tools to manipulate metadata store.
> -
>
> Key: HADOOP-13650
> URL: https://issues.apache.org/jira/browse/HADOOP-13650
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HADOOP-13650-HADOOP-13345.000.patch,
> HADOOP-13650-HADOOP-13345.001.patch, HADOOP-13650-HADOOP-13345.002.patch,
> HADOOP-13650-HADOOP-13345.003.patch, HADOOP-13650-HADOOP-13345.004.patch
>
>
> Similar systems like EMRFS has the CLI tools to manipulate the metadata
> store, i.e., create or delete metadata store, or {{import}}, {{sync}} the
> file metadata between metadata store and S3.
> http://docs.aws.amazon.com//ElasticMapReduce/latest/ReleaseGuide/emrfs-cli-reference.html
> S3Guard should offer similar functionality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13929) ADLS should not check in contract-test-options.xml
[
https://issues.apache.org/jira/browse/HADOOP-13929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812470#comment-15812470
]
Chris Nauroth commented on HADOOP-13929:
[~jzhuge], there are a lot of nice clean-ups here. Thank you! Unfortunately,
I've lost my configurations for live testing against ADL, so I'm going to need
to recreate that. Meanwhile, [~vishwajeet.dusane] and [~ASikaria], if you get
a test run completed with this patch before me, please comment and let me know.
> ADLS should not check in contract-test-options.xml
> --
>
> Key: HADOOP-13929
> URL: https://issues.apache.org/jira/browse/HADOOP-13929
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/adl, test
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: John Zhuge
> Fix For: 3.0.0-alpha2
>
> Attachments: HADOOP-13929.001.patch, HADOOP-13929.002.patch,
> HADOOP-13929.003.patch, HADOOP-13929.004.patch, HADOOP-13929.005.patch,
> HADOOP-13929.006.patch, HADOOP-13929.007.patch, HADOOP-13929.008.patch
>
>
> Should not check in the file {{contract-test-options.xml}}. Make sure the
> file is excluded by {{.gitignore}}. Make sure ADLS {{index.md}} provides a
> complete example of this file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13946) Document how HDFS updates timestamps in the FS spec; compare with object stores
[
https://issues.apache.org/jira/browse/HADOOP-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812450#comment-15812450
]
Chris Nauroth commented on HADOOP-13946:
bq. creation time? That of mkdir()?
I think I'm confused by the mentions of creation time. We have mtime and atime
in {{FileStatus}}. AFAIK, the inode data structures in the NameNode don't
track a separate notion of creation time, just mtime and atime. Is there
something I've missed?
bq. what operations on child entries update the modtime? mkdir, create, delete,
rename. And which don't?
Yes, and to this please also add concat.
bq. chmod and set time calls?
chmod (and setacl) and setTimes do not alter any modification times, neither on
the target path itself nor its parent.
> Document how HDFS updates timestamps in the FS spec; compare with object
> stores
> ---
>
> Key: HADOOP-13946
> URL: https://issues.apache.org/jira/browse/HADOOP-13946
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation, fs
>Affects Versions: 2.7.3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HADOOP-13946-001.patch, HADOOP-13946-002.patch
>
>
> SPARK-17159 shows that the behavior of when HDFS updates timestamps isn't
> well documented. Document these in the FS spec.
> I'm not going to add tests for this, as it is so very dependent on FS
> implementations, as in "POSIX filesystems may behave differently from HDFS".
> If someone knows what happens there, their contribution is welcome.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13908) S3Guard: Existing tables may not be initialized correctly in DynamoDBMetadataStore
[
https://issues.apache.org/jira/browse/HADOOP-13908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812406#comment-15812406
]
Chris Nauroth commented on HADOOP-13908:
[~liuml07], it looks like we'll need to rebase this patch after a commit today.
Once that's available, I'd be happy to do a final test run and commit the
patch.
> S3Guard: Existing tables may not be initialized correctly in
> DynamoDBMetadataStore
> --
>
> Key: HADOOP-13908
> URL: https://issues.apache.org/jira/browse/HADOOP-13908
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-13908-HADOOP-13345.000.patch,
> HADOOP-13908-HADOOP-13345.001.patch, HADOOP-13908-HADOOP-13345.002.patch,
> HADOOP-13908-HADOOP-13345.002.patch, HADOOP-13908-HADOOP-13345.003.patch,
> HADOOP-13908-HADOOP-13345.004.patch, HADOOP-13908-HADOOP-13345.005.patch
>
>
> This was based on discussion in [HADOOP-13455]. Though we should not create
> table unless the config {{fs.s3a.s3guard.ddb.table.create}} is set true, we
> still have to get the existing table in
> {{DynamoDBMetadataStore#initialize()}} and wait for its becoming active,
> before any table/item operations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13336) S3A to support per-bucket configuration
[ https://issues.apache.org/jira/browse/HADOOP-13336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812397#comment-15812397 ] Chris Nauroth commented on HADOOP-13336: I'm in favor of the approach, and there are already plenty of reviewers commenting, so I'll focus my code review energies elsewhere. Thanks, Steve! > S3A to support per-bucket configuration > --- > > Key: HADOOP-13336 > URL: https://issues.apache.org/jira/browse/HADOOP-13336 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13336-006.patch, HADOOP-13336-007.patch, > HADOOP-13336-HADOOP-13345-001.patch, HADOOP-13336-HADOOP-13345-002.patch, > HADOOP-13336-HADOOP-13345-003.patch, HADOOP-13336-HADOOP-13345-004.patch, > HADOOP-13336-HADOOP-13345-005.patch, HADOOP-13336-HADOOP-13345-006.patch > > > S3a now supports different regions, by way of declaring the endpoint —but you > can't do things like read in one region, write back in another (e.g. a distcp > backup), because only one region can be specified in a configuration. > If s3a supported region declaration in the URL, e.g. s3a://b1.frankfurt > s3a://b2.seol , then this would be possible. > Swift does this with a full filesystem binding/config: endpoints, username, > etc, in the XML file. Would we need to do that much? It'd be simpler > initially to use a domain suffix of a URL to set the region of a bucket from > the domain and have the aws library sort the details out itself, maybe with > some config options for working with non-AWS infra -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13336) S3A to support per-bucket configuration
[
https://issues.apache.org/jira/browse/HADOOP-13336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812375#comment-15812375
]
Larry McCay commented on HADOOP-13336:
--
Hi [~ste...@apache.org] -
This looks like a good approach - couple comments:
* In propagateBucketOptions it seems that there may be a redundant loop:
{noformat}
+ for (String unmodifiable : UNMODIFIABLE_PREFIX) {
+if (stripped.startsWith(unmodifiable)) {
+ //tell user off
+ LOG.debug("Ignoring bucket option {}", key);
+ skip = true;
+}
+ }
+ for (String unmodifiable : UNMODIFIABLE_VALUE) {
+if (stripped.equals(unmodifiable)) {
+ //tell user off
+ LOG.debug("Ignoring bucket option {}", key);
+ skip = true;
+}
+ }
{noformat}
* would it also be better to have the log message mention why it is being
ignored?
* would the following code in the same method result in passwords being logged
by any chance:
{noformat}
+ // propagate the value, building a new origin field.
+ // to track overwrites, the generic key is overwritten even if
+ // already matches the new one.
+ if (!skip) {
+final String generic = FS_S3A_PREFIX + stripped;
+LOG.debug("Setting {} to value of {} : {} (was {})",
+generic, key, value, source.get(generic));
+dest.set(generic, value, key);
+ }
{noformat}
* I don't see any evidence that this will work with credential provider API
through the use of Configuration.getPassword. Am I missing some other nuance
here?
> S3A to support per-bucket configuration
> ---
>
> Key: HADOOP-13336
> URL: https://issues.apache.org/jira/browse/HADOOP-13336
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13336-006.patch, HADOOP-13336-007.patch,
> HADOOP-13336-HADOOP-13345-001.patch, HADOOP-13336-HADOOP-13345-002.patch,
> HADOOP-13336-HADOOP-13345-003.patch, HADOOP-13336-HADOOP-13345-004.patch,
> HADOOP-13336-HADOOP-13345-005.patch, HADOOP-13336-HADOOP-13345-006.patch
>
>
> S3a now supports different regions, by way of declaring the endpoint —but you
> can't do things like read in one region, write back in another (e.g. a distcp
> backup), because only one region can be specified in a configuration.
> If s3a supported region declaration in the URL, e.g. s3a://b1.frankfurt
> s3a://b2.seol , then this would be possible.
> Swift does this with a full filesystem binding/config: endpoints, username,
> etc, in the XML file. Would we need to do that much? It'd be simpler
> initially to use a domain suffix of a URL to set the region of a bucket from
> the domain and have the aws library sort the details out itself, maybe with
> some config options for working with non-AWS infra
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13961) mvn install fails on trunk
[
https://issues.apache.org/jira/browse/HADOOP-13961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15810772#comment-15810772
]
John Zhuge edited comment on HADOOP-13961 at 1/9/17 5:44 PM:
-
Reverting HADOOP-13597 does not fix my {{mvn clean}} issue.
was (Author: jzhuge):
Reverting HADOOP-13597 does not fix the issue.
> mvn install fails on trunk
> --
>
> Key: HADOOP-13961
> URL: https://issues.apache.org/jira/browse/HADOOP-13961
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha2
>Reporter: Karthik Kambatla
>Assignee: John Zhuge
>Priority: Blocker
>
> mvn install fails for me on trunk on a new environment with the following:
> {noformat}
> [ERROR] Failed to execute goal on project hadoop-hdfs: Could not resolve
> dependencies for project
> org.apache.hadoop:hadoop-hdfs:jar:3.0.0-alpha2-SNAPSHOT: Could not find
> artifact
> org.apache.hadoop:hadoop-kms:jar:classes:3.0.0-alpha2-20161228.102554-925 in
> apache.snapshots.https
> (https://repository.apache.org/content/repositories/snapshots) -> [Help 1]
> {noformat}
> This works on an existing dev setup, likely because I have the jar in my m2
> cache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13961) mvn install fails on trunk
[
https://issues.apache.org/jira/browse/HADOOP-13961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812362#comment-15812362
]
John Zhuge commented on HADOOP-13961:
-
Thanks [~sjlee0]. I have reproduced the issue with just {{mvn install}} and
will fix it.
> mvn install fails on trunk
> --
>
> Key: HADOOP-13961
> URL: https://issues.apache.org/jira/browse/HADOOP-13961
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha2
>Reporter: Karthik Kambatla
>Assignee: John Zhuge
>Priority: Blocker
>
> mvn install fails for me on trunk on a new environment with the following:
> {noformat}
> [ERROR] Failed to execute goal on project hadoop-hdfs: Could not resolve
> dependencies for project
> org.apache.hadoop:hadoop-hdfs:jar:3.0.0-alpha2-SNAPSHOT: Could not find
> artifact
> org.apache.hadoop:hadoop-kms:jar:classes:3.0.0-alpha2-20161228.102554-925 in
> apache.snapshots.https
> (https://repository.apache.org/content/repositories/snapshots) -> [Help 1]
> {noformat}
> This works on an existing dev setup, likely because I have the jar in my m2
> cache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-13961) mvn install fails on trunk
[
https://issues.apache.org/jira/browse/HADOOP-13961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge reassigned HADOOP-13961:
---
Assignee: John Zhuge
> mvn install fails on trunk
> --
>
> Key: HADOOP-13961
> URL: https://issues.apache.org/jira/browse/HADOOP-13961
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha2
>Reporter: Karthik Kambatla
>Assignee: John Zhuge
>Priority: Blocker
>
> mvn install fails for me on trunk on a new environment with the following:
> {noformat}
> [ERROR] Failed to execute goal on project hadoop-hdfs: Could not resolve
> dependencies for project
> org.apache.hadoop:hadoop-hdfs:jar:3.0.0-alpha2-SNAPSHOT: Could not find
> artifact
> org.apache.hadoop:hadoop-kms:jar:classes:3.0.0-alpha2-20161228.102554-925 in
> apache.snapshots.https
> (https://repository.apache.org/content/repositories/snapshots) -> [Help 1]
> {noformat}
> This works on an existing dev setup, likely because I have the jar in my m2
> cache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13589) S3Guard: Allow execution of all S3A integration tests with S3Guard enabled.
[ https://issues.apache.org/jira/browse/HADOOP-13589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13589: Attachment: HADOOP-13589-HADOOP-13345-001.patch Patch 001 Adds profiles: s3guard, dynamo and non-auth, to enable s3guard, switch from local to dynamo and from authoritative to non-authoritative. The last two only do anything useful unless s3guard is enabled. * if s3guard is set via the test runner properties, s3guard is set up with the local or dynamo db metastore and auth options, with ddb.create=true set to create the db. * testing section in s3guard updated. Test: s3 ireland, with local and dynamo, auth & non-auth, serial and parallel test runs. > S3Guard: Allow execution of all S3A integration tests with S3Guard enabled. > --- > > Key: HADOOP-13589 > URL: https://issues.apache.org/jira/browse/HADOOP-13589 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Chris Nauroth >Assignee: Steve Loughran > Attachments: HADOOP-13589-HADOOP-13345-001.patch > > > With S3Guard enabled, S3A must continue to be functionally correct. The best > way to verify this is to execute our existing S3A integration tests in a mode > with S3A enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13786) Add S3Guard committer for zero-rename commits to consistent S3 endpoints
[ https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812342#comment-15812342 ] Steve Loughran commented on HADOOP-13786: - Making this standalone, giving title but making clear it's for any consistent endpoint. And about to discard all work in patch 1 except the algorithm doc > Add S3Guard committer for zero-rename commits to consistent S3 endpoints > > > Key: HADOOP-13786 > URL: https://issues.apache.org/jira/browse/HADOOP-13786 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: HADOOP-13345 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13786-HADOOP-13345-001.patch > > > A goal of this code is "support O(1) commits to S3 repositories in the > presence of failures". Implement it, including whatever is needed to > demonstrate the correctness of the algorithm. (that is, assuming that s3guard > provides a consistent view of the presence/absence of blobs, show that we can > commit directly). > I consider ourselves free to expose the blobstore-ness of the s3 output > streams (ie. not visible until the close()), if we need to use that to allow > us to abort commit operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13433) Race in UGI.reloginFromKeytab
[
https://issues.apache.org/jira/browse/HADOOP-13433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812310#comment-15812310
]
stack commented on HADOOP-13433:
Should the test case be integrated into the patch [~Apache9]?
Have you deployed this fix on your clusters?
Patch LGTM.
Any opinion mighty [~steve_l]?
> Race in UGI.reloginFromKeytab
> -
>
> Key: HADOOP-13433
> URL: https://issues.apache.org/jira/browse/HADOOP-13433
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Attachments: HADOOP-13433-v1.patch, HADOOP-13433-v2.patch,
> HADOOP-13433.patch, HBASE-13433-testcase-v3.patch
>
>
> This is a problem that has troubled us for several years. For our HBase
> cluster, sometimes the RS will be stuck due to
> {noformat}
> 2016-06-20,03:44:12,936 INFO org.apache.hadoop.ipc.SecureClient: Exception
> encountered while connecting to the server :
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: The ticket
> isn't for us (35) - BAD TGS SERVER NAME)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194)
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:140)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupSaslConnection(SecureClient.java:187)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.access$700(SecureClient.java:95)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:325)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:322)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1781)
> at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
> at org.apache.hadoop.hbase.security.User.call(User.java:607)
> at org.apache.hadoop.hbase.security.User.access$700(User.java:51)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:461)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstreams(SecureClient.java:321)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1164)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1004)
> at
> org.apache.hadoop.hbase.ipc.SecureRpcEngine$Invoker.invoke(SecureRpcEngine.java:107)
> at $Proxy24.replicateLogEntries(Unknown Source)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:962)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.runLoop(ReplicationSource.java:466)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:515)
> Caused by: GSSException: No valid credentials provided (Mechanism level: The
> ticket isn't for us (35) - BAD TGS SERVER NAME)
> at
> sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:663)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:180)
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175)
> ... 23 more
> Caused by: KrbException: The ticket isn't for us (35) - BAD TGS SERVER NAME
> at sun.security.krb5.KrbTgsRep.(KrbTgsRep.java:64)
> at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:185)
> at
> sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:294)
> at
> sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:106)
> at
> sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:557)
> at
> sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:594)
> ... 26 more
> Caused by: KrbException: Identifier doesn't match expected value (906)
> at sun.security.krb5.internal.KDCRep.init(KDCRep.java:133)
> at sun.security.krb5.internal.TGSRep.init(TGSRep.java:58)
> at sun.security.krb5.internal.TGSRep.(TGSRep.java:53)
> at sun.security.krb5.KrbTgsRep.(KrbTgsRep.java:46)
>
[jira] [Commented] (HADOOP-13961) mvn install fails on trunk
[
https://issues.apache.org/jira/browse/HADOOP-13961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812263#comment-15812263
]
Sangjin Lee commented on HADOOP-13961:
--
I don't know about the hadoop-common issue [~jzhuge] is seeing, but I can
reproduce the original problem pretty easily. With a clean trunk and an empty
maven local cache the build fails:
{panel}
[ERROR] Failed to execute goal on project hadoop-hdfs: Could not resolve
dependencies for project
org.apache.hadoop:hadoop-hdfs:jar:3.0.0-alpha2-SNAPSHOT: The following
artifacts could not be resolved:
org.apache.hadoop:hadoop-kms:jar:classes:3.0.0-alpha2-SNAPSHOT,
org.apache.hadoop:hadoop-kms:jar:tests:3.0.0-alpha2-SNAPSHOT: Could not find
artifact org.apache.hadoop:hadoop-kms:jar:classes:3.0.0-alpha2-SNAPSHOT in
apache.snapshots.https
(https://repository.apache.org/content/repositories/snapshots) -> [Help 1]
{panel}
And the problem does go away if I revert HADOOP-13597.
IMO the cause seems pretty clear. With HADOOP-13597, it appears that we stopped
installing hadoop-kms-classes.jr and hadoop-kms-tests.jar
(https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-kms/3.0.0-alpha2-SNAPSHOT/).
But there are projects that depend on those classifiers (hadoop-hdfs being
one).
> mvn install fails on trunk
> --
>
> Key: HADOOP-13961
> URL: https://issues.apache.org/jira/browse/HADOOP-13961
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha2
>Reporter: Karthik Kambatla
>Priority: Blocker
>
> mvn install fails for me on trunk on a new environment with the following:
> {noformat}
> [ERROR] Failed to execute goal on project hadoop-hdfs: Could not resolve
> dependencies for project
> org.apache.hadoop:hadoop-hdfs:jar:3.0.0-alpha2-SNAPSHOT: Could not find
> artifact
> org.apache.hadoop:hadoop-kms:jar:classes:3.0.0-alpha2-20161228.102554-925 in
> apache.snapshots.https
> (https://repository.apache.org/content/repositories/snapshots) -> [Help 1]
> {noformat}
> This works on an existing dev setup, likely because I have the jar in my m2
> cache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13961) mvn install fails on trunk
[
https://issues.apache.org/jira/browse/HADOOP-13961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812239#comment-15812239
]
Allen Wittenauer commented on HADOOP-13961:
---
Note that the Hadoop-trunk-Commit is having issues (on certain nodes?) which
directly impacts whats in the nexus.
> mvn install fails on trunk
> --
>
> Key: HADOOP-13961
> URL: https://issues.apache.org/jira/browse/HADOOP-13961
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha2
>Reporter: Karthik Kambatla
>Priority: Blocker
>
> mvn install fails for me on trunk on a new environment with the following:
> {noformat}
> [ERROR] Failed to execute goal on project hadoop-hdfs: Could not resolve
> dependencies for project
> org.apache.hadoop:hadoop-hdfs:jar:3.0.0-alpha2-SNAPSHOT: Could not find
> artifact
> org.apache.hadoop:hadoop-kms:jar:classes:3.0.0-alpha2-20161228.102554-925 in
> apache.snapshots.https
> (https://repository.apache.org/content/repositories/snapshots) -> [Help 1]
> {noformat}
> This works on an existing dev setup, likely because I have the jar in my m2
> cache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13961) mvn install fails on trunk
[
https://issues.apache.org/jira/browse/HADOOP-13961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812235#comment-15812235
]
Allen Wittenauer commented on HADOOP-13961:
---
That's a normal failure due to the weird way the Hadoop build works. You can't
clean without doing an install first since hadoop-maven-plugins is built on the
fly and not pushed to nexus repo.
> mvn install fails on trunk
> --
>
> Key: HADOOP-13961
> URL: https://issues.apache.org/jira/browse/HADOOP-13961
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha2
>Reporter: Karthik Kambatla
>Priority: Blocker
>
> mvn install fails for me on trunk on a new environment with the following:
> {noformat}
> [ERROR] Failed to execute goal on project hadoop-hdfs: Could not resolve
> dependencies for project
> org.apache.hadoop:hadoop-hdfs:jar:3.0.0-alpha2-SNAPSHOT: Could not find
> artifact
> org.apache.hadoop:hadoop-kms:jar:classes:3.0.0-alpha2-20161228.102554-925 in
> apache.snapshots.https
> (https://repository.apache.org/content/repositories/snapshots) -> [Help 1]
> {noformat}
> This works on an existing dev setup, likely because I have the jar in my m2
> cache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13962) Update ADLS SDK to 2.1.4
[ https://issues.apache.org/jira/browse/HADOOP-13962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812231#comment-15812231 ] Steve Loughran commented on HADOOP-13962: - does this change any dependencies? > Update ADLS SDK to 2.1.4 > > > Key: HADOOP-13962 > URL: https://issues.apache.org/jira/browse/HADOOP-13962 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Affects Versions: 3.0.0-alpha2 >Reporter: John Zhuge >Assignee: John Zhuge > > ADLS has multiple upgrades since the version 2.0.11 we are using: 2.1.1, > 2.1.2, and 2.1.4. Change list: > https://github.com/Azure/azure-data-lake-store-java/blob/master/CHANGES.md. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13962) Update ADLS SDK to 2.1.4
John Zhuge created HADOOP-13962: --- Summary: Update ADLS SDK to 2.1.4 Key: HADOOP-13962 URL: https://issues.apache.org/jira/browse/HADOOP-13962 Project: Hadoop Common Issue Type: Improvement Components: fs/adl Affects Versions: 3.0.0-alpha2 Reporter: John Zhuge Assignee: John Zhuge ADLS has multiple upgrades since the version 2.0.11 we are using: 2.1.1, 2.1.2, and 2.1.4. Change list: https://github.com/Azure/azure-data-lake-store-java/blob/master/CHANGES.md. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13961) mvn install fails on trunk
[
https://issues.apache.org/jira/browse/HADOOP-13961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15810772#comment-15810772
]
John Zhuge edited comment on HADOOP-13961 at 1/9/17 4:03 PM:
-
Reverting HADOOP-13597 does not fix the issue.
was (Author: jzhuge):
The issue is not fixed after reverting HADOOP-13597.
> mvn install fails on trunk
> --
>
> Key: HADOOP-13961
> URL: https://issues.apache.org/jira/browse/HADOOP-13961
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha2
>Reporter: Karthik Kambatla
>Priority: Blocker
>
> mvn install fails for me on trunk on a new environment with the following:
> {noformat}
> [ERROR] Failed to execute goal on project hadoop-hdfs: Could not resolve
> dependencies for project
> org.apache.hadoop:hadoop-hdfs:jar:3.0.0-alpha2-SNAPSHOT: Could not find
> artifact
> org.apache.hadoop:hadoop-kms:jar:classes:3.0.0-alpha2-20161228.102554-925 in
> apache.snapshots.https
> (https://repository.apache.org/content/repositories/snapshots) -> [Help 1]
> {noformat}
> This works on an existing dev setup, likely because I have the jar in my m2
> cache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13903) KMS does not provide any useful debug information
[
https://issues.apache.org/jira/browse/HADOOP-13903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812007#comment-15812007
]
Hadoop QA commented on HADOOP-13903:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
41s{color} | {color:red} hadoop-common-project/hadoop-kms generated 4 new + 0
unchanged - 0 fixed = 4 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 18s{color}
| {color:red} hadoop-kms in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
37s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 75m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-common-project/hadoop-kms |
| | Possible null pointer dereference of op in
org.apache.hadoop.crypto.key.kms.server.KMSAudit.op(KMSAuditLogger$OpStatus,
Object, UserGroupInformation, String, String, String) Dereferenced at
KMSAudit.java:op in
org.apache.hadoop.crypto.key.kms.server.KMSAudit.op(KMSAuditLogger$OpStatus,
Object, UserGroupInformation, String, String, String) Dereferenced at
KMSAudit.java:[line 224] |
| | Non-virtual method call in
org.apache.hadoop.crypto.key.kms.server.KMSAudit.error(UserGroupInformation,
String, String, String) passes null for non-null parameter of
op(KMSAuditLogger$OpStatus, Object, UserGroupInformation, String, String,
String) At KMSAudit.java:String, String, String) passes null for non-null
parameter of op(KMSAuditLogger$OpStatus, Object, UserGroupInformation, String,
String, String) At KMSAudit.java:[line 249] |
| | Non-virtual method call in
[jira] [Work started] (HADOOP-13589) S3Guard: Allow execution of all S3A integration tests with S3Guard enabled.
[ https://issues.apache.org/jira/browse/HADOOP-13589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-13589 started by Steve Loughran. --- > S3Guard: Allow execution of all S3A integration tests with S3Guard enabled. > --- > > Key: HADOOP-13589 > URL: https://issues.apache.org/jira/browse/HADOOP-13589 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Chris Nauroth >Assignee: Steve Loughran > > With S3Guard enabled, S3A must continue to be functionally correct. The best > way to verify this is to execute our existing S3A integration tests in a mode > with S3A enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13805) UGI.getCurrentUser() fails if user does not have a keytab associated
[
https://issues.apache.org/jira/browse/HADOOP-13805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15811981#comment-15811981
]
Alejandro Abdelnur commented on HADOOP-13805:
-
[~xiaochen],
Unless I'm missing something, I don't think patch 5 will do the trick for the
scenario where I've seen this issue to pop up.
{{getCurrentUser()}} creates a new UGI using the
{{UserGroupInformation(Subject)}} constructor, in your proposed change, the
availability of the keytab is determined by inspecting the given subject. If
the given Subject has a keytab but is not owned/created to the UGI (my usecase)
it will create UGI with the isKeytab flag set to true and this will fail in
the same way as without patch 5.
Somehow, we have to make sure that if a UGI is created with an external
Subject, any UGI derived from it it should not say that it has a keytab.
> UGI.getCurrentUser() fails if user does not have a keytab associated
>
>
> Key: HADOOP-13805
> URL: https://issues.apache.org/jira/browse/HADOOP-13805
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.8.0, 2.9.0, 3.0.0-alpha2
>Reporter: Alejandro Abdelnur
>Assignee: Xiao Chen
> Attachments: HADOOP-13805.01.patch, HADOOP-13805.02.patch,
> HADOOP-13805.03.patch, HADOOP-13805.04.patch, HADOOP-13805.05.patch
>
>
> HADOOP-13558 intention was to avoid UGI from trying to renew the TGT when the
> UGI is created from an existing Subject as in that case the keytab is not
> 'own' by UGI but by the creator of the Subject.
> In HADOOP-13558 we introduced a new private UGI constructor
> {{UserGroupInformation(Subject subject, final boolean externalKeyTab)}} and
> we use with TRUE only when doing a {{UGI.loginUserFromSubject()}}.
> The problem is, when we call {{UGI.getCurrentUser()}}, and UGI was created
> via a Subject (via the {{UGI.loginUserFromSubject()}} method), we call {{new
> UserGroupInformation(subject)}} which will delegate to
> {{UserGroupInformation(Subject subject, final boolean externalKeyTab)}} and
> that will use externalKeyTab == *FALSE*.
> Then the UGI returned by {{UGI.getCurrentUser()}} will attempt to login using
> a non-existing keytab if the TGT expired.
> This problem is experienced in {{KMSClientProvider}} when used by the HDFS
> filesystem client accessing an an encryption zone.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13345) S3Guard: Improved Consistency for S3A
[
https://issues.apache.org/jira/browse/HADOOP-13345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15811874#comment-15811874
]
Steve Loughran commented on HADOOP-13345:
-
-1 to anything working with user:pass in URIs. IT's wrong because it
contaminates the logs with secrets. With HADOOP-13336 you can define per-bucket
secrets in core-site.xml, or the command line with {{-D
fs.s3a.bucket.stevel-new.aws.secret.id=mykey}}; they'll get picked up in he FS
instance.
This means that the bucket URI will be needed for setup, but it doesn't have to
involve instantiating the FS, just:
{code}
bucket = new URI(fsName).getHost()
conf = propagateBucketOptions(origConf, bucket)
s3ClientFactoryClass = conf.getClass(
S3_CLIENT_FACTORY_IMPL, DEFAULT_S3_CLIENT_FACTORY_IMPL,
S3ClientFactory.class);
s3Client = ReflectionUtils.newInstance(s3ClientFactoryClass, conf)
.createS3Client(name, uri);
{code}
Thats all. But we will need that bucket name if there's any need to go near
bucket-related options
> S3Guard: Improved Consistency for S3A
> -
>
> Key: HADOOP-13345
> URL: https://issues.apache.org/jira/browse/HADOOP-13345
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HADOOP-13345.prototype1.patch,
> S3C-ConsistentListingonS3-Design.pdf, S3GuardImprovedConsistencyforS3A.pdf,
> S3GuardImprovedConsistencyforS3AV2.pdf, s3c.001.patch
>
>
> This issue proposes S3Guard, a new feature of S3A, to provide an option for a
> stronger consistency model than what is currently offered. The solution
> coordinates with a strongly consistent external store to resolve
> inconsistencies caused by the S3 eventual consistency model.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13786) Add S3Guard committer for zero-rename commits to consistent S3 endpoints
[ https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13786: Summary: Add S3Guard committer for zero-rename commits to consistent S3 endpoints (was: add output committer which uses s3guard for consistent commits to S3) > Add S3Guard committer for zero-rename commits to consistent S3 endpoints > > > Key: HADOOP-13786 > URL: https://issues.apache.org/jira/browse/HADOOP-13786 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: HADOOP-13345 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13786-HADOOP-13345-001.patch > > > A goal of this code is "support O(1) commits to S3 repositories in the > presence of failures". Implement it, including whatever is needed to > demonstrate the correctness of the algorithm. (that is, assuming that s3guard > provides a consistent view of the presence/absence of blobs, show that we can > commit directly). > I consider ourselves free to expose the blobstore-ness of the s3 output > streams (ie. not visible until the close()), if we need to use that to allow > us to abort commit operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13786) add output committer which uses s3guard for consistent commits to S3
[ https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13786: Issue Type: Bug (was: Sub-task) Parent: (was: HADOOP-13345) > add output committer which uses s3guard for consistent commits to S3 > > > Key: HADOOP-13786 > URL: https://issues.apache.org/jira/browse/HADOOP-13786 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: HADOOP-13345 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13786-HADOOP-13345-001.patch > > > A goal of this code is "support O(1) commits to S3 repositories in the > presence of failures". Implement it, including whatever is needed to > demonstrate the correctness of the algorithm. (that is, assuming that s3guard > provides a consistent view of the presence/absence of blobs, show that we can > commit directly). > I consider ourselves free to expose the blobstore-ness of the s3 output > streams (ie. not visible until the close()), if we need to use that to allow > us to abort commit operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13912) S3a Multipart Committer (avoid rename)
[
https://issues.apache.org/jira/browse/HADOOP-13912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15811840#comment-15811840
]
Steve Loughran commented on HADOOP-13912:
-
We need a name for this? S3guard committer? I know it's being designed to work
directly with a consistent S3 instance, but it all comes together. Calling it
the s3guard one will discourage people trying to use it without s3guard turned
on
> S3a Multipart Committer (avoid rename)
> --
>
> Key: HADOOP-13912
> URL: https://issues.apache.org/jira/browse/HADOOP-13912
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
>Reporter: Thomas Demoor
>Assignee: Thomas Demoor
>
> Object stores do not have an efficient rename operation, which is used by the
> Hadoop FileOutputCommitter to atomically promote the "winning" attempt out of
> the multiple (speculative) attempts to the final path. These slow job commits
> are one of the main friction points when using object stores in Hadoop.There
> have been quite some attempts at resolving this: HADOOP-9565, Apache Spark
> DirectOutputCommitters, ... but they have proven not to be robust in face of
> adversity (network partitions, ...).
> The current ticket proposes to do the atomic commit by using the S3 Multipart
> API, which allows multiple concurrent uploads on the same objectname, each in
> its own "temporary space, identified by the UploadId which is returned as a
> response to InitiateMultipartUpload. Every attempt writes directly to the
> final {{outputPath}}. Data is uploaded using Put Part and as a response an
> ETag for the part is returned and stored. The CompleteMultipartUpload is
> postponed. Instead, we persist the UploadId (using a _temporary subdir or
> elsewhere) and the ETags. When a certain "job" wins
> {{CompleteMultipartUpload}} is called for each of its files using the proper
> list of Part ETags.
> Completing a MultipartUpload is a metadata only operation (internally in S3)
> and is thus orders of magnitude faster than the rename-based approach which
> moves all the data.
> Required work:
> * Expose the multipart initiate and complete calls in S3AOutputStream to
> S3AFilesystem
> * Use these multipart calls in a custom committer as described above. I
> propose to build on the S3ACommitter [~ste...@apache.org] is doing for
> HADOOP-13786
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13903) KMS does not provide any useful debug information
[ https://issues.apache.org/jira/browse/HADOOP-13903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tristan Stevens updated HADOOP-13903: - Attachment: HADOOP-13903-6.patch > KMS does not provide any useful debug information > - > > Key: HADOOP-13903 > URL: https://issues.apache.org/jira/browse/HADOOP-13903 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.9.0, 3.0.0-alpha2 >Reporter: Tristan Stevens >Assignee: Tristan Stevens >Priority: Minor > Attachments: HADOOP-13903-2.patch, HADOOP-13903-3.patch, > HADOOP-13903-5-branch2.7.patch, HADOOP-13903-5.patch, HADOOP-13903-6.patch, > HADOOP-13903-6.patch, HADOOP-13903-branch2.7-6.patch, HADOOP-13903.patch > > > At the moment there is no debug or trace level logs generated for KMS > authorisation decisions. In order for users to understand what is going on in > given scenarios this would be invaluable. > Code should endeavour to keep as much work off the sunny-day-code-path as > much as possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13876) S3Guard: better support for multi-bucket access including read-only
[
https://issues.apache.org/jira/browse/HADOOP-13876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15811816#comment-15811816
]
Steve Loughran commented on HADOOP-13876:
-
This will also surface on a {{-Dscale}} test run, as that also hits the public,
read-only landsat dataset.
With HADOOP-13336 you get to control the bucket options individually. For
{{ITestS3AAWSCredentialsProvider}}, it will simply be possible to disable the
metastore for the relevant read-only buckets. All you need to do is set
{{fs.s3a.bucket.landsat-pds.metastore=org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore}}
On that topic, can we support some shorter names for the standard set of
stores, "none", "local", and "dynamo"?
> S3Guard: better support for multi-bucket access including read-only
> ---
>
> Key: HADOOP-13876
> URL: https://issues.apache.org/jira/browse/HADOOP-13876
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Aaron Fabbri
>
> HADOOP-13449 adds support for DynamoDBMetadataStore.
> The code currently supports two options for choosing DynamoDB table names:
> 1. Use name of each s3 bucket and auto-create a DynamoDB table for each.
> 2. Configure a table name in the {{fs.s3a.s3guard.ddb.table}} parameter.
> One of the issues is with accessing read-only buckets. If a user accesses a
> read-only bucket with credentials that do not have DynamoDB write
> permissions, they will get errors when trying to access the read-only bucket.
> This manifests causes test failures for {{ITestS3AAWSCredentialsProvider}}.
> Goals for this JIRA:
> - Fix {{ITestS3AAWSCredentialsProvider}} in a way that makes sense for the
> real use-case.
> - Allow for a "one DynamoDB table per cluster" configuration with a way to
> chose which credentials are used for DynamoDB.
> - Document limitations etc. in the s3guard.md site doc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13336) S3A to support per-bucket configuration
[
https://issues.apache.org/jira/browse/HADOOP-13336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15811795#comment-15811795
]
Hadoop QA commented on HADOOP-13336:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} hadoop-tools/hadoop-aws: The patch generated 0 new +
30 unchanged - 1 fixed = 30 total (was 31) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
20s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13336 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12846320/HADOOP-13336-007.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 10847234dc95 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / db490ec |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11398/testReport/ |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11398/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> S3A to support per-bucket configuration
> ---
>
> Key: HADOOP-13336
> URL: https://issues.apache.org/jira/browse/HADOOP-13336
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13336-006.patch, HADOOP-13336-007.patch,
> HADOOP-13336-HADOOP-13345-001.patch, HADOOP-13336-HADOOP-13345-002.patch,
>
[jira] [Commented] (HADOOP-13903) KMS does not provide any useful debug information
[
https://issues.apache.org/jira/browse/HADOOP-13903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15811775#comment-15811775
]
Hadoop QA commented on HADOOP-13903:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HADOOP-13903 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-13903 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12846322/HADOOP-13903-branch2.7-6.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11399/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> KMS does not provide any useful debug information
> -
>
> Key: HADOOP-13903
> URL: https://issues.apache.org/jira/browse/HADOOP-13903
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.9.0, 3.0.0-alpha2
>Reporter: Tristan Stevens
>Assignee: Tristan Stevens
>Priority: Minor
> Attachments: HADOOP-13903-2.patch, HADOOP-13903-3.patch,
> HADOOP-13903-5-branch2.7.patch, HADOOP-13903-5.patch, HADOOP-13903-6.patch,
> HADOOP-13903-branch2.7-6.patch, HADOOP-13903.patch
>
>
> At the moment there is no debug or trace level logs generated for KMS
> authorisation decisions. In order for users to understand what is going on in
> given scenarios this would be invaluable.
> Code should endeavour to keep as much work off the sunny-day-code-path as
> much as possible.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13903) KMS does not provide any useful debug information
[ https://issues.apache.org/jira/browse/HADOOP-13903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tristan Stevens updated HADOOP-13903: - Attachment: HADOOP-13903-branch2.7-6.patch HADOOP-13903-6.patch Updated both patches to resolve whitespace issue and to add Javadoc for public method with Object. Also, changed some of the Object signatures to String where possible. > KMS does not provide any useful debug information > - > > Key: HADOOP-13903 > URL: https://issues.apache.org/jira/browse/HADOOP-13903 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.9.0, 3.0.0-alpha2 >Reporter: Tristan Stevens >Assignee: Tristan Stevens >Priority: Minor > Attachments: HADOOP-13903-2.patch, HADOOP-13903-3.patch, > HADOOP-13903-5-branch2.7.patch, HADOOP-13903-5.patch, HADOOP-13903-6.patch, > HADOOP-13903-branch2.7-6.patch, HADOOP-13903.patch > > > At the moment there is no debug or trace level logs generated for KMS > authorisation decisions. In order for users to understand what is going on in > given scenarios this would be invaluable. > Code should endeavour to keep as much work off the sunny-day-code-path as > much as possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13336) S3A to support per-bucket configuration
[ https://issues.apache.org/jira/browse/HADOOP-13336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13336: Status: Patch Available (was: Open) > S3A to support per-bucket configuration > --- > > Key: HADOOP-13336 > URL: https://issues.apache.org/jira/browse/HADOOP-13336 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13336-006.patch, HADOOP-13336-007.patch, > HADOOP-13336-HADOOP-13345-001.patch, HADOOP-13336-HADOOP-13345-002.patch, > HADOOP-13336-HADOOP-13345-003.patch, HADOOP-13336-HADOOP-13345-004.patch, > HADOOP-13336-HADOOP-13345-005.patch, HADOOP-13336-HADOOP-13345-006.patch > > > S3a now supports different regions, by way of declaring the endpoint —but you > can't do things like read in one region, write back in another (e.g. a distcp > backup), because only one region can be specified in a configuration. > If s3a supported region declaration in the URL, e.g. s3a://b1.frankfurt > s3a://b2.seol , then this would be possible. > Swift does this with a full filesystem binding/config: endpoints, username, > etc, in the XML file. Would we need to do that much? It'd be simpler > initially to use a domain suffix of a URL to set the region of a bucket from > the domain and have the aws library sort the details out itself, maybe with > some config options for working with non-AWS infra -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13336) S3A to support per-bucket configuration
[ https://issues.apache.org/jira/browse/HADOOP-13336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13336: Attachment: HADOOP-13336-007.patch Patch 007: latest test fixes, including the checkstyle ones > S3A to support per-bucket configuration > --- > > Key: HADOOP-13336 > URL: https://issues.apache.org/jira/browse/HADOOP-13336 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13336-006.patch, HADOOP-13336-007.patch, > HADOOP-13336-HADOOP-13345-001.patch, HADOOP-13336-HADOOP-13345-002.patch, > HADOOP-13336-HADOOP-13345-003.patch, HADOOP-13336-HADOOP-13345-004.patch, > HADOOP-13336-HADOOP-13345-005.patch, HADOOP-13336-HADOOP-13345-006.patch > > > S3a now supports different regions, by way of declaring the endpoint —but you > can't do things like read in one region, write back in another (e.g. a distcp > backup), because only one region can be specified in a configuration. > If s3a supported region declaration in the URL, e.g. s3a://b1.frankfurt > s3a://b2.seol , then this would be possible. > Swift does this with a full filesystem binding/config: endpoints, username, > etc, in the XML file. Would we need to do that much? It'd be simpler > initially to use a domain suffix of a URL to set the region of a bucket from > the domain and have the aws library sort the details out itself, maybe with > some config options for working with non-AWS infra -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13336) S3A to support per-bucket configuration
[ https://issues.apache.org/jira/browse/HADOOP-13336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13336: Status: Open (was: Patch Available) cancelling patch; trunk branch was 1 commit behind > S3A to support per-bucket configuration > --- > > Key: HADOOP-13336 > URL: https://issues.apache.org/jira/browse/HADOOP-13336 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13336-006.patch, HADOOP-13336-007.patch, > HADOOP-13336-HADOOP-13345-001.patch, HADOOP-13336-HADOOP-13345-002.patch, > HADOOP-13336-HADOOP-13345-003.patch, HADOOP-13336-HADOOP-13345-004.patch, > HADOOP-13336-HADOOP-13345-005.patch, HADOOP-13336-HADOOP-13345-006.patch > > > S3a now supports different regions, by way of declaring the endpoint —but you > can't do things like read in one region, write back in another (e.g. a distcp > backup), because only one region can be specified in a configuration. > If s3a supported region declaration in the URL, e.g. s3a://b1.frankfurt > s3a://b2.seol , then this would be possible. > Swift does this with a full filesystem binding/config: endpoints, username, > etc, in the XML file. Would we need to do that much? It'd be simpler > initially to use a domain suffix of a URL to set the region of a bucket from > the domain and have the aws library sort the details out itself, maybe with > some config options for working with non-AWS infra -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13336) S3A to support per-bucket configuration
[
https://issues.apache.org/jira/browse/HADOOP-13336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15811713#comment-15811713
]
Hadoop QA commented on HADOOP-13336:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 11s{color} | {color:orange} hadoop-tools/hadoop-aws: The patch generated 2
new + 30 unchanged - 1 fixed = 32 total (was 31) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
19s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13336 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12846315/HADOOP-13336-006.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 21fe0e95d5e2 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / db490ec |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11397/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-aws.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11397/testReport/ |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11397/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> S3A to support per-bucket configuration
> ---
>
> Key: HADOOP-13336
> URL: https://issues.apache.org/jira/browse/HADOOP-13336
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments:
[jira] [Commented] (HADOOP-11487) FileNotFound on distcp to s3n/s3a due to creation inconsistency
[
https://issues.apache.org/jira/browse/HADOOP-11487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15811707#comment-15811707
]
Steve Loughran commented on HADOOP-11487:
-
Linking to HADOOP-13950, as it may be a lower cost solution.
AWS caches negative GETs on a path, and FS.create() does an existence check as
part of its getFileStatus call, which is always done to look for the dest being
a directory.
if overwrite=true, we don't care if a dir exists, so only need the 2 directory
probes, not the direct path HEAD, so will skip the possibility of a -ve HEAD
result being cached. This *may* be all that is needed
> FileNotFound on distcp to s3n/s3a due to creation inconsistency
>
>
> Key: HADOOP-11487
> URL: https://issues.apache.org/jira/browse/HADOOP-11487
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, fs/s3
>Affects Versions: 2.7.2
>Reporter: Paulo Motta
>Assignee: John Zhuge
>
> I'm trying to copy a large amount of files from HDFS to S3 via distcp and I'm
> getting the following exception:
> {code:java}
> 2015-01-16 20:53:18,187 ERROR [main]
> org.apache.hadoop.tools.mapred.CopyMapper: Failure in copying
> hdfs://10.165.35.216/hdfsFolder/file.gz to s3n://s3-bucket/file.gz
> java.io.FileNotFoundException: No such file or directory
> 's3n://s3-bucket/file.gz'
> at
> org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:445)
> at
> org.apache.hadoop.tools.util.DistCpUtils.preserve(DistCpUtils.java:187)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:233)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:45)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> 2015-01-16 20:53:18,276 WARN [main] org.apache.hadoop.mapred.YarnChild:
> Exception running child : java.io.FileNotFoundException: No such file or
> directory 's3n://s3-bucket/file.gz'
> at
> org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:445)
> at
> org.apache.hadoop.tools.util.DistCpUtils.preserve(DistCpUtils.java:187)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:233)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:45)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> {code}
> However, when I try hadoop fs -ls s3n://s3-bucket/file.gz the file is there.
> So probably due to Amazon's S3 eventual consistency the job failure.
> In my opinion, in order to fix this problem NativeS3FileSystem.getFileStatus
> must use fs.s3.maxRetries property in order to avoid failures like this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13336) S3A to support per-bucket configuration
[ https://issues.apache.org/jira/browse/HADOOP-13336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13336: Status: Patch Available (was: Open) > S3A to support per-bucket configuration > --- > > Key: HADOOP-13336 > URL: https://issues.apache.org/jira/browse/HADOOP-13336 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13336-006.patch, > HADOOP-13336-HADOOP-13345-001.patch, HADOOP-13336-HADOOP-13345-002.patch, > HADOOP-13336-HADOOP-13345-003.patch, HADOOP-13336-HADOOP-13345-004.patch, > HADOOP-13336-HADOOP-13345-005.patch, HADOOP-13336-HADOOP-13345-006.patch > > > S3a now supports different regions, by way of declaring the endpoint —but you > can't do things like read in one region, write back in another (e.g. a distcp > backup), because only one region can be specified in a configuration. > If s3a supported region declaration in the URL, e.g. s3a://b1.frankfurt > s3a://b2.seol , then this would be possible. > Swift does this with a full filesystem binding/config: endpoints, username, > etc, in the XML file. Would we need to do that much? It'd be simpler > initially to use a domain suffix of a URL to set the region of a bucket from > the domain and have the aws library sort the details out itself, maybe with > some config options for working with non-AWS infra -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13336) S3A to support per-bucket configuration
[ https://issues.apache.org/jira/browse/HADOOP-13336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13336: Status: Open (was: Patch Available) > S3A to support per-bucket configuration > --- > > Key: HADOOP-13336 > URL: https://issues.apache.org/jira/browse/HADOOP-13336 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13336-006.patch, > HADOOP-13336-HADOOP-13345-001.patch, HADOOP-13336-HADOOP-13345-002.patch, > HADOOP-13336-HADOOP-13345-003.patch, HADOOP-13336-HADOOP-13345-004.patch, > HADOOP-13336-HADOOP-13345-005.patch, HADOOP-13336-HADOOP-13345-006.patch > > > S3a now supports different regions, by way of declaring the endpoint —but you > can't do things like read in one region, write back in another (e.g. a distcp > backup), because only one region can be specified in a configuration. > If s3a supported region declaration in the URL, e.g. s3a://b1.frankfurt > s3a://b2.seol , then this would be possible. > Swift does this with a full filesystem binding/config: endpoints, username, > etc, in the XML file. Would we need to do that much? It'd be simpler > initially to use a domain suffix of a URL to set the region of a bucket from > the domain and have the aws library sort the details out itself, maybe with > some config options for working with non-AWS infra -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13336) S3A to support per-bucket configuration
[ https://issues.apache.org/jira/browse/HADOOP-13336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13336: Attachment: HADOOP-13336-006.patch Patch 006 against trunk. Main diff is in testing, plus some changes to S3AFS aren't needed/valid > S3A to support per-bucket configuration > --- > > Key: HADOOP-13336 > URL: https://issues.apache.org/jira/browse/HADOOP-13336 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13336-006.patch, > HADOOP-13336-HADOOP-13345-001.patch, HADOOP-13336-HADOOP-13345-002.patch, > HADOOP-13336-HADOOP-13345-003.patch, HADOOP-13336-HADOOP-13345-004.patch, > HADOOP-13336-HADOOP-13345-005.patch, HADOOP-13336-HADOOP-13345-006.patch > > > S3a now supports different regions, by way of declaring the endpoint —but you > can't do things like read in one region, write back in another (e.g. a distcp > backup), because only one region can be specified in a configuration. > If s3a supported region declaration in the URL, e.g. s3a://b1.frankfurt > s3a://b2.seol , then this would be possible. > Swift does this with a full filesystem binding/config: endpoints, username, > etc, in the XML file. Would we need to do that much? It'd be simpler > initially to use a domain suffix of a URL to set the region of a bucket from > the domain and have the aws library sort the details out itself, maybe with > some config options for working with non-AWS infra -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13960) Initialize DynamoDBMetadataStore without associated S3AFileSystem
[
https://issues.apache.org/jira/browse/HADOOP-13960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HADOOP-13960:
---
Resolution: Fixed
Fix Version/s: HADOOP-13345
Status: Resolved (was: Patch Available)
+1. Thanks [~liuml07]. LGTM.
Committed to feature branch.
> Initialize DynamoDBMetadataStore without associated S3AFileSystem
> -
>
> Key: HADOOP-13960
> URL: https://issues.apache.org/jira/browse/HADOOP-13960
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Fix For: HADOOP-13345
>
> Attachments: HADOOP-13960-HADOOP-13345.000.patch
>
>
> Per the discussion in [HADOOP-13650], it's helpful to initialize a
> DynamoDBMetadataStore object without S3AFileSystem. In the current code, we
> can achieve this via {{DynamoDBMetadataStore#initialize(Configuration)}}.
> However, users still have to provide the associated S3AFileSystem URI in the
> configuration, by means of either setting the {{fs.defaultFS}} in
> configuration file or {{-fs s3://bucket}} command line parameter. Setting the
> default FileSystem in configuration seems not necessary as the command line
> is to manipulate metadata store, e.g. command line tools on an existing HDFS
> cluster.
> This JIRA is to track the effort of initializing a DynamoDBMetadataStore
> without associating any S3 buckets, so that S3AFileSystem is not needed.
> Users have to specify in configuration the DynamoDB endpoints (for region)
> and table name along with credentials, AWS client settings.
> See [~eddyxu] and [~liuml07]'s comments in [HADOOP-13650] for more details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
