[jira] [Commented] (HADOOP-13433) Race in UGI.reloginFromKeytab
[
https://issues.apache.org/jira/browse/HADOOP-13433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851188#comment-15851188
]
Hadoop QA commented on HADOOP-13433:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m
31s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
25s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
14s{color} | {color:green} branch-2.7 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
5s{color} | {color:green} branch-2.7 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s{color} | {color:green} branch-2.7 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
58s{color} | {color:red} hadoop-common-project/hadoop-common in branch-2.7 has
3 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} branch-2.7 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} branch-2.7 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
45s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
49s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
49s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 29s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 2 new + 80 unchanged - 4 fixed = 82 total (was 84) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 4749 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 2m
28s{color} | {color:red} The patch 98 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 24m 35s{color}
| {color:red} hadoop-common in the patch failed with JDK v1.7.0_121. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 1m
0s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}114m 44s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_121 Failed junit tests | hadoop.util.bloom.TestBloomFilters |
| | hadoop.ha.TestZKFailoverControllerStress |
| | hadoop.ipc.TestDecayRpcScheduler |
| | hadoop.ipc.TestCallQueueManager |
| | hadoop.metrics2.impl.TestGangliaMetrics |
| JDK v1.8.0_121
[jira] [Commented] (HADOOP-14044) Synchronization issue in delegation token cancel functionality
[
https://issues.apache.org/jira/browse/HADOOP-14044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851144#comment-15851144
]
Xiao Chen commented on HADOOP-14044:
Thanks [~hgadre] for revving! +1 on patch 3, will commit end of Friday.
Could you please set target version? 2.9.0 for branch-2, 3.0.0-alpha3 for
trunk. Please check http://hadoop.apache.org/releases.html for earlier versions.
> Synchronization issue in delegation token cancel functionality
> --
>
> Key: HADOOP-14044
> URL: https://issues.apache.org/jira/browse/HADOOP-14044
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Hrishikesh Gadre
>Assignee: Hrishikesh Gadre
> Attachments: dt_fail.log, dt_success.log, HADOOP-14044-001.patch,
> HADOOP-14044-002.patch, HADOOP-14044-003.patch
>
>
> We are using Hadoop delegation token authentication functionality in Apache
> Solr. As part of the integration testing, I found following issue with the
> delegation token cancelation functionality.
> Consider a setup with 2 Solr servers (S1 and S2) which are configured to use
> delegation token functionality backed by Zookeeper. Now invoke following
> steps,
> [Step 1] Send a request to S1 to create a delegation token.
> (Delegation token DT is created successfully)
> [Step 2] Send a request to cancel DT to S2
> (DT is canceled successfully. client receives HTTP 200 response)
> [Step 3] Send a request to cancel DT to S2 again
> (DT cancelation fails. client receives HTTP 404 response)
> [Step 4] Send a request to cancel DT to S1
> At this point we get two different responses.
> - DT cancelation fails. client receives HTTP 404 response
> - DT cancelation succeeds. client receives HTTP 200 response
> Also as per the current implementation, each server maintains an in_memory
> cache of current tokens which is updated using the ZK watch mechanism. e.g.
> the ZK watch on S1 will ensure that the in_memory cache is synchronized after
> step 2.
> After investigation, I found the root cause for this behavior is due to the
> race condition between step 4 and the firing of ZK watch on S1. Whenever the
> watch fires before the step 4 - we get HTTP 404 response (as expected). When
> that is not the case - we get HTTP 200 response along with following ERROR
> message in the log,
> {noformat}
> Attempted to remove a non-existing znode /ZKDTSMTokensRoot/DT_XYZ
> {noformat}
> From client perspective, the server *should* return HTTP 404 error when the
> cancel request is sent out for an invalid token.
> Ref: Here is the relevant Solr unit test for reference,
> https://github.com/apache/lucene-solr/blob/746786636404cdb8ce505ed0ed02b8d9144ab6c4/solr/core/src/test/org/apache/solr/cloud/TestSolrCloudWithDelegationTokens.java#L285
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13433) Race in UGI.reloginFromKeytab
[
https://issues.apache.org/jira/browse/HADOOP-13433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-13433:
---
Attachment: HADOOP-13433-branch-2.8.patch
Thanks for the backport patches, [~Apache9]!
Seems jenkins only kicked
https://builds.apache.org/job/PreCommit-HADOOP-Build/11567/ when the 2
attachments were uploaded together, for 2.7.
Downloaded an reattached the branch-2.8 patch to trigger jenkins (don't know
how to kick the job for that patch otherwise..)
> Race in UGI.reloginFromKeytab
> -
>
> Key: HADOOP-13433
> URL: https://issues.apache.org/jira/browse/HADOOP-13433
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: HADOOP-13433-branch-2.7.patch,
> HADOOP-13433-branch-2.8.patch, HADOOP-13433-branch-2.8.patch,
> HADOOP-13433-branch-2.patch, HADOOP-13433.patch, HADOOP-13433-v1.patch,
> HADOOP-13433-v2.patch, HADOOP-13433-v4.patch, HADOOP-13433-v5.patch,
> HADOOP-13433-v6.patch, HBASE-13433-testcase-v3.patch
>
>
> This is a problem that has troubled us for several years. For our HBase
> cluster, sometimes the RS will be stuck due to
> {noformat}
> 2016-06-20,03:44:12,936 INFO org.apache.hadoop.ipc.SecureClient: Exception
> encountered while connecting to the server :
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: The ticket
> isn't for us (35) - BAD TGS SERVER NAME)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194)
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:140)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupSaslConnection(SecureClient.java:187)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.access$700(SecureClient.java:95)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:325)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:322)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1781)
> at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
> at org.apache.hadoop.hbase.security.User.call(User.java:607)
> at org.apache.hadoop.hbase.security.User.access$700(User.java:51)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:461)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstreams(SecureClient.java:321)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1164)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1004)
> at
> org.apache.hadoop.hbase.ipc.SecureRpcEngine$Invoker.invoke(SecureRpcEngine.java:107)
> at $Proxy24.replicateLogEntries(Unknown Source)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:962)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.runLoop(ReplicationSource.java:466)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:515)
> Caused by: GSSException: No valid credentials provided (Mechanism level: The
> ticket isn't for us (35) - BAD TGS SERVER NAME)
> at
> sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:663)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:180)
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175)
> ... 23 more
> Caused by: KrbException: The ticket isn't for us (35) - BAD TGS SERVER NAME
> at sun.security.krb5.KrbTgsRep.(KrbTgsRep.java:64)
> at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:185)
> at
> sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:294)
> at
> sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:106)
> at
>
[jira] [Updated] (HADOOP-13433) Race in UGI.reloginFromKeytab
[
https://issues.apache.org/jira/browse/HADOOP-13433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HADOOP-13433:
---
Status: Patch Available (was: Reopened)
> Race in UGI.reloginFromKeytab
> -
>
> Key: HADOOP-13433
> URL: https://issues.apache.org/jira/browse/HADOOP-13433
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 3.0.0-alpha1, 2.6.5, 2.7.3, 2.8.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: HADOOP-13433-branch-2.7.patch,
> HADOOP-13433-branch-2.8.patch, HADOOP-13433-branch-2.patch,
> HADOOP-13433.patch, HADOOP-13433-v1.patch, HADOOP-13433-v2.patch,
> HADOOP-13433-v4.patch, HADOOP-13433-v5.patch, HADOOP-13433-v6.patch,
> HBASE-13433-testcase-v3.patch
>
>
> This is a problem that has troubled us for several years. For our HBase
> cluster, sometimes the RS will be stuck due to
> {noformat}
> 2016-06-20,03:44:12,936 INFO org.apache.hadoop.ipc.SecureClient: Exception
> encountered while connecting to the server :
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: The ticket
> isn't for us (35) - BAD TGS SERVER NAME)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194)
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:140)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupSaslConnection(SecureClient.java:187)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.access$700(SecureClient.java:95)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:325)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:322)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1781)
> at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
> at org.apache.hadoop.hbase.security.User.call(User.java:607)
> at org.apache.hadoop.hbase.security.User.access$700(User.java:51)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:461)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstreams(SecureClient.java:321)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1164)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1004)
> at
> org.apache.hadoop.hbase.ipc.SecureRpcEngine$Invoker.invoke(SecureRpcEngine.java:107)
> at $Proxy24.replicateLogEntries(Unknown Source)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:962)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.runLoop(ReplicationSource.java:466)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:515)
> Caused by: GSSException: No valid credentials provided (Mechanism level: The
> ticket isn't for us (35) - BAD TGS SERVER NAME)
> at
> sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:663)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:180)
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175)
> ... 23 more
> Caused by: KrbException: The ticket isn't for us (35) - BAD TGS SERVER NAME
> at sun.security.krb5.KrbTgsRep.(KrbTgsRep.java:64)
> at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:185)
> at
> sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:294)
> at
> sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:106)
> at
> sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:557)
> at
> sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:594)
> ... 26 more
> Caused by: KrbException: Identifier doesn't match expected value (906)
> at sun.security.krb5.internal.KDCRep.init(KDCRep.java:133)
> at sun.security.krb5.internal.TGSRep.init(TGSRep.java:58)
>
[jira] [Assigned] (HADOOP-14052) Fix dead link in KMS document
[
https://issues.apache.org/jira/browse/HADOOP-14052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton reassigned HADOOP-14052:
-
Assignee: Christina Vu
> Fix dead link in KMS document
> -
>
> Key: HADOOP-14052
> URL: https://issues.apache.org/jira/browse/HADOOP-14052
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.0.0-alpha2
>Reporter: Akira Ajisaka
>Assignee: Christina Vu
>Priority: Minor
> Labels: newbie
>
> The link for Rollover Key section is broken.
> {noformat:title=./hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm}
> This is usually useful after a [Rollover](Rollover_Key) of an encryption key.
> {noformat}
> (Rollover_Key) should be (#Rollover_Key)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14053) Update the link to HTrace SpanReceivers
[
https://issues.apache.org/jira/browse/HADOOP-14053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HADOOP-14053:
---
Labels: newbie (was: )
> Update the link to HTrace SpanReceivers
> ---
>
> Key: HADOOP-14053
> URL: https://issues.apache.org/jira/browse/HADOOP-14053
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation
>Reporter: Akira Ajisaka
>Priority: Minor
> Labels: newbie
>
> Apache HTrace developer guide was moved to the different page. The link for
> Span Receiver should be updated as well.
> {noformat:title=Tracing.md}
> by using implementation of
> [SpanReceiver](http://htrace.incubator.apache.org/#Span_Receivers)
> {noformat}
> The new link is
> http://htrace.incubator.apache.org/developer_guide.html#SpanReceivers
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14053) Update the link to HTrace SpanReceivers
Akira Ajisaka created HADOOP-14053:
--
Summary: Update the link to HTrace SpanReceivers
Key: HADOOP-14053
URL: https://issues.apache.org/jira/browse/HADOOP-14053
Project: Hadoop Common
Issue Type: Improvement
Components: documentation
Reporter: Akira Ajisaka
Priority: Minor
Apache HTrace developer guide was moved to the different page. The link for
Span Receiver should be updated as well.
{noformat:title=Tracing.md}
by using implementation of
[SpanReceiver](http://htrace.incubator.apache.org/#Span_Receivers)
{noformat}
The new link is
http://htrace.incubator.apache.org/developer_guide.html#SpanReceivers
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14052) Fix dead link in KMS document
[
https://issues.apache.org/jira/browse/HADOOP-14052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HADOOP-14052:
---
Description:
The link for Rollover Key section is broken.
{noformat:title=./hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm}
This is usually useful after a [Rollover](Rollover_Key) of an encryption key.
{noformat}
(Rollover_Key) should be (#Rollover_Key)
was:
The link for Rollover Key section is broken.
{noformat:title=./hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm}
This is usually useful after a [Rollover](Rollover_Key) of an encryption key.
{noformat}
(Rollover_key) should be (#Rollover_key)
> Fix dead link in KMS document
> -
>
> Key: HADOOP-14052
> URL: https://issues.apache.org/jira/browse/HADOOP-14052
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.0.0-alpha2
>Reporter: Akira Ajisaka
> Labels: newbie
>
> The link for Rollover Key section is broken.
> {noformat:title=./hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm}
> This is usually useful after a [Rollover](Rollover_Key) of an encryption key.
> {noformat}
> (Rollover_Key) should be (#Rollover_Key)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14052) Fix dead link in KMS document
[
https://issues.apache.org/jira/browse/HADOOP-14052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HADOOP-14052:
---
Priority: Minor (was: Major)
> Fix dead link in KMS document
> -
>
> Key: HADOOP-14052
> URL: https://issues.apache.org/jira/browse/HADOOP-14052
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.0.0-alpha2
>Reporter: Akira Ajisaka
>Priority: Minor
> Labels: newbie
>
> The link for Rollover Key section is broken.
> {noformat:title=./hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm}
> This is usually useful after a [Rollover](Rollover_Key) of an encryption key.
> {noformat}
> (Rollover_Key) should be (#Rollover_Key)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14052) Fix dead link in KMS document
Akira Ajisaka created HADOOP-14052:
--
Summary: Fix dead link in KMS document
Key: HADOOP-14052
URL: https://issues.apache.org/jira/browse/HADOOP-14052
Project: Hadoop Common
Issue Type: Bug
Components: documentation
Affects Versions: 3.0.0-alpha2
Reporter: Akira Ajisaka
The link for Rollover Key section is broken.
{noformat:title=./hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm}
This is usually useful after a [Rollover](Rollover_Key) of an encryption key.
{noformat}
(Rollover_key) should be (#Rollover_key)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13433) Race in UGI.reloginFromKeytab
[
https://issues.apache.org/jira/browse/HADOOP-13433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HADOOP-13433:
---
Attachment: HADOOP-13433-branch-2.7.patch
HADOOP-13433-branch-2.8.patch
Patches for branch-2.8 and branch-2.7. The patch for branch-2.7 could also be
applied to branch-2.6.
> Race in UGI.reloginFromKeytab
> -
>
> Key: HADOOP-13433
> URL: https://issues.apache.org/jira/browse/HADOOP-13433
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: HADOOP-13433-branch-2.7.patch,
> HADOOP-13433-branch-2.8.patch, HADOOP-13433-branch-2.patch,
> HADOOP-13433.patch, HADOOP-13433-v1.patch, HADOOP-13433-v2.patch,
> HADOOP-13433-v4.patch, HADOOP-13433-v5.patch, HADOOP-13433-v6.patch,
> HBASE-13433-testcase-v3.patch
>
>
> This is a problem that has troubled us for several years. For our HBase
> cluster, sometimes the RS will be stuck due to
> {noformat}
> 2016-06-20,03:44:12,936 INFO org.apache.hadoop.ipc.SecureClient: Exception
> encountered while connecting to the server :
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: The ticket
> isn't for us (35) - BAD TGS SERVER NAME)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194)
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:140)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupSaslConnection(SecureClient.java:187)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.access$700(SecureClient.java:95)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:325)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:322)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1781)
> at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
> at org.apache.hadoop.hbase.security.User.call(User.java:607)
> at org.apache.hadoop.hbase.security.User.access$700(User.java:51)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:461)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstreams(SecureClient.java:321)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1164)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1004)
> at
> org.apache.hadoop.hbase.ipc.SecureRpcEngine$Invoker.invoke(SecureRpcEngine.java:107)
> at $Proxy24.replicateLogEntries(Unknown Source)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:962)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.runLoop(ReplicationSource.java:466)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:515)
> Caused by: GSSException: No valid credentials provided (Mechanism level: The
> ticket isn't for us (35) - BAD TGS SERVER NAME)
> at
> sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:663)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:180)
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175)
> ... 23 more
> Caused by: KrbException: The ticket isn't for us (35) - BAD TGS SERVER NAME
> at sun.security.krb5.KrbTgsRep.(KrbTgsRep.java:64)
> at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:185)
> at
> sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:294)
> at
> sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:106)
> at
> sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:557)
> at
> sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:594)
> ... 26 more
> Caused by: KrbException: Identifier doesn't match expected
[jira] [Reopened] (HADOOP-13433) Race in UGI.reloginFromKeytab
[
https://issues.apache.org/jira/browse/HADOOP-13433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang reopened HADOOP-13433:
Reopen to attach patches for branch-2.8 and branch-2.7.
> Race in UGI.reloginFromKeytab
> -
>
> Key: HADOOP-13433
> URL: https://issues.apache.org/jira/browse/HADOOP-13433
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: HADOOP-13433-branch-2.patch, HADOOP-13433.patch,
> HADOOP-13433-v1.patch, HADOOP-13433-v2.patch, HADOOP-13433-v4.patch,
> HADOOP-13433-v5.patch, HADOOP-13433-v6.patch, HBASE-13433-testcase-v3.patch
>
>
> This is a problem that has troubled us for several years. For our HBase
> cluster, sometimes the RS will be stuck due to
> {noformat}
> 2016-06-20,03:44:12,936 INFO org.apache.hadoop.ipc.SecureClient: Exception
> encountered while connecting to the server :
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: The ticket
> isn't for us (35) - BAD TGS SERVER NAME)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194)
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:140)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupSaslConnection(SecureClient.java:187)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.access$700(SecureClient.java:95)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:325)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:322)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1781)
> at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
> at org.apache.hadoop.hbase.security.User.call(User.java:607)
> at org.apache.hadoop.hbase.security.User.access$700(User.java:51)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:461)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstreams(SecureClient.java:321)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1164)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1004)
> at
> org.apache.hadoop.hbase.ipc.SecureRpcEngine$Invoker.invoke(SecureRpcEngine.java:107)
> at $Proxy24.replicateLogEntries(Unknown Source)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:962)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.runLoop(ReplicationSource.java:466)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:515)
> Caused by: GSSException: No valid credentials provided (Mechanism level: The
> ticket isn't for us (35) - BAD TGS SERVER NAME)
> at
> sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:663)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:180)
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175)
> ... 23 more
> Caused by: KrbException: The ticket isn't for us (35) - BAD TGS SERVER NAME
> at sun.security.krb5.KrbTgsRep.(KrbTgsRep.java:64)
> at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:185)
> at
> sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:294)
> at
> sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:106)
> at
> sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:557)
> at
> sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:594)
> ... 26 more
> Caused by: KrbException: Identifier doesn't match expected value (906)
> at sun.security.krb5.internal.KDCRep.init(KDCRep.java:133)
> at sun.security.krb5.internal.TGSRep.init(TGSRep.java:58)
> at
[jira] [Commented] (HADOOP-14017) User friendly name for ADLS user and group
[
https://issues.apache.org/jira/browse/HADOOP-14017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850952#comment-15850952
]
John Zhuge commented on HADOOP-14017:
-
Could we make it possible to display either GUID or user-friendly name? My use
case is {{-ls -i}} to be introduced by HDFS-10233.
> User friendly name for ADLS user and group
> --
>
> Key: HADOOP-14017
> URL: https://issues.apache.org/jira/browse/HADOOP-14017
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/adl
>Affects Versions: 3.0.0-alpha3
>Reporter: John Zhuge
>Assignee: John Zhuge
>
> ADLS displays GUID whenever user or group displayed, e.g., {{ls}},
> {{getfacl}}.
> ADLS requires GUID whenever user or group input is needed, e.g., {{setfacl}},
> {{chown}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14017) User friendly names for ADLS user and group
[
https://issues.apache.org/jira/browse/HADOOP-14017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HADOOP-14017:
Description:
ADLS displays GUID whenever user or group displayed, e.g., {{ls}}, {{getfacl}}.
ADLS requires GUID whenever user or group input is needed, e.g., {{setfacl}},
{{chown}}.
was:
Track the effort to integrate ADLS ACL which models after HDFS ACL. Both are
based on POSIX ACL.
Of course this will go too far without AuthN integration of some sort.
> User friendly names for ADLS user and group
> ---
>
> Key: HADOOP-14017
> URL: https://issues.apache.org/jira/browse/HADOOP-14017
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/adl
>Affects Versions: 3.0.0-alpha3
>Reporter: John Zhuge
>Assignee: John Zhuge
>
> ADLS displays GUID whenever user or group displayed, e.g., {{ls}},
> {{getfacl}}.
> ADLS requires GUID whenever user or group input is needed, e.g., {{setfacl}},
> {{chown}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14017) User friendly name for ADLS user and group
[
https://issues.apache.org/jira/browse/HADOOP-14017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HADOOP-14017:
Summary: User friendly name for ADLS user and group (was: User friendly
names for ADLS user and group)
> User friendly name for ADLS user and group
> --
>
> Key: HADOOP-14017
> URL: https://issues.apache.org/jira/browse/HADOOP-14017
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/adl
>Affects Versions: 3.0.0-alpha3
>Reporter: John Zhuge
>Assignee: John Zhuge
>
> ADLS displays GUID whenever user or group displayed, e.g., {{ls}},
> {{getfacl}}.
> ADLS requires GUID whenever user or group input is needed, e.g., {{setfacl}},
> {{chown}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14017) User friendly names for ADLS user and group
[ https://issues.apache.org/jira/browse/HADOOP-14017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-14017: Summary: User friendly names for ADLS user and group (was: Integrate ADLS ACL) > User friendly names for ADLS user and group > --- > > Key: HADOOP-14017 > URL: https://issues.apache.org/jira/browse/HADOOP-14017 > Project: Hadoop Common > Issue Type: Bug > Components: fs/adl >Affects Versions: 3.0.0-alpha3 >Reporter: John Zhuge >Assignee: John Zhuge > > Track the effort to integrate ADLS ACL which models after HDFS ACL. Both are > based on POSIX ACL. > Of course this will go too far without AuthN integration of some sort. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13433) Race in UGI.reloginFromKeytab
[
https://issues.apache.org/jira/browse/HADOOP-13433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850895#comment-15850895
]
Duo Zhang commented on HADOOP-13433:
Let me prepare patches for branch-2.8 and branch-2.7.
{quote}
I guess this is a pretty corner case, so it would not be a big concern.
{quote}
Yes, this rarely happens so I do not think it is a big deal. And the service
tickets are generated with the old tgt, so theoretically they should be
destroyed.
Thanks.
> Race in UGI.reloginFromKeytab
> -
>
> Key: HADOOP-13433
> URL: https://issues.apache.org/jira/browse/HADOOP-13433
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: HADOOP-13433-branch-2.patch, HADOOP-13433.patch,
> HADOOP-13433-v1.patch, HADOOP-13433-v2.patch, HADOOP-13433-v4.patch,
> HADOOP-13433-v5.patch, HADOOP-13433-v6.patch, HBASE-13433-testcase-v3.patch
>
>
> This is a problem that has troubled us for several years. For our HBase
> cluster, sometimes the RS will be stuck due to
> {noformat}
> 2016-06-20,03:44:12,936 INFO org.apache.hadoop.ipc.SecureClient: Exception
> encountered while connecting to the server :
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: The ticket
> isn't for us (35) - BAD TGS SERVER NAME)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194)
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:140)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupSaslConnection(SecureClient.java:187)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.access$700(SecureClient.java:95)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:325)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:322)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1781)
> at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
> at org.apache.hadoop.hbase.security.User.call(User.java:607)
> at org.apache.hadoop.hbase.security.User.access$700(User.java:51)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:461)
> at
> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstreams(SecureClient.java:321)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1164)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1004)
> at
> org.apache.hadoop.hbase.ipc.SecureRpcEngine$Invoker.invoke(SecureRpcEngine.java:107)
> at $Proxy24.replicateLogEntries(Unknown Source)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:962)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.runLoop(ReplicationSource.java:466)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:515)
> Caused by: GSSException: No valid credentials provided (Mechanism level: The
> ticket isn't for us (35) - BAD TGS SERVER NAME)
> at
> sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:663)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:180)
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175)
> ... 23 more
> Caused by: KrbException: The ticket isn't for us (35) - BAD TGS SERVER NAME
> at sun.security.krb5.KrbTgsRep.(KrbTgsRep.java:64)
> at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:185)
> at
> sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:294)
> at
> sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:106)
> at
> sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:557)
> at
>
[jira] [Commented] (HADOOP-14044) Synchronization issue in delegation token cancel functionality
[
https://issues.apache.org/jira/browse/HADOOP-14044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850894#comment-15850894
]
Hadoop QA commented on HADOOP-14044:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
55s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 59m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-14044 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12850725/HADOOP-14044-003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 57fe2804ea40 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 0914fcc |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11566/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11566/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Synchronization issue in delegation token cancel functionality
> --
>
> Key: HADOOP-14044
> URL: https://issues.apache.org/jira/browse/HADOOP-14044
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Hrishikesh Gadre
>Assignee: Hrishikesh Gadre
> Attachments: dt_fail.log, dt_success.log, HADOOP-14044-001.patch,
> HADOOP-14044-002.patch,
[jira] [Commented] (HADOOP-13866) Upgrade netty-all to 4.1.1.Final
[
https://issues.apache.org/jira/browse/HADOOP-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850837#comment-15850837
]
Xiao Chen commented on HADOOP-13866:
Thanks Ted, removed 2.9 and linked to HADOOP-14043.
I think this is ready to be committed to trunk. Any objections?
> Upgrade netty-all to 4.1.1.Final
>
>
> Key: HADOOP-13866
> URL: https://issues.apache.org/jira/browse/HADOOP-13866
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Blocker
> Attachments: HADOOP-13866.v1.patch, HADOOP-13866.v2.patch,
> HADOOP-13866.v3.patch, HADOOP-13866.v4.patch, HADOOP-13866.v6.patch,
> HADOOP-13866.v7.patch, HADOOP-13866.v8.patch, HADOOP-13866.v8.patch,
> HADOOP-13866.v8.patch
>
>
> netty-all 4.1.1.Final is stable release which we should upgrade to.
> See bottom of HADOOP-12927 for related discussion.
> This issue was discovered since hbase 2.0 uses 4.1.1.Final of netty.
> When launching mapreduce job from hbase, /grid/0/hadoop/yarn/local/
> usercache/hbase/appcache/application_1479850535804_0008/container_e01_1479850535804_0008_01_05/mr-framework/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar
> (from hdfs) is ahead of 4.1.1.Final jar (from hbase) on the classpath.
> Resulting in the following exception:
> {code}
> 2016-12-01 20:17:26,678 WARN [Default-IPC-NioEventLoopGroup-1-1]
> io.netty.util.concurrent.DefaultPromise: An exception was thrown by
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete()
> java.lang.NoSuchMethodError:
> io.netty.buffer.ByteBuf.retainedDuplicate()Lio/netty/buffer/ByteBuf;
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:272)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:262)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> at
> io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13866) Upgrade netty-all to 4.1.1.Final
[
https://issues.apache.org/jira/browse/HADOOP-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-13866:
---
Target Version/s: 3.0.0-beta1 (was: 2.9.0, 3.0.0-beta1)
> Upgrade netty-all to 4.1.1.Final
>
>
> Key: HADOOP-13866
> URL: https://issues.apache.org/jira/browse/HADOOP-13866
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Blocker
> Attachments: HADOOP-13866.v1.patch, HADOOP-13866.v2.patch,
> HADOOP-13866.v3.patch, HADOOP-13866.v4.patch, HADOOP-13866.v6.patch,
> HADOOP-13866.v7.patch, HADOOP-13866.v8.patch, HADOOP-13866.v8.patch,
> HADOOP-13866.v8.patch
>
>
> netty-all 4.1.1.Final is stable release which we should upgrade to.
> See bottom of HADOOP-12927 for related discussion.
> This issue was discovered since hbase 2.0 uses 4.1.1.Final of netty.
> When launching mapreduce job from hbase, /grid/0/hadoop/yarn/local/
> usercache/hbase/appcache/application_1479850535804_0008/container_e01_1479850535804_0008_01_05/mr-framework/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar
> (from hdfs) is ahead of 4.1.1.Final jar (from hbase) on the classpath.
> Resulting in the following exception:
> {code}
> 2016-12-01 20:17:26,678 WARN [Default-IPC-NioEventLoopGroup-1-1]
> io.netty.util.concurrent.DefaultPromise: An exception was thrown by
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete()
> java.lang.NoSuchMethodError:
> io.netty.buffer.ByteBuf.retainedDuplicate()Lio/netty/buffer/ByteBuf;
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:272)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:262)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> at
> io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13866) Upgrade netty-all to 4.1.1.Final
[
https://issues.apache.org/jira/browse/HADOOP-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850826#comment-15850826
]
Ted Yu commented on HADOOP-13866:
-
2.9 can temporarily be dropped.
If HADOOP-14043 gets into 2.9, we can pull this in.
> Upgrade netty-all to 4.1.1.Final
>
>
> Key: HADOOP-13866
> URL: https://issues.apache.org/jira/browse/HADOOP-13866
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Blocker
> Attachments: HADOOP-13866.v1.patch, HADOOP-13866.v2.patch,
> HADOOP-13866.v3.patch, HADOOP-13866.v4.patch, HADOOP-13866.v6.patch,
> HADOOP-13866.v7.patch, HADOOP-13866.v8.patch, HADOOP-13866.v8.patch,
> HADOOP-13866.v8.patch
>
>
> netty-all 4.1.1.Final is stable release which we should upgrade to.
> See bottom of HADOOP-12927 for related discussion.
> This issue was discovered since hbase 2.0 uses 4.1.1.Final of netty.
> When launching mapreduce job from hbase, /grid/0/hadoop/yarn/local/
> usercache/hbase/appcache/application_1479850535804_0008/container_e01_1479850535804_0008_01_05/mr-framework/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar
> (from hdfs) is ahead of 4.1.1.Final jar (from hbase) on the classpath.
> Resulting in the following exception:
> {code}
> 2016-12-01 20:17:26,678 WARN [Default-IPC-NioEventLoopGroup-1-1]
> io.netty.util.concurrent.DefaultPromise: An exception was thrown by
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete()
> java.lang.NoSuchMethodError:
> io.netty.buffer.ByteBuf.retainedDuplicate()Lio/netty/buffer/ByteBuf;
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:272)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:262)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> at
> io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14041) CLI command to prune old metadata
[ https://issues.apache.org/jira/browse/HADOOP-14041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850820#comment-15850820 ] Sean Mackrory commented on HADOOP-14041: Been thinking about it some more and cleaning up directories is very tricky. One problem is that since we don't put a mod_time on directories (presumably just because S3 doesn't?) so it's impossible to distinguish between a directory that has existed for a long time and has had all of it's contents pruned, vs. a directory that was just created recently and had no contents to prune (yet). Putting a mod_time on a directory could be done in 2 days: we could just use that as a creation time, or a time when it's list of children changed. If it's only used for deciding when to prune old metadata, using it as creation time allows us to clean very old directories that don't have more recent children without the overhead of updating it every time we add or modify a child. But that might be a bit of a departure from the meaning expressed by "modification time". I'm thinking a couple of things: 1) For now, I think I'll just prune directories that did have contents, but are now completely empty post-prune. Later, maybe we can add mod_time for directories and clean up directories that are old enough to be pruned and are empty, even though they didn't have children removed in the prune. The more I think about it, the more I think that will be rare and not worth adding mod_time to all directories just to clean it up more nicely. 2) Having thought about the gap between identifying files to prune and which directories to prune, it's probably better to do this in very small batches. It's okay for this prune command to take a longer time to run because we're making many round trips. The benefit of that is we minimize the window in which files can get created in a directory that is being cleaned up and might be considered empty. It also minimized impact on other workloads. So ultimately I'm thinking the best way to do this is to clean up directories that did have children but had them all pruned (and THEIR parents if the same is now true of the parent directory), and to do this in very small batches or even individually. The more I think about it, it's probably not worth adding mod_time to directories to handle this any more completely. Would love to hear others' input, though. > CLI command to prune old metadata > - > > Key: HADOOP-14041 > URL: https://issues.apache.org/jira/browse/HADOOP-14041 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Sean Mackrory >Assignee: Sean Mackrory > Attachments: HADOOP-14041-HADOOP-13345.001.patch > > > Add a CLI command that allows users to specify an age at which to prune > metadata that hasn't been modified for an extended period of time. Since the > primary use-case targeted at the moment is list consistency, it would make > sense (especially when authoritative=false) to prune metadata that is > expected to have become consistent a long time ago. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13866) Upgrade netty-all to 4.1.1.Final
[
https://issues.apache.org/jira/browse/HADOOP-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850812#comment-15850812
]
Xiao Chen commented on HADOOP-13866:
So should we remove 2.9.0 from target versions? Thanks.
> Upgrade netty-all to 4.1.1.Final
>
>
> Key: HADOOP-13866
> URL: https://issues.apache.org/jira/browse/HADOOP-13866
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Blocker
> Attachments: HADOOP-13866.v1.patch, HADOOP-13866.v2.patch,
> HADOOP-13866.v3.patch, HADOOP-13866.v4.patch, HADOOP-13866.v6.patch,
> HADOOP-13866.v7.patch, HADOOP-13866.v8.patch, HADOOP-13866.v8.patch,
> HADOOP-13866.v8.patch
>
>
> netty-all 4.1.1.Final is stable release which we should upgrade to.
> See bottom of HADOOP-12927 for related discussion.
> This issue was discovered since hbase 2.0 uses 4.1.1.Final of netty.
> When launching mapreduce job from hbase, /grid/0/hadoop/yarn/local/
> usercache/hbase/appcache/application_1479850535804_0008/container_e01_1479850535804_0008_01_05/mr-framework/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar
> (from hdfs) is ahead of 4.1.1.Final jar (from hbase) on the classpath.
> Resulting in the following exception:
> {code}
> 2016-12-01 20:17:26,678 WARN [Default-IPC-NioEventLoopGroup-1-1]
> io.netty.util.concurrent.DefaultPromise: An exception was thrown by
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete()
> java.lang.NoSuchMethodError:
> io.netty.buffer.ByteBuf.retainedDuplicate()Lio/netty/buffer/ByteBuf;
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:272)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:262)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> at
> io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13866) Upgrade netty-all to 4.1.1.Final
[
https://issues.apache.org/jira/browse/HADOOP-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850808#comment-15850808
]
Ted Yu commented on HADOOP-13866:
-
That's true - if downstream project(s) uses netty API available in 4.1.0.Beta5
but not in 4.1.1
> Upgrade netty-all to 4.1.1.Final
>
>
> Key: HADOOP-13866
> URL: https://issues.apache.org/jira/browse/HADOOP-13866
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Blocker
> Attachments: HADOOP-13866.v1.patch, HADOOP-13866.v2.patch,
> HADOOP-13866.v3.patch, HADOOP-13866.v4.patch, HADOOP-13866.v6.patch,
> HADOOP-13866.v7.patch, HADOOP-13866.v8.patch, HADOOP-13866.v8.patch,
> HADOOP-13866.v8.patch
>
>
> netty-all 4.1.1.Final is stable release which we should upgrade to.
> See bottom of HADOOP-12927 for related discussion.
> This issue was discovered since hbase 2.0 uses 4.1.1.Final of netty.
> When launching mapreduce job from hbase, /grid/0/hadoop/yarn/local/
> usercache/hbase/appcache/application_1479850535804_0008/container_e01_1479850535804_0008_01_05/mr-framework/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar
> (from hdfs) is ahead of 4.1.1.Final jar (from hbase) on the classpath.
> Resulting in the following exception:
> {code}
> 2016-12-01 20:17:26,678 WARN [Default-IPC-NioEventLoopGroup-1-1]
> io.netty.util.concurrent.DefaultPromise: An exception was thrown by
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete()
> java.lang.NoSuchMethodError:
> io.netty.buffer.ByteBuf.retainedDuplicate()Lio/netty/buffer/ByteBuf;
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:272)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:262)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> at
> io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13998) initial s3guard preview
[ https://issues.apache.org/jira/browse/HADOOP-13998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850796#comment-15850796 ] Aaron Fabbri commented on HADOOP-13998: --- [~aw] no.. thanks for noticing. I'll fix that (filed HADOOP-14051). > initial s3guard preview > --- > > Key: HADOOP-13998 > URL: https://issues.apache.org/jira/browse/HADOOP-13998 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Steve Loughran > > JIRA to link in all the things we think are needed for a preview/merge into > trunk -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14051) S3Guard: link docs from index, fix typos
Aaron Fabbri created HADOOP-14051: - Summary: S3Guard: link docs from index, fix typos Key: HADOOP-14051 URL: https://issues.apache.org/jira/browse/HADOOP-14051 Project: Hadoop Common Issue Type: Sub-task Reporter: Aaron Fabbri Assignee: Aaron Fabbri JIRA for a quick round of s3guard.md documentation improvements. - Link from index.md - Make a pass and fix typos / outdated stuff / spelling etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14044) Synchronization issue in delegation token cancel functionality
[
https://issues.apache.org/jira/browse/HADOOP-14044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hrishikesh Gadre updated HADOOP-14044:
--
Attachment: HADOOP-14044-003.patch
[~xiaochen] Thanks for the review. Here is an updated patch which addresses
these comments.
> Synchronization issue in delegation token cancel functionality
> --
>
> Key: HADOOP-14044
> URL: https://issues.apache.org/jira/browse/HADOOP-14044
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Hrishikesh Gadre
>Assignee: Hrishikesh Gadre
> Attachments: dt_fail.log, dt_success.log, HADOOP-14044-001.patch,
> HADOOP-14044-002.patch, HADOOP-14044-003.patch
>
>
> We are using Hadoop delegation token authentication functionality in Apache
> Solr. As part of the integration testing, I found following issue with the
> delegation token cancelation functionality.
> Consider a setup with 2 Solr servers (S1 and S2) which are configured to use
> delegation token functionality backed by Zookeeper. Now invoke following
> steps,
> [Step 1] Send a request to S1 to create a delegation token.
> (Delegation token DT is created successfully)
> [Step 2] Send a request to cancel DT to S2
> (DT is canceled successfully. client receives HTTP 200 response)
> [Step 3] Send a request to cancel DT to S2 again
> (DT cancelation fails. client receives HTTP 404 response)
> [Step 4] Send a request to cancel DT to S1
> At this point we get two different responses.
> - DT cancelation fails. client receives HTTP 404 response
> - DT cancelation succeeds. client receives HTTP 200 response
> Also as per the current implementation, each server maintains an in_memory
> cache of current tokens which is updated using the ZK watch mechanism. e.g.
> the ZK watch on S1 will ensure that the in_memory cache is synchronized after
> step 2.
> After investigation, I found the root cause for this behavior is due to the
> race condition between step 4 and the firing of ZK watch on S1. Whenever the
> watch fires before the step 4 - we get HTTP 404 response (as expected). When
> that is not the case - we get HTTP 200 response along with following ERROR
> message in the log,
> {noformat}
> Attempted to remove a non-existing znode /ZKDTSMTokensRoot/DT_XYZ
> {noformat}
> From client perspective, the server *should* return HTTP 404 error when the
> cancel request is sent out for an invalid token.
> Ref: Here is the relevant Solr unit test for reference,
> https://github.com/apache/lucene-solr/blob/746786636404cdb8ce505ed0ed02b8d9144ab6c4/solr/core/src/test/org/apache/solr/cloud/TestSolrCloudWithDelegationTokens.java#L285
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14050) Add process name to kms process
[
https://issues.apache.org/jira/browse/HADOOP-14050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850762#comment-15850762
]
John Zhuge commented on HADOOP-14050:
-
Ignore my previous comment, the command line already has {{-Dproc_kms}}.
> Add process name to kms process
> ---
>
> Key: HADOOP-14050
> URL: https://issues.apache.org/jira/browse/HADOOP-14050
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms, scripts
>Affects Versions: 2.7.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Minor
> Attachments: HADOOP-14050-branch-2.patch
>
>
> Like other hadoop daemons, we should start the kms process with
> -Dproc_ (e.g. -Dproc_kms).
> This will help ops script to easily grep the process with this name.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14050) Add process name to kms process
[
https://issues.apache.org/jira/browse/HADOOP-14050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850757#comment-15850757
]
John Zhuge commented on HADOOP-14050:
-
[~shahrs87] Thanks for the contribution. Do you plan to fix trunk? In
{{hadoop-kms.sh}}.
> Add process name to kms process
> ---
>
> Key: HADOOP-14050
> URL: https://issues.apache.org/jira/browse/HADOOP-14050
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms, scripts
>Affects Versions: 2.7.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Minor
> Attachments: HADOOP-14050-branch-2.patch
>
>
> Like other hadoop daemons, we should start the kms process with
> -Dproc_ (e.g. -Dproc_kms).
> This will help ops script to easily grep the process with this name.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14028) S3A block output streams don't delete temporary files in multipart uploads
[
https://issues.apache.org/jira/browse/HADOOP-14028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850746#comment-15850746
]
Hadoop QA commented on HADOOP-14028:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
16s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} branch-2.8 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} branch-2.8 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} branch-2.8 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} branch-2.8 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 12s{color} | {color:orange} hadoop-tools/hadoop-aws: The patch generated 5
new + 26 unchanged - 2 fixed = 31 total (was 28) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
21s{color} | {color:green} hadoop-aws in the patch passed with JDK v1.7.0_121.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:5af2af1 |
| JIRA Issue | HADOOP-14028 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12850704/HADOOP-14028-branch-2.8-003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux ae17cd972161 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | branch-2.8 / 4e423ed |
| Default Java | 1.7.0_121 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_121
[jira] [Commented] (HADOOP-13998) initial s3guard preview
[ https://issues.apache.org/jira/browse/HADOOP-13998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850733#comment-15850733 ] Allen Wittenauer commented on HADOOP-13998: --- Is the s3guard documentation actually linked from the main index? > initial s3guard preview > --- > > Key: HADOOP-13998 > URL: https://issues.apache.org/jira/browse/HADOOP-13998 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Steve Loughran > > JIRA to link in all the things we think are needed for a preview/merge into > trunk -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13866) Upgrade netty-all to 4.1.1.Final
[
https://issues.apache.org/jira/browse/HADOOP-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850715#comment-15850715
]
Xiao Chen commented on HADOOP-13866:
bq. There is no code change needed for the upgrade.
Understand this patch doesn't have any code change. But we had to revert
HDFS-8377 which means the version bump potentially requires code changes for
downstream, right?
> Upgrade netty-all to 4.1.1.Final
>
>
> Key: HADOOP-13866
> URL: https://issues.apache.org/jira/browse/HADOOP-13866
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Blocker
> Attachments: HADOOP-13866.v1.patch, HADOOP-13866.v2.patch,
> HADOOP-13866.v3.patch, HADOOP-13866.v4.patch, HADOOP-13866.v6.patch,
> HADOOP-13866.v7.patch, HADOOP-13866.v8.patch, HADOOP-13866.v8.patch,
> HADOOP-13866.v8.patch
>
>
> netty-all 4.1.1.Final is stable release which we should upgrade to.
> See bottom of HADOOP-12927 for related discussion.
> This issue was discovered since hbase 2.0 uses 4.1.1.Final of netty.
> When launching mapreduce job from hbase, /grid/0/hadoop/yarn/local/
> usercache/hbase/appcache/application_1479850535804_0008/container_e01_1479850535804_0008_01_05/mr-framework/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar
> (from hdfs) is ahead of 4.1.1.Final jar (from hbase) on the classpath.
> Resulting in the following exception:
> {code}
> 2016-12-01 20:17:26,678 WARN [Default-IPC-NioEventLoopGroup-1-1]
> io.netty.util.concurrent.DefaultPromise: An exception was thrown by
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete()
> java.lang.NoSuchMethodError:
> io.netty.buffer.ByteBuf.retainedDuplicate()Lio/netty/buffer/ByteBuf;
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:272)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:262)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> at
> io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14047) Require admin to access KMS instrumentation servlets
[
https://issues.apache.org/jira/browse/HADOOP-14047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850713#comment-15850713
]
Xiao Chen commented on HADOOP-14047:
Thanks for the patch [~jzhuge]. LGTM, +1 pending figuring out about the
{{/logs}} as discussed in HDFS-10860, and possibly updating the configs/docs
here.
> Require admin to access KMS instrumentation servlets
>
>
> Key: HADOOP-14047
> URL: https://issues.apache.org/jira/browse/HADOOP-14047
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 3.0.0-alpha3
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Attachments: HADOOP-14047.001.patch, HADOOP-14047.002.patch
>
>
> Discovered during HDFS-10860 review. To require admin to access KMS
> instrumentation servlets, {{HttpServer2#setACL}} must be called. Add
> configuration property {{hadoop.httpfs.http.administrators}}, similar to
> {{dfs.cluster.administrators}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13866) Upgrade netty-all to 4.1.1.Final
[
https://issues.apache.org/jira/browse/HADOOP-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850706#comment-15850706
]
Ted Yu commented on HADOOP-13866:
-
There is no code change needed for the upgrade.
> Upgrade netty-all to 4.1.1.Final
>
>
> Key: HADOOP-13866
> URL: https://issues.apache.org/jira/browse/HADOOP-13866
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Blocker
> Attachments: HADOOP-13866.v1.patch, HADOOP-13866.v2.patch,
> HADOOP-13866.v3.patch, HADOOP-13866.v4.patch, HADOOP-13866.v6.patch,
> HADOOP-13866.v7.patch, HADOOP-13866.v8.patch, HADOOP-13866.v8.patch,
> HADOOP-13866.v8.patch
>
>
> netty-all 4.1.1.Final is stable release which we should upgrade to.
> See bottom of HADOOP-12927 for related discussion.
> This issue was discovered since hbase 2.0 uses 4.1.1.Final of netty.
> When launching mapreduce job from hbase, /grid/0/hadoop/yarn/local/
> usercache/hbase/appcache/application_1479850535804_0008/container_e01_1479850535804_0008_01_05/mr-framework/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar
> (from hdfs) is ahead of 4.1.1.Final jar (from hbase) on the classpath.
> Resulting in the following exception:
> {code}
> 2016-12-01 20:17:26,678 WARN [Default-IPC-NioEventLoopGroup-1-1]
> io.netty.util.concurrent.DefaultPromise: An exception was thrown by
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete()
> java.lang.NoSuchMethodError:
> io.netty.buffer.ByteBuf.retainedDuplicate()Lio/netty/buffer/ByteBuf;
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:272)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:262)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> at
> io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13866) Upgrade netty-all to 4.1.1.Final
[
https://issues.apache.org/jira/browse/HADOOP-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850693#comment-15850693
]
Xiao Chen edited comment on HADOOP-13866 at 2/2/17 11:02 PM:
-
Thanks [~tedyu] for the continued work and testing here.
Regarding Andrew's question:
{quote} is this an incompatible change?{quote}
It seems yet, but could you confirm?
I'm +1 on the 0.4KB patch 8 (the one without the whitespace file change) for
trunk pending this.
was (Author: xiaochen):
Thanks [~tedyu] for the continued work and testing here.
Regarding Andrew's question:
{quote} is this an incompatible change?{quote}
This version change from {{4.1.0.Beta5}} to {{4.1.1.Final}} seems didn't need a
code change. But could you confirm?
I'm +1 on the 0.4KB patch 8 (the one without the whitespace file change) for
trunk pending this.
> Upgrade netty-all to 4.1.1.Final
>
>
> Key: HADOOP-13866
> URL: https://issues.apache.org/jira/browse/HADOOP-13866
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Blocker
> Attachments: HADOOP-13866.v1.patch, HADOOP-13866.v2.patch,
> HADOOP-13866.v3.patch, HADOOP-13866.v4.patch, HADOOP-13866.v6.patch,
> HADOOP-13866.v7.patch, HADOOP-13866.v8.patch, HADOOP-13866.v8.patch,
> HADOOP-13866.v8.patch
>
>
> netty-all 4.1.1.Final is stable release which we should upgrade to.
> See bottom of HADOOP-12927 for related discussion.
> This issue was discovered since hbase 2.0 uses 4.1.1.Final of netty.
> When launching mapreduce job from hbase, /grid/0/hadoop/yarn/local/
> usercache/hbase/appcache/application_1479850535804_0008/container_e01_1479850535804_0008_01_05/mr-framework/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar
> (from hdfs) is ahead of 4.1.1.Final jar (from hbase) on the classpath.
> Resulting in the following exception:
> {code}
> 2016-12-01 20:17:26,678 WARN [Default-IPC-NioEventLoopGroup-1-1]
> io.netty.util.concurrent.DefaultPromise: An exception was thrown by
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete()
> java.lang.NoSuchMethodError:
> io.netty.buffer.ByteBuf.retainedDuplicate()Lio/netty/buffer/ByteBuf;
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:272)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:262)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> at
> io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13866) Upgrade netty-all to 4.1.1.Final
[
https://issues.apache.org/jira/browse/HADOOP-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850693#comment-15850693
]
Xiao Chen edited comment on HADOOP-13866 at 2/2/17 11:02 PM:
-
Thanks [~tedyu] for the continued work and testing here.
Regarding Andrew's question:
{quote} is this an incompatible change?{quote}
It seems yes, but could you confirm?
I'm +1 on the 0.4KB patch 8 (the one without the whitespace file change) for
trunk pending this.
was (Author: xiaochen):
Thanks [~tedyu] for the continued work and testing here.
Regarding Andrew's question:
{quote} is this an incompatible change?{quote}
It seems yet, but could you confirm?
I'm +1 on the 0.4KB patch 8 (the one without the whitespace file change) for
trunk pending this.
> Upgrade netty-all to 4.1.1.Final
>
>
> Key: HADOOP-13866
> URL: https://issues.apache.org/jira/browse/HADOOP-13866
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Blocker
> Attachments: HADOOP-13866.v1.patch, HADOOP-13866.v2.patch,
> HADOOP-13866.v3.patch, HADOOP-13866.v4.patch, HADOOP-13866.v6.patch,
> HADOOP-13866.v7.patch, HADOOP-13866.v8.patch, HADOOP-13866.v8.patch,
> HADOOP-13866.v8.patch
>
>
> netty-all 4.1.1.Final is stable release which we should upgrade to.
> See bottom of HADOOP-12927 for related discussion.
> This issue was discovered since hbase 2.0 uses 4.1.1.Final of netty.
> When launching mapreduce job from hbase, /grid/0/hadoop/yarn/local/
> usercache/hbase/appcache/application_1479850535804_0008/container_e01_1479850535804_0008_01_05/mr-framework/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar
> (from hdfs) is ahead of 4.1.1.Final jar (from hbase) on the classpath.
> Resulting in the following exception:
> {code}
> 2016-12-01 20:17:26,678 WARN [Default-IPC-NioEventLoopGroup-1-1]
> io.netty.util.concurrent.DefaultPromise: An exception was thrown by
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete()
> java.lang.NoSuchMethodError:
> io.netty.buffer.ByteBuf.retainedDuplicate()Lio/netty/buffer/ByteBuf;
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:272)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:262)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> at
> io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13866) Upgrade netty-all to 4.1.1.Final
[
https://issues.apache.org/jira/browse/HADOOP-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850693#comment-15850693
]
Xiao Chen commented on HADOOP-13866:
Thanks [~tedyu] for the continued work and testing here.
Regarding Andrew's question:
{quote} is this an incompatible change?{quote}
This version change from {{4.1.0.Beta5}} to {{4.1.1.Final}} seems didn't need a
code change. But could you confirm?
I'm +1 on the 0.4KB patch 8 (the one without the whitespace file change)
pending this.
> Upgrade netty-all to 4.1.1.Final
>
>
> Key: HADOOP-13866
> URL: https://issues.apache.org/jira/browse/HADOOP-13866
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Blocker
> Attachments: HADOOP-13866.v1.patch, HADOOP-13866.v2.patch,
> HADOOP-13866.v3.patch, HADOOP-13866.v4.patch, HADOOP-13866.v6.patch,
> HADOOP-13866.v7.patch, HADOOP-13866.v8.patch, HADOOP-13866.v8.patch,
> HADOOP-13866.v8.patch
>
>
> netty-all 4.1.1.Final is stable release which we should upgrade to.
> See bottom of HADOOP-12927 for related discussion.
> This issue was discovered since hbase 2.0 uses 4.1.1.Final of netty.
> When launching mapreduce job from hbase, /grid/0/hadoop/yarn/local/
> usercache/hbase/appcache/application_1479850535804_0008/container_e01_1479850535804_0008_01_05/mr-framework/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar
> (from hdfs) is ahead of 4.1.1.Final jar (from hbase) on the classpath.
> Resulting in the following exception:
> {code}
> 2016-12-01 20:17:26,678 WARN [Default-IPC-NioEventLoopGroup-1-1]
> io.netty.util.concurrent.DefaultPromise: An exception was thrown by
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete()
> java.lang.NoSuchMethodError:
> io.netty.buffer.ByteBuf.retainedDuplicate()Lio/netty/buffer/ByteBuf;
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:272)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:262)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> at
> io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13866) Upgrade netty-all to 4.1.1.Final
[
https://issues.apache.org/jira/browse/HADOOP-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850693#comment-15850693
]
Xiao Chen edited comment on HADOOP-13866 at 2/2/17 11:01 PM:
-
Thanks [~tedyu] for the continued work and testing here.
Regarding Andrew's question:
{quote} is this an incompatible change?{quote}
This version change from {{4.1.0.Beta5}} to {{4.1.1.Final}} seems didn't need a
code change. But could you confirm?
I'm +1 on the 0.4KB patch 8 (the one without the whitespace file change) for
trunk pending this.
was (Author: xiaochen):
Thanks [~tedyu] for the continued work and testing here.
Regarding Andrew's question:
{quote} is this an incompatible change?{quote}
This version change from {{4.1.0.Beta5}} to {{4.1.1.Final}} seems didn't need a
code change. But could you confirm?
I'm +1 on the 0.4KB patch 8 (the one without the whitespace file change)
pending this.
> Upgrade netty-all to 4.1.1.Final
>
>
> Key: HADOOP-13866
> URL: https://issues.apache.org/jira/browse/HADOOP-13866
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Blocker
> Attachments: HADOOP-13866.v1.patch, HADOOP-13866.v2.patch,
> HADOOP-13866.v3.patch, HADOOP-13866.v4.patch, HADOOP-13866.v6.patch,
> HADOOP-13866.v7.patch, HADOOP-13866.v8.patch, HADOOP-13866.v8.patch,
> HADOOP-13866.v8.patch
>
>
> netty-all 4.1.1.Final is stable release which we should upgrade to.
> See bottom of HADOOP-12927 for related discussion.
> This issue was discovered since hbase 2.0 uses 4.1.1.Final of netty.
> When launching mapreduce job from hbase, /grid/0/hadoop/yarn/local/
> usercache/hbase/appcache/application_1479850535804_0008/container_e01_1479850535804_0008_01_05/mr-framework/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar
> (from hdfs) is ahead of 4.1.1.Final jar (from hbase) on the classpath.
> Resulting in the following exception:
> {code}
> 2016-12-01 20:17:26,678 WARN [Default-IPC-NioEventLoopGroup-1-1]
> io.netty.util.concurrent.DefaultPromise: An exception was thrown by
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete()
> java.lang.NoSuchMethodError:
> io.netty.buffer.ByteBuf.retainedDuplicate()Lio/netty/buffer/ByteBuf;
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:272)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:262)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> at
> io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14044) Synchronization issue in delegation token cancel functionality
[
https://issues.apache.org/jira/browse/HADOOP-14044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850688#comment-15850688
]
Xiao Chen commented on HADOOP-14044:
Thanks for the patch [~hgadre] and the manual tests, looks pretty good to me. I
agree it's not worth to write a unit test, since that would introduce a whole
lot of test controls (or too many mocks).
Suggest to add {{synchronized}} keyword to the new
{{syncLocalCacheWithZkState}}, since we're handling {{currentTokens}} there. I
understand the current code is fine because the caller method {{cancelToken}}
is synchronized, but adding it would be more future proof.
And a super trivial nit: I think we can just name the new method
{{syncLocalCacheWithZk}}.
> Synchronization issue in delegation token cancel functionality
> --
>
> Key: HADOOP-14044
> URL: https://issues.apache.org/jira/browse/HADOOP-14044
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Hrishikesh Gadre
>Assignee: Hrishikesh Gadre
> Attachments: dt_fail.log, dt_success.log, HADOOP-14044-001.patch,
> HADOOP-14044-002.patch
>
>
> We are using Hadoop delegation token authentication functionality in Apache
> Solr. As part of the integration testing, I found following issue with the
> delegation token cancelation functionality.
> Consider a setup with 2 Solr servers (S1 and S2) which are configured to use
> delegation token functionality backed by Zookeeper. Now invoke following
> steps,
> [Step 1] Send a request to S1 to create a delegation token.
> (Delegation token DT is created successfully)
> [Step 2] Send a request to cancel DT to S2
> (DT is canceled successfully. client receives HTTP 200 response)
> [Step 3] Send a request to cancel DT to S2 again
> (DT cancelation fails. client receives HTTP 404 response)
> [Step 4] Send a request to cancel DT to S1
> At this point we get two different responses.
> - DT cancelation fails. client receives HTTP 404 response
> - DT cancelation succeeds. client receives HTTP 200 response
> Also as per the current implementation, each server maintains an in_memory
> cache of current tokens which is updated using the ZK watch mechanism. e.g.
> the ZK watch on S1 will ensure that the in_memory cache is synchronized after
> step 2.
> After investigation, I found the root cause for this behavior is due to the
> race condition between step 4 and the firing of ZK watch on S1. Whenever the
> watch fires before the step 4 - we get HTTP 404 response (as expected). When
> that is not the case - we get HTTP 200 response along with following ERROR
> message in the log,
> {noformat}
> Attempted to remove a non-existing znode /ZKDTSMTokensRoot/DT_XYZ
> {noformat}
> From client perspective, the server *should* return HTTP 404 error when the
> cancel request is sent out for an invalid token.
> Ref: Here is the relevant Solr unit test for reference,
> https://github.com/apache/lucene-solr/blob/746786636404cdb8ce505ed0ed02b8d9144ab6c4/solr/core/src/test/org/apache/solr/cloud/TestSolrCloudWithDelegationTokens.java#L285
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem
[ https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned HADOOP-13075: --- Assignee: Steve Moist > Add support for SSE-KMS and SSE-C in s3a filesystem > --- > > Key: HADOOP-13075 > URL: https://issues.apache.org/jira/browse/HADOOP-13075 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Andrew Olson >Assignee: Steve Moist > > S3 provides 3 types of server-side encryption [1], > * SSE-S3 (Amazon S3-Managed Keys) [2] > * SSE-KMS (AWS KMS-Managed Keys) [3] > * SSE-C (Customer-Provided Keys) [4] > Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 > (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5]. > With native support in aws-java-sdk already available it should be fairly > straightforward [6],[7] to support the other two types of SSE with some > additional fs.s3a configuration properties. > [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html > [2] > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html > [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html > [4] > http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html > [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html > [6] > http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java > [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem
[ https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13075: Assignee: (was: Steve Loughran) Affects Version/s: 2.8.0 Status: Patch Available (was: Open) > Add support for SSE-KMS and SSE-C in s3a filesystem > --- > > Key: HADOOP-13075 > URL: https://issues.apache.org/jira/browse/HADOOP-13075 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Andrew Olson > > S3 provides 3 types of server-side encryption [1], > * SSE-S3 (Amazon S3-Managed Keys) [2] > * SSE-KMS (AWS KMS-Managed Keys) [3] > * SSE-C (Customer-Provided Keys) [4] > Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 > (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5]. > With native support in aws-java-sdk already available it should be fairly > straightforward [6],[7] to support the other two types of SSE with some > additional fs.s3a configuration properties. > [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html > [2] > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html > [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html > [4] > http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html > [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html > [6] > http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java > [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem
[ https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13075: Status: Open (was: Patch Available) > Add support for SSE-KMS and SSE-C in s3a filesystem > --- > > Key: HADOOP-13075 > URL: https://issues.apache.org/jira/browse/HADOOP-13075 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Andrew Olson >Assignee: Steve Loughran > > S3 provides 3 types of server-side encryption [1], > * SSE-S3 (Amazon S3-Managed Keys) [2] > * SSE-KMS (AWS KMS-Managed Keys) [3] > * SSE-C (Customer-Provided Keys) [4] > Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 > (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5]. > With native support in aws-java-sdk already available it should be fairly > straightforward [6],[7] to support the other two types of SSE with some > additional fs.s3a configuration properties. > [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html > [2] > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html > [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html > [4] > http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html > [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html > [6] > http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java > [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem
[
https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850616#comment-15850616
]
ASF GitHub Bot commented on HADOOP-13075:
-
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/183#discussion_r99235609
--- Diff:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java
---
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.contract.s3a.S3AContract;
+import org.junit.Ignore;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+
+import java.io.IOException;
+import java.net.URI;
+
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.*;
+
+/**
+ * Test whether or not encryption settings propagate by choosing an invalid
+ * one. We expect the S3AFileSystem to fail to initialize.
+ */
+@Ignore
+public class ITestS3AEncryptionAlgorithmValidation
+extends AbstractS3ATestBase {
+
+ @Rule
+ public ExpectedException expectedException = ExpectedException.none();
+
+ @Test
+ public void testEncryptionAlgorithmSetToDES() throws Throwable {
+expectedException.expect(IOException.class);
+expectedException.expectMessage("Unknown Server Side algorithm DES");
+
+Configuration conf = super.createConfiguration();
+//DES is an invalid encryption algorithm
+conf.set(Constants.SERVER_SIDE_ENCRYPTION_ALGORITHM, "DES");
+S3AContract contract = (S3AContract) createContract(conf);
+contract.init();
+//skip tests if they aren't enabled
+assumeEnabled();
+//extract the test FS
+FileSystem fileSystem = contract.getTestFileSystem();
+assertNotNull("null filesystem", fileSystem);
+URI fsURI = fileSystem.getUri();
+LOG.info("Test filesystem = {} implemented by {}", fsURI, fileSystem);
+assertEquals("wrong filesystem of " + fsURI,
+contract.getScheme(), fsURI.getScheme());
+fileSystem.initialize(fsURI, conf);
+
+ }
+
+ @Test
+ public void testEncryptionAlgorithmSSECWithNoEncryptionKey() throws
+Throwable {
+expectedException.expect(IllegalArgumentException.class);
+expectedException.expectMessage("The value of property " +
+"fs.s3a.server-side-encryption-key must not be null");
+
+Configuration conf = super.createConfiguration();
+//SSE-C must be configured with an encryption key
+conf.set(Constants.SERVER_SIDE_ENCRYPTION_ALGORITHM,
S3AEncryptionMethods
+.SSE_C.getMethod());
+conf.set(Constants.SERVER_SIDE_ENCRYPTION_KEY, null);
+S3AContract contract = (S3AContract) createContract(conf);
+contract.init();
+//skip tests if they aren't enabled
+assumeEnabled();
+//extract the test FS
+FileSystem fileSystem = contract.getTestFileSystem();
+assertNotNull("null filesystem", fileSystem);
+URI fsURI = fileSystem.getUri();
+LOG.info("Test filesystem = {} implemented by {}", fsURI, fileSystem);
+assertEquals("wrong filesystem of " + fsURI,
+contract.getScheme(), fsURI.getScheme());
+fileSystem.initialize(fsURI, conf);
+ }
+
+ @Test
+ public void testEncryptionAlgorithmSSECWithBlankEncryptionKey() throws
+Throwable {
+expectedException.expect(IOException.class);
+expectedException.expectMessage("SSE-C is enabled and no " +
--- End diff --
I do like shared constants here, with the constant in the production code,
test reading it. Stops the tests being brittle to change in the message. A flaw
with
[jira] [Commented] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem
[
https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850610#comment-15850610
]
ASF GitHub Bot commented on HADOOP-13075:
-
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/183#discussion_r99235308
--- Diff:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java
---
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.contract.s3a.S3AContract;
+import org.junit.Ignore;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+
+import java.io.IOException;
+import java.net.URI;
+
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.*;
+
+/**
+ * Test whether or not encryption settings propagate by choosing an invalid
+ * one. We expect the S3AFileSystem to fail to initialize.
+ */
+@Ignore
+public class ITestS3AEncryptionAlgorithmValidation
+extends AbstractS3ATestBase {
+
+ @Rule
+ public ExpectedException expectedException = ExpectedException.none();
+
+ @Test
+ public void testEncryptionAlgorithmSetToDES() throws Throwable {
+expectedException.expect(IOException.class);
+expectedException.expectMessage("Unknown Server Side algorithm DES");
+
+Configuration conf = super.createConfiguration();
+//DES is an invalid encryption algorithm
+conf.set(Constants.SERVER_SIDE_ENCRYPTION_ALGORITHM, "DES");
+S3AContract contract = (S3AContract) createContract(conf);
+contract.init();
+//skip tests if they aren't enabled
+assumeEnabled();
+//extract the test FS
+FileSystem fileSystem = contract.getTestFileSystem();
+assertNotNull("null filesystem", fileSystem);
+URI fsURI = fileSystem.getUri();
+LOG.info("Test filesystem = {} implemented by {}", fsURI, fileSystem);
+assertEquals("wrong filesystem of " + fsURI,
+contract.getScheme(), fsURI.getScheme());
+fileSystem.initialize(fsURI, conf);
+
+ }
+
+ @Test
+ public void testEncryptionAlgorithmSSECWithNoEncryptionKey() throws
+Throwable {
+expectedException.expect(IllegalArgumentException.class);
+expectedException.expectMessage("The value of property " +
+"fs.s3a.server-side-encryption-key must not be null");
+
--- End diff --
We're generally moving towards `LambdaTestUtils.intercept()`, because if a
closure fails, intercept() will print what came back. Less important for voids
though, and on branch-2 & java7, not so compelling.
> Add support for SSE-KMS and SSE-C in s3a filesystem
> ---
>
> Key: HADOOP-13075
> URL: https://issues.apache.org/jira/browse/HADOOP-13075
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Andrew Olson
>Assignee: Steve Loughran
>
> S3 provides 3 types of server-side encryption [1],
> * SSE-S3 (Amazon S3-Managed Keys) [2]
> * SSE-KMS (AWS KMS-Managed Keys) [3]
> * SSE-C (Customer-Provided Keys) [4]
> Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3
> (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5].
> With native support in aws-java-sdk already available it should be fairly
> straightforward [6],[7] to support the other two types of SSE with some
> additional fs.s3a configuration properties.
> [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html
> [2]
> http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html
> [3]
[jira] [Updated] (HADOOP-14028) S3A block output streams don't delete temporary files in multipart uploads
[ https://issues.apache.org/jira/browse/HADOOP-14028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14028: Target Version/s: 2.8.0 > S3A block output streams don't delete temporary files in multipart uploads > -- > > Key: HADOOP-14028 > URL: https://issues.apache.org/jira/browse/HADOOP-14028 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: JDK 8 + ORC 1.3.0 + hadoop-aws 3.0.0-alpha2 >Reporter: Seth Fitzsimmons >Assignee: Steve Loughran > Attachments: HADOOP-14028-branch-2-001.patch, > HADOOP-14028-branch-2.8-002.patch, HADOOP-14028-branch-2.8-003.patch > > > I have `fs.s3a.fast.upload` enabled with 3.0.0-alpha2 (it's exactly what I > was looking for after running into the same OOM problems) and don't see it > cleaning up the disk-cached blocks. > I'm generating a ~50GB file on an instance with ~6GB free when the process > starts. My expectation is that local copies of the blocks would be deleted > after those parts finish uploading, but I'm seeing more than 15 blocks in > /tmp (and none of them have been deleted thus far). > I see that DiskBlock deletes temporary files when closed, but is it closed > after individual blocks have finished uploading or when the entire file has > been fully written to the FS (full upload completed, including all parts)? > As a temporary workaround to avoid running out of space, I'm listing files, > sorting by atime, and deleting anything older than the first 20: `ls -ut | > tail -n +21 | xargs rm` > Steve Loughran says: > > They should be deleted as soon as the upload completes; the close() call > > that the AWS httpclient makes on the input stream triggers the deletion. > > Though there aren't tests for it, as I recall. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14028) S3A block output streams don't delete temporary files in multipart uploads
[ https://issues.apache.org/jira/browse/HADOOP-14028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14028: Status: Patch Available (was: Open) > S3A block output streams don't delete temporary files in multipart uploads > -- > > Key: HADOOP-14028 > URL: https://issues.apache.org/jira/browse/HADOOP-14028 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: JDK 8 + ORC 1.3.0 + hadoop-aws 3.0.0-alpha2 >Reporter: Seth Fitzsimmons >Assignee: Steve Loughran > Attachments: HADOOP-14028-branch-2-001.patch, > HADOOP-14028-branch-2.8-002.patch, HADOOP-14028-branch-2.8-003.patch > > > I have `fs.s3a.fast.upload` enabled with 3.0.0-alpha2 (it's exactly what I > was looking for after running into the same OOM problems) and don't see it > cleaning up the disk-cached blocks. > I'm generating a ~50GB file on an instance with ~6GB free when the process > starts. My expectation is that local copies of the blocks would be deleted > after those parts finish uploading, but I'm seeing more than 15 blocks in > /tmp (and none of them have been deleted thus far). > I see that DiskBlock deletes temporary files when closed, but is it closed > after individual blocks have finished uploading or when the entire file has > been fully written to the FS (full upload completed, including all parts)? > As a temporary workaround to avoid running out of space, I'm listing files, > sorting by atime, and deleting anything older than the first 20: `ls -ut | > tail -n +21 | xargs rm` > Steve Loughran says: > > They should be deleted as soon as the upload completes; the close() call > > that the AWS httpclient makes on the input stream triggers the deletion. > > Though there aren't tests for it, as I recall. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14028) S3A block output streams don't delete temporary files in multipart uploads
[ https://issues.apache.org/jira/browse/HADOOP-14028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14028: Attachment: HADOOP-14028-branch-2.8-003.patch HADOOP-14028 Patch 003 pulls in some of the changes coming in to HADOOP-13786 just to move the put/multipart put operations into the right place, and to do more diagnostics on block operations, plus the various checkstyle complaints. Based on some feedback from the SDK team, we must explicitly close the blocks on multipart, releasing the files. We do not need to do this for the putObject call. The disk block is now modified to delete the file in its close() as well as the stream: whichever closes first wins. > S3A block output streams don't delete temporary files in multipart uploads > -- > > Key: HADOOP-14028 > URL: https://issues.apache.org/jira/browse/HADOOP-14028 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: JDK 8 + ORC 1.3.0 + hadoop-aws 3.0.0-alpha2 >Reporter: Seth Fitzsimmons >Assignee: Steve Loughran > Attachments: HADOOP-14028-branch-2-001.patch, > HADOOP-14028-branch-2.8-002.patch, HADOOP-14028-branch-2.8-003.patch > > > I have `fs.s3a.fast.upload` enabled with 3.0.0-alpha2 (it's exactly what I > was looking for after running into the same OOM problems) and don't see it > cleaning up the disk-cached blocks. > I'm generating a ~50GB file on an instance with ~6GB free when the process > starts. My expectation is that local copies of the blocks would be deleted > after those parts finish uploading, but I'm seeing more than 15 blocks in > /tmp (and none of them have been deleted thus far). > I see that DiskBlock deletes temporary files when closed, but is it closed > after individual blocks have finished uploading or when the entire file has > been fully written to the FS (full upload completed, including all parts)? > As a temporary workaround to avoid running out of space, I'm listing files, > sorting by atime, and deleting anything older than the first 20: `ls -ut | > tail -n +21 | xargs rm` > Steve Loughran says: > > They should be deleted as soon as the upload completes; the close() call > > that the AWS httpclient makes on the input stream triggers the deletion. > > Though there aren't tests for it, as I recall. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14028) S3A block output streams don't delete temporary files in multipart uploads
[ https://issues.apache.org/jira/browse/HADOOP-14028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14028: Status: Open (was: Patch Available) > S3A block output streams don't delete temporary files in multipart uploads > -- > > Key: HADOOP-14028 > URL: https://issues.apache.org/jira/browse/HADOOP-14028 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: JDK 8 + ORC 1.3.0 + hadoop-aws 3.0.0-alpha2 >Reporter: Seth Fitzsimmons >Assignee: Steve Loughran > Attachments: HADOOP-14028-branch-2-001.patch, > HADOOP-14028-branch-2.8-002.patch, HADOOP-14028-branch-2.8-003.patch > > > I have `fs.s3a.fast.upload` enabled with 3.0.0-alpha2 (it's exactly what I > was looking for after running into the same OOM problems) and don't see it > cleaning up the disk-cached blocks. > I'm generating a ~50GB file on an instance with ~6GB free when the process > starts. My expectation is that local copies of the blocks would be deleted > after those parts finish uploading, but I'm seeing more than 15 blocks in > /tmp (and none of them have been deleted thus far). > I see that DiskBlock deletes temporary files when closed, but is it closed > after individual blocks have finished uploading or when the entire file has > been fully written to the FS (full upload completed, including all parts)? > As a temporary workaround to avoid running out of space, I'm listing files, > sorting by atime, and deleting anything older than the first 20: `ls -ut | > tail -n +21 | xargs rm` > Steve Loughran says: > > They should be deleted as soon as the upload completes; the close() call > > that the AWS httpclient makes on the input stream triggers the deletion. > > Though there aren't tests for it, as I recall. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem
[
https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850588#comment-15850588
]
ASF GitHub Bot commented on HADOOP-13075:
-
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/183#discussion_r99232624
--- Diff:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java
---
@@ -98,24 +101,26 @@
*/
private long contentRangeStart;
- public S3AInputStream(String bucket,
- String key,
- long contentLength,
- AmazonS3 client,
- FileSystem.Statistics stats,
- S3AInstrumentation instrumentation,
- long readahead,
- S3AInputPolicy inputPolicy) {
-Preconditions.checkArgument(StringUtils.isNotEmpty(bucket), "No
Bucket");
-Preconditions.checkArgument(StringUtils.isNotEmpty(key), "No Key");
-Preconditions.checkArgument(contentLength >= 0 , "Negative content
length");
-this.bucket = bucket;
-this.key = key;
+ public S3AInputStream(S3ObjectAttributes s3Attributes,
+long contentLength,
+AmazonS3 client,
+FileSystem.Statistics stats,
+S3AInstrumentation instrumentation,
+long readahead,
+S3AInputPolicy inputPolicy) {
+
Preconditions.checkArgument(StringUtils.isNotEmpty(s3Attributes.getBucket()),
"No Bucket");
--- End diff --
I wouldn't adjust the indentation, because it amplifies the diff and makes
merging harder. Lines 105-110 should be the same as the original.
> Add support for SSE-KMS and SSE-C in s3a filesystem
> ---
>
> Key: HADOOP-13075
> URL: https://issues.apache.org/jira/browse/HADOOP-13075
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Andrew Olson
>Assignee: Steve Loughran
>
> S3 provides 3 types of server-side encryption [1],
> * SSE-S3 (Amazon S3-Managed Keys) [2]
> * SSE-KMS (AWS KMS-Managed Keys) [3]
> * SSE-C (Customer-Provided Keys) [4]
> Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3
> (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5].
> With native support in aws-java-sdk already available it should be fairly
> straightforward [6],[7] to support the other two types of SSE with some
> additional fs.s3a configuration properties.
> [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html
> [2]
> http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html
> [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html
> [4]
> http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html
> [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html
> [6]
> http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java
> [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13786) Add S3Guard committer for zero-rename commits to consistent S3 endpoints
[
https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850578#comment-15850578
]
Hadoop QA commented on HADOOP-13786:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
11s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
55s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
34s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
6s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
7s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
6s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
51s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
54s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 12m 18s{color}
| {color:red} root generated 3 new + 700 unchanged - 0 fixed = 703 total (was
700) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 37s{color} | {color:orange} root: The patch generated 80 new + 83 unchanged
- 4 fixed = 163 total (was 87) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
30s{color} | {color:red} hadoop-aws in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 52 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
54s{color} | {color:red} hadoop-tools/hadoop-aws generated 3 new + 0 unchanged
- 0 fixed = 3 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
25s{color} | {color:red} hadoop-aws in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 21s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
12s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
51s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}105m 22s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-tools/hadoop-aws |
| | org.apache.hadoop.fs.s3a.S3AFileSystem$WriteOperationHelper defines
equals and uses Object.hashCode() At S3AFileSystem.java:Object.hashCode() At
S3AFileSystem.java:[line 2483] |
| | Using pointer equality to compare a
java.util.ArrayList
with a FileCommitActions$CommitFileOutcome in
org.apache.hadoop.fs.s3a.commit.FileCommitActions$CommitAllFilesOutcome.add(FileCommitActions$CommitFileOutcome)
At FileCommitActions.java:a
[jira] [Commented] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem
[
https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850575#comment-15850575
]
ASF GitHub Bot commented on HADOOP-13075:
-
Github user steveloughran commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/183#discussion_r99231072
--- Diff:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
---
@@ -1788,6 +1817,83 @@ public void progressChanged(ProgressEvent
progressEvent) {
}
}
+ protected void setOptionalMultipartUploadRequestParameters(
+ InitiateMultipartUploadRequest req) {
+switch (serverSideEncryptionAlgorithm) {
+case SSE_KMS:
+ req.setSSEAwsKeyManagementParams(generateSSEAwsKeyParams());
+ break;
+case SSE_C:
+ if
(StringUtils.isNotBlank(S3AUtils.getServerSideEncryptionKey(getConf( {
--- End diff --
S3aUtils is already statically imported; no need to qualify.
> Add support for SSE-KMS and SSE-C in s3a filesystem
> ---
>
> Key: HADOOP-13075
> URL: https://issues.apache.org/jira/browse/HADOOP-13075
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Andrew Olson
>Assignee: Steve Loughran
>
> S3 provides 3 types of server-side encryption [1],
> * SSE-S3 (Amazon S3-Managed Keys) [2]
> * SSE-KMS (AWS KMS-Managed Keys) [3]
> * SSE-C (Customer-Provided Keys) [4]
> Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3
> (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5].
> With native support in aws-java-sdk already available it should be fairly
> straightforward [6],[7] to support the other two types of SSE with some
> additional fs.s3a configuration properties.
> [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html
> [2]
> http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html
> [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html
> [4]
> http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html
> [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html
> [6]
> http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java
> [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem
[ https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850570#comment-15850570 ] Steve Loughran commented on HADOOP-13075: - I was thinking of a test which submits a file with one key, and tries to read it with another, expecting a failure/the wrong data. Is that possible if a new FS instance is created for the read and the write? > Add support for SSE-KMS and SSE-C in s3a filesystem > --- > > Key: HADOOP-13075 > URL: https://issues.apache.org/jira/browse/HADOOP-13075 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Andrew Olson >Assignee: Steve Loughran > > S3 provides 3 types of server-side encryption [1], > * SSE-S3 (Amazon S3-Managed Keys) [2] > * SSE-KMS (AWS KMS-Managed Keys) [3] > * SSE-C (Customer-Provided Keys) [4] > Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 > (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5]. > With native support in aws-java-sdk already available it should be fairly > straightforward [6],[7] to support the other two types of SSE with some > additional fs.s3a configuration properties. > [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html > [2] > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html > [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html > [4] > http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html > [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html > [6] > http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java > [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14044) Synchronization issue in delegation token cancel functionality
[
https://issues.apache.org/jira/browse/HADOOP-14044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850471#comment-15850471
]
Hadoop QA commented on HADOOP-14044:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 7s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 59m 32s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ha.TestZKFailoverController |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-14044 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12850664/HADOOP-14044-002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 500b91fa3ec3 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 0914fcc |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11563/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11563/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11563/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Synchronization issue in delegation token cancel functionality
> --
>
> Key: HADOOP-14044
> URL: https://issues.apache.org/jira/browse/HADOOP-14044
> Project: Hadoop
[jira] [Commented] (HADOOP-13866) Upgrade netty-all to 4.1.1.Final
[
https://issues.apache.org/jira/browse/HADOOP-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850432#comment-15850432
]
Ted Yu commented on HADOOP-13866:
-
By swapping in netty-all-4.1.1.Final.jar to hadoop directories where
netty-3.6.2.Final.jar used to reside, I was able to run
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList without triggering
NoSuchMethodException.
> Upgrade netty-all to 4.1.1.Final
>
>
> Key: HADOOP-13866
> URL: https://issues.apache.org/jira/browse/HADOOP-13866
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Blocker
> Attachments: HADOOP-13866.v1.patch, HADOOP-13866.v2.patch,
> HADOOP-13866.v3.patch, HADOOP-13866.v4.patch, HADOOP-13866.v6.patch,
> HADOOP-13866.v7.patch, HADOOP-13866.v8.patch, HADOOP-13866.v8.patch,
> HADOOP-13866.v8.patch
>
>
> netty-all 4.1.1.Final is stable release which we should upgrade to.
> See bottom of HADOOP-12927 for related discussion.
> This issue was discovered since hbase 2.0 uses 4.1.1.Final of netty.
> When launching mapreduce job from hbase, /grid/0/hadoop/yarn/local/
> usercache/hbase/appcache/application_1479850535804_0008/container_e01_1479850535804_0008_01_05/mr-framework/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar
> (from hdfs) is ahead of 4.1.1.Final jar (from hbase) on the classpath.
> Resulting in the following exception:
> {code}
> 2016-12-01 20:17:26,678 WARN [Default-IPC-NioEventLoopGroup-1-1]
> io.netty.util.concurrent.DefaultPromise: An exception was thrown by
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete()
> java.lang.NoSuchMethodError:
> io.netty.buffer.ByteBuf.retainedDuplicate()Lio/netty/buffer/ByteBuf;
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:272)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:262)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> at
> io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14028) S3A block output streams don't delete temporary files in multipart uploads
[ https://issues.apache.org/jira/browse/HADOOP-14028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850389#comment-15850389 ] Steve Loughran commented on HADOOP-14028: - It's confirmed that the multipart part upload does not close streams; the docs will make it clearer in future. So yes, we do need to close it on a multipart commit, and no, not on a single part one. code will be updated accordingly > S3A block output streams don't delete temporary files in multipart uploads > -- > > Key: HADOOP-14028 > URL: https://issues.apache.org/jira/browse/HADOOP-14028 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: JDK 8 + ORC 1.3.0 + hadoop-aws 3.0.0-alpha2 >Reporter: Seth Fitzsimmons >Assignee: Steve Loughran > Attachments: HADOOP-14028-branch-2-001.patch, > HADOOP-14028-branch-2.8-002.patch > > > I have `fs.s3a.fast.upload` enabled with 3.0.0-alpha2 (it's exactly what I > was looking for after running into the same OOM problems) and don't see it > cleaning up the disk-cached blocks. > I'm generating a ~50GB file on an instance with ~6GB free when the process > starts. My expectation is that local copies of the blocks would be deleted > after those parts finish uploading, but I'm seeing more than 15 blocks in > /tmp (and none of them have been deleted thus far). > I see that DiskBlock deletes temporary files when closed, but is it closed > after individual blocks have finished uploading or when the entire file has > been fully written to the FS (full upload completed, including all parts)? > As a temporary workaround to avoid running out of space, I'm listing files, > sorting by atime, and deleting anything older than the first 20: `ls -ut | > tail -n +21 | xargs rm` > Steve Loughran says: > > They should be deleted as soon as the upload completes; the close() call > > that the AWS httpclient makes on the input stream triggers the deletion. > > Though there aren't tests for it, as I recall. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13786) Add S3Guard committer for zero-rename commits to consistent S3 endpoints
[ https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13786: Status: Patch Available (was: Open) > Add S3Guard committer for zero-rename commits to consistent S3 endpoints > > > Key: HADOOP-13786 > URL: https://issues.apache.org/jira/browse/HADOOP-13786 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: HADOOP-13345 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13786-HADOOP-13345-001.patch, > HADOOP-13786-HADOOP-13345-002.patch, HADOOP-13786-HADOOP-13345-003.patch, > HADOOP-13786-HADOOP-13345-004.patch > > > A goal of this code is "support O(1) commits to S3 repositories in the > presence of failures". Implement it, including whatever is needed to > demonstrate the correctness of the algorithm. (that is, assuming that s3guard > provides a consistent view of the presence/absence of blobs, show that we can > commit directly). > I consider ourselves free to expose the blobstore-ness of the s3 output > streams (ie. not visible until the close()), if we need to use that to allow > us to abort commit operations. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13786) Add S3Guard committer for zero-rename commits to consistent S3 endpoints
[
https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13786:
Attachment: HADOOP-13786-HADOOP-13345-004.patch
Patch 004
this is passing the tests in a suite derived from
{{org.apache.hadoop.mapreduce.lib.output.TestFileOutputCommitter}};still
looking at ways to simulate failure conditions and semantics of failure we want.
Essentially: once a pending commit has happened, there is no *retry*. Meaning:
when a task has committed once, it should fail from then on, which it does with
an FNFE on the task attempt dir.
Similarly you can only commit a job once, even if all the job does is delete
all child directories.
One change in this patch is the need to support pending subtrees, eg. map
output to the directory part-/index and part-/data in the destination
dir; this has been done by adding the notion of a {{__base}} path element in
the pending tree. When a {{__base}} path is a parent. the destination path is
the parent of the __pending dir, with all children under {{__base}} retained.
With each task attempt dir being
{{dest/__pending/$app/$app-attempt/$task_attempt/__base}}, this ensures that
all data created in the task working dir ends up under the destination in the
same directory tree.
issues:
* what about cleaning up __pending? Job commit?
* need to stop someone creating a path {{__base/__pending}} and so sneak in
pending stuff/get very confused. Actually, stop __pending under __pending.
> Add S3Guard committer for zero-rename commits to consistent S3 endpoints
>
>
> Key: HADOOP-13786
> URL: https://issues.apache.org/jira/browse/HADOOP-13786
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13786-HADOOP-13345-001.patch,
> HADOOP-13786-HADOOP-13345-002.patch, HADOOP-13786-HADOOP-13345-003.patch,
> HADOOP-13786-HADOOP-13345-004.patch
>
>
> A goal of this code is "support O(1) commits to S3 repositories in the
> presence of failures". Implement it, including whatever is needed to
> demonstrate the correctness of the algorithm. (that is, assuming that s3guard
> provides a consistent view of the presence/absence of blobs, show that we can
> commit directly).
> I consider ourselves free to expose the blobstore-ness of the s3 output
> streams (ie. not visible until the close()), if we need to use that to allow
> us to abort commit operations.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13786) Add S3Guard committer for zero-rename commits to consistent S3 endpoints
[ https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13786: Status: Open (was: Patch Available) > Add S3Guard committer for zero-rename commits to consistent S3 endpoints > > > Key: HADOOP-13786 > URL: https://issues.apache.org/jira/browse/HADOOP-13786 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: HADOOP-13345 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-13786-HADOOP-13345-001.patch, > HADOOP-13786-HADOOP-13345-002.patch, HADOOP-13786-HADOOP-13345-003.patch > > > A goal of this code is "support O(1) commits to S3 repositories in the > presence of failures". Implement it, including whatever is needed to > demonstrate the correctness of the algorithm. (that is, assuming that s3guard > provides a consistent view of the presence/absence of blobs, show that we can > commit directly). > I consider ourselves free to expose the blobstore-ness of the s3 output > streams (ie. not visible until the close()), if we need to use that to allow > us to abort commit operations. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14017) Integrate ADLS ACL
[
https://issues.apache.org/jira/browse/HADOOP-14017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850322#comment-15850322
]
Vishwajeet Dusane commented on HADOOP-14017:
Support for AAD user friendly name would be required to {{setfacl}} or
{{getfacl}} but not enough for integration with Hadoop U/G. Would you like to
file specific JIRA for integration with Hadoop users and groups?
> Integrate ADLS ACL
> --
>
> Key: HADOOP-14017
> URL: https://issues.apache.org/jira/browse/HADOOP-14017
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/adl
>Affects Versions: 3.0.0-alpha3
>Reporter: John Zhuge
>Assignee: John Zhuge
>
> Track the effort to integrate ADLS ACL which models after HDFS ACL. Both are
> based on POSIX ACL.
> Of course this will go too far without AuthN integration of some sort.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14044) Synchronization issue in delegation token cancel functionality
[
https://issues.apache.org/jira/browse/HADOOP-14044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hrishikesh Gadre updated HADOOP-14044:
--
Attachment: HADOOP-14044-002.patch
[~xiaochen] Thanks for the feedback. Here is a patch implementing this
approach. I verified this patch manually on a real cluster. For this I had to
disable Zookeeper watch to ensure that the inconsistency between local cache
and the ZK state can be reproduced.
Please take a look and let me have your feedback. Due to concurrency issue, I
think it would be difficult to write a unit test for this scenario.
> Synchronization issue in delegation token cancel functionality
> --
>
> Key: HADOOP-14044
> URL: https://issues.apache.org/jira/browse/HADOOP-14044
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Hrishikesh Gadre
>Assignee: Hrishikesh Gadre
> Attachments: dt_fail.log, dt_success.log, HADOOP-14044-001.patch,
> HADOOP-14044-002.patch
>
>
> We are using Hadoop delegation token authentication functionality in Apache
> Solr. As part of the integration testing, I found following issue with the
> delegation token cancelation functionality.
> Consider a setup with 2 Solr servers (S1 and S2) which are configured to use
> delegation token functionality backed by Zookeeper. Now invoke following
> steps,
> [Step 1] Send a request to S1 to create a delegation token.
> (Delegation token DT is created successfully)
> [Step 2] Send a request to cancel DT to S2
> (DT is canceled successfully. client receives HTTP 200 response)
> [Step 3] Send a request to cancel DT to S2 again
> (DT cancelation fails. client receives HTTP 404 response)
> [Step 4] Send a request to cancel DT to S1
> At this point we get two different responses.
> - DT cancelation fails. client receives HTTP 404 response
> - DT cancelation succeeds. client receives HTTP 200 response
> Also as per the current implementation, each server maintains an in_memory
> cache of current tokens which is updated using the ZK watch mechanism. e.g.
> the ZK watch on S1 will ensure that the in_memory cache is synchronized after
> step 2.
> After investigation, I found the root cause for this behavior is due to the
> race condition between step 4 and the firing of ZK watch on S1. Whenever the
> watch fires before the step 4 - we get HTTP 404 response (as expected). When
> that is not the case - we get HTTP 200 response along with following ERROR
> message in the log,
> {noformat}
> Attempted to remove a non-existing znode /ZKDTSMTokensRoot/DT_XYZ
> {noformat}
> From client perspective, the server *should* return HTTP 404 error when the
> cancel request is sent out for an invalid token.
> Ref: Here is the relevant Solr unit test for reference,
> https://github.com/apache/lucene-solr/blob/746786636404cdb8ce505ed0ed02b8d9144ab6c4/solr/core/src/test/org/apache/solr/cloud/TestSolrCloudWithDelegationTokens.java#L285
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14048) The REDO operation of AtomicRename of folder doesn't create a placeholder blob for destination folder
[ https://issues.apache.org/jira/browse/HADOOP-14048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850313#comment-15850313 ] NITIN VERMA commented on HADOOP-14048: -- Hi [~liuml07] Could you please help in reviewing this change and committing it to Hadoop Azure ? Thanks! Nitin. > The REDO operation of AtomicRename of folder doesn't create a placeholder > blob for destination folder > - > > Key: HADOOP-14048 > URL: https://issues.apache.org/jira/browse/HADOOP-14048 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/azure >Reporter: NITIN VERMA >Assignee: NITIN VERMA > Attachments: HADOOP-14048.patch > > > While doing manual testing, I realized that the crash recovery of > AtomicRename operation of a folder in AzureNativeFileSystem doesn't create a > placeholder property blob for destination folder. Due to this bug, the > destination folder can not be renamed again. > Below is how I tested this: > 1. Create a test directory as "/test/A" > 2. Create 15 block blobs in "/test/A" folder. > 3. Run "hadoop fs -mv /test/A /test/B" command and crash it as soon as > /test/A-RenamePending.json file is created. > 4. Now run "hadoop fs -lsr /test" command, which should complete the pending > rename operation (redo) as a part of crash recovery. > 5. The REDO method copies the pending files from source folder to destination > folder (by consulting A-RenamePending.json file), but it doesn't create a > 0-byte property blob for /test/B folder, which is a bug as that folder will > not be usable for many operations. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13929) ADLS connector should not check in contract-test-options.xml
[
https://issues.apache.org/jira/browse/HADOOP-13929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850298#comment-15850298
]
Vishwajeet Dusane commented on HADOOP-13929:
Thanks [~jzhuge] for clarification.
> ADLS connector should not check in contract-test-options.xml
>
>
> Key: HADOOP-13929
> URL: https://issues.apache.org/jira/browse/HADOOP-13929
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/adl, test
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: John Zhuge
> Fix For: 3.0.0-alpha3
>
> Attachments: HADOOP-13929.001.patch, HADOOP-13929.002.patch,
> HADOOP-13929.003.patch, HADOOP-13929.004.patch, HADOOP-13929.005.patch,
> HADOOP-13929.006.patch, HADOOP-13929.007.patch, HADOOP-13929.008.patch,
> HADOOP-13929.009.patch, HADOOP-13929.010.patch, HADOOP-13929.011.patch
>
>
> Should not check in the file {{contract-test-options.xml}}. Make sure the
> file is excluded by {{.gitignore}}. Make sure ADLS {{index.md}} provides a
> complete example of this file.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14050) Add process name to kms process
[
https://issues.apache.org/jira/browse/HADOOP-14050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850219#comment-15850219
]
Hadoop QA commented on HADOOP-14050:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
45s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
9s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
7s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
17s{color} | {color:green} hadoop-kms in the patch passed with JDK v1.7.0_121.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 9m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:b59b8b7 |
| JIRA Issue | HADOOP-14050 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12850655/HADOOP-14050-branch-2.patch
|
| Optional Tests | asflicense mvnsite unit shellcheck shelldocs |
| uname | Linux 712f1d74c985 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | branch-2 / c699ce7 |
| shellcheck | v0.4.5 |
| JDK v1.7.0_121 Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11562/testReport/ |
| modules | C: hadoop-common-project/hadoop-kms U:
hadoop-common-project/hadoop-kms |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11562/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add process name to kms process
> ---
>
> Key: HADOOP-14050
> URL: https://issues.apache.org/jira/browse/HADOOP-14050
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms, scripts
>Affects Versions: 2.7.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Minor
> Attachments: HADOOP-14050-branch-2.patch
>
>
> Like other hadoop daemons, we should start the kms process with
> -Dproc_ (e.g. -Dproc_kms).
> This will help ops script to easily grep the process with this name.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14041) CLI command to prune old metadata
[ https://issues.apache.org/jira/browse/HADOOP-14041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory updated HADOOP-14041: --- Attachment: HADOOP-14041-HADOOP-13345.001.patch Attaching a patch that adds prune(timestamp) to the MetadataStore interface and existing implementations, a CLI tool, and tests for all of that. prune() takes a UTC timestamp as returned by System.currentTimeMillis() and should trim everything with a modification time older than that. The CLI tool determines the timestamp by taking the current time and subtracting various lengths of time. One tricky thing is you can specify minutes with -M, and all the time ranges are in caps so that doesn't clash with -m for specifying the metastore URL. One thing that probably needs more work is what to do about directories. The local implementation will delete its record of a directory if all the files it tracks in that directory get pruned. I should at least do the equivalent for the DynamoDB implementation, but since there's been some special consideration for handling empty directories that may warrant some more thought. I know [~fabbri]'s been thinking about the nuances of empty directories - any thoughts on that? All tests pass except as currently documented in other JIRAs. I did for a time have a lot of tests fail at the assertion of type S3AFileStatus in PathMetadataDynamoDBTranslation.pathMetadataToItem. Indeed, we do have a lot of instances of FileStatus (S3AFileStatus' parent class) flying around S3Guard, so I'm surprised I don't get it consistently, but today all the tests are passing. I can't see how anything I've changed while working on this patch would impact it. So just throwing this out there in case others have seen it or have any insight. > CLI command to prune old metadata > - > > Key: HADOOP-14041 > URL: https://issues.apache.org/jira/browse/HADOOP-14041 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Sean Mackrory >Assignee: Sean Mackrory > Attachments: HADOOP-14041-HADOOP-13345.001.patch > > > Add a CLI command that allows users to specify an age at which to prune > metadata that hasn't been modified for an extended period of time. Since the > primary use-case targeted at the moment is list consistency, it would make > sense (especially when authoritative=false) to prune metadata that is > expected to have become consistent a long time ago. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14050) Add process name to kms process
[ https://issues.apache.org/jira/browse/HADOOP-14050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated HADOOP-14050: Status: Patch Available (was: Open) Applies cleanly to branch-2.7/branch-2.8. [~aw]: do you mind reviewing this tiny patch ? > Add process name to kms process > --- > > Key: HADOOP-14050 > URL: https://issues.apache.org/jira/browse/HADOOP-14050 > Project: Hadoop Common > Issue Type: Improvement > Components: kms, scripts >Affects Versions: 2.7.0 >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah >Priority: Minor > Attachments: HADOOP-14050-branch-2.patch > > > Like other hadoop daemons, we should start the kms process with > -Dproc_ (e.g. -Dproc_kms). > This will help ops script to easily grep the process with this name. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14050) Add process name to kms process
[ https://issues.apache.org/jira/browse/HADOOP-14050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated HADOOP-14050: Attachment: HADOOP-14050-branch-2.patch > Add process name to kms process > --- > > Key: HADOOP-14050 > URL: https://issues.apache.org/jira/browse/HADOOP-14050 > Project: Hadoop Common > Issue Type: Improvement > Components: kms, scripts >Affects Versions: 2.7.0 >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah >Priority: Minor > Attachments: HADOOP-14050-branch-2.patch > > > Like other hadoop daemons, we should start the kms process with > -Dproc_ (e.g. -Dproc_kms). > This will help ops script to easily grep the process with this name. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14050) Add process name to kms process
[ https://issues.apache.org/jira/browse/HADOOP-14050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850144#comment-15850144 ] Rushabh S Shah commented on HADOOP-14050: - bq. FYI, HADOOP-13597 did this for 3.0.0. Thanks! I did notice that but still we need this change for brach-2 and branch-2.8 (and maybe for branch-2.7). > Add process name to kms process > --- > > Key: HADOOP-14050 > URL: https://issues.apache.org/jira/browse/HADOOP-14050 > Project: Hadoop Common > Issue Type: Improvement > Components: kms, scripts >Affects Versions: 2.7.0 >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah >Priority: Minor > > Like other hadoop daemons, we should start the kms process with > -Dproc_ (e.g. -Dproc_kms). > This will help ops script to easily grep the process with this name. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14050) Add process name to kms process
[ https://issues.apache.org/jira/browse/HADOOP-14050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850133#comment-15850133 ] Allen Wittenauer commented on HADOOP-14050: --- FYI, HADOOP-13597 did this for 3.0.0. > Add process name to kms process > --- > > Key: HADOOP-14050 > URL: https://issues.apache.org/jira/browse/HADOOP-14050 > Project: Hadoop Common > Issue Type: Improvement > Components: kms, scripts >Affects Versions: 2.7.0 >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah >Priority: Minor > > Like other hadoop daemons, we should start the kms process with > -Dproc_ (e.g. -Dproc_kms). > This will help ops script to easily grep the process with this name. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14050) Add process name to kms process
Rushabh S Shah created HADOOP-14050: --- Summary: Add process name to kms process Key: HADOOP-14050 URL: https://issues.apache.org/jira/browse/HADOOP-14050 Project: Hadoop Common Issue Type: Improvement Components: kms, scripts Affects Versions: 2.7.0 Reporter: Rushabh S Shah Assignee: Rushabh S Shah Priority: Minor Like other hadoop daemons, we should start the kms process with -Dproc_ (e.g. -Dproc_kms). This will help ops script to easily grep the process with this name. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12760) sun.misc.Cleaner has moved to a new location in OpenJDK 9
[
https://issues.apache.org/jira/browse/HADOOP-12760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849865#comment-15849865
]
Hadoop QA commented on HADOOP-12760:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
43s{color} | {color:green} root generated 0 new + 694 unchanged - 6 fixed = 694
total (was 700) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 30s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 11 new + 131 unchanged - 0 fixed = 142 total (was 131) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 12s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 56s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.TestKDiag |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-12760 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12850623/HADOOP-12760.01.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 708a530ff995 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 3433f57 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11561/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11561/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11561/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11561/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was
[jira] [Created] (HADOOP-14049) Honour AclBit flag associated to file/folder permission for Azure datalake account
Vishwajeet Dusane created HADOOP-14049:
--
Summary: Honour AclBit flag associated to file/folder permission
for Azure datalake account
Key: HADOOP-14049
URL: https://issues.apache.org/jira/browse/HADOOP-14049
Project: Hadoop Common
Issue Type: New Feature
Components: fs/adl
Affects Versions: 3.0.0-alpha3
Reporter: Vishwajeet Dusane
ADLS persist AclBit information on a file/folder. Since Java SDK 2.1.4 - AclBit
value can be retrieved using {{DirectoryEntry.aclBit}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14049) Honour AclBit flag associated to file/folder permission for Azure datalake account
[
https://issues.apache.org/jira/browse/HADOOP-14049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vishwajeet Dusane reassigned HADOOP-14049:
--
Assignee: Vishwajeet Dusane
> Honour AclBit flag associated to file/folder permission for Azure datalake
> account
> --
>
> Key: HADOOP-14049
> URL: https://issues.apache.org/jira/browse/HADOOP-14049
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/adl
>Affects Versions: 3.0.0-alpha3
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
>
> ADLS persist AclBit information on a file/folder. Since Java SDK 2.1.4 -
> AclBit value can be retrieved using {{DirectoryEntry.aclBit}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12760) sun.misc.Cleaner has moved to a new location in OpenJDK 9
[ https://issues.apache.org/jira/browse/HADOOP-12760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-12760: --- Attachment: HADOOP-12760.01.patch Thanks Uwe. Updated the patch. > sun.misc.Cleaner has moved to a new location in OpenJDK 9 > - > > Key: HADOOP-12760 > URL: https://issues.apache.org/jira/browse/HADOOP-12760 > Project: Hadoop Common > Issue Type: Bug >Reporter: Chris Hegarty >Assignee: Akira Ajisaka >Priority: Minor > Attachments: HADOOP-12760.00.patch, HADOOP-12760.01.patch > > > This is a heads-up: there are upcoming changes in JDK 9 that will require, at > least, a small update to org.apache.hadoop.crypto.CryptoStreamUtils & > org.apache.hadoop.io.nativeio.NativeIO. > OpenJDK issue no. 8148117: "Move sun.misc.Cleaner to jdk.internal.ref" [1], > will move the Cleaner class from sun.misc to jdk.internal.ref. There is > ongoing discussion about the possibility of providing a public supported API, > maybe in the JDK 9 timeframe, for releasing NIO direct buffer native memory, > see the core-libs-dev mail thread [2]. At the very least CryptoStreamUtils & > NativeIO [3] should be updated to have knowledge of the new location of the > JDK Cleaner. > [1] https://bugs.openjdk.java.net/browse/JDK-8148117 > [2] > http://mail.openjdk.java.net/pipermail/core-libs-dev/2016-January/038243.html > [3] https://github.com/apache/hadoop/search?utf8=✓=sun.misc.Cleaner -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13998) initial s3guard preview
[ https://issues.apache.org/jira/browse/HADOOP-13998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849666#comment-15849666 ] Aaron Fabbri edited comment on HADOOP-13998 at 2/2/17 8:56 AM: --- All of the dependencies for this have patches available for review. I would like to start on empty directory handling improvements, but would prefer to merge the feature branch to trunk first to avoid having to maintain more S3AFileSystem diffs. *I'm proposing we merge* HADOOP-13345 to trunk as soon as we get the dependent JIRAs linked here committed. I'll provide a summary of where we are at below. I look forward to feedback from [~ste...@apache.org], [~cnauroth], [~eddyxu], [~mackrorysd], and the rest of the community. The main feature we want for the initial version is listing consistency, and we've accomplished that. For testing, we have completed (off the top of my head): - List consistency tests with failure injection. (HADOOP-13793) This integration test forces a delay in visibility of certain files by wrapping the AWS S3 client. It asserts listing is consistent. The test fails without S3Guard, and succeeds with it. - All existing S3 integration tests with and without S3Guard. The filesystem contract tests have been invaluable here. (HADOOP-13589 makes these very easy to run). - MetadataStore contract tests that ensure that the API semantics of the DynamoDB and in-memory reference implementations are correct. - MetadataStore scale tests that can be used to force DynamoDB service throttling and ensure we are robust to that. - Unit tests for different parts of the S3Guard logic. In addition to this upstream testing, my colleagues have run a couple of our in-house test harnesses against S3Guard. This includes Hive, Spark, and a number of other components. All the testing is looking great so far. Edit: Here is a [link to current s3guard documentation|https://github.com/apache/hadoop/blob/HADOOP-13345/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md] was (Author: fabbri): All of the dependencies for this have patches available for review. I would like to start on empty directory handling improvements, but would prefer to merge the feature branch to trunk first to avoid having to maintain more S3AFileSystem diffs. *I'm proposing we merge* HADOOP-13345 to trunk as soon as we get the dependent JIRAs linked here committed. I'll provide a summary of where we are at below. I look forward to feedback from [~ste...@apache.org], [~cnauroth], [~eddyxu], [~mackrorysd], and the rest of the community. The main feature we want for the initial version is listing consistency, and we've accomplished that. For testing, we have completed (off the top of my head): - List consistency tests with failure injection. (HADOOP-13793) This integration test forces a delay in visibility of certain files by wrapping the AWS S3 client. It asserts listing is consistent. The test fails without S3Guard, and succeeds with it. - All existing S3 integration tests with and without S3Guard. The filesystem contract tests have been invaluable here. (HADOOP-13589 makes these very easy to run). - MetadataStore contract tests that ensure that the API semantics of the DynamoDB and in-memory reference implementations are correct. - MetadataStore scale tests that can be used to force DynamoDB service throttling and ensure we are robust to that. - Unit tests for different parts of the S3Guard logic. In addition to this upstream testing, my colleagues have run a couple of our in-house test harnesses against S3Guard. This includes Hive, Spark, and a number of other components. All the testing is looking great so far. > initial s3guard preview > --- > > Key: HADOOP-13998 > URL: https://issues.apache.org/jira/browse/HADOOP-13998 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Steve Loughran > > JIRA to link in all the things we think are needed for a preview/merge into > trunk -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13998) initial s3guard preview
[ https://issues.apache.org/jira/browse/HADOOP-13998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849666#comment-15849666 ] Aaron Fabbri commented on HADOOP-13998: --- All of the dependencies for this have patches available for review. I would like to start on empty directory handling improvements, but would prefer to merge the feature branch to trunk first to avoid having to maintain more S3AFileSystem diffs. *I'm proposing we merge* HADOOP-13345 to trunk as soon as we get the dependent JIRAs linked here committed. I'll provide a summary of where we are at below. I look forward to feedback from [~ste...@apache.org], [~cnauroth], [~eddyxu], [~mackrorysd], and the rest of the community. The main feature we want for the initial version is listing consistency, and we've accomplished that. For testing, we have completed (off the top of my head): - List consistency tests with failure injection. (HADOOP-13793) This integration test forces a delay in visibility of certain files by wrapping the AWS S3 client. It asserts listing is consistent. The test fails without S3Guard, and succeeds with it. - All existing S3 integration tests with and without S3Guard. The filesystem contract tests have been invaluable here. (HADOOP-13589 makes these very easy to run). - MetadataStore contract tests that ensure that the API semantics of the DynamoDB and in-memory reference implementations are correct. - MetadataStore scale tests that can be used to force DynamoDB service throttling and ensure we are robust to that. - Unit tests for different parts of the S3Guard logic. In addition to this upstream testing, my colleagues have run a couple of our in-house test harnesses against S3Guard. This includes Hive, Spark, and a number of other components. All the testing is looking great so far. > initial s3guard preview > --- > > Key: HADOOP-13998 > URL: https://issues.apache.org/jira/browse/HADOOP-13998 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Steve Loughran > > JIRA to link in all the things we think are needed for a preview/merge into > trunk -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14013) S3Guard: fix multi-bucket integration tests
[
https://issues.apache.org/jira/browse/HADOOP-14013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Fabbri updated HADOOP-14013:
--
Attachment: HADOOP-14013-HADOOP-13345.001.patch
Attaching v1 patch, based on suggestions from [~steve_l] and [~liuml07].
Also removes example s3guard test settings from the test core-site.xml since we
have the maven profiles now.
Tested via
{noformat}
mvn clean verify -Ddynamo -Ds3guard -Dit.test=ITestS3AAWSCredentialsProvider
-Dtest=none
{noformat}
Without the patch we see the familiar failure:
{noformat}
ITestS3AAWSCredentialsProvider.testAnonymousProvider:132 » AWSServiceIO initTa..
{noformat}
With the patch it succeeds, as expected.
> S3Guard: fix multi-bucket integration tests
> ---
>
> Key: HADOOP-14013
> URL: https://issues.apache.org/jira/browse/HADOOP-14013
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Aaron Fabbri
>Assignee: Aaron Fabbri
> Attachments: HADOOP-14013-HADOOP-13345.001.patch
>
>
> Clone of HADOOP-13876 to address multi-bucket test failures separately.
> One of the issues is with accessing read-only buckets. If a user accesses a
> read-only bucket with credentials that do not have DynamoDB write
> permissions, they will get errors when trying to access the read-only bucket.
> This manifests causes test failures for {{ITestS3AAWSCredentialsProvider}}.
> Goals for this JIRA:
> - Fix {{ITestS3AAWSCredentialsProvider}} in a way that makes sense for the
> real use-case.
> - Document limitations etc. in the s3guard.md site doc.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14013) S3Guard: fix multi-bucket integration tests
[
https://issues.apache.org/jira/browse/HADOOP-14013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Fabbri updated HADOOP-14013:
--
Status: Patch Available (was: Open)
> S3Guard: fix multi-bucket integration tests
> ---
>
> Key: HADOOP-14013
> URL: https://issues.apache.org/jira/browse/HADOOP-14013
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Aaron Fabbri
>Assignee: Aaron Fabbri
> Attachments: HADOOP-14013-HADOOP-13345.001.patch
>
>
> Clone of HADOOP-13876 to address multi-bucket test failures separately.
> One of the issues is with accessing read-only buckets. If a user accesses a
> read-only bucket with credentials that do not have DynamoDB write
> permissions, they will get errors when trying to access the read-only bucket.
> This manifests causes test failures for {{ITestS3AAWSCredentialsProvider}}.
> Goals for this JIRA:
> - Fix {{ITestS3AAWSCredentialsProvider}} in a way that makes sense for the
> real use-case.
> - Document limitations etc. in the s3guard.md site doc.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14048) The REDO operation of AtomicRename of folder doesn't create a placeholder blob for destination folder
[
https://issues.apache.org/jira/browse/HADOOP-14048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849632#comment-15849632
]
Hadoop QA commented on HADOOP-14048:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
21s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-14048 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12850600/HADOOP-14048.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 68c88aef74c8 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24
10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 2a942ee |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11559/testReport/ |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11559/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> The REDO operation of AtomicRename of folder doesn't create a placeholder
> blob for destination folder
> -
>
> Key: HADOOP-14048
> URL: https://issues.apache.org/jira/browse/HADOOP-14048
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/azure
>Reporter: NITIN VERMA
>Assignee: NITIN VERMA
> Attachments:
