[jira] [Commented] (HADOOP-15419) Should not obtain delegationTokens from all namenodes when using ViewFS

[Tao Jie (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457290#comment-16457290
 ] 

Tao Jie commented on HADOOP-15419:
--

[~xkrogen] thank you for your comment. I understand your concerns. Actually I 
don't want to break any existing logic or assumptions for viewfs. We just want 
add another option for the user which somehow like a {{hint}} to viewfs which 
could help to reduce request to the cluster.  

> Should not obtain delegationTokens from all namenodes when using ViewFS
> ---
>
> Key: HADOOP-15419
> URL: https://issues.apache.org/jira/browse/HADOOP-15419
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 3.0.2
>Reporter: Tao Jie
>Priority: Major
>
> Today when submit a job to a viewfs cluster, the client will try to obtain 
> delegation token from all namenodes under the viewfs while only one namespace 
> is actually used in this job. It would create many unnecessary rpc call to 
> the whole cluster.
> In viewfs situation, we can just obtain delegation token from specific 
> namenode rather than all namenodes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15356) Make HTTP timeout configurable in ADLS Connector

[genericqa (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457255#comment-16457255
 ] 

genericqa commented on HADOOP-15356:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 28m 
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 32s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
17s{color} | {color:red} hadoop-azure-datalake in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 
16s{color} | {color:red} hadoop-azure-datalake in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 16s{color} 
| {color:red} hadoop-azure-datalake in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 10s{color} | {color:orange} hadoop-tools/hadoop-azure-datalake: The patch 
generated 7 new + 0 unchanged - 0 fixed = 7 total (was 0) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m 
17s{color} | {color:red} hadoop-azure-datalake in the patch failed. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 18s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
17s{color} | {color:red} hadoop-azure-datalake in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 19s{color} 
| {color:red} hadoop-azure-datalake in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
21s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m 19s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HADOOP-15356 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12921083/HADOOP-15356.002.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 5e56be562fd9 3.13.0-137-generic #186-Ubuntu SMP Mon Dec 4 
19:09:19 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / f469628 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| mvninstall | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/14533/artifact/out/patch-mvninstall-hadoop-tools_hadoop-azure-datalake.txt
 |
| compile | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/14533/artifact/out/patch-compile-hadoop-tools_hadoop-azure-datalake.txt
 |
| javac | 

[jira] [Commented] (HADOOP-15239) S3ABlockOutputStream.flush() be no-op when stream closed

[Aaron Fabbri (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457223#comment-16457223
 ] 

Aaron Fabbri commented on HADOOP-15239:
---

+1, LGTM. Will commit after some testing.

> S3ABlockOutputStream.flush() be no-op when stream closed
> 
>
> Key: HADOOP-15239
> URL: https://issues.apache.org/jira/browse/HADOOP-15239
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 2.9.0, 2.8.3, 2.7.5, 3.0.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Trivial
> Attachments: HADOOP-15239.001.patch, HADOOP-15239.002.patch
>
>
> when you call flush() on a closed S3A output stream, you get a stack trace. 
> This can cause problems in code with race conditions across threads, e.g. 
> FLINK-8543. 
> we could make it log@warn "stream closed" rather than raise an IOE. It's just 
> a hint, after all.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15422) s3guard doesn't init when the secrets are in the s3a URI

[Aaron Fabbri (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457217#comment-16457217
 ] 

Aaron Fabbri commented on HADOOP-15422:
---

{quote}If the AWS secrets are in the login, S3guard doesn't list the root dir
{quote}
This is your punishment for putting secrets in your URI.

I'm kidding, of course.  I will try to get to this Monday if not sooner.  Thank 
you for the patch.

> s3guard doesn't init when the secrets are in the s3a URI
> 
>
> Key: HADOOP-15422
> URL: https://issues.apache.org/jira/browse/HADOOP-15422
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-15422-001.patch
>
>
> If the AWS secrets are in the login, S3guard doesn't list the root dir



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15356) Make HTTP timeout configurable in ADLS Connector

[Sean Mackrory (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457204#comment-16457204
 ] 

Sean Mackrory commented on HADOOP-15356:


Took a stab at the tests as suggested, without the SDK upgrade. Unfortunately 
this patch would require a way to retrieve the effective timeout (this.timeout 
in the AdlStoreClient class) as it's used in the other operations, so we'll 
depend on an SDK upgrade. I tried adding it locally to the SDK and the tests 
pass - but if we could add something like this officiall, that would be great 
[~ASikaria].

> Make HTTP timeout configurable in ADLS Connector
> 
>
> Key: HADOOP-15356
> URL: https://issues.apache.org/jira/browse/HADOOP-15356
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Reporter: Atul Sikaria
>Assignee: Atul Sikaria
>Priority: Major
> Attachments: HADOOP-15356.001.patch, HADOOP-15356.002.patch
>
>
> Currently the HTTP timeout for the connections to ADLS are not configurable 
> in Hadoop. This patch enables the timeouts to be configurable based on a 
> core-site config setting. Also, up the ADLS SDK version to 2.2.8, that has 
> default value of 60 seconds - any optimizations to that setting can now be 
> done in Hadoop through core-site.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15356) Make HTTP timeout configurable in ADLS Connector

[Sean Mackrory (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Mackrory updated HADOOP-15356:
---
Attachment: HADOOP-15356.002.patch

> Make HTTP timeout configurable in ADLS Connector
> 
>
> Key: HADOOP-15356
> URL: https://issues.apache.org/jira/browse/HADOOP-15356
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Reporter: Atul Sikaria
>Assignee: Atul Sikaria
>Priority: Major
> Attachments: HADOOP-15356.001.patch, HADOOP-15356.002.patch
>
>
> Currently the HTTP timeout for the connections to ADLS are not configurable 
> in Hadoop. This patch enables the timeouts to be configurable based on a 
> core-site config setting. Also, up the ADLS SDK version to 2.2.8, that has 
> default value of 60 seconds - any optimizations to that setting can now be 
> done in Hadoop through core-site.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15422) s3guard doesn't init when the secrets are in the s3a URI

[genericqa (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457194#comment-16457194
 ] 

genericqa commented on HADOOP-15422:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 4 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  2m 
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 30m 
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 37m 
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  3m 
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
15m 49s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 26m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 26m 
35s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
3m  7s{color} | {color:orange} root: The patch generated 2 new + 24 unchanged - 
0 fixed = 26 total (was 24) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m  8s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  8m 
49s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  4m 
46s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
36s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}154m 21s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HADOOP-15422 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12921057/HADOOP-15422-001.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 6452ee71942e 3.13.0-137-generic #186-Ubuntu SMP Mon Dec 4 
19:09:19 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 92c5331 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| checkstyle | 

[jira] [Commented] (HADOOP-15418) Hadoop KMSAuthenticationFilter needs to use getPropsByPrefix instead of iterator to avoid ConcurrentModificationException

[Suma Shivaprasad (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457181#comment-16457181
 ] 

Suma Shivaprasad commented on HADOOP-15418:
---

[~lqjack] Thanks for the patch . Can you add a UT to test this change.

> Hadoop KMSAuthenticationFilter needs to use getPropsByPrefix instead of 
> iterator to avoid ConcurrentModificationException
> -
>
> Key: HADOOP-15418
> URL: https://issues.apache.org/jira/browse/HADOOP-15418
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
>Priority: Major
>
> The issue is similar to what was fixed in HADOOP-15411. Fixing this in 
> KMSAuthenticationFilter as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15426) S3guard throttle event on delete => 400 error code => exception

[Steve Loughran (JIRA)]

Steve Loughran created HADOOP-15426:
---

 Summary: S3guard throttle event on delete => 400 error code => 
exception
 Key: HADOOP-15426
 URL: https://issues.apache.org/jira/browse/HADOOP-15426
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs/s3
Affects Versions: 3.2.0
Reporter: Steve Loughran


managed to create on a parallel test run
{code}
org.apache.hadoop.fs.s3a.AWSServiceThrottledException: delete on 
s3a://hwdev-steve-ireland-new/fork-0005/test/existing-dir/existing-file: 
com.amazonaws.services.dynamodbv2.model.ProvisionedThroughputExceededException: 
The level of configured provisioned throughput for the table was exceeded. 
Consider increasing your provisioning level with the UpdateTable API. (Service: 
AmazonDynamoDBv2; Status Code: 400; Error Code: 
ProvisionedThroughputExceededException; Request ID: 
RDM3370REDBBJQ0SLCLOFC8G43VV4KQNSO5AEMVJF66Q9ASUAAJG): The level of configured 
provisioned throughput for the table was exceeded. Consider increasing your 
provisioning level with the UpdateTable API. (Service: AmazonDynamoDBv2; Status 
Code: 400; Error Code: ProvisionedThroughputExceededException; Request ID: 
RDM3370REDBBJQ0SLCLOFC8G43VV4KQNSO5AEMVJF66Q9ASUAAJG)
at 

{code}

We should be able to handle this. 400 "bad things happened" error though, not 
the 503 from S3.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15426) S3guard throttle event on delete => 400 error code => exception

[Steve Loughran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457017#comment-16457017
 ] 

Steve Loughran commented on HADOOP-15426:
-

Full stack
{code}
org.apache.hadoop.fs.s3a.AWSServiceThrottledException: delete on 
s3a://hwdev-steve-ireland-new/fork-0005/test/existing-dir/existing-file: 
com.amazonaws.services.dynamodbv2.model.ProvisionedThroughputExceededException: 
The level of configured provisioned throughput for the table was exceeded. 
Consider increasing your provisioning level with the UpdateTable API. (Service: 
AmazonDynamoDBv2; Status Code: 400; Error Code: 
ProvisionedThroughputExceededException; Request ID: 
RDM3370REDBBJQ0SLCLOFC8G43VV4KQNSO5AEMVJF66Q9ASUAAJG): The level of configured 
provisioned throughput for the table was exceeded. Consider increasing your 
provisioning level with the UpdateTable API. (Service: AmazonDynamoDBv2; Status 
Code: 400; Error Code: ProvisionedThroughputExceededException; Request ID: 
RDM3370REDBBJQ0SLCLOFC8G43VV4KQNSO5AEMVJF66Q9ASUAAJG)
at 
org.apache.hadoop.fs.s3a.S3AUtils.translateDynamoDBException(S3AUtils.java:383)
at 
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:181)
at 
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:145)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.delete(S3AFileSystem.java:1713)
at 
org.apache.hadoop.fs.s3a.ITestS3GuardEmptyDirs.testEmptyDirs(ITestS3GuardEmptyDirs.java:76)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at 
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: 
com.amazonaws.services.dynamodbv2.model.ProvisionedThroughputExceededException: 
The level of configured provisioned throughput for the table was exceeded. 
Consider increasing your provisioning level with the UpdateTable API. (Service: 
AmazonDynamoDBv2; Status Code: 400; Error Code: 
ProvisionedThroughputExceededException; Request ID: 
RDM3370REDBBJQ0SLCLOFC8G43VV4KQNSO5AEMVJF66Q9ASUAAJG)
at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1639)
at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304)
at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)
at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
at 
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
at 
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
at 
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:2925)
at 
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:2901)
at 
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.executeBatchWriteItem(AmazonDynamoDBClient.java:605)
at 
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.batchWriteItem(AmazonDynamoDBClient.java:581)
at 
com.amazonaws.services.dynamodbv2.document.internal.BatchWriteItemImpl.doBatchWriteItem(BatchWriteItemImpl.java:111)
at 
com.amazonaws.services.dynamodbv2.document.internal.BatchWriteItemImpl.batchWriteItemUnprocessed(BatchWriteItemImpl.java:64)
at 
com.amazonaws.services.dynamodbv2.document.DynamoDB.batchWriteItemUnprocessed(DynamoDB.java:189)
at 
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.processBatchWriteRequest(DynamoDBMetadataStore.java:637)
at 

[jira] [Updated] (HADOOP-15422) s3guard doesn't init when the secrets are in the s3a URI

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-15422:

Status: Patch Available  (was: Open)

Patch 001

patches up URIs before DDB uses them, so there's no mismatch in comparisons.

* new test works in the IDE
* full test run I get failures in: test local prune, assumed roles. Unsure if 
they are related

I'm having doubts about this, as in "is my CLI Setup broken", so that DDB is 
authenticating, but then the fs listing stuff is failing just due to the URI 
mismatch.

> s3guard doesn't init when the secrets are in the s3a URI
> 
>
> Key: HADOOP-15422
> URL: https://issues.apache.org/jira/browse/HADOOP-15422
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-15422-001.patch
>
>
> If the AWS secrets are in the login, S3guard doesn't list the root dir



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15422) s3guard doesn't init when the secrets are in the s3a URI

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-15422:

Attachment: HADOOP-15422-001.patch

> s3guard doesn't init when the secrets are in the s3a URI
> 
>
> Key: HADOOP-15422
> URL: https://issues.apache.org/jira/browse/HADOOP-15422
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-15422-001.patch
>
>
> If the AWS secrets are in the login, S3guard doesn't list the root dir



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15422) s3guard doesn't init when the secrets are in the s3a URI

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-15422:

Summary: s3guard doesn't init when the secrets are in the s3a URI  (was: 
s3guard doesn't list root dir when the secrets are in the s3a URI)

> s3guard doesn't init when the secrets are in the s3a URI
> 
>
> Key: HADOOP-15422
> URL: https://issues.apache.org/jira/browse/HADOOP-15422
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Priority: Minor
>
> If the AWS secrets are in the login, S3guard doesn't list the root dir



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15392) S3A Metrics in S3AInstrumentation Cause Memory Leaks in HBase Export

[Aaron Fabbri (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456971#comment-16456971
 ] 

Aaron Fabbri commented on HADOOP-15392:
---

Hey [~Krizek] thanks for working with the community on this. CDH ships a pretty 
recent version of upstream S3A.  [~mackrorysd] works with me. He did try to 
reproduce this on recent CDH distribution. I've been following the thread here 
and am stumped based on evidence so far.

> S3A Metrics in S3AInstrumentation Cause Memory Leaks in HBase Export
> 
>
> Key: HADOOP-15392
> URL: https://issues.apache.org/jira/browse/HADOOP-15392
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Voyta
>Priority: Blocker
>
> While using HBase S3A Export Snapshot utility we started to experience memory 
> leaks of the process after version upgrade.
> By running code analysis we traced the cause to revision 
> 6555af81a26b0b72ec3bee7034e01f5bd84b1564 that added the following static 
> reference (singleton):
> private static MetricsSystem metricsSystem = null;
> When application uses S3AFileSystem instance that is not closed immediately 
> metrics are accumulated in this instance and memory grows without any limit.
>  
> Expectation:
>  * It would be nice to have an option to disable metrics completely as this 
> is not needed for Export Snapshot utility.
>  * Usage of S3AFileSystem should not contain any static object that can grow 
> indefinitely.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15425) CopyFilesMapper.doCopyFile hangs with misconfigured sizeBuf

[John Doe (JIRA)]

John Doe created HADOOP-15425:
-

 Summary: CopyFilesMapper.doCopyFile hangs with misconfigured 
sizeBuf
 Key: HADOOP-15425
 URL: https://issues.apache.org/jira/browse/HADOOP-15425
 Project: Hadoop Common
  Issue Type: Bug
  Components: tools
Affects Versions: 2.5.0
Reporter: John Doe


When the sizeBuf is configured to be 0, the for loop in 
DistCpV1$CopyFilesMapper.doCopyFile function hangs endlessly.
This is because when the buf.size (i.e., sizeBuf) is 0, the bytesRead will 
always be 0 by invoking bytesRead=in.read(buf).
Here is the code snippet.

{code:java}
sizeBuf = job.getInt("copy.buf.size", 128 * 1024); //when copy.buf.size = 0
buffer = new byte[sizeBuf];

private long doCopyFile(FileStatus srcstat, Path tmpfile, Path absdst, 
Reporter reporter) throws IOException {
  FSDataInputStream in = null;
  FSDataOutputStream out = null;
  long bytesCopied = 0L;
  try {
Path srcPath = srcstat.getPath();
// open src file
in = srcPath.getFileSystem(job).open(srcPath);
reporter.incrCounter(Counter.BYTESEXPECTED, srcstat.getLen());
// open tmp file
out = create(tmpfile, reporter, srcstat);
LOG.info("Copying file " + srcPath + " of size " + srcstat.getLen() + " 
bytes...");

// copy file
for(int bytesRead; (bytesRead = in.read(buffer)) >= 0; ) {
  out.write(buffer, 0, bytesRead);
  bytesCopied += bytesRead;
  reporter.setStatus(... );
}
  } finally {
checkAndClose(in);
checkAndClose(out);
  }
  return bytesCopied;
}
{code}





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15419) Should not obtain delegationTokens from all namenodes when using ViewFS

[Erik Krogen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456916#comment-16456916
 ] 

Erik Krogen commented on HADOOP-15419:
--

[~Tao Jie], I echo [~hexiaoqiao]'s concerns. Your recent proposal sounds like 
it is defeating the point of ViewFS, which is to hide the physical location 
details of data. If you have to be aware of physical locations of all of the 
data you are reading, and set configurations accordingly, it seems to me that 
you have lost the advantages of ViewFS. If you aware you are only using a 
single physical cluster, why not just bypass ViewFS and use the URI directly..?

> Should not obtain delegationTokens from all namenodes when using ViewFS
> ---
>
> Key: HADOOP-15419
> URL: https://issues.apache.org/jira/browse/HADOOP-15419
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 3.0.2
>Reporter: Tao Jie
>Priority: Major
>
> Today when submit a job to a viewfs cluster, the client will try to obtain 
> delegation token from all namenodes under the viewfs while only one namespace 
> is actually used in this job. It would create many unnecessary rpc call to 
> the whole cluster.
> In viewfs situation, we can just obtain delegation token from specific 
> namenode rather than all namenodes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15424) XDR.ensureFreeSpace hangs when a corrupted bytebuffer passed into the constructor

[John Doe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Doe updated HADOOP-15424:
--
Description: 
When a corrupted bytebuffer passed into the constructor, i.e., the 
bytebuffer.capacity = 0, the while loop in XDR.ensureFreeSpace function hangs 
endlessly. 
This is because the loop stride (newCapacity) is always 0, making the loop 
index (newRemaining) always less than the upper bound (size).
Here is the code snippet.
{code:java}
  public XDR(ByteBuffer buf, State state) {
this.buf = buf;
this.state = state;
  }

  private void ensureFreeSpace(int size) {
Preconditions.checkState(state == State.WRITING);
if (buf.remaining() < size) {
  int newCapacity = buf.capacity() * 2;
  int newRemaining = buf.capacity() + buf.remaining();

  while (newRemaining < size) {
newRemaining += newCapacity;
newCapacity *= 2;
  }

  ByteBuffer newbuf = ByteBuffer.allocate(newCapacity);
  buf.flip();
  newbuf.put(buf);
  buf = newbuf;
}
  }
{code}
 

  was:
When a corrupted bytebuffer passed into the constructor, i.e., the 
bytebuffer.capacity = 0, the while loop in XDR.ensureFreeSpace function hangs 
endlessly. 
This is because the loop stride (newCapacity) is always 0, making the loop 
index (newRemaining) always less than the upper bound (size).
Here is the code snippet.

{code:java}
  public XDR(ByteBuffer buf, State state) {
this.buf = buf;
this.state = state;
  }

  private void ensureFreeSpace(int size) {
Preconditions.checkState(state == State.WRITING);
if (buf.remaining() < size) {
  int newCapacity = buf.capacity() * 2;
  int newRemaining = buf.capacity() + buf.remaining();

  while (newRemaining < size) {
newRemaining += newCapacity;
newCapacity *= 2;
  }

  ByteBuffer newbuf = ByteBuffer.allocate(newCapacity);
  buf.flip();
  newbuf.put(buf);
  buf = newbuf;
}
  }
{code}

 


> XDR.ensureFreeSpace hangs when a corrupted bytebuffer passed into the 
> constructor
> -
>
> Key: HADOOP-15424
> URL: https://issues.apache.org/jira/browse/HADOOP-15424
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: nfs
>Affects Versions: 2.5.0
>Reporter: John Doe
>Priority: Major
>
> When a corrupted bytebuffer passed into the constructor, i.e., the 
> bytebuffer.capacity = 0, the while loop in XDR.ensureFreeSpace function hangs 
> endlessly. 
> This is because the loop stride (newCapacity) is always 0, making the loop 
> index (newRemaining) always less than the upper bound (size).
> Here is the code snippet.
> {code:java}
>   public XDR(ByteBuffer buf, State state) {
> this.buf = buf;
> this.state = state;
>   }
>   private void ensureFreeSpace(int size) {
> Preconditions.checkState(state == State.WRITING);
> if (buf.remaining() < size) {
>   int newCapacity = buf.capacity() * 2;
>   int newRemaining = buf.capacity() + buf.remaining();
>   while (newRemaining < size) {
> newRemaining += newCapacity;
> newCapacity *= 2;
>   }
>   ByteBuffer newbuf = ByteBuffer.allocate(newCapacity);
>   buf.flip();
>   newbuf.put(buf);
>   buf = newbuf;
> }
>   }
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15424) XDR.ensureFreeSpace hangs when a corrupted bytebuffer passed into the constructor

[John Doe (JIRA)]

John Doe created HADOOP-15424:
-

 Summary: XDR.ensureFreeSpace hangs when a corrupted bytebuffer 
passed into the constructor
 Key: HADOOP-15424
 URL: https://issues.apache.org/jira/browse/HADOOP-15424
 Project: Hadoop Common
  Issue Type: Bug
  Components: nfs
Affects Versions: 2.5.0
Reporter: John Doe


When a corrupted bytebuffer passed into the constructor, i.e., the 
bytebuffer.capacity = 0, the while loop in XDR.ensureFreeSpace function hangs 
endlessly. 
This is because the loop stride (newCapacity) is always 0, making the loop 
index (newRemaining) always less than the upper bound (size).
Here is the code snippet.

{code:java}
  public XDR(ByteBuffer buf, State state) {
this.buf = buf;
this.state = state;
  }

  private void ensureFreeSpace(int size) {
Preconditions.checkState(state == State.WRITING);
if (buf.remaining() < size) {
  int newCapacity = buf.capacity() * 2;
  int newRemaining = buf.capacity() + buf.remaining();

  while (newRemaining < size) {
newRemaining += newCapacity;
newCapacity *= 2;
  }

  ByteBuffer newbuf = ByteBuffer.allocate(newCapacity);
  buf.flip();
  newbuf.put(buf);
  buf = newbuf;
}
  }
{code}

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15423) Use single hash Path -> tuple(DirListingMetadata, PathMetadata) in LocalMetadataStore

[lqjack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456846#comment-16456846
 ] 

lqjack commented on HADOOP-15423:
-

[~gabor.bota] What about provide only one map 

private LruHashMap metaMap;

interface PathMetadataable {

  boolean pathMetadata();


  PathMetadata   pathInfo();


  DirListingMetadata dirInfo();
}

> Use single hash Path -> tuple(DirListingMetadata, PathMetadata) in 
> LocalMetadataStore
> -
>
> Key: HADOOP-15423
> URL: https://issues.apache.org/jira/browse/HADOOP-15423
> Project: Hadoop Common
>  Issue Type: Sub-task
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Minor
>
> Right now the s3guard.LocalMetadataStore uses two HashMap in the 
> implementation - one for the file and one for the dir hash.
> {code:java}
>   /** Contains directories and files. */
>   private LruHashMap fileHash;
>   /** Contains directory listings. */
>   private LruHashMap dirHash;
> {code}
> It would be nice to have only one hash instead of these two for storing the 
> values. An idea for the implementation would be to have a class with nullable 
> fields:
> {code:java}
>   static class LocalMetaEntry {
> @Nullable
> public PathMetadata pathMetadata;
> @Nullable
> public DirListingMetadata dirListingMetadata;
>   }
> {code}
> or a Pair (tuple):
> {code:java}
> Pair metaEntry;
> {code}
> And only one hash/cache for these elements.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15415) copyBytes hangs when the configuration file is corrupted

[John Doe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Doe updated HADOOP-15415:
--
Affects Version/s: 2.5.0

> copyBytes hangs when the configuration file is corrupted
> 
>
> Key: HADOOP-15415
> URL: https://issues.apache.org/jira/browse/HADOOP-15415
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common
>Affects Versions: 0.23.0, 2.5.0, 3.1.0
>Reporter: John Doe
>Priority: Minor
>
> The third parameter,  buffSize, is read from the configuration files or 
> user-specified.
> When the configuration file is corrupted or the user configures with a wrong 
> value, i.e., 0, the bytesRead will always be 0, making the while loop's 
> condition always true, hanging IOUtils.copyBytes() endlessly.
> Here is the snippet of the code. There are four copyBytes in the following 
> code. The 3rd and 4th copyBytes calls the 1st one. The 1st one calls the 2nd 
> one. Hang happens in the while loop of the second copyBytes function.
>  
> {code:java}
> //1st copyBytes
>   public static void copyBytes(InputStream in, OutputStream out, int 
> buffSize, boolean close) 
> throws IOException {
> try {
>   copyBytes(in, out, buffSize);
>   if(close) {
> out.close();
> out = null;
> in.close();
> in = null;
>   }
> } finally {
>   if(close) {
> closeStream(out);
> closeStream(in);
>   }
> }
>   }
>   
> //2nd copyBytes
>   public static void copyBytes(InputStream in, OutputStream out, int 
> buffSize) 
> throws IOException {
> PrintStream ps = out instanceof PrintStream ? (PrintStream)out : null;
> byte buf[] = new byte[buffSize];
> int bytesRead = in.read(buf);
> while (bytesRead >= 0) {
>   out.write(buf, 0, bytesRead);
>   if ((ps != null) && ps.checkError()) {
> throw new IOException("Unable to write to output stream.");
>   }
>   bytesRead = in.read(buf);
> }
>   }
> //3rd copyBytes
>   public static void copyBytes(InputStream in, OutputStream out, 
> Configuration conf)
> throws IOException {
> copyBytes(in, out, conf.getInt("io.file.buffer.size", 4096), true);
>   }
>   
> //4th copyBytes
>   public static void copyBytes(InputStream in, OutputStream out, 
> Configuration conf, boolean close)
> throws IOException {
> copyBytes(in, out, conf.getInt("io.file.buffer.size", 4096),  close);
>   }
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15418) Hadoop KMSAuthenticationFilter needs to use getPropsByPrefix instead of iterator to avoid ConcurrentModificationException

[lqjack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456840#comment-16456840
 ] 

lqjack commented on HADOOP-15418:
-

https://github.com/apache/hadoop/pull/369

> Hadoop KMSAuthenticationFilter needs to use getPropsByPrefix instead of 
> iterator to avoid ConcurrentModificationException
> -
>
> Key: HADOOP-15418
> URL: https://issues.apache.org/jira/browse/HADOOP-15418
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
>Priority: Major
>
> The issue is similar to what was fixed in HADOOP-15411. Fixing this in 
> KMSAuthenticationFilter as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15418) Hadoop KMSAuthenticationFilter needs to use getPropsByPrefix instead of iterator to avoid ConcurrentModificationException

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456838#comment-16456838
 ] 

ASF GitHub Bot commented on HADOOP-15418:
-

GitHub user lqjack opened a pull request:

https://github.com/apache/hadoop/pull/369

HADOOP-15418

fix ConcurrentModifyException

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lqjack/hadoop HADOOP-15418

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hadoop/pull/369.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #369


commit 22b095f90e349eabc41ba3177d8c1b6e094dcc42
Author: lqjaclee 
Date:   2018-04-27T17:55:18Z

HADOOP-15418

fix ConcurrentModifyException




> Hadoop KMSAuthenticationFilter needs to use getPropsByPrefix instead of 
> iterator to avoid ConcurrentModificationException
> -
>
> Key: HADOOP-15418
> URL: https://issues.apache.org/jira/browse/HADOOP-15418
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
>Priority: Major
>
> The issue is similar to what was fixed in HADOOP-15411. Fixing this in 
> KMSAuthenticationFilter as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15422) s3guard doesn't list root dir when the secrets are in the s3a URI

[Steve Loughran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456826#comment-16456826
 ] 

Steve Loughran commented on HADOOP-15422:
-

fsshell doesn't print a stack on IllegalArgumentException; added a logDebug 
statement for that
{code:java}
at 
com.google.common.base.Preconditions.checkArgument(Preconditions.java:115)
at 
org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata.checkChildPath(DirListingMetadata.java:281)
at 
org.apache.hadoop.fs.s3a.s3guard.DirListingMetadata.(DirListingMetadata.java:80)
at 
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.lambda$listChildren$3(DynamoDBMetadataStore.java:523)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109)
at 
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.listChildren(DynamoDBMetadataStore.java:508)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(S3AFileSystem.java:1896)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$listStatus$9(S3AFileSystem.java:1868)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus(S3AFileSystem.java:1868)
at 
org.apache.hadoop.fs.shell.PathData.getDirectoryContents(PathData.java:269)
at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:439)
at org.apache.hadoop.fs.shell.Ls.processPathArgument(Ls.java:226)
at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:286)
at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:270)
at 
org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:120)
at org.apache.hadoop.fs.shell.Command.run(Command.java:177)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:328)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:392)
Usage: hadoop fs [generic options]

 {code}

> s3guard doesn't list root dir when the secrets are in the s3a URI
> -
>
> Key: HADOOP-15422
> URL: https://issues.apache.org/jira/browse/HADOOP-15422
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Priority: Minor
>
> If the AWS secrets are in the login, S3guard doesn't list the root dir



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15380) TestViewFileSystemLocalFileSystem#testTrashRoot leaves an unnecessary file

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456811#comment-16456811
 ] 

ASF GitHub Bot commented on HADOOP-15380:
-

GitHub user lqjack opened a pull request:

https://github.com/apache/hadoop/pull/368

HADOOP-15380

move crc file to trash if exist

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lqjack/hadoop HADOOP-15380

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hadoop/pull/368.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #368


commit 402ff637bfac298e54d90d2d539667764b2fc47c
Author: lqjaclee 
Date:   2018-04-27T17:32:04Z

HADOOP-15380

move crc file to trash if exist




> TestViewFileSystemLocalFileSystem#testTrashRoot leaves an unnecessary file
> --
>
> Key: HADOOP-15380
> URL: https://issues.apache.org/jira/browse/HADOOP-15380
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Andras Bokor
>Assignee: Andras Bokor
>Priority: Major
>
> After running
> {code}mvn test -Dtest=TestViewFileSystemLocalFileSystem#testTrashRoot
> git status{code}
> Git reports an untracked file: 
> {{hadoop-common-project/hadoop-common/.debug.log.crc}}
> It seems some cleanup issue.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15417) s3: retrieveBlock hangs when the configuration file is corrupted

[John Doe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Doe updated HADOOP-15417:
--
Component/s: common

> s3: retrieveBlock hangs when the configuration file is corrupted
> 
>
> Key: HADOOP-15417
> URL: https://issues.apache.org/jira/browse/HADOOP-15417
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common, fs/s3
>Affects Versions: 0.23.0, 2.5.0
>Reporter: John Doe
>Priority: Major
>
> The bufferSize is read from the configuration files.
> When the configuration file is corrupted, i.e.,bufferSize=0, the numRead will 
> always be 0, making the while loop's condition always true, hanging 
> Jets3tFileSystemStore.retrieveBlock() endlessly.
> Here is the snippet of the code. 
> {code:java}
>   private int bufferSize;
>   this.bufferSize = conf.getInt( 
> S3FileSystemConfigKeys.S3_STREAM_BUFFER_SIZE_KEY, 
> S3FileSystemConfigKeys.S3_STREAM_BUFFER_SIZE_DEFAULT);
>   public File retrieveBlock(Block block, long byteRangeStart)
> throws IOException {
> File fileBlock = null;
> InputStream in = null;
> OutputStream out = null;
> try {
>   fileBlock = newBackupFile();
>   in = get(blockToKey(block), byteRangeStart);
>   out = new BufferedOutputStream(new FileOutputStream(fileBlock));
>   byte[] buf = new byte[bufferSize];
>   int numRead;
>   while ((numRead = in.read(buf)) >= 0) {
> out.write(buf, 0, numRead);
>   }
>   return fileBlock;
> } catch (IOException e) {
>   ...
> } finally {
>   ...
> }
>   }
> {code}
> Similar case: 
> [Hadoop-15415|https://issues.apache.org/jira/browse/HADOOP-15415].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15380) TestViewFileSystemLocalFileSystem#testTrashRoot leaves an unnecessary file

[lqjack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456812#comment-16456812
 ] 

lqjack commented on HADOOP-15380:
-

https://github.com/apache/hadoop/pull/368

> TestViewFileSystemLocalFileSystem#testTrashRoot leaves an unnecessary file
> --
>
> Key: HADOOP-15380
> URL: https://issues.apache.org/jira/browse/HADOOP-15380
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Andras Bokor
>Assignee: Andras Bokor
>Priority: Major
>
> After running
> {code}mvn test -Dtest=TestViewFileSystemLocalFileSystem#testTrashRoot
> git status{code}
> Git reports an untracked file: 
> {{hadoop-common-project/hadoop-common/.debug.log.crc}}
> It seems some cleanup issue.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15417) s3: retrieveBlock hangs when the configuration file is corrupted

[John Doe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Doe updated HADOOP-15417:
--
Affects Version/s: 2.5.0

> s3: retrieveBlock hangs when the configuration file is corrupted
> 
>
> Key: HADOOP-15417
> URL: https://issues.apache.org/jira/browse/HADOOP-15417
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common, fs/s3
>Affects Versions: 0.23.0, 2.5.0
>Reporter: John Doe
>Priority: Major
>
> The bufferSize is read from the configuration files.
> When the configuration file is corrupted, i.e.,bufferSize=0, the numRead will 
> always be 0, making the while loop's condition always true, hanging 
> Jets3tFileSystemStore.retrieveBlock() endlessly.
> Here is the snippet of the code. 
> {code:java}
>   private int bufferSize;
>   this.bufferSize = conf.getInt( 
> S3FileSystemConfigKeys.S3_STREAM_BUFFER_SIZE_KEY, 
> S3FileSystemConfigKeys.S3_STREAM_BUFFER_SIZE_DEFAULT);
>   public File retrieveBlock(Block block, long byteRangeStart)
> throws IOException {
> File fileBlock = null;
> InputStream in = null;
> OutputStream out = null;
> try {
>   fileBlock = newBackupFile();
>   in = get(blockToKey(block), byteRangeStart);
>   out = new BufferedOutputStream(new FileOutputStream(fileBlock));
>   byte[] buf = new byte[bufferSize];
>   int numRead;
>   while ((numRead = in.read(buf)) >= 0) {
> out.write(buf, 0, numRead);
>   }
>   return fileBlock;
> } catch (IOException e) {
>   ...
> } finally {
>   ...
> }
>   }
> {code}
> Similar case: 
> [Hadoop-15415|https://issues.apache.org/jira/browse/HADOOP-15415].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15382) Log kinit output in credential renewal thread

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456809#comment-16456809
 ] 

Hudson commented on HADOOP-15382:
-

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14083 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/14083/])
HADOOP-15382. Log kinit output in credential renewal thread. Contributed 
(weichiu: rev bff3d7b0cf073ccc061db30af6d52fa4a9f21c05)
* (edit) 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java


> Log kinit output in credential renewal thread
> -
>
> Key: HADOOP-15382
> URL: https://issues.apache.org/jira/browse/HADOOP-15382
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: security
>Reporter: Wei-Chiu Chuang
>Assignee: Gabor Bota
>Priority: Minor
> Fix For: 3.2.0
>
> Attachments: HADOOP-15382.001.patch, HADOOP-15382.002.patch
>
>
> We currently run kinit command in a thread to renew kerberos credentials 
> periodically. 
> {code:java}
> Shell.execCommand(cmd, "-R");
> if (LOG.isDebugEnabled()) {
>   LOG.debug("renewed ticket");
> }
> {code}
> It seems useful to log the output of the kinit too.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15382) Log kinit output in credential renewal thread

[Wei-Chiu Chuang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HADOOP-15382:
-
   Resolution: Fixed
Fix Version/s: 3.2.0
   Status: Resolved  (was: Patch Available)

Thanks [~gabor.bota]! Pushed rev 002 in trunk.

> Log kinit output in credential renewal thread
> -
>
> Key: HADOOP-15382
> URL: https://issues.apache.org/jira/browse/HADOOP-15382
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: security
>Reporter: Wei-Chiu Chuang
>Assignee: Gabor Bota
>Priority: Minor
> Fix For: 3.2.0
>
> Attachments: HADOOP-15382.001.patch, HADOOP-15382.002.patch
>
>
> We currently run kinit command in a thread to renew kerberos credentials 
> periodically. 
> {code:java}
> Shell.execCommand(cmd, "-R");
> if (LOG.isDebugEnabled()) {
>   LOG.debug("renewed ticket");
> }
> {code}
> It seems useful to log the output of the kinit too.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15422) s3guard doesn't list root dir when the secrets are in the s3a URI

[Steve Loughran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456658#comment-16456658
 ] 

Steve Loughran commented on HADOOP-15422:
-

FWIW, comparing the s3guarded vs unguarded, when you do get a listing of a 
directory back, the unguarded one includes the secrets (bad) & when guarded 
they are stripped (good). It's an observable difference in outcome though

> s3guard doesn't list root dir when the secrets are in the s3a URI
> -
>
> Key: HADOOP-15422
> URL: https://issues.apache.org/jira/browse/HADOOP-15422
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Priority: Minor
>
> If the AWS secrets are in the login, S3guard doesn't list the root dir



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-15422) s3guard doesn't list root dir when the secrets are in the s3a URI

[Steve Loughran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456625#comment-16456625
 ] 

Steve Loughran edited comment on HADOOP-15422 at 4/27/18 3:41 PM:
--

The first stack was a spurious failure, but with the corrected credentials, an 
ls  of the root directory fails saying the child path must be a child of the 
path-with-secrets-inline

{code}
Listing table hwdev-steve-2 in region us-west-1: s3a://A..A:X..X@hwdev-steve-2/
-ls: childPath s3a://hwdev-steve-2/cloud-integration must be a child of 
s3a://A..A:X..X@hwdev-steve-2/
{code}

. Listing of child paths which exist/don't exist pass



Other commands (get, diff) work and don't include secrets in their listing

This is probably related to our work to strip out secrets from URIs. We don't 
want them in the database, or in logs

{code}


>  hadoop fs -ls "s3a://A..A:X..X@hwdev-steve-2/"

...

18/04/27 15:21:49 DEBUG s3guard.DynamoDBClientFactory: Creating DynamoDB client 
in region us-west-1
18/04/27 15:21:49 DEBUG s3guard.DynamoDBMetadataStore: Binding to table 
hwdev-steve-2
18/04/27 15:21:49 DEBUG s3a.AWSCredentialProviderList: Using credentials from 
BasicAWSCredentialsProvider
18/04/27 15:21:49 DEBUG s3guard.DynamoDBMetadataStore: Table state: 
{AttributeDefinitions: [{AttributeName: child,AttributeType: S}, 
{AttributeName: parent,AttributeType: S}],TableName: hwdev-steve-2,KeySchema: 
[{AttributeName: parent,KeyType: HASH}, {AttributeName: child,KeyType: 
RANGE}],TableStatus: ACTIVE,CreationDateTime: Wed Dec 06 14:25:57 UTC 
2017,ProvisionedThroughput: {LastIncreaseDateTime: Thu Apr 05 13:27:34 UTC 
2018,LastDecreaseDateTime: Wed Feb 28 14:11:15 UTC 2018,NumberOfDecreasesToday: 
0,ReadCapacityUnits: 30,WriteCapacityUnits: 30},TableSizeBytes: 
59156,ItemCount: 432,TableArn: 
arn:aws:dynamodb:us-west-1:980678866538:table/hwdev-steve-2,TableId: 
11eb665e-746c-4255-8cc1-6890b0bbea24,}
18/04/27 15:21:49 DEBUG s3guard.DynamoDBMetadataStore: Using existing DynamoDB 
table hwdev-steve-2 in region us-west-1 created Wed Dec 06 14:25:39 UTC 2017
18/04/27 15:21:49 DEBUG s3a.S3AFileSystem: Using metadata store 
DynamoDBMetadataStore{region=us-west-1, tableName=hwdev-steve-2}, 
authoritative=false
18/04/27 15:21:49 DEBUG s3a.S3AUtils: Value of fs.s3a.multipart.purge.age is 
86400
18/04/27 15:21:49 DEBUG s3a.S3AStorageStatistics: op_glob_status += 1  ->  1
18/04/27 15:21:49 DEBUG s3a.S3AStorageStatistics: op_get_file_status += 1  ->  1
18/04/27 15:21:49 DEBUG s3a.S3AFileSystem: Getting path status for 
s3a://A..A:X..X@hwdev-steve-2/  ()
18/04/27 15:21:49 DEBUG s3guard.DynamoDBMetadataStore: Get from table 
hwdev-steve-2 in region us-west-1: s3a://A..A:X..X@hwdev-steve-2/
18/04/27 15:21:49 DEBUG s3a.S3AFileSystem: List status for path: 
s3a://A..A:X..X@hwdev-steve-2/
18/04/27 15:21:49 DEBUG s3a.S3AStorageStatistics: op_list_status += 1  ->  1
18/04/27 15:21:49 DEBUG s3a.S3AStorageStatistics: op_get_file_status += 1  ->  2
18/04/27 15:21:49 DEBUG s3a.S3AFileSystem: Getting path status for 
s3a://A..A:X..X@hwdev-steve-2/  ()
18/04/27 15:21:49 DEBUG s3guard.DynamoDBMetadataStore: Get from table 
hwdev-steve-2 in region us-west-1: s3a://A..A:X..X@hwdev-steve-2/
18/04/27 15:21:49 DEBUG s3guard.DynamoDBMetadataStore: Listing table 
hwdev-steve-2 in region us-west-1: s3a://A..A:X..X@hwdev-steve-2/
-ls: childPath s3a://hwdev-steve-2/cloud-integration must be a child of 
s3a://A..A:X..X@hwdev-steve-2/

..
Usage: hadoop fs [generic options]
[-appendToFile  ... ]
[-cat [-ignoreCrc]  ...]
[-checksum  ...]
[-chgrp [-R] GROUP PATH...]
[-chmod [-R]  PATH...]
[-chown [-R] [OWNER][:[GROUP]] PATH...]
[-copyFromLocal [-f] [-p] [-l] [-d] [-t ]  ... 
]
[-copyToLocal [-f] [-p] [-ignoreCrc] [-crc]  ... ]

? echo $?
255

{code}

Works for a child entry which exists

{code}
[root@ctr-e138-1518143905142-264443-01-06 ~]# hadoop fs -ls 
"s3a://A..A:X..X@hwdev-steve-2/cloud-integration"
18/04/27 15:24:29 DEBUG s3a.S3AFileSystem: Initializing S3AFileSystem for 
hwdev-steve-2
18/04/27 15:24:29 DEBUG s3a.S3AUtils: Propagating entries under 
fs.s3a.bucket.hwdev-steve-2.
18/04/27 15:24:29 WARN s3native.S3xLoginHelper: The Filesystem URI contains 
login details. This is insecure and may be unsupported in future.
18/04/27 15:24:29 DEBUG s3a.S3AUtils: For URI 
s3a://hwdev-steve-2//cloud-integration, using credentials 
AWSCredentialProviderList: BasicAWSCredentialsProvider 
EnvironmentVariableCredentialsProvider 
com.amazonaws.auth.InstanceProfileCredentialsProvider@14008db3
...
18/04/27 15:24:30 DEBUG s3guard.S3Guard: Using DynamoDBMetadataStore metadata 
store for s3a filesystem
18/04/27 15:24:30 DEBUG s3guard.DynamoDBMetadataStore: Inferring DynamoDB 
region from S3 bucket: us-west-1
18/04/27 15:24:30 DEBUG s3guard.DynamoDBMetadataStore: Creating DynamoDB 

[jira] [Commented] (HADOOP-15422) s3guard doesn't list root dir when the secrets are in the s3a URI

[Steve Loughran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456625#comment-16456625
 ] 

Steve Loughran commented on HADOOP-15422:
-

The first stack was a spurious failure, but with the corrected credentials, an 
ls  of the root directory fails saying the child path must be a child of the 
path-with-secrets-inline

{code}
Listing table hwdev-steve-2 in region us-west-1: s3a://A..A:X..X@hwdev-steve-2/
-ls: childPath s3a://hwdev-steve-2/cloud-integration must be a child of 
s3a://A..A:X..X@hwdev-steve-2/
{code}

. Listing of child paths which exist/don't exist pass



Other commands (get, diff) work and don't include secrets in their listing

This is probably related to our work to strip out secrets from URIs. We don't 
want them in the database, or in logs
{code}


>  hadoop fs -ls "s3a://A..A:X..X@hwdev-steve-2/"

...

{code}
18/04/27 15:21:49 DEBUG s3guard.DynamoDBClientFactory: Creating DynamoDB client 
in region us-west-1
18/04/27 15:21:49 DEBUG s3guard.DynamoDBMetadataStore: Binding to table 
hwdev-steve-2
18/04/27 15:21:49 DEBUG s3a.AWSCredentialProviderList: Using credentials from 
BasicAWSCredentialsProvider
18/04/27 15:21:49 DEBUG s3guard.DynamoDBMetadataStore: Table state: 
{AttributeDefinitions: [{AttributeName: child,AttributeType: S}, 
{AttributeName: parent,AttributeType: S}],TableName: hwdev-steve-2,KeySchema: 
[{AttributeName: parent,KeyType: HASH}, {AttributeName: child,KeyType: 
RANGE}],TableStatus: ACTIVE,CreationDateTime: Wed Dec 06 14:25:57 UTC 
2017,ProvisionedThroughput: {LastIncreaseDateTime: Thu Apr 05 13:27:34 UTC 
2018,LastDecreaseDateTime: Wed Feb 28 14:11:15 UTC 2018,NumberOfDecreasesToday: 
0,ReadCapacityUnits: 30,WriteCapacityUnits: 30},TableSizeBytes: 
59156,ItemCount: 432,TableArn: 
arn:aws:dynamodb:us-west-1:980678866538:table/hwdev-steve-2,TableId: 
11eb665e-746c-4255-8cc1-6890b0bbea24,}
18/04/27 15:21:49 DEBUG s3guard.DynamoDBMetadataStore: Using existing DynamoDB 
table hwdev-steve-2 in region us-west-1 created Wed Dec 06 14:25:39 UTC 2017
18/04/27 15:21:49 DEBUG s3a.S3AFileSystem: Using metadata store 
DynamoDBMetadataStore{region=us-west-1, tableName=hwdev-steve-2}, 
authoritative=false
18/04/27 15:21:49 DEBUG s3a.S3AUtils: Value of fs.s3a.multipart.purge.age is 
86400
18/04/27 15:21:49 DEBUG s3a.S3AStorageStatistics: op_glob_status += 1  ->  1
18/04/27 15:21:49 DEBUG s3a.S3AStorageStatistics: op_get_file_status += 1  ->  1
18/04/27 15:21:49 DEBUG s3a.S3AFileSystem: Getting path status for 
s3a://A..A:X..X@hwdev-steve-2/  ()
18/04/27 15:21:49 DEBUG s3guard.DynamoDBMetadataStore: Get from table 
hwdev-steve-2 in region us-west-1: s3a://A..A:X..X@hwdev-steve-2/
18/04/27 15:21:49 DEBUG s3a.S3AFileSystem: List status for path: 
s3a://A..A:X..X@hwdev-steve-2/
18/04/27 15:21:49 DEBUG s3a.S3AStorageStatistics: op_list_status += 1  ->  1
18/04/27 15:21:49 DEBUG s3a.S3AStorageStatistics: op_get_file_status += 1  ->  2
18/04/27 15:21:49 DEBUG s3a.S3AFileSystem: Getting path status for 
s3a://A..A:X..X@hwdev-steve-2/  ()
18/04/27 15:21:49 DEBUG s3guard.DynamoDBMetadataStore: Get from table 
hwdev-steve-2 in region us-west-1: s3a://A..A:X..X@hwdev-steve-2/
18/04/27 15:21:49 DEBUG s3guard.DynamoDBMetadataStore: Listing table 
hwdev-steve-2 in region us-west-1: s3a://A..A:X..X@hwdev-steve-2/
-ls: childPath s3a://hwdev-steve-2/cloud-integration must be a child of 
s3a://A..A:X..X@hwdev-steve-2/

..
Usage: hadoop fs [generic options]
[-appendToFile  ... ]
[-cat [-ignoreCrc]  ...]
[-checksum  ...]
[-chgrp [-R] GROUP PATH...]
[-chmod [-R]  PATH...]
[-chown [-R] [OWNER][:[GROUP]] PATH...]
[-copyFromLocal [-f] [-p] [-l] [-d] [-t ]  ... 
]
[-copyToLocal [-f] [-p] [-ignoreCrc] [-crc]  ... ]

? echo $?
255

{code}

Works for a child entry which exists

{code}
[root@ctr-e138-1518143905142-264443-01-06 ~]# hadoop fs -ls 
"s3a://A..A:X..X@hwdev-steve-2/cloud-integration"
18/04/27 15:24:29 DEBUG s3a.S3AFileSystem: Initializing S3AFileSystem for 
hwdev-steve-2
18/04/27 15:24:29 DEBUG s3a.S3AUtils: Propagating entries under 
fs.s3a.bucket.hwdev-steve-2.
18/04/27 15:24:29 WARN s3native.S3xLoginHelper: The Filesystem URI contains 
login details. This is insecure and may be unsupported in future.
18/04/27 15:24:29 DEBUG s3a.S3AUtils: For URI 
s3a://hwdev-steve-2//cloud-integration, using credentials 
AWSCredentialProviderList: BasicAWSCredentialsProvider 
EnvironmentVariableCredentialsProvider 
com.amazonaws.auth.InstanceProfileCredentialsProvider@14008db3
...
18/04/27 15:24:30 DEBUG s3guard.S3Guard: Using DynamoDBMetadataStore metadata 
store for s3a filesystem
18/04/27 15:24:30 DEBUG s3guard.DynamoDBMetadataStore: Inferring DynamoDB 
region from S3 bucket: us-west-1
18/04/27 15:24:30 DEBUG s3guard.DynamoDBMetadataStore: Creating DynamoDB client 
class 

[jira] [Issue Comment Deleted] (HADOOP-15422) s3guard doesn't list root dir when the secrets are in the s3a URI

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-15422:

Comment: was deleted

(was: {code}
18/04/26 19:50:28 WARN s3native.S3xLoginHelper: The Filesystem URI contains 
login details. This is insecure and may be unsupported in future.
18/04/26 19:50:32 ERROR s3guard.S3Guard: Failed to instantiate metadata store 
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore defined in 
fs.s3a.metadatastore.impl: java.net.SocketTimeoutException: initTable on 
hwdev-steve-new: com.amazonaws.AmazonClientException: No AWS Credentials 
provided by BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider 
InstanceProfileCredentialsProvider : com.amazonaws.SdkClientException: Unable 
to load credentials from service endpoint
java.net.SocketTimeoutException: initTable on hwdev-steve-ireland: 
com.amazonaws.AmazonClientException: No AWS Credentials provided by 
BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider 
InstanceProfileCredentialsProvider : com.amazonaws.SdkClientException: Unable 
to load credentials from service endpoint
{code})

> s3guard doesn't list root dir when the secrets are in the s3a URI
> -
>
> Key: HADOOP-15422
> URL: https://issues.apache.org/jira/browse/HADOOP-15422
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Priority: Minor
>
> If the AWS secrets are in the login, S3guard doesn't list the root dir



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15422) s3guard doesn't list root dir when the secrets are in the s3a URI

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-15422:

Description: If the AWS secrets are in the login, S3guard doesn't list the 
root dir  (was: If the AWS secrets are in the login, S3guard doesn't init. 
Presumably this is related to the credential chain setup
{code})

> s3guard doesn't list root dir when the secrets are in the s3a URI
> -
>
> Key: HADOOP-15422
> URL: https://issues.apache.org/jira/browse/HADOOP-15422
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Priority: Minor
>
> If the AWS secrets are in the login, S3guard doesn't list the root dir



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15422) s3guard doesn't list root dir when the secrets are in the s3a URI

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-15422:

Summary: s3guard doesn't list root dir when the secrets are in the s3a URI  
(was: s3guard doesnt init when the secrets are in the s3a URI)

> s3guard doesn't list root dir when the secrets are in the s3a URI
> -
>
> Key: HADOOP-15422
> URL: https://issues.apache.org/jira/browse/HADOOP-15422
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Priority: Minor
>
> If the AWS secrets are in the login, S3guard doesn't init. Presumably this is 
> related to the credential chain setup
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15423) Use single hash Path -> tuple(DirListingMetadata, PathMetadata) in LocalMetadataStore

[Gabor Bota (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gabor Bota updated HADOOP-15423:

Description: 
Right now the s3guard.LocalMetadataStore uses two HashMap in the implementation 
- one for the file and one for the dir hash.
{code:java}
  /** Contains directories and files. */
  private LruHashMap fileHash;

  /** Contains directory listings. */
  private LruHashMap dirHash;
{code}

It would be nice to have only one hash instead of these two for storing the 
values. An idea for the implementation would be to have a class with nullable 
fields:

{code:java}
  static class LocalMetaEntry {
@Nullable
public PathMetadata pathMetadata;
@Nullable
public DirListingMetadata dirListingMetadata;
  }
{code}

or a Pair (tuple):

{code:java}
Pair metaEntry;
{code}

And only one hash/cache for these elements.

  was:
Right now the s3guard.LocalMetadataStore uses two HashMap in the implementation 
one for the file and one for the dir hash.
{code:java}
  /** Contains directories and files. */
  private LruHashMap fileHash;

  /** Contains directory listings. */
  private LruHashMap dirHash;
{code}

It would be nice to have only one hash instead of these two for storing the 
values. An idea for the implementation would be to have a class with nullable 
fields:

{code:java}
  static class LocalMetaEntry {
@Nullable
public PathMetadata pathMetadata;
@Nullable
public DirListingMetadata dirListingMetadata;
  }
{code}

or a Pair (tuple):

{code:java}
Pair metaEntry;
{code}

And only one hash/cache for these elements.


> Use single hash Path -> tuple(DirListingMetadata, PathMetadata) in 
> LocalMetadataStore
> -
>
> Key: HADOOP-15423
> URL: https://issues.apache.org/jira/browse/HADOOP-15423
> Project: Hadoop Common
>  Issue Type: Sub-task
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Minor
>
> Right now the s3guard.LocalMetadataStore uses two HashMap in the 
> implementation - one for the file and one for the dir hash.
> {code:java}
>   /** Contains directories and files. */
>   private LruHashMap fileHash;
>   /** Contains directory listings. */
>   private LruHashMap dirHash;
> {code}
> It would be nice to have only one hash instead of these two for storing the 
> values. An idea for the implementation would be to have a class with nullable 
> fields:
> {code:java}
>   static class LocalMetaEntry {
> @Nullable
> public PathMetadata pathMetadata;
> @Nullable
> public DirListingMetadata dirListingMetadata;
>   }
> {code}
> or a Pair (tuple):
> {code:java}
> Pair metaEntry;
> {code}
> And only one hash/cache for these elements.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-13756) LocalMetadataStore#put(DirListingMetadata) should also put file metadata into fileHash.

[Gabor Bota (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456485#comment-16456485
 ] 

Gabor Bota edited comment on HADOOP-13756 at 4/27/18 2:01 PM:
--

Thanks [~fabbri]. I've created 
https://issues.apache.org/jira/browse/HADOOP-15423 for the Path -> 
tuple(DirListingMetadata, PathMetadata) change.


was (Author: gabor.bota):
Thanks [~fabbri]. I've issed https://issues.apache.org/jira/browse/HADOOP-15423 
for the Path -> tuple(DirListingMetadata, PathMetadata) change.

> LocalMetadataStore#put(DirListingMetadata) should also put file metadata into 
> fileHash.
> ---
>
> Key: HADOOP-13756
> URL: https://issues.apache.org/jira/browse/HADOOP-13756
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3, test
>Affects Versions: 3.0.0-beta1
>Reporter: Lei (Eddy) Xu
>Assignee: Gabor Bota
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HADOOP-13756.001.patch
>
>
> {{LocalMetadataStore#put(DirListingMetadata)}} only puts the metadata into 
> {{dirHash}}, thus all {{FileStatus}} s are missing from 
> {{LocalMedataStore#fileHash()}}, which makes it confuse to use.
> So in the current way, to correctly put file status into the store (and also 
> set {{authoriative}} flag), you need to run  {code}
> List metas = new ArrayList();
> boolean authorizative = true;
> for (S3AFileStatus status : files) {
>PathMetadata meta = new PathMetadata(status);
>store.put(meta);
> }
> DirListingMetadata dirMeta = new DirMeta(parent, metas, authorizative);
> store.put(dirMeta);
> {code}
> Since solely calling {{store.put(dirMeta)}} is not correct, and calling 
> {{store.put(dirMeta);}} after putting all sub-file {{FileStatuss}} does the 
> repetitive jobs. Can we just use a {{put(PathMetadata)}} and a 
> {{get/setAuthorative()}}   in the MetadataStore interface instead?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13756) LocalMetadataStore#put(DirListingMetadata) should also put file metadata into fileHash.

[Gabor Bota (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456485#comment-16456485
 ] 

Gabor Bota commented on HADOOP-13756:
-

Thanks [~fabbri]. I've issed https://issues.apache.org/jira/browse/HADOOP-15423 
for the Path -> tuple(DirListingMetadata, PathMetadata) change.

> LocalMetadataStore#put(DirListingMetadata) should also put file metadata into 
> fileHash.
> ---
>
> Key: HADOOP-13756
> URL: https://issues.apache.org/jira/browse/HADOOP-13756
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3, test
>Affects Versions: 3.0.0-beta1
>Reporter: Lei (Eddy) Xu
>Assignee: Gabor Bota
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HADOOP-13756.001.patch
>
>
> {{LocalMetadataStore#put(DirListingMetadata)}} only puts the metadata into 
> {{dirHash}}, thus all {{FileStatus}} s are missing from 
> {{LocalMedataStore#fileHash()}}, which makes it confuse to use.
> So in the current way, to correctly put file status into the store (and also 
> set {{authoriative}} flag), you need to run  {code}
> List metas = new ArrayList();
> boolean authorizative = true;
> for (S3AFileStatus status : files) {
>PathMetadata meta = new PathMetadata(status);
>store.put(meta);
> }
> DirListingMetadata dirMeta = new DirMeta(parent, metas, authorizative);
> store.put(dirMeta);
> {code}
> Since solely calling {{store.put(dirMeta)}} is not correct, and calling 
> {{store.put(dirMeta);}} after putting all sub-file {{FileStatuss}} does the 
> repetitive jobs. Can we just use a {{put(PathMetadata)}} and a 
> {{get/setAuthorative()}}   in the MetadataStore interface instead?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15423) Use single hash Path -> tuple(DirListingMetadata, PathMetadata) in LocalMetadataStore

[Gabor Bota (JIRA)]

Gabor Bota created HADOOP-15423:
---

 Summary: Use single hash Path -> tuple(DirListingMetadata, 
PathMetadata) in LocalMetadataStore
 Key: HADOOP-15423
 URL: https://issues.apache.org/jira/browse/HADOOP-15423
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Gabor Bota
Assignee: Gabor Bota


Right now the s3guard.LocalMetadataStore uses two HashMap in the implementation 
one for the file and one for the dir hash.
{code:java}
  /** Contains directories and files. */
  private LruHashMap fileHash;

  /** Contains directory listings. */
  private LruHashMap dirHash;
{code}

It would be nice to have only one hash instead of these two for storing the 
values. An idea for the implementation would be to have a class with nullable 
fields:

{code:java}
  static class LocalMetaEntry {
@Nullable
public PathMetadata pathMetadata;
@Nullable
public DirListingMetadata dirListingMetadata;
  }
{code}

or a Pair (tuple):

{code:java}
Pair metaEntry;
{code}

And only one hash/cache for these elements.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15361) RawLocalFileSystem should use Java nio framework for rename

[genericqa (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456365#comment-16456365
 ] 

genericqa commented on HADOOP-15361:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 4 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m 
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m 
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 58s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
55s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 27m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
49s{color} | {color:green} hadoop-common-project/hadoop-common: The patch 
generated 0 new + 67 unchanged - 2 fixed = 67 total (was 69) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m  3s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  8m 40s{color} 
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
35s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}121m  7s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.viewfs.TestViewFsTrash |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HADOOP-15361 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12920997/HADOOP-15361.04.patch 
|
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux d89e6f574a20 3.13.0-137-generic #186-Ubuntu SMP Mon Dec 4 
19:09:19 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 84ecfe3 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| unit | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/14531/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/14531/testReport/ |
| Max. process+thread count | 1399 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U: 
hadoop-common-project/hadoop-common |
| Console output | 

[jira] [Updated] (HADOOP-15416) s3guard diff assert failure if source path not found

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-15416:

Description: Got an illegal argument exception trying to do a s3guard diff 
in a test run. Underlying cause: directory in supplied s3a path didn't exist  
(was: Got an illegal argument exception trying to do a s3guard diff in a test 
run. Works OK on the command line (now))

> s3guard diff assert failure if source path not found
> 
>
> Key: HADOOP-15416
> URL: https://issues.apache.org/jira/browse/HADOOP-15416
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.1.0
> Environment: s3a with fault injection turned on
>Reporter: Steve Loughran
>Priority: Minor
>
> Got an illegal argument exception trying to do a s3guard diff in a test run. 
> Underlying cause: directory in supplied s3a path didn't exist



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15416) s3guard diff assert failure if source path not found

[Steve Loughran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456287#comment-16456287
 ] 

Steve Loughran commented on HADOOP-15416:
-

triggers if the source path idoesn't exist

{code}
hadoop s3guard diff s3a://hwdev-steve-new/no-such-dir
18/04/27 12:18:24 INFO s3guard.S3GuardTool: Metadata store 
DynamoDBMetadataStore{region=us-west-1, tableName=hwdev-steve-new} is 
initialized.
java.lang.IllegalArgumentException
at 
com.google.common.base.Preconditions.checkArgument(Preconditions.java:72)
at 
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$Diff.compareDir(S3GuardTool.java:808)
at 
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$Diff.compareRoot(S3GuardTool.java:868)
at 
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$Diff.run(S3GuardTool.java:890)
at 
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:350)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at 
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:1477)
at 
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.main(S3GuardTool.java:1486)
18/04/27 12:18:24 INFO util.ExitUtil: Exiting with status -1: 
java.lang.IllegalArgumentException
{code}

works OK if you list a bucket with no trailing /: {{hadoop s3guard diff 
s3a://hwdev-steve-new}}

> s3guard diff assert failure if source path not found
> 
>
> Key: HADOOP-15416
> URL: https://issues.apache.org/jira/browse/HADOOP-15416
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.1.0
> Environment: s3a with fault injection turned on
>Reporter: Steve Loughran
>Priority: Minor
>
> Got an illegal argument exception trying to do a s3guard diff in a test run. 
> Works OK on the command line (now)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15416) s3guard diff assert failure if source path not found

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-15416:

Summary: s3guard diff assert failure if source path not found  (was: 
s3guard diff assert failure)

> s3guard diff assert failure if source path not found
> 
>
> Key: HADOOP-15416
> URL: https://issues.apache.org/jira/browse/HADOOP-15416
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.1.0
> Environment: s3a with fault injection turned on
>Reporter: Steve Loughran
>Priority: Minor
>
> Got an illegal argument exception trying to do a s3guard diff in a test run. 
> Works OK on the command line (now)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15361) RawLocalFileSystem should use Java nio framework for rename

[Andras Bokor (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456240#comment-16456240
 ] 

Andras Bokor commented on HADOOP-15361:
---

Reattaching patch 03. I cannot reproduce the failing UT and logs are no longer 
available on Jenkins.

> RawLocalFileSystem should use Java nio framework for rename
> ---
>
> Key: HADOOP-15361
> URL: https://issues.apache.org/jira/browse/HADOOP-15361
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Andras Bokor
>Assignee: Andras Bokor
>Priority: Major
>  Labels: incompatibleChange
> Attachments: HADOOP-15361.01.patch, HADOOP-15361.02.patch, 
> HADOOP-15361.03.patch, HADOOP-15361.04.patch
>
>
> Currently RawLocalFileSystem uses a fallback logic for cross-volume renames. 
> The fallback logic is a copy-on-fail logic so when rename fails it copies the 
> source then delete it.
>  An additional fallback logic was needed for Windows to provide POSIX rename 
> behavior.
> Due to the fallback logic RawLocalFileSystem does not pass the contract tests 
> (HADOOP-13082).
> With using Java nio framework both could be eliminated since it is not 
> platform dependent and provides cross-volume rename.
> In addition the fallback logic for Windows is not correct since Java io 
> overrides the destination only if the source is also a directory but 
> handleEmptyDstDirectoryOnWindows method checks only the destination. That 
> means rename allows to override a directory with a file on Windows but not on 
> Unix.
> File#renameTo and Files#move are not 100% compatible:
>  If the source is a directory and the destination is an empty directory 
> File#renameTo overrides the source but Files#move is does not. We have to use 
> {{StandardCopyOption.REPLACE_EXISTING}} but it overrides the destination even 
> if the source or the destination is a file. So to make them compatible we 
> have to check that the either the source or the destination is a directory 
> before we add the copy option.
> I think the correct strategy is
>  * Where the contract test passed so far it should pass after this
>  * Where the contract test failed because of Java specific think and not 
> because of the fallback logic we should keep the original behavior.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15361) RawLocalFileSystem should use Java nio framework for rename

[Andras Bokor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Bokor updated HADOOP-15361:
--
Attachment: HADOOP-15361.04.patch

> RawLocalFileSystem should use Java nio framework for rename
> ---
>
> Key: HADOOP-15361
> URL: https://issues.apache.org/jira/browse/HADOOP-15361
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Andras Bokor
>Assignee: Andras Bokor
>Priority: Major
>  Labels: incompatibleChange
> Attachments: HADOOP-15361.01.patch, HADOOP-15361.02.patch, 
> HADOOP-15361.03.patch, HADOOP-15361.04.patch
>
>
> Currently RawLocalFileSystem uses a fallback logic for cross-volume renames. 
> The fallback logic is a copy-on-fail logic so when rename fails it copies the 
> source then delete it.
>  An additional fallback logic was needed for Windows to provide POSIX rename 
> behavior.
> Due to the fallback logic RawLocalFileSystem does not pass the contract tests 
> (HADOOP-13082).
> With using Java nio framework both could be eliminated since it is not 
> platform dependent and provides cross-volume rename.
> In addition the fallback logic for Windows is not correct since Java io 
> overrides the destination only if the source is also a directory but 
> handleEmptyDstDirectoryOnWindows method checks only the destination. That 
> means rename allows to override a directory with a file on Windows but not on 
> Unix.
> File#renameTo and Files#move are not 100% compatible:
>  If the source is a directory and the destination is an empty directory 
> File#renameTo overrides the source but Files#move is does not. We have to use 
> {{StandardCopyOption.REPLACE_EXISTING}} but it overrides the destination even 
> if the source or the destination is a file. So to make them compatible we 
> have to check that the either the source or the destination is a directory 
> before we add the copy option.
> I think the correct strategy is
>  * Where the contract test passed so far it should pass after this
>  * Where the contract test failed because of Java specific think and not 
> because of the fallback logic we should keep the original behavior.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15422) s3guard doesnt init when the secrets are in the s3a URI

[Steve Loughran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456232#comment-16456232
 ] 

Steve Loughran commented on HADOOP-15422:
-

{code}
18/04/26 19:50:28 WARN s3native.S3xLoginHelper: The Filesystem URI contains 
login details. This is insecure and may be unsupported in future.
18/04/26 19:50:32 ERROR s3guard.S3Guard: Failed to instantiate metadata store 
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore defined in 
fs.s3a.metadatastore.impl: java.net.SocketTimeoutException: initTable on 
hwdev-steve-new: com.amazonaws.AmazonClientException: No AWS Credentials 
provided by BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider 
InstanceProfileCredentialsProvider : com.amazonaws.SdkClientException: Unable 
to load credentials from service endpoint
java.net.SocketTimeoutException: initTable on hwdev-steve-ireland: 
com.amazonaws.AmazonClientException: No AWS Credentials provided by 
BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider 
InstanceProfileCredentialsProvider : com.amazonaws.SdkClientException: Unable 
to load credentials from service endpoint
{code}

> s3guard doesnt init when the secrets are in the s3a URI
> ---
>
> Key: HADOOP-15422
> URL: https://issues.apache.org/jira/browse/HADOOP-15422
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Priority: Minor
>
> If the AWS secrets are in the login, S3guard doesn't init. Presumably this is 
> related to the credential chain setup
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15422) s3guard doesnt init when the secrets are in the s3a URI

[Steve Loughran (JIRA)]

Steve Loughran created HADOOP-15422:
---

 Summary: s3guard doesnt init when the secrets are in the s3a URI
 Key: HADOOP-15422
 URL: https://issues.apache.org/jira/browse/HADOOP-15422
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs/s3
Affects Versions: 3.1.0
Reporter: Steve Loughran


If the AWS secrets are in the login, S3guard doesn't init. Presumably this is 
related to the credential chain setup
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14188) Remove the usage of org.mockito.internal.util.reflection.Whitebox

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456220#comment-16456220
 ] 

Hudson commented on HADOOP-14188:
-

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14078 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/14078/])
HADOOP-14188. Remove the usage of (aajisaka: rev 
84ecfe3cebe32ae048eae4b1bf02f6cdfd1fafe9)
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingInvalidateBlock.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHASafeMode.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSOutputStream.java
* (edit) 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreamsWithOpensslAesCtrCryptoCodec.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCommitBlockSynchronization.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfoStriped.java
* (edit) 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalFileSystem.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestBootstrapStandbyWithQJM.java
* (add) 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/Whitebox.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
* (edit) 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestStatsDMetrics.java
* (edit) 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestReencryptionHandler.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManagerSafeMode.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeManager.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddStripedBlockInFBR.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDeleteRace.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestFileWithSnapshotFeature.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFSForHA.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestComputeInvalidateWork.java
* (edit) 
hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestSequentialBlockGroupId.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReconstructStripedBlocksWithRackAwareness.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDFSUpgradeWithHA.java
* (edit) 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServer.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestRenameWithSnapshots.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestReencryption.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZonesWithKMS.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestKeyManager.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestHostFileManager.java
* (edit) 
hadoop-common-project/hadoop-nfs/src/test/java/org/apache/hadoop/portmap/TestPortmap.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainersLauncher.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockPoolManager.java
* (edit) 

[jira] [Commented] (HADOOP-15409) S3AFileSystem.verifyBucketExists to move to s3.doesBucketExistV2

[Steve Loughran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456210#comment-16456210
 ] 

Steve Loughran commented on HADOOP-15409:
-

Which S3 endpoint have you tested this against? 

> S3AFileSystem.verifyBucketExists to move to s3.doesBucketExistV2
> 
>
> Key: HADOOP-15409
> URL: https://issues.apache.org/jira/browse/HADOOP-15409
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Priority: Major
>
> in S3AFileSystem.initialize(), we check for the bucket existing with 
> verifyBucketExists(), which calls s3.doesBucketExist(). But that doesn't 
> check for auth issues. 
> s3. doesBucketExistV2() does at least validate credentials, and should be 
> switched to. This will help things fail faster 
> See SPARK-24000



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15421) Stabilise/formalise the JSON _SUCCESS format used in the S3A committers

[Steve Loughran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456209#comment-16456209
 ] 

Steve Loughran commented on HADOOP-15421:
-

+ [~rdblue]. ~[~jzhuge] I know iceberg has its own format which uses unique 
filenames to avoid update inconsistency, but they might have some suggestions 
here.

Current format: 
https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/files/SuccessData.java

There's always been a version marker in the file, to allow us to switch to a 
new format & let tests discover this by checking the version field alone...

> Stabilise/formalise the JSON _SUCCESS format used in the S3A committers
> ---
>
> Key: HADOOP-15421
> URL: https://issues.apache.org/jira/browse/HADOOP-15421
> Project: Hadoop Common
>  Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Priority: Major
>
> the S3A committers rely on an atomic PUT to save a JSON summary of the job to 
> the dest FS, containing files, statistics, etc. This is for internal testing, 
> but it turns out to be useful for spark integration testing, Hive, etc.
> IBM's stocator also generated a manifest.
> Proposed: come up with (an extensible) design that we are happy with as a 
> long lived format.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14178) Move Mockito up to version 2.x

[Akira Ajisaka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka updated HADOOP-14178:
---
Status: Open  (was: Patch Available)

> Move Mockito up to version 2.x
> --
>
> Key: HADOOP-14178
> URL: https://issues.apache.org/jira/browse/HADOOP-14178
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: test
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HADOOP-14178.001.patch, HADOOP-14178.002.patch, 
> HADOOP-14178.003.patch, HADOOP-14178.004.patch, HADOOP-14178.005-wip.patch, 
> HADOOP-14178.005-wip2.patch, HADOOP-14178.005-wip3.patch, 
> HADOOP-14178.005-wip4.patch, HADOOP-14178.005-wip5.patch, 
> HADOOP-14178.005-wip6.patch, HADOOP-14178.005.patch, HADOOP-14178.006.patch, 
> HADOOP-14178.007.patch, HADOOP-14178.008.patch, HADOOP-14178.009.patch, 
> HADOOP-14178.010.patch, HADOOP-14178.011.patch, HADOOP-14178.012.patch
>
>
> I don't know when Hadoop picked up Mockito, but it has been frozen at 1.8.5 
> since the switch to maven in 2011. 
> Mockito is now at version 2.1, [with lots of Java 8 
> support|https://github.com/mockito/mockito/wiki/What%27s-new-in-Mockito-2]. 
> That' s not just defining actions as closures, but in supporting Optional 
> types, mocking methods in interfaces, etc. 
> It's only used for testing, and, *provided there aren't regressions*, cost of 
> upgrade is low. The good news: test tools usually come with good test 
> coverage. The bad: mockito does go deep into java bytecodes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14188) Remove the usage of org.mockito.internal.util.reflection.Whitebox

[Akira Ajisaka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-14188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka updated HADOOP-14188:
---
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 3.2.0
   Status: Resolved  (was: Patch Available)

Committed this to trunk. Thanks [~ehiggs]!

> Remove the usage of org.mockito.internal.util.reflection.Whitebox
> -
>
> Key: HADOOP-14188
> URL: https://issues.apache.org/jira/browse/HADOOP-14188
> Project: Hadoop Common
>  Issue Type: Test
>  Components: test
>Reporter: Akira Ajisaka
>Assignee: Ewan Higgs
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HADOOP-14188.01.patch, HADOOP-14188.02.patch, 
> HADOOP-14188.03.patch, HADOOP-14188.04.patch, HADOOP-14188.05.patch, 
> HADOOP-14188.06.patch, HADOOP-14188.07.patch, HADOOP-14188.08.patch, 
> HADOOP-14188.09.patch, HADOOP-14188.10.patch, HADOOP-14188.11.patch, 
> HADOOP-14188.12.patch
>
>
> org.mockito.internal.util.reflection.Whitebox was removed in Mockito 2.1, so 
> we need to remove the usage to upgrade Mockito. Getter/setter method can be 
> used instead of this hack.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14188) Remove the usage of org.mockito.internal.util.reflection.Whitebox

[Akira Ajisaka (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-14188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456207#comment-16456207
 ] 

Akira Ajisaka commented on HADOOP-14188:


+1, the test failures are not related to the patch. Committing this.

> Remove the usage of org.mockito.internal.util.reflection.Whitebox
> -
>
> Key: HADOOP-14188
> URL: https://issues.apache.org/jira/browse/HADOOP-14188
> Project: Hadoop Common
>  Issue Type: Test
>  Components: test
>Reporter: Akira Ajisaka
>Assignee: Ewan Higgs
>Priority: Major
> Attachments: HADOOP-14188.01.patch, HADOOP-14188.02.patch, 
> HADOOP-14188.03.patch, HADOOP-14188.04.patch, HADOOP-14188.05.patch, 
> HADOOP-14188.06.patch, HADOOP-14188.07.patch, HADOOP-14188.08.patch, 
> HADOOP-14188.09.patch, HADOOP-14188.10.patch, HADOOP-14188.11.patch, 
> HADOOP-14188.12.patch
>
>
> org.mockito.internal.util.reflection.Whitebox was removed in Mockito 2.1, so 
> we need to remove the usage to upgrade Mockito. Getter/setter method can be 
> used instead of this hack.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15421) Stabilise/formalise the JSON _SUCCESS format used in the S3A committers

[Steve Loughran (JIRA)]

Steve Loughran created HADOOP-15421:
---

 Summary: Stabilise/formalise the JSON _SUCCESS format used in the 
S3A committers
 Key: HADOOP-15421
 URL: https://issues.apache.org/jira/browse/HADOOP-15421
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 3.2.0
Reporter: Steve Loughran


the S3A committers rely on an atomic PUT to save a JSON summary of the job to 
the dest FS, containing files, statistics, etc. This is for internal testing, 
but it turns out to be useful for spark integration testing, Hive, etc.

IBM's stocator also generated a manifest.

Proposed: come up with (an extensible) design that we are happy with as a long 
lived format.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15415) copyBytes hangs when the configuration file is corrupted

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-15415:

Affects Version/s: 3.1.0

> copyBytes hangs when the configuration file is corrupted
> 
>
> Key: HADOOP-15415
> URL: https://issues.apache.org/jira/browse/HADOOP-15415
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common
>Affects Versions: 0.23.0, 3.1.0
>Reporter: John Doe
>Priority: Major
>
> The third parameter,  buffSize, is read from the configuration files or 
> user-specified.
> When the configuration file is corrupted or the user configures with a wrong 
> value, i.e., 0, the bytesRead will always be 0, making the while loop's 
> condition always true, hanging IOUtils.copyBytes() endlessly.
> Here is the snippet of the code. There are four copyBytes in the following 
> code. The 3rd and 4th copyBytes calls the 1st one. The 1st one calls the 2nd 
> one. Hang happens in the while loop of the second copyBytes function.
>  
> {code:java}
> //1st copyBytes
>   public static void copyBytes(InputStream in, OutputStream out, int 
> buffSize, boolean close) 
> throws IOException {
> try {
>   copyBytes(in, out, buffSize);
>   if(close) {
> out.close();
> out = null;
> in.close();
> in = null;
>   }
> } finally {
>   if(close) {
> closeStream(out);
> closeStream(in);
>   }
> }
>   }
>   
> //2nd copyBytes
>   public static void copyBytes(InputStream in, OutputStream out, int 
> buffSize) 
> throws IOException {
> PrintStream ps = out instanceof PrintStream ? (PrintStream)out : null;
> byte buf[] = new byte[buffSize];
> int bytesRead = in.read(buf);
> while (bytesRead >= 0) {
>   out.write(buf, 0, bytesRead);
>   if ((ps != null) && ps.checkError()) {
> throw new IOException("Unable to write to output stream.");
>   }
>   bytesRead = in.read(buf);
> }
>   }
> //3rd copyBytes
>   public static void copyBytes(InputStream in, OutputStream out, 
> Configuration conf)
> throws IOException {
> copyBytes(in, out, conf.getInt("io.file.buffer.size", 4096), true);
>   }
>   
> //4th copyBytes
>   public static void copyBytes(InputStream in, OutputStream out, 
> Configuration conf, boolean close)
> throws IOException {
> copyBytes(in, out, conf.getInt("io.file.buffer.size", 4096),  close);
>   }
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15415) copyBytes hangs when the configuration file is corrupted

[Steve Loughran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456192#comment-16456192
 ] 

Steve Loughran commented on HADOOP-15415:
-

This is in the latest hadoop versions. Can you submit a patch with a Junit 
test? Thanks

> copyBytes hangs when the configuration file is corrupted
> 
>
> Key: HADOOP-15415
> URL: https://issues.apache.org/jira/browse/HADOOP-15415
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common
>Affects Versions: 0.23.0, 3.1.0
>Reporter: John Doe
>Priority: Minor
>
> The third parameter,  buffSize, is read from the configuration files or 
> user-specified.
> When the configuration file is corrupted or the user configures with a wrong 
> value, i.e., 0, the bytesRead will always be 0, making the while loop's 
> condition always true, hanging IOUtils.copyBytes() endlessly.
> Here is the snippet of the code. There are four copyBytes in the following 
> code. The 3rd and 4th copyBytes calls the 1st one. The 1st one calls the 2nd 
> one. Hang happens in the while loop of the second copyBytes function.
>  
> {code:java}
> //1st copyBytes
>   public static void copyBytes(InputStream in, OutputStream out, int 
> buffSize, boolean close) 
> throws IOException {
> try {
>   copyBytes(in, out, buffSize);
>   if(close) {
> out.close();
> out = null;
> in.close();
> in = null;
>   }
> } finally {
>   if(close) {
> closeStream(out);
> closeStream(in);
>   }
> }
>   }
>   
> //2nd copyBytes
>   public static void copyBytes(InputStream in, OutputStream out, int 
> buffSize) 
> throws IOException {
> PrintStream ps = out instanceof PrintStream ? (PrintStream)out : null;
> byte buf[] = new byte[buffSize];
> int bytesRead = in.read(buf);
> while (bytesRead >= 0) {
>   out.write(buf, 0, bytesRead);
>   if ((ps != null) && ps.checkError()) {
> throw new IOException("Unable to write to output stream.");
>   }
>   bytesRead = in.read(buf);
> }
>   }
> //3rd copyBytes
>   public static void copyBytes(InputStream in, OutputStream out, 
> Configuration conf)
> throws IOException {
> copyBytes(in, out, conf.getInt("io.file.buffer.size", 4096), true);
>   }
>   
> //4th copyBytes
>   public static void copyBytes(InputStream in, OutputStream out, 
> Configuration conf, boolean close)
> throws IOException {
> copyBytes(in, out, conf.getInt("io.file.buffer.size", 4096),  close);
>   }
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15415) copyBytes hangs when the configuration file is corrupted

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-15415:

Priority: Minor  (was: Major)

> copyBytes hangs when the configuration file is corrupted
> 
>
> Key: HADOOP-15415
> URL: https://issues.apache.org/jira/browse/HADOOP-15415
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common
>Affects Versions: 0.23.0, 3.1.0
>Reporter: John Doe
>Priority: Minor
>
> The third parameter,  buffSize, is read from the configuration files or 
> user-specified.
> When the configuration file is corrupted or the user configures with a wrong 
> value, i.e., 0, the bytesRead will always be 0, making the while loop's 
> condition always true, hanging IOUtils.copyBytes() endlessly.
> Here is the snippet of the code. There are four copyBytes in the following 
> code. The 3rd and 4th copyBytes calls the 1st one. The 1st one calls the 2nd 
> one. Hang happens in the while loop of the second copyBytes function.
>  
> {code:java}
> //1st copyBytes
>   public static void copyBytes(InputStream in, OutputStream out, int 
> buffSize, boolean close) 
> throws IOException {
> try {
>   copyBytes(in, out, buffSize);
>   if(close) {
> out.close();
> out = null;
> in.close();
> in = null;
>   }
> } finally {
>   if(close) {
> closeStream(out);
> closeStream(in);
>   }
> }
>   }
>   
> //2nd copyBytes
>   public static void copyBytes(InputStream in, OutputStream out, int 
> buffSize) 
> throws IOException {
> PrintStream ps = out instanceof PrintStream ? (PrintStream)out : null;
> byte buf[] = new byte[buffSize];
> int bytesRead = in.read(buf);
> while (bytesRead >= 0) {
>   out.write(buf, 0, bytesRead);
>   if ((ps != null) && ps.checkError()) {
> throw new IOException("Unable to write to output stream.");
>   }
>   bytesRead = in.read(buf);
> }
>   }
> //3rd copyBytes
>   public static void copyBytes(InputStream in, OutputStream out, 
> Configuration conf)
> throws IOException {
> copyBytes(in, out, conf.getInt("io.file.buffer.size", 4096), true);
>   }
>   
> //4th copyBytes
>   public static void copyBytes(InputStream in, OutputStream out, 
> Configuration conf, boolean close)
> throws IOException {
> copyBytes(in, out, conf.getInt("io.file.buffer.size", 4096),  close);
>   }
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15417) s3: retrieveBlock hangs when the configuration file is corrupted

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-15417:

Summary: s3: retrieveBlock hangs when the configuration file is 
corrupted  (was: retrieveBlock hangs when the configuration file is corrupted)
Description: 
The bufferSize is read from the configuration files.

When the configuration file is corrupted, i.e.,bufferSize=0, the numRead will 
always be 0, making the while loop's condition always true, hanging 
Jets3tFileSystemStore.retrieveBlock() endlessly.

Here is the snippet of the code. 


{code:java}
  private int bufferSize;

  this.bufferSize = conf.getInt( 
S3FileSystemConfigKeys.S3_STREAM_BUFFER_SIZE_KEY, 
S3FileSystemConfigKeys.S3_STREAM_BUFFER_SIZE_DEFAULT);

  public File retrieveBlock(Block block, long byteRangeStart)
throws IOException {
File fileBlock = null;
InputStream in = null;
OutputStream out = null;
try {
  fileBlock = newBackupFile();
  in = get(blockToKey(block), byteRangeStart);
  out = new BufferedOutputStream(new FileOutputStream(fileBlock));
  byte[] buf = new byte[bufferSize];
  int numRead;
  while ((numRead = in.read(buf)) >= 0) {
out.write(buf, 0, numRead);
  }
  return fileBlock;
} catch (IOException e) {
  ...
} finally {
  ...
}
  }
{code}

Similar case: [Hadoop-15415|https://issues.apache.org/jira/browse/HADOOP-15415].


  was:

The bufferSize is read from the configuration files.

When the configuration file is corrupted, i.e.,bufferSize=0, the numRead will 
always be 0, making the while loop's condition always true, hanging 
Jets3tFileSystemStore.retrieveBlock() endlessly.

Here is the snippet of the code. 


{code:java}
  private int bufferSize;

  this.bufferSize = conf.getInt( 
S3FileSystemConfigKeys.S3_STREAM_BUFFER_SIZE_KEY, 
S3FileSystemConfigKeys.S3_STREAM_BUFFER_SIZE_DEFAULT);

  public File retrieveBlock(Block block, long byteRangeStart)
throws IOException {
File fileBlock = null;
InputStream in = null;
OutputStream out = null;
try {
  fileBlock = newBackupFile();
  in = get(blockToKey(block), byteRangeStart);
  out = new BufferedOutputStream(new FileOutputStream(fileBlock));
  byte[] buf = new byte[bufferSize];
  int numRead;
  while ((numRead = in.read(buf)) >= 0) {
out.write(buf, 0, numRead);
  }
  return fileBlock;
} catch (IOException e) {
  ...
} finally {
  ...
}
  }
{code}

Similar case: [Hadoop-15415|https://issues.apache.org/jira/browse/HADOOP-15415].


Component/s: (was: common)
 fs/s3

> s3: retrieveBlock hangs when the configuration file is corrupted
> 
>
> Key: HADOOP-15417
> URL: https://issues.apache.org/jira/browse/HADOOP-15417
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs/s3
>Affects Versions: 0.23.0
>Reporter: John Doe
>Priority: Major
>
> The bufferSize is read from the configuration files.
> When the configuration file is corrupted, i.e.,bufferSize=0, the numRead will 
> always be 0, making the while loop's condition always true, hanging 
> Jets3tFileSystemStore.retrieveBlock() endlessly.
> Here is the snippet of the code. 
> {code:java}
>   private int bufferSize;
>   this.bufferSize = conf.getInt( 
> S3FileSystemConfigKeys.S3_STREAM_BUFFER_SIZE_KEY, 
> S3FileSystemConfigKeys.S3_STREAM_BUFFER_SIZE_DEFAULT);
>   public File retrieveBlock(Block block, long byteRangeStart)
> throws IOException {
> File fileBlock = null;
> InputStream in = null;
> OutputStream out = null;
> try {
>   fileBlock = newBackupFile();
>   in = get(blockToKey(block), byteRangeStart);
>   out = new BufferedOutputStream(new FileOutputStream(fileBlock));
>   byte[] buf = new byte[bufferSize];
>   int numRead;
>   while ((numRead = in.read(buf)) >= 0) {
> out.write(buf, 0, numRead);
>   }
>   return fileBlock;
> } catch (IOException e) {
>   ...
> } finally {
>   ...
> }
>   }
> {code}
> Similar case: 
> [Hadoop-15415|https://issues.apache.org/jira/browse/HADOOP-15415].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-15417) retrieveBlock hangs when the configuration file is corrupted

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HADOOP-15417.
-
Resolution: Won't Fix

> retrieveBlock hangs when the configuration file is corrupted
> 
>
> Key: HADOOP-15417
> URL: https://issues.apache.org/jira/browse/HADOOP-15417
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common
>Affects Versions: 0.23.0
>Reporter: John Doe
>Priority: Major
>
> The bufferSize is read from the configuration files.
> When the configuration file is corrupted, i.e.,bufferSize=0, the numRead will 
> always be 0, making the while loop's condition always true, hanging 
> Jets3tFileSystemStore.retrieveBlock() endlessly.
> Here is the snippet of the code. 
> {code:java}
>   private int bufferSize;
>   this.bufferSize = conf.getInt( 
> S3FileSystemConfigKeys.S3_STREAM_BUFFER_SIZE_KEY, 
> S3FileSystemConfigKeys.S3_STREAM_BUFFER_SIZE_DEFAULT);
>   public File retrieveBlock(Block block, long byteRangeStart)
> throws IOException {
> File fileBlock = null;
> InputStream in = null;
> OutputStream out = null;
> try {
>   fileBlock = newBackupFile();
>   in = get(blockToKey(block), byteRangeStart);
>   out = new BufferedOutputStream(new FileOutputStream(fileBlock));
>   byte[] buf = new byte[bufferSize];
>   int numRead;
>   while ((numRead = in.read(buf)) >= 0) {
> out.write(buf, 0, numRead);
>   }
>   return fileBlock;
> } catch (IOException e) {
>   ...
> } finally {
>   ...
> }
>   }
> {code}
> Similar case: 
> [Hadoop-15415|https://issues.apache.org/jira/browse/HADOOP-15415].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15417) retrieveBlock hangs when the configuration file is corrupted

[Steve Loughran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456189#comment-16456189
 ] 

Steve Loughran commented on HADOOP-15417:
-

John. I was wondering if you really meant 0.23, but then I saw that this really 
is a piece of S3FileSystem, which we have now deleted entirely as of 
HADOOP-12609 in 2016.

This going to be wontfix. 

I don't think anyone is going to look at issues filed against 0.23; branch-2 
and hadoop-3.1+ are where problems get replicated and fixed, then backported, 
at best, to hadoop-2.8, or, depending on severity, 2.7

Can you check out the latest hadoop code and see if you can write tests to 
replicate there?

thanks

> retrieveBlock hangs when the configuration file is corrupted
> 
>
> Key: HADOOP-15417
> URL: https://issues.apache.org/jira/browse/HADOOP-15417
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: common
>Affects Versions: 0.23.0
>Reporter: John Doe
>Priority: Major
>
> The bufferSize is read from the configuration files.
> When the configuration file is corrupted, i.e.,bufferSize=0, the numRead will 
> always be 0, making the while loop's condition always true, hanging 
> Jets3tFileSystemStore.retrieveBlock() endlessly.
> Here is the snippet of the code. 
> {code:java}
>   private int bufferSize;
>   this.bufferSize = conf.getInt( 
> S3FileSystemConfigKeys.S3_STREAM_BUFFER_SIZE_KEY, 
> S3FileSystemConfigKeys.S3_STREAM_BUFFER_SIZE_DEFAULT);
>   public File retrieveBlock(Block block, long byteRangeStart)
> throws IOException {
> File fileBlock = null;
> InputStream in = null;
> OutputStream out = null;
> try {
>   fileBlock = newBackupFile();
>   in = get(blockToKey(block), byteRangeStart);
>   out = new BufferedOutputStream(new FileOutputStream(fileBlock));
>   byte[] buf = new byte[bufferSize];
>   int numRead;
>   while ((numRead = in.read(buf)) >= 0) {
> out.write(buf, 0, numRead);
>   }
>   return fileBlock;
> } catch (IOException e) {
>   ...
> } finally {
>   ...
> }
>   }
> {code}
> Similar case: 
> [Hadoop-15415|https://issues.apache.org/jira/browse/HADOOP-15415].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15380) TestViewFileSystemLocalFileSystem#testTrashRoot leaves an unnecessary file

[Andras Bokor (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456183#comment-16456183
 ] 

Andras Bokor commented on HADOOP-15380:
---

It's not a test issue but a problem with LocalFilesystem. ChecksumFileSystem 
has no method for rename(Path, Path, Options.Rename...) so FilterFilesystem's 
method will be called which does not handle crc files. HADOOP-15388 will solve 
this.

> TestViewFileSystemLocalFileSystem#testTrashRoot leaves an unnecessary file
> --
>
> Key: HADOOP-15380
> URL: https://issues.apache.org/jira/browse/HADOOP-15380
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Andras Bokor
>Assignee: Andras Bokor
>Priority: Major
>
> After running
> {code}mvn test -Dtest=TestViewFileSystemLocalFileSystem#testTrashRoot
> git status{code}
> Git reports an untracked file: 
> {{hadoop-common-project/hadoop-common/.debug.log.crc}}
> It seems some cleanup issue.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15392) S3A Metrics in S3AInstrumentation Cause Memory Leaks in HBase Export

[Voyta (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456141#comment-16456141
 ] 

Voyta commented on HADOOP-15392:


[~mackrorysd] Thank you for your investigation. It should not be any of those 
as MapReduce is running in YARN and default file system and cache is not in S3. 
Although, it's worth mentioning we run Cloudera-forked version of Hadoop/HBase 
(version of Cloudera Manager 5.14.0). So far I didn't see any difference in the 
code in question.

Maybe, [~fabbri] should be able to clarify if there is any alteration in the 
forked version of this code?

> S3A Metrics in S3AInstrumentation Cause Memory Leaks in HBase Export
> 
>
> Key: HADOOP-15392
> URL: https://issues.apache.org/jira/browse/HADOOP-15392
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Voyta
>Priority: Blocker
>
> While using HBase S3A Export Snapshot utility we started to experience memory 
> leaks of the process after version upgrade.
> By running code analysis we traced the cause to revision 
> 6555af81a26b0b72ec3bee7034e01f5bd84b1564 that added the following static 
> reference (singleton):
> private static MetricsSystem metricsSystem = null;
> When application uses S3AFileSystem instance that is not closed immediately 
> metrics are accumulated in this instance and memory grows without any limit.
>  
> Expectation:
>  * It would be nice to have an option to disable metrics completely as this 
> is not needed for Export Snapshot utility.
>  * Usage of S3AFileSystem should not contain any static object that can grow 
> indefinitely.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Assigned] (HADOOP-15420) s3guard ITestS3GuardToolLocal failures in diff tests

[Gabor Bota (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gabor Bota reassigned HADOOP-15420:
---

Assignee: Gabor Bota

> s3guard ITestS3GuardToolLocal failures in diff tests
> 
>
> Key: HADOOP-15420
> URL: https://issues.apache.org/jira/browse/HADOOP-15420
> Project: Hadoop Common
>  Issue Type: Sub-task
>Reporter: Aaron Fabbri
>Assignee: Gabor Bota
>Priority: Minor
>
> Noticed this when testing the patch for HADOOP-13756.
>  
> {code:java}
> [ERROR] Failures:
> [ERROR]   
> ITestS3GuardToolLocal>AbstractS3GuardToolTestBase.testPruneCommandCLI:221->AbstractS3GuardToolTestBase.testPruneCommand:201->AbstractS3GuardToolTestBase.assertMetastoreListingCount:214->Assert.assertEquals:555->Assert.assertEquals:118->Assert.failNotEquals:743->Assert.fail:88
>  Pruned children count 
> [PathMetadata{fileStatus=S3AFileStatus{path=s3a://bucket-new/test/testPruneCommandCLI/stale;
>  isDirectory=false; length=100; replication=1; blocksize=512; 
> modification_time=1524798258286; access_time=0; owner=hdfs; group=hdfs; 
> permission=rw-rw-rw-; isSymlink=false; hasAcl=false; isEncrypted=false; 
> isErasureCoded=false} isEmptyDirectory=FALSE; isEmptyDirectory=UNKNOWN; 
> isDeleted=false}, 
> PathMetadata{fileStatus=S3AFileStatus{path=s3a://bucket-new/test/testPruneCommandCLI/fresh;
>  isDirectory=false; length=100; replication=1; blocksize=512; 
> modification_time=1524798262583; access_time=0; owner=hdfs; group=hdfs; 
> permission=rw-rw-rw-; isSymlink=false; hasAcl=false; isEncrypted=false; 
> isErasureCoded=false} isEmptyDirectory=FALSE; isEmptyDirectory=UNKNOWN; 
> isDeleted=false}] expected:<1> but was:<2>{code}
>  
> Looking through the code, I'm noticing a couple of issues.
>  
> 1. {{testDiffCommand()}} is in {{ITestS3GuardToolLocal}}, but it should 
> really be running for all MetadataStore implementations.  Seems like it 
> should live in {{AbstractS3GuardToolTestBase}}.
> 2. {{AbstractS3GuardToolTestBase#createFile()}} seems wrong. When 
> {{onMetadataStore}} is false, it does a {{ContractTestUtils.touch(file)}}, 
> but the fs is initialized with a MetadataStore present, so seem like the fs 
> will still put the file in the MetadataStore?
> There are other tests which explicitly go around the MetadataStore by using 
> {{fs.setMetadataStore(nullMS)}}, e.g. ITestS3AInconsistency. We should do 
> something similar in {{AbstractS3GuardToolTestBase#createFile()}}, minding 
> any issues with parallel test runs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15419) Should not obtain delegationTokens from all namenodes when using ViewFS

[Tao Jie (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456005#comment-16456005
 ] 

Tao Jie commented on HADOOP-15419:
--

[~hexiaoqiao] thank you for your comment.
{quote}
In fact, some request path may be found after task running (for one simple 
instance: hard code path in MapReduce/Spark job), if we don't obtain delegation 
token for that NameNode, the Job will be fail due to all tasks can not pass 
authentication.
{quote}
Agree. For the cluster, it is not easy to know which namenodes a certain job 
would access. I think the mechanism could be more flexible and obtaining tokens 
from all namenodes seems to be too crude.
1, We can have a option maybe {{fs.viewfs.use.specific.filesystem}}, only when 
this option is true, the following logic works.
2, When submit a mr/spark job, if the input/output path is a viewfs path, 
instead of obtaining token from all namenode, we would visit and fetch token 
from only a SET of filesystem.
3, The raw filesystem of the input/output path should be in the SET
4, We may have a global option like {{fs.viewfs.global.filesystem}} which 
defines filesystems that all jobs may visit(Eg. the filesystem of tmp dir, 
scratch dir), and it should be added into the SET
5, Job-level option like {{fs.viewfs.additional.filesystem}} which defiles 
extra filesystem that the certain job need.
Since obtaining delegation tokens happens on the client side, the effect of the 
modification would be controllable.  
Any thought?

> Should not obtain delegationTokens from all namenodes when using ViewFS
> ---
>
> Key: HADOOP-15419
> URL: https://issues.apache.org/jira/browse/HADOOP-15419
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 3.0.2
>Reporter: Tao Jie
>Priority: Major
>
> Today when submit a job to a viewfs cluster, the client will try to obtain 
> delegation token from all namenodes under the viewfs while only one namespace 
> is actually used in this job. It would create many unnecessary rpc call to 
> the whole cluster.
> In viewfs situation, we can just obtain delegation token from specific 
> namenode rather than all namenodes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15414) Job submit not work well on HDFS Federation with Transparent Encryption feature

[He Xiaoqiao (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455997#comment-16455997
 ] 

He Xiaoqiao commented on HADOOP-15414:
--

[~daryn],[~xiaochen], suggest that fix it using current patch (or another more 
graceful solution to quick fix it) since this problem exist in almost all 
versions.

> Job submit not work well on HDFS Federation with Transparent Encryption 
> feature
> ---
>
> Key: HADOOP-15414
> URL: https://issues.apache.org/jira/browse/HADOOP-15414
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs
>Reporter: He Xiaoqiao
>Priority: Major
> Attachments: HADOOP-15414-trunk.001.patch
>
>
> When submit sample MapReduce job WordCount which read/write path under 
> encryption zone on HDFS Federation in security mode to YARN, task throws 
> exception as below:
> {code:java}
> 18/04/26 16:07:26 INFO mapreduce.Job: Task Id : attempt_JOBID_m_TASKID_0, 
> Status : FAILED
> Error: java.io.IOException: 
> org.apache.hadoop.security.authentication.client.AuthenticationException: 
> GSSException: No valid credentials provided (Mechanism level: Failed to find 
> any Kerberos tgt)
> at 
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:489)
> at 
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:776)
> at 
> org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388)
> at 
> org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1468)
> at 
> org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1538)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:306)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300)
> at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:300)
> at org.apache.hadoop.fs.FilterFileSystem.open(FilterFileSystem.java:161)
> at 
> org.apache.hadoop.fs.viewfs.ChRootedFileSystem.open(ChRootedFileSystem.java:258)
> at 
> org.apache.hadoop.fs.viewfs.ViewFileSystem.open(ViewFileSystem.java:424)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:793)
> at 
> org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:85)
> at 
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:552)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:823)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1690)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> Caused by: 
> org.apache.hadoop.security.authentication.client.AuthenticationException: 
> GSSException: No valid credentials provided (Mechanism level: Failed to find 
> any Kerberos tgt)
> at 
> org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:332)
> at 
> org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205)
> at 
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:128)
> at 
> org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215)
> at 
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:322)
> at 
> org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:483)
> at 
> org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:478)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1690)
> at 
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:478)
> ... 21 more
> Caused by: GSSException: No valid credentials provided (Mechanism level: 
> Failed to find any Kerberos tgt)
> at 
> sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
> at 
>