[jira] [Resolved] (HADOOP-16251) ABFS: add FSMainOperationsBaseTest
[ https://issues.apache.org/jira/browse/HADOOP-16251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri resolved HADOOP-16251. --- Resolution: Fixed Fix Version/s: 3.2.1 > ABFS: add FSMainOperationsBaseTest > -- > > Key: HADOOP-16251 > URL: https://issues.apache.org/jira/browse/HADOOP-16251 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Da Zhou >Assignee: Da Zhou >Priority: Major > Fix For: 3.2.1 > > > Just happened to see > "hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FSMainOperationsBaseTest.java", > ABFS could inherit this test to increase its test coverage. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16291) HDFS Permissions Guide appears incorrect about getFileStatus()/getFileInfo()
Aaron Fabbri created HADOOP-16291: - Summary: HDFS Permissions Guide appears incorrect about getFileStatus()/getFileInfo() Key: HADOOP-16291 URL: https://issues.apache.org/jira/browse/HADOOP-16291 Project: Hadoop Common Issue Type: Bug Components: documentation Reporter: Aaron Fabbri Fix some errors in the HDFS Permissions doc. Noticed this when reviewing HADOOP-16251. The FS Permissions [documentation|https://hadoop.apache.org/docs/r3.2.0/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html] seems to mark a lot of permissions as Not Applicable (N/A) when that is not the case. In particular getFileInfo (getFileStatus) checks READ permission bit [here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java#L3202-L3204], as it should. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15780) S3Guard: document how to deal with non-S3Guard processes writing data to S3Guarded buckets
Aaron Fabbri created HADOOP-15780: - Summary: S3Guard: document how to deal with non-S3Guard processes writing data to S3Guarded buckets Key: HADOOP-15780 URL: https://issues.apache.org/jira/browse/HADOOP-15780 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 3.2.0 Reporter: Aaron Fabbri Our general policy for S3Guard is this: All modifiers of a bucket that is configured for use with S3Guard, must use S3Guard. Otherwise, the MetadataStore will not be properly updated as the S3 bucket changes and problems will arise. There are limited circumstances in which may be safe to have an external (non-s3guard) process writing data. There are also scenarios where it definitely breaks things. I think we should start by documenting the cases that this works / does not work for. After we've enumerated that, we can suggest enhancements as needed to make this sort of configuration easier to use. To get the ball rolling, some things that do not work: - Deleting a path *p* with S3Guard, then writing a new file at path *p* without S3guard (will still have delete marker in S3Guard, making the file appear to be deleted but still visible in S3 due to false "eventual consistency") (as [~ste...@apache.org] and I have discussed) - When fs.s3a.metadatastore.authoritative is true, adding files to directories without S3Guard, then listing with S3Guard may exclude externally-written files from listings. (Note, there are also S3A interop issues with other non-S3A clients even without S3Guard, due to the unique way S3A interprets empty directory markers). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15779) S3guard: add inconsistency detection metrics
Aaron Fabbri created HADOOP-15779: - Summary: S3guard: add inconsistency detection metrics Key: HADOOP-15779 URL: https://issues.apache.org/jira/browse/HADOOP-15779 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Affects Versions: 3.2.0 Reporter: Aaron Fabbri S3Guard uses a trailing log of metadata changes made to an S3 bucket to add consistency to the eventually-consistent AWS S3 service. We should add some metrics that are incremented when we detect inconsistency (eventual consistency) in S3. I'm thinking at least two counters: (1) getFileStatus() (HEAD) inconsistency detected, and (2) listing inconsistency detected. We may want to further separate into categories (present / not present etc.) This is related to Auth. Mode and TTL work that is ongoing, so let me outline how I think this should all evolve: This should happen after HADOOP-15621 (TTL for dynamo MetadataStore), since that will change *when* we query both S3 and the MetadataStore (e.g. Dynamo) for metadata. There I suggest that: # Prune time is different than TTL. Prune time: "how long until inconsistency is no longer a problem" . TTL time "how long a MetadataStore entry is considered authoritative before refresh" # Prune expired: delete entries (when hadoop CLI prune command is run). TTL Expired: entries become non-authoritative. # Prune implemented in each MetadataStore, but TTL filtering happens in S3A. Once we have this, S3A will be consulting both S3 and MetadataStore depending on configuration and/or age of the entry in the MetadataStore. Today HEAD/getFileStatus() is always short-circuit (skips S3 if MetadataStore returns results). I think S3A should consult both when TTL is stale, and invoke a callback on inconsistency that increments the new metrics. For listing, we already are comparing both sources of truth (except when S3A auth mode is on and a directory is marked authoritative in MS), so it would be pretty simple to invoke a callback on inconsistency and bump some metrics. Comments / suggestions / questions welcomed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15754) s3guard: testDynamoTableTagging should clear existing config
Aaron Fabbri created HADOOP-15754:
-
Summary: s3guard: testDynamoTableTagging should clear existing
config
Key: HADOOP-15754
URL: https://issues.apache.org/jira/browse/HADOOP-15754
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Affects Versions: 3.2.0
Reporter: Aaron Fabbri
I recently committed HADOOP-14734 which adds support for tagging Dynamo DB
tables for S3Guard when they are created.
Later, when testing another patch, I hit a test failure because I still had a
tag option set in my test configuration (auth-keys.xml) that was adding my own
table tag.
{noformat}
[ERROR]
testDynamoTableTagging(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB)
Time elapsed: 13.384 s <<< FAILURE!
java.lang.AssertionError: expected:<2> but was:<3>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB.testDynamoTableTagging(ITestS3GuardToolDynamoDB.java:129)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74){noformat}
I think the solution is just to clear any tag.* options set in the
configuration at the beginning of the test.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15621) s3guard: implement time-based (TTL) expiry for DynamoDB Metadata Store
Aaron Fabbri created HADOOP-15621: - Summary: s3guard: implement time-based (TTL) expiry for DynamoDB Metadata Store Key: HADOOP-15621 URL: https://issues.apache.org/jira/browse/HADOOP-15621 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.0.0-beta1 Reporter: Aaron Fabbri Assignee: Gabor Bota Fix For: 3.2.0 LocalMetadataStore is primarily a reference implementation for testing. It may be useful in narrow circumstances where the workload can tolerate short-term lack of inter-node consistency: Being in-memory, one JVM/node's LocalMetadataStore will not see another node's changes to the underlying filesystem. To put a bound on the time during which this inconsistency may occur, we should implement time-based (a.k.a. Time To Live / TTL) expiration for LocalMetadataStore -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15525) s3a: clarify / improve support for mixed ACL buckets
Aaron Fabbri created HADOOP-15525:
-
Summary: s3a: clarify / improve support for mixed ACL buckets
Key: HADOOP-15525
URL: https://issues.apache.org/jira/browse/HADOOP-15525
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Affects Versions: 3.0.0
Reporter: Aaron Fabbri
Scenario: customer wants to only give a Hadoop cluster access to a subtree of
an S3 bucket.
For example, assume Hadoop uses some IAM identity "hadoop", which they wish to
grant full permission to everything under the following path:
s3a://bucket/a/b/c/hadoop-dir
they don't want hadoop user to be able to read/list/delete anything outside of
the hadoop-dir "subdir"
Problems:
To implement the "directory structure on flat key space" emulation logic we use
to present a Hadoop FS on top of a blob store, we need to create / delete /
list ancestors of {{hadoop-dir}}.
I'd like us to either (1) document a workaround (example IAM ACLs) that gets
this basic functionality, and/or (2) make improvements to make this less
painful.
We've discussed some of these issues before but I didn't see a dedicated JIRA.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-11697) Use larger value for fs.s3a.connection.timeout.
[
https://issues.apache.org/jira/browse/HADOOP-11697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Fabbri resolved HADOOP-11697.
---
Resolution: Duplicate
Fix Version/s: 3.0.0
> Use larger value for fs.s3a.connection.timeout.
> ---
>
> Key: HADOOP-11697
> URL: https://issues.apache.org/jira/browse/HADOOP-11697
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 2.6.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Minor
> Labels: s3
> Fix For: 3.0.0
>
> Attachments: HADOOP-11697.001.patch, HDFS-7908.000.patch
>
>
> The default value of {{fs.s3a.connection.timeout}} is {{5}} milliseconds.
> It causes many {{SocketTimeoutException}} when uploading large files using
> {{hadoop fs -put}}.
> Also, the units for {{fs.s3a.connection.timeout}} and
> {{fs.s3a.connection.estaablish.timeout}} are milliseconds. For s3
> connections, I think it is not necessary to have sub-seconds timeout value.
> Thus I suggest to change the time unit to seconds, to easy sys admin's job.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15444) ITestS3GuardToolDynamo should only run with -Ddynamo
Aaron Fabbri created HADOOP-15444:
-
Summary: ITestS3GuardToolDynamo should only run with -Ddynamo
Key: HADOOP-15444
URL: https://issues.apache.org/jira/browse/HADOOP-15444
Project: Hadoop Common
Issue Type: Sub-task
Affects Versions: 3.0.0
Reporter: Aaron Fabbri
If you run S3A integration tests with just {{-Ds3guard}} and not {{-Ddynamo}},
then I do not think that ITestS3GuardToolDynamo should run.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15420) s3guard ITestS3GuardToolLocal failures in diff tests
Aaron Fabbri created HADOOP-15420:
-
Summary: s3guard ITestS3GuardToolLocal failures in diff tests
Key: HADOOP-15420
URL: https://issues.apache.org/jira/browse/HADOOP-15420
Project: Hadoop Common
Issue Type: Improvement
Reporter: Aaron Fabbri
Noticed this when testing the patch for HADOOP-13756.
{code:java}
[ERROR] Failures:
[ERROR]
ITestS3GuardToolLocal>AbstractS3GuardToolTestBase.testPruneCommandCLI:221->AbstractS3GuardToolTestBase.testPruneCommand:201->AbstractS3GuardToolTestBase.assertMetastoreListingCount:214->Assert.assertEquals:555->Assert.assertEquals:118->Assert.failNotEquals:743->Assert.fail:88
Pruned children count
[PathMetadata{fileStatus=S3AFileStatus{path=s3a://bucket-new/test/testPruneCommandCLI/stale;
isDirectory=false; length=100; replication=1; blocksize=512;
modification_time=1524798258286; access_time=0; owner=hdfs; group=hdfs;
permission=rw-rw-rw-; isSymlink=false; hasAcl=false; isEncrypted=false;
isErasureCoded=false} isEmptyDirectory=FALSE; isEmptyDirectory=UNKNOWN;
isDeleted=false},
PathMetadata{fileStatus=S3AFileStatus{path=s3a://bucket-new/test/testPruneCommandCLI/fresh;
isDirectory=false; length=100; replication=1; blocksize=512;
modification_time=1524798262583; access_time=0; owner=hdfs; group=hdfs;
permission=rw-rw-rw-; isSymlink=false; hasAcl=false; isEncrypted=false;
isErasureCoded=false} isEmptyDirectory=FALSE; isEmptyDirectory=UNKNOWN;
isDeleted=false}] expected:<1> but was:<2>{code}
Looking through the code, I'm noticing a couple of issues.
1. {{testDiffCommand()}} is in {{ITestS3GuardToolLocal}}, but it should really
be running for all MetadataStore implementations. Seem like it should live in
{{AbstractS3GuardToolTestBase}}.
2. {{AbstractS3GuardToolTestBase#createFile()}} seems wrong. When
{{onMetadataStore}} is false, it does a {{ContractTestUtils.touch(file)}}, but
the fs is initialized with a MetadataStore present, so won't the fs put the
file in the MetadataStore?
There are other tests which explicitly go around the MetadataStore by using
{{fs.setMetadataStore(nullMS)}}, e.g. ITestS3AInconsistency. We should do
something similar in {{AbstractS3GuardToolTestBase#createFile()}}, minding any
issues with parallel test runs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15400) Improve S3Guard documentation on Authoritative Mode implemenation
Aaron Fabbri created HADOOP-15400:
-
Summary: Improve S3Guard documentation on Authoritative Mode
implemenation
Key: HADOOP-15400
URL: https://issues.apache.org/jira/browse/HADOOP-15400
Project: Hadoop Common
Issue Type: Improvement
Components: fs/s3
Affects Versions: 3.0.1
Reporter: Aaron Fabbri
Part of the design of S3Guard is support for skipping the call to S3
listObjects and serving directory listings out of the MetadataStore under
certain circumstances. This feature is called "authoritative" mode. I've
talked to many people about this feature and it seems to be universally
confusing.
I suggest we improve / add a section to the s3guard.md site docs elaborating on
what Authoritative Mode is.
It is *not* treating the MetadataStore (e.g. dynamodb) as the source of truth
in general.
It *is* the ability to short-circuit S3 list objects and serve listings from
the MetadataStore in some circumstances:
For S3A to skip S3's list objects on some *path*, and serve it directly from
the MetadataStore, the following things must all be true:
# The MetadataStore implementation persists the bit
{{DirListingMetadata.isAuthorititative}} set when calling
{{MetadataStore#put(DirListingMetadata)}}
# The S3A client is configured to allow metadatastore to be authoritative
source of a directory listing (fs.s3a.metadatastore.authoritative=true).
# The MetadataStore has a full listing for *path* stored in it. This only
happens if the FS client (s3a) explicitly has stored a full directory listing
with {{DirListingMetadata.isAuthorititative=true}} before the said listing
request happens.
Note that #1 only currently happens in LocalMetadataStore. Adding support to
DynamoDBMetadataStore is covered in HADOOP-14154.
Also, the multiple uses of the word "authoritative" are confusing. Two meanings
are used:
1. In the FS client configuration fs.s3a.metadatastore.authoritative
- Behavior of S3A code (not MetadataStore)
- "S3A is allowed to skip S3.list() when it has full listing from
MetadataStore"
2. MetadataStore
When storing a dir listing, can set a bit isAuthoritative
1 : "full contents of directory"
0 : "may not be full listing"
Note that a MetadataStore *MAY* persist this bit. (not *MUST*).
We should probably rename the {{DirListingMetadata.isAuthorititative}} to
{{.fullListing}} or at least put a comment where it is used to clarify its
meaning.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15207) Hadoop performance Issues
[
https://issues.apache.org/jira/browse/HADOOP-15207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Fabbri resolved HADOOP-15207.
---
Resolution: Invalid
Can someone please ban this user? Similar SEO spam has been pasted on other
sites I can see. We should remove the link and/or delete the Jira too.
> Hadoop performance Issues
> -
>
> Key: HADOOP-15207
> URL: https://issues.apache.org/jira/browse/HADOOP-15207
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: HADOOP-13345
>Reporter: nicole wells
>Priority: Minor
>
> I am doing a hadoop project where I am working with 100MB, 500MB, 1GB files.
> A multinode hadoop cluster with 4 nodes is implemented for the purpose. The
> time taken for running the mapreduce program in multinode cluster is much
> larger than the time taken in running single node cluster setup. Also, it is
> shocking to observe that the basic Java program(without [Hadoop
> BigData|https://mindmajix.com/hadoop-training]) finishes the operation faster
> than both the single and multi node clusters. Here is the code for the mapper
> class:
>
> {code:java}
> public class myMapperClass extends MapReduceBase implements
> Mapper
> {
> private final static IntWritable one = new IntWritable(1);
> private final static IntWritable two = new IntWritable(2);
> private final static IntWritable three = new IntWritable(3);
> private final static IntWritable four = new IntWritable(4);
> private final static IntWritable five = new IntWritable(5);
> private final static IntWritable six = new IntWritable(6);
> private final static IntWritable seven = new IntWritable(7);
> private final static IntWritable eight = new IntWritable(8);
> private final static IntWritable nine= new IntWritable(9);
> private Text srcIP,srcIPN;
> private Text dstIP,dstIPN;
> private Text srcPort,srcPortN;
> private Text dstPort,dstPortN;
> private Text counter1,counter2,counter3,counter4,counter5 ;
> //private Text total_records;
> int ddos_line = 0;
> //map method that performs the tokenizer job and framing the initial key
> value pairs
> @Override
> public void map(LongWritable key, Text value, OutputCollector IntWritable> output, Reporter reporter) throws IOException
> {
> String line1 = value.toString();
> ddos_line++;
> int pos1=0;
> int lineno=0;
> int[] count = {10, 10, 10, 10, 10};
> int[] lineIndex = {0, 0, 0, 0, 0};
> for(int i=0;i<9;i++)
> {
> pos1 = line1.indexOf("|",pos1+1);
> }
> srcIP = new Text( line1.substring(0,line1.indexOf("|")) );
> String srcIPP = srcIP.toString();
> dstIP = new Text(line1.substring(
> srcIPP.length()+1,line1.indexOf("|",srcIPP.length()+1)) ) ;
> srcPort = new Text( line1.substring(pos1+1,line1.indexOf("|",pos1+1))
> );
> pos1 = line1.indexOf("|",pos1+1);
> dstPort = new Text( line1.substring(pos1+1,line1.indexOf("|",pos1+1))
> );
> //BufferedReader br = new BufferedReader(new
> FileReader("/home/yogi/Desktop/normal_small"));
> FileSystem fs = FileSystem.get(new Configuration());
> FileStatus[] status = fs.listStatus(new
> Path("hdfs://master:54310/usr/local/hadoop/input/normal_small"));
> BufferedReader br=new BufferedReader(new
> InputStreamReader(fs.open(status[0].getPath(;
> String line=br.readLine();
> lineno++;
> boolean bool = true;
> while (bool) {
> for(int i=0; i<5;i++)
> {
> if(bool==false)
> break;
> int pos=0;
> int temp;
> for(int j=0;j<9;j++)
> {
> pos = line.indexOf("|",pos+1);
> }
> srcIPN = new Text( line.substring(0,line.indexOf("|")) );
> String srcIPP2 = srcIPN.toString();
> dstIPN = new Text(line.substring(
> srcIPP2.length()+1,line.indexOf("|",srcIPP2.length()+1)) ) ;
> srcPortN = new Text(
> line.substring(pos+1,line.indexOf("|",pos+1)) );
> pos = line.indexOf("|",pos+1);
> dstPortN = new Text(
> line.substring(pos+1,line.indexOf("|",pos+1)) );
> if(srcIP.equals(srcIPN) && dstIP.equals(dstIPN))
> {
> int tmp, tmp2;
> tmp = Integer.parseInt(srcPort.toString()) -
> Integer.parseInt(srcPortN.toString());
> if(tmp<0)
> tmp*=-1;
> tmp2 = Integer.parseInt(dstPort.toString()) -
> Integer.parseInt(dstPortN.toString());
[jira] [Created] (HADOOP-15040) AWS SDK NPE bug spams logs w/ Yarn Log Aggregation
Aaron Fabbri created HADOOP-15040:
-
Summary: AWS SDK NPE bug spams logs w/ Yarn Log Aggregation
Key: HADOOP-15040
URL: https://issues.apache.org/jira/browse/HADOOP-15040
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Affects Versions: 3.0.0-beta1
Reporter: Aaron Fabbri
Assignee: Aaron Fabbri
My colleagues working with Yarn log aggregation found that they were getting
this message spammed in their logs when they used an s3a:// URI for logs
(yarn.nodemanager.remote-app-log-dir):
{noformat}
getting attribute Region of com.amazonaws.management:type=AwsSdkMetrics threw
an exception
javax.management.RuntimeMBeanException: java.lang.NullPointerException
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839)
at
Caused by: java.lang.NullPointerException
at com.amazonaws.metrics.AwsSdkMetrics.getRegion(AwsSdkMetrics.java:729)
at com.amazonaws.metrics.MetricAdmin.getRegion(MetricAdmin.java:67)
at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
{noformat}
This happens even though the aws sdk cloudwatch metrics reporting was disabled
(default), which is a bug.
I filed a [github issue|https://github.com/aws/aws-sdk-java/issues/1375|] and
it looks like a fix should be coming around SDK release 1.11.229 or so.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14936) S3Guard: remove "experimental" from documentation
Aaron Fabbri created HADOOP-14936: - Summary: S3Guard: remove "experimental" from documentation Key: HADOOP-14936 URL: https://issues.apache.org/jira/browse/HADOOP-14936 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 3.0.0-beta1 Reporter: Aaron Fabbri Priority: Minor I think it is time to remove the "experimental feature" designation in the site docs for S3Guard. Discuss. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14746) Cut S3AOutputStream
[
https://issues.apache.org/jira/browse/HADOOP-14746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Fabbri resolved HADOOP-14746.
---
Resolution: Duplicate
Fix Version/s: 3.0.0-beta1
Resolving this.. it was covered by HADOOP-14738
> Cut S3AOutputStream
> ---
>
> Key: HADOOP-14746
> URL: https://issues.apache.org/jira/browse/HADOOP-14746
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Priority: Minor
> Fix For: 3.0.0-beta1
>
>
> We've been happy with the new S3A BlockOutputStream, with better scale,
> performance, instrumentation & recovery. I propose cutting the
> older{{S3AOutputStream}} code entirely.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14927) ITestS3GuardTool failures in testDestroyNoBucket()
Aaron Fabbri created HADOOP-14927:
-
Summary: ITestS3GuardTool failures in testDestroyNoBucket()
Key: HADOOP-14927
URL: https://issues.apache.org/jira/browse/HADOOP-14927
Project: Hadoop Common
Issue Type: Bug
Components: s3
Affects Versions: 3.0.0-alpha3
Reporter: Aaron Fabbri
Priority: Minor
Hit this when testing for the Hadoop 3.0.0-beta1 RC0.
{noformat}
hadoop-3.0.0-beta1-src/hadoop-tools/hadoop-aws$ mvn clean verify
-Dit.test="ITestS3GuardTool*" -Dtest=none -Ds3guard -Ddynamo
...
Failed tests:
ITestS3GuardToolDynamoDB>AbstractS3GuardToolTestBase.testDestroyNoBucket:228
Expected an exception, got 0
ITestS3GuardToolLocal>AbstractS3GuardToolTestBase.testDestroyNoBucket:228
Expected an exception, got 0
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14633) S3Guard: optimize create codepath
Aaron Fabbri created HADOOP-14633: - Summary: S3Guard: optimize create codepath Key: HADOOP-14633 URL: https://issues.apache.org/jira/browse/HADOOP-14633 Project: Hadoop Common Issue Type: Sub-task Environment: Reporter: Aaron Fabbri Assignee: Aaron Fabbri Priority: Minor Following up on HADOOP-14457, a couple of things to do that will improve create performance as I mentioned in the comment [here|https://issues.apache.org/jira/browse/HADOOP-14457?focusedCommentId=16078465=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16078465] -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14548) S3Guard: issues running parallel tests w/ S3N
Aaron Fabbri created HADOOP-14548: - Summary: S3Guard: issues running parallel tests w/ S3N Key: HADOOP-14548 URL: https://issues.apache.org/jira/browse/HADOOP-14548 Project: Hadoop Common Issue Type: Sub-task Reporter: Aaron Fabbri In general, running S3Guard and parallel tests with S3A and S3N contract tests enabled is asking for trouble: S3Guard code assumes there are not other non-S3Guard clients modifying the bucket. Goal of this JIRA is to: - Discuss current failures running `mvn verify -Dparallel-tests -Ds3guard -Ddynamo` with S3A and S3N contract tests configured. - Identify any failures here that are worth looking into. - Document (or enforce) that people should not do this, or should expect failures if they do. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14527) ITestS3GuardListConsistency is too slow
Aaron Fabbri created HADOOP-14527:
-
Summary: ITestS3GuardListConsistency is too slow
Key: HADOOP-14527
URL: https://issues.apache.org/jira/browse/HADOOP-14527
Project: Hadoop Common
Issue Type: Sub-task
Affects Versions: HADOOP-13345
Reporter: Aaron Fabbri
Assignee: Aaron Fabbri
Priority: Minor
I'm really glad to see folks adopting the inconsistency injection stuff and
adding test cases to ITestS3GuardListConsistency. That test class has become
very slow, however, mostly due to {{Thread.sleep()}} calls that wait for the
inconsistency injection timers to expire.
I will take a stab at speeding up this test class. As is it takes about 8
minutes to run.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14509) InconsistentAmazonS3Client adds extra paths to listStatus() after delete.
Aaron Fabbri created HADOOP-14509:
-
Summary: InconsistentAmazonS3Client adds extra paths to
listStatus() after delete.
Key: HADOOP-14509
URL: https://issues.apache.org/jira/browse/HADOOP-14509
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Aaron Fabbri
Assignee: Aaron Fabbri
I identified a potential issue in code that simulates list-after-delete
inconsistency when code reviewing HADOOP-13760. It appeared to work for the
existing test cases but now that we are using the inconsistency injection code
for general testing (e.g. HADOOP-14488) we need to make sure this stuff is
correct.
Deliverable is to make sure {{InconsistentAmazonS3Client#restoreListObjects()}}
is correct.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14506) Add create() contract test that verifies ancestor dir creation
Aaron Fabbri created HADOOP-14506:
-
Summary: Add create() contract test that verifies ancestor dir
creation
Key: HADOOP-14506
URL: https://issues.apache.org/jira/browse/HADOOP-14506
Project: Hadoop Common
Issue Type: Sub-task
Components: fs
Affects Versions: HADOOP-13345
Reporter: Aaron Fabbri
Priority: Minor
The semantics of {{FileSystem#create()}} appear to include implicit creation of
any parent or ancestor directories in the Path being created, unlike Posix
filesystems, which expect the parent directory to already exist.
S3AFileSystem *does* implicitly create ancestor directories. It does *not*
currently enforce that any ancestors are directories (HADOOP-13221).
Deliverable for this JIRA is just a test case added to
{{AbstractContractCreateTest}} which verifies that missing ancestors are
created by {{create()}}.
Pulling this test dev work out into separate jira from HADOOP-14457. Targeting
S3Guard branch for now, but we could move this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14505) simplify mkdirs() after S3Guard delete tracking change
Aaron Fabbri created HADOOP-14505: - Summary: simplify mkdirs() after S3Guard delete tracking change Key: HADOOP-14505 URL: https://issues.apache.org/jira/browse/HADOOP-14505 Project: Hadoop Common Issue Type: Sub-task Reporter: Aaron Fabbri Priority: Minor I noticed after reviewing the S3Guard delete tracking changes for HADOOP-13760, that mkdirs() can probably be simplified, replacing the use of checkPathForDirectory() with a simple getFileStatus(). Creating a separate JIRA so these changes can be reviewed / tested in isolation. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14504) ProvidedFileStatusIterator#next() may throw IndexOutOfBoundsException
Aaron Fabbri created HADOOP-14504: - Summary: ProvidedFileStatusIterator#next() may throw IndexOutOfBoundsException Key: HADOOP-14504 URL: https://issues.apache.org/jira/browse/HADOOP-14504 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HADOOP-13345 Reporter: Aaron Fabbri Assignee: Aaron Fabbri [~mackrorysd] noticed this as part of his work on HADOOP-14457. We are splitting that JIRA into smaller patches so we will address this separately. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14468) S3Guard: make short-circuit getFileStatus() configurable
Aaron Fabbri created HADOOP-14468:
-
Summary: S3Guard: make short-circuit getFileStatus() configurable
Key: HADOOP-14468
URL: https://issues.apache.org/jira/browse/HADOOP-14468
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Aaron Fabbri
Assignee: Aaron Fabbri
Currently, when S3Guard is enabled, getFileStatus() will skip S3 if it gets a
result from the MetadataStore (e.g. dynamodb) first.
I would like to add a new parameter
{{fs.s3a.metadatastore.getfilestatus.authoritative}} which, when true, keeps
the current behavior. When false, S3AFileSystem will check both S3 and the
MetadataStore.
I'm not sure yet if we want to have this behavior the same for all callers of
getFileStatus(), or if we only want to check both S3 and MetadataStore for some
internal callers such as open().
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14467) S3Guard: Improve FNFE message when opening a stream
Aaron Fabbri created HADOOP-14467: - Summary: S3Guard: Improve FNFE message when opening a stream Key: HADOOP-14467 URL: https://issues.apache.org/jira/browse/HADOOP-14467 Project: Hadoop Common Issue Type: Sub-task Reporter: Aaron Fabbri Assignee: Aaron Fabbri Following up on the [discussion on HADOOP-13345|https://issues.apache.org/jira/browse/HADOOP-13345?focusedCommentId=16030050=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16030050], because S3Guard can serve getFileStatus() from the MetadataStore without doing a HEAD on S3, a FileNotFound error on a file due to S3 GET inconsistency does not happen on open(), but on the first read of the stream. We may add retries to the S3 client in the future, but for now we should have an exception message that indicates this may be due to inconsistency (assuming it isn't a more straightforward case like someone deleting the object out from under you). This is expected to be a rare case, since the S3 service is now mostly consistent for GET. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14428) s3a: mkdir appears to be broken
Aaron Fabbri created HADOOP-14428: - Summary: s3a: mkdir appears to be broken Key: HADOOP-14428 URL: https://issues.apache.org/jira/browse/HADOOP-14428 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Affects Versions: 3.0.0-alpha2, HADOOP-13345 Reporter: Aaron Fabbri Priority: Blocker Reproduction is: hadoop fs -mkdir s3a://my-bucket/dir/ hadoop fs -ls s3a://my-bucket/dir/ ls: `s3a://my-bucket/dir/': No such file or directory I believe this is a regression from HADOOP-14255. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13981) S3Guard CLI: Add documentation
[ https://issues.apache.org/jira/browse/HADOOP-13981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri resolved HADOOP-13981. --- Resolution: Fixed Documentation was added as part of S3Guard CLI JIRAs, resolving. > S3Guard CLI: Add documentation > -- > > Key: HADOOP-13981 > URL: https://issues.apache.org/jira/browse/HADOOP-13981 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri > > I believe we still need documentation for the new S3Guard CLI commands. > Synopsis of all the commands and some examples would be great. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14391) s3a: auto-detect region for bucket and use right endpoint
Aaron Fabbri created HADOOP-14391:
-
Summary: s3a: auto-detect region for bucket and use right endpoint
Key: HADOOP-14391
URL: https://issues.apache.org/jira/browse/HADOOP-14391
Project: Hadoop Common
Issue Type: Improvement
Components: s3
Affects Versions: 3.0.0-alpha2
Reporter: Aaron Fabbri
Specifying the S3A endpoint ({{fs.s3a.endpoint}}) is
- *required* for regions which only support v4 authentication
- A good practice for all regions.
The user experience of having to configure endpoints is not great. Often it is
neglected and leads to additional cost, reduced performance, or failures for v4
auth regions.
I want to explore an option which, when enabled, auto-detects the region for an
s3 bucket and uses the proper endpoint. Not sure if this is possible or anyone
has looked into it yet.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14323) ITestS3GuardListConsistency failure w/ Local, authoritative metadata store
Aaron Fabbri created HADOOP-14323:
-
Summary: ITestS3GuardListConsistency failure w/ Local,
authoritative metadata store
Key: HADOOP-14323
URL: https://issues.apache.org/jira/browse/HADOOP-14323
Project: Hadoop Common
Issue Type: Sub-task
Components: s3
Affects Versions: HADOOP-13345
Reporter: Aaron Fabbri
Priority: Minor
When doing some testing for HADOOP-14266 I noticed this test failure:
{noformat}
java.lang.NullPointerException: null
at
org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.testListStatusWriteBack(ITestS3GuardListConsistency.java:317)
{noformat}
I was running with LocalMetadataStore and
{{fs.s3a.metadatastore.authoritative}} set to true. I haven't been testing
this mode recently so not sure if this case ever worked. Lower priority but we
should fix it.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14283) S3A may hang due to bug in AWS SDK 1.11.86
Aaron Fabbri created HADOOP-14283: - Summary: S3A may hang due to bug in AWS SDK 1.11.86 Key: HADOOP-14283 URL: https://issues.apache.org/jira/browse/HADOOP-14283 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Affects Versions: 3.0.0-alpha2 Reporter: Aaron Fabbri Assignee: Aaron Fabbri Priority: Critical We hit a hang bug when testing S3A with parallel renames. I narrowed this down to the newer AWS Java SDK. It only happens under load, and appears to be a failure to wake up a waiting thread on timeout/error. I've created a github issue here: https://github.com/aws/aws-sdk-java/issues/1102 I can post a Hadoop scale test which reliably reproduces this after some cleanup. I have posted an SDK-only test here which reproduces the issue without Hadoop: https://github.com/ajfabbri/awstest I have a support ticket open and am working with Amazon on this bug so I'll take this issue. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14144) s3guard: CLI import does not yield an empty diff.
Aaron Fabbri created HADOOP-14144:
-
Summary: s3guard: CLI import does not yield an empty diff.
Key: HADOOP-14144
URL: https://issues.apache.org/jira/browse/HADOOP-14144
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Aaron Fabbri
Priority: Minor
I expected the following steps to yield zero diff from `hadoop s3guard diff`
command.
(1) hadoop s3guard init ... (create fresh table)
(2) hadoop s3guard import (fresh table, existing bucket with data in it)
(3) hadoop s3guard diff ..
Instead I still get a non-zero diff on step #3, and also noticed some entries
are printed twice.
{noformat}
dude@computer:~/Code/hadoop$ hadoop s3guard diff -meta dynamodb://dude-dev
-region us-west-2 s3a://dude-dev
S3 D s3a://fabbri-dev/user/fabbri/test/parentdirdest
S3 D s3a://fabbri-dev/user/fabbri/test/parentdirdest
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14096) s3guard: regression in dirListingUnion
Aaron Fabbri created HADOOP-14096:
-
Summary: s3guard: regression in dirListingUnion
Key: HADOOP-14096
URL: https://issues.apache.org/jira/browse/HADOOP-14096
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: HADOOP-13345
Reporter: Aaron Fabbri
Priority: Critical
Just noticed HADOOP-14020 introduced a bug in S3Guard#dirListingUnion.
The offending change is here:
{noformat}
- if (dirMeta.get(s.getPath()) == null) {
-dirMeta.put(s);
- }
+ changed = changed || dirMeta.put(s);
+}
+
{noformat}
hint: Logical OR is a short-circuit operator.
Easy fix, but should probably come with a unit test for dirListingUnion().
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14087) S3A typo in pom.xml test exclusions
Aaron Fabbri created HADOOP-14087: - Summary: S3A typo in pom.xml test exclusions Key: HADOOP-14087 URL: https://issues.apache.org/jira/browse/HADOOP-14087 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: HADOOP-13345 Reporter: Aaron Fabbri Assignee: Aaron Fabbri When you overload DDB, you get error messages warning of throttling, [as documented by AWS|http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Programming.Errors.html#Programming.Errors.MessagesAndCodes] Reduce load on DDB by doing a table lookup before the create, then, in table create/delete operations and in get/put actions, recognise the error codes and retry using an appropriate retry policy (exponential backoff + ultimate failure) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14051) S3Guard: link docs from index, fix typos
Aaron Fabbri created HADOOP-14051: - Summary: S3Guard: link docs from index, fix typos Key: HADOOP-14051 URL: https://issues.apache.org/jira/browse/HADOOP-14051 Project: Hadoop Common Issue Type: Sub-task Reporter: Aaron Fabbri Assignee: Aaron Fabbri JIRA for a quick round of s3guard.md documentation improvements. - Link from index.md - Make a pass and fix typos / outdated stuff / spelling etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14036) S3Guard: intermittent duplicate item keys failure
Aaron Fabbri created HADOOP-14036:
-
Summary: S3Guard: intermittent duplicate item keys failure
Key: HADOOP-14036
URL: https://issues.apache.org/jira/browse/HADOOP-14036
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Aaron Fabbri
I see this occasionally when running integration tests with -Ds3guard -Ddynamo:
{noformat}
testRenameToDirWithSamePrefixAllowed(org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract)
Time elapsed: 2.756 sec <<< ERROR!
org.apache.hadoop.fs.s3a.AWSServiceIOException: move:
com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: Provided list
of item keys contains duplicates (Service: AmazonDynamoDBv2; Status Code: 400;
Error Code: ValidationException; Request ID:
QSBVQV69279UGOB4AJ4NO9Q86VVV4KQNSO5AEMVJF66Q9ASUAAJG): Provided list of item
keys contains duplicates (Service: AmazonDynamoDBv2; Status Code: 400; Error
Code: ValidationException; Request ID:
QSBVQV69279UGOB4AJ4NO9Q86VVV4KQNSO5AEMVJF66Q9ASUAAJG)
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:178)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.move(DynamoDBMetadataStore.java:408)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerRename(S3AFileSystem.java:869)
at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:662)
at
org.apache.hadoop.fs.FileSystemContractBaseTest.rename(FileSystemContractBaseTest.java:525)
at
org.apache.hadoop.fs.FileSystemContractBaseTest.testRenameToDirWithSamePrefixAllowed(FileSystemContractBaseTest.java:669)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcces
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14013) S3Guard: fix multi-bucket integration tests
Aaron Fabbri created HADOOP-14013:
-
Summary: S3Guard: fix multi-bucket integration tests
Key: HADOOP-14013
URL: https://issues.apache.org/jira/browse/HADOOP-14013
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: HADOOP-13345
Reporter: Aaron Fabbri
Assignee: Aaron Fabbri
HADOOP-13449 adds support for DynamoDBMetadataStore.
The code currently supports two options for choosing DynamoDB table names:
1. Use name of each s3 bucket and auto-create a DynamoDB table for each.
2. Configure a table name in the {{fs.s3a.s3guard.ddb.table}} parameter.
One of the issues is with accessing read-only buckets. If a user accesses a
read-only bucket with credentials that do not have DynamoDB write permissions,
they will get errors when trying to access the read-only bucket. This
manifests causes test failures for {{ITestS3AAWSCredentialsProvider}}.
Goals for this JIRA:
- Fix {{ITestS3AAWSCredentialsProvider}} in a way that makes sense for the real
use-case.
- Allow for a "one DynamoDB table per cluster" configuration with a way to
chose which credentials are used for DynamoDB.
- Document limitations etc. in the s3guard.md site doc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13995) s3guard cli: make tests easier to run and address failure
Aaron Fabbri created HADOOP-13995: - Summary: s3guard cli: make tests easier to run and address failure Key: HADOOP-13995 URL: https://issues.apache.org/jira/browse/HADOOP-13995 Project: Hadoop Common Issue Type: Sub-task Reporter: Aaron Fabbri Assignee: Sean Mackrory Following up on HADOOP-13650, we should: - Make it clearer which config parameters need to be set for test to succeed, and provide good defaults. - Address any remaining test failures. - Change TestS3GuardTool to an ITest -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13981) S3Guard CLI: Add documentation
Aaron Fabbri created HADOOP-13981: - Summary: S3Guard CLI: Add documentation Key: HADOOP-13981 URL: https://issues.apache.org/jira/browse/HADOOP-13981 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Reporter: Aaron Fabbri Assignee: Aaron Fabbri I believe we still need documentation for the new S3Guard CLI commands. Synopsis of all the commands and some examples would be great. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13980) S3Guard CLI: Add fsck check command
Aaron Fabbri created HADOOP-13980: - Summary: S3Guard CLI: Add fsck check command Key: HADOOP-13980 URL: https://issues.apache.org/jira/browse/HADOOP-13980 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Reporter: Aaron Fabbri Assignee: Aaron Fabbri As discussed in HADOOP-13650, we want to add an S3Guard CLI command which compares S3 with MetadataStore, and returns a failure status if any invariants are violated. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13914) s3guard: improve S3AFileStatus#isEmptyDirectory handling
Aaron Fabbri created HADOOP-13914: - Summary: s3guard: improve S3AFileStatus#isEmptyDirectory handling Key: HADOOP-13914 URL: https://issues.apache.org/jira/browse/HADOOP-13914 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HADOOP-13345 Reporter: Aaron Fabbri As discussed in HADOOP-13449, proper support for the isEmptyDirectory() flag stored in S3AFileStatus is missing from DynamoDBMetadataStore. The approach taken by LocalMetadataStore is not suitable for the DynamoDB implementation, and also sacrifices good code separation to minimize S3AFileSystem changes pre-merge to trunk. I will attach a design doc that attempts to clearly explain the problem and preferred solution. I suggest we do this work after merging the HADOOP-13345 branch to trunk, but am open to suggestions. I can also attach a patch of a integration test that exercises the missing case and demonstrates a failure with DynamoDBMetadataStore. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13886) s3guard: ITestS3AFileOperationCost.testFakeDirectoryDeletion failure
Aaron Fabbri created HADOOP-13886: - Summary: s3guard: ITestS3AFileOperationCost.testFakeDirectoryDeletion failure Key: HADOOP-13886 URL: https://issues.apache.org/jira/browse/HADOOP-13886 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Reporter: Aaron Fabbri testFakeDirectoryDeletion(org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost) Time elapsed: 10.011 sec <<< FAILURE! java.lang.AssertionError: after rename(srcFilePath, destFilePath): directories_created expected:<1> but was:<0> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.apache.hadoop.fs.s3a.S3ATestUtils$MetricDiff.assertDiffEquals(S3ATestUtils.java:431) at org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost.testFakeDirectoryDeletion(ITestS3AFileOperationCost.java:254) More details to follow in comments. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13877) S3Guard: fix TestDynamoDBMetadataStore when fs.s3a.s3guard.ddb.table is set
Aaron Fabbri created HADOOP-13877:
-
Summary: S3Guard: fix TestDynamoDBMetadataStore when
fs.s3a.s3guard.ddb.table is set
Key: HADOOP-13877
URL: https://issues.apache.org/jira/browse/HADOOP-13877
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: HADOOP-13345
Reporter: Aaron Fabbri
Assignee: Aaron Fabbri
I see a couple of failures in the DynamoDB MetadataStore unit test when I set
{{fs.s3a.s3guard.ddb.table}} in my test/resources/core-site.xml.
I have a fix already, so I'll take this JIRA.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13876) S3Guard: better support for multi-bucket access including read-only
Aaron Fabbri created HADOOP-13876:
-
Summary: S3Guard: better support for multi-bucket access including
read-only
Key: HADOOP-13876
URL: https://issues.apache.org/jira/browse/HADOOP-13876
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: HADOOP-13345
Reporter: Aaron Fabbri
HADOOP-13449 adds support for DynamoDBMetadataStore.
The code currently supports two options for choosing DynamoDB table names:
1. Use name of each s3 bucket and auto-create a DynamoDB table for each.
2. Configure a table name in the {{fs.s3a.s3guard.ddb.table}} parameter.
One of the issues is with accessing read-only buckets. If a user accesses a
read-only bucket with credentials that do not have DynamoDB write permissions,
they will get errors when trying to access the read-only bucket. This
manifests causes test failures for {{ITestS3AAWSCredentialsProvider}}.
Goals for this JIRA:
- Fix {{ITestS3AAWSCredentialsProvider}} in a way that makes sense for the real
use-case.
- Allow for a "one DynamoDB table per cluster" configuration with a way to
chose which credentials are used for DynamoDB.
- Document limitations etc. in the s3guard.md site doc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13793) s3guard: add inconsistency injection, integration tests
Aaron Fabbri created HADOOP-13793: - Summary: s3guard: add inconsistency injection, integration tests Key: HADOOP-13793 URL: https://issues.apache.org/jira/browse/HADOOP-13793 Project: Hadoop Common Issue Type: Sub-task Reporter: Aaron Fabbri Many of us share concerns that testing the consistency features of S3Guard will be difficult if we depend on the rare and unpredictable occurrence of actual inconsistency in S3 to exercise those code paths. I think we should have a mechanism for injecting failure to force exercising of the consistency codepaths in S3Guard. Requirements: - Integration tests that cause S3A to see the types of inconsistency we address with S3Guard. - These are deterministic integration tests. Unit tests are possible as well, if we were to stub out the S3Client. That may be less bang for the buck, though. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13761) S3Guard: implement retries
Aaron Fabbri created HADOOP-13761: - Summary: S3Guard: implement retries Key: HADOOP-13761 URL: https://issues.apache.org/jira/browse/HADOOP-13761 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Reporter: Aaron Fabbri Following the S3AFileSystem integration patch in HADOOP-13651, we need to add retry logic. In HADOOP-13651, I added TODO comments in most of the places retry loops are needed, including: - open(path). If MetadataStore reflects recent create/move of file path, but we fail to read it from S3, retry. - delete(path). If deleteObject() on S3 fails, but MetadataStore shows the file exists, retry. - rename(src,dest). If source path is not visible in S3 yet, retry. - listFiles(). Skip for now. Not currently implemented in S3Guard. I will create a separate JIRA for this as it will likely require interface changes (i.e. prefix or subtree scan). We may miss some cases initially and we should do failure injection testing to make sure we're covered. Failure injection tests can be a separate JIRA to make this easier to review. We also need basic configuration parameters around retry policy. There should be a way to specify maximum retry duration, as some applications would prefer to receive an error eventually, than waiting indefinitely. We should also be keeping statistics when inconsistency is detected and we enter a retry loop. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13760) S3Guard: add delete tracking
Aaron Fabbri created HADOOP-13760:
-
Summary: S3Guard: add delete tracking
Key: HADOOP-13760
URL: https://issues.apache.org/jira/browse/HADOOP-13760
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Reporter: Aaron Fabbri
Following the S3AFileSystem integration patch in HADOOP-13651, we need to add
delete tracking.
Current behavior on delete is to remove the metadata from the MetadataStore.
To make deletes consistent, we need to add a {{isDeleted}} flag to
{{PathMetadata}} and check it when returning results from functions like
{{getFileStatus()}} and {{listStatus()}}. In HADOOP-13651, I added TODO
comments in most of the places these new conditions are needed. The work does
not look to bad.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13651) S3Guard: S3AFileSystem Integration with MetadataStore
Aaron Fabbri created HADOOP-13651: - Summary: S3Guard: S3AFileSystem Integration with MetadataStore Key: HADOOP-13651 URL: https://issues.apache.org/jira/browse/HADOOP-13651 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Reporter: Aaron Fabbri Modify S3AFileSystem et al. to optionally use a MetadataStore for metadata consistency and caching. Implementation should have minimal overhead when no MetadataStore is configured. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13649) s3guard: implement time-based (TTL) expiry for LocalMetadataStore
Aaron Fabbri created HADOOP-13649: - Summary: s3guard: implement time-based (TTL) expiry for LocalMetadataStore Key: HADOOP-13649 URL: https://issues.apache.org/jira/browse/HADOOP-13649 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Reporter: Aaron Fabbri Assignee: Aaron Fabbri LocalMetadataStore is primarily a reference implementation for testing. It may be useful in narrow circumstances where the workload can tolerate short-term lack of inter-node consistency: Being in-memory, one JVM/node's LocalMetadataStore will not see another node's changes to the underlying filesystem. To put a bound on the time during which this inconsistency may occur, we should implement time-based (a.k.a. Time To Live / TTL) expiration for LocalMetadataStore -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13631) S3Guard: implement move() for LocalMetadataStore, add unit tests
Aaron Fabbri created HADOOP-13631: - Summary: S3Guard: implement move() for LocalMetadataStore, add unit tests Key: HADOOP-13631 URL: https://issues.apache.org/jira/browse/HADOOP-13631 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Reporter: Aaron Fabbri Assignee: Aaron Fabbri Building on HADOOP-13573 and HADOOP-13452, implement move() in LocalMetadataStore and associated MetadataStore unit tests. (Making this a separate JIRA to break up work into decent-sized and reviewable chunks.) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13573) S3Guard: create basic contract tests for MetadataStore implementations
Aaron Fabbri created HADOOP-13573: - Summary: S3Guard: create basic contract tests for MetadataStore implementations Key: HADOOP-13573 URL: https://issues.apache.org/jira/browse/HADOOP-13573 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 2.9.0 Reporter: Aaron Fabbri Assignee: Aaron Fabbri We should have some contract-style unit tests for the MetadataStore interface to validate that the different implementations provide correct semantics. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13559) Remove close() within try-with-resources
Aaron Fabbri created HADOOP-13559: - Summary: Remove close() within try-with-resources Key: HADOOP-13559 URL: https://issues.apache.org/jira/browse/HADOOP-13559 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.8.0, 2.9.0 Reporter: Aaron Fabbri Assignee: Aaron Fabbri Priority: Minor My colleague noticed that HADOOP-12994 introduced to places where close() was still called manually within a try-with-resources block. I'll attach a patch to remove the manual close() calls. These extra calls to close() are probably safe, as InputStream is a Closable, not AutoClosable (the later does not specify close() as idempotent). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13476) CredentialProviderFactory fails at class loading from libhdfs (JNI)
Aaron Fabbri created HADOOP-13476:
-
Summary: CredentialProviderFactory fails at class loading from
libhdfs (JNI)
Key: HADOOP-13476
URL: https://issues.apache.org/jira/browse/HADOOP-13476
Project: Hadoop Common
Issue Type: Bug
Components: fs/s3
Affects Versions: 2.8.0, 2.9.0, 3.0.0-alpha2
Reporter: Aaron Fabbri
Assignee: Aaron Fabbri
This bug was discovered when trying to run Impala (libhdfs.so) with s3a and
Java KeyStore credentials. Because JNI threads have a different classloader
(bootstrap), we fail to load JavaKeyStoreProvider.
{quote}
15:11:42.658087 26310 jni-util.cc:166] java.util.ServiceConfigurationError:
org.apache.hadoop.security.alias.CredentialProviderFactory: Provider
org.apache.hadoop.security.alias.JavaKeyStoreProvider$Factory not found
at java.util.ServiceLoader.fail(ServiceLoader.java:231)
at java.util.ServiceLoader.access$300(ServiceLoader.java:181)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:365)
at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
at
org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:57)
at
org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:1950)
at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:1930)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getAWSAccessKeys(S3AFileSystem.java:366)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getAWSCredentialsProvider(S3AFileSystem.java:415)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:176)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13351) TestDFSClientSocketSize buffer size tests are flaky
Aaron Fabbri created HADOOP-13351:
-
Summary: TestDFSClientSocketSize buffer size tests are flaky
Key: HADOOP-13351
URL: https://issues.apache.org/jira/browse/HADOOP-13351
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.8.0, 3.0.0-alpha1
Reporter: Aaron Fabbri
{{TestDFSClientSocketSize}} has two tests that assert that a value that was set
via {{java.net.Socket#setSendBufferSize}} is equal to the value subsequently
returned by {{java.net.Socket#getSendBufferSize}}.
These tests are flaky when we run them. The occasionally fail.
This is expected behavior, actually, because {{Socket#setSendBufferSize()}}[is
only a
hint|https://docs.oracle.com/javase/7/docs/api/java/net/Socket.html#setSendBufferSize(int)].
(Similar to how the underlying libc {{setsockopt(SO_SNDBUF)}} works).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13230) s3a's use of fake empty directory blobs does not interoperate with other s3 tools
Aaron Fabbri created HADOOP-13230: - Summary: s3a's use of fake empty directory blobs does not interoperate with other s3 tools Key: HADOOP-13230 URL: https://issues.apache.org/jira/browse/HADOOP-13230 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Affects Versions: 2.9.0 Reporter: Aaron Fabbri Users of s3a may not realize that, in some cases, it does not interoperate well with other s3 tools, such as the AWS CLI. (See HIVE-13778, IMPALA-3558). Specifically, if a user: - Creates an empty directory with hadoop fs -mkdir s3a://bucket/path - Copies data into that directory via another tool, i.e. aws cli. - Tries to access the data in that directory with any Hadoop software. Then the last step fails because the fake empty directory blob that s3a wrote in the first step, causes s3a (listStatus() etc.) to continue to treat that directory as empty, even though the second step was supposed to populate the directory with data. I wanted to document this fact for users. We may mark this as not-fix, "by design".. May also be interesting to brainstorm solutions and/or a config option to change the behavior if folks care. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-12267) s3a failure due to integer overflow bug in AWS SDK
Aaron Fabbri created HADOOP-12267: - Summary: s3a failure due to integer overflow bug in AWS SDK Key: HADOOP-12267 URL: https://issues.apache.org/jira/browse/HADOOP-12267 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Affects Versions: 2.6.0 Reporter: Aaron Fabbri Assignee: Aaron Fabbri Under high load writing to Amazon AWS S3 storage, a client can be throttled enough to encounter 24 retries in a row. The amazon http client code (in aws-java-sdk jar) has a bug in its exponential backoff retry code, that causes integer overflow, and a call to Thread.sleep() with a negative value, which causes client to bail out with an exception (see below). Bug has been fixed in aws-java-sdk: https://github.com/aws/aws-sdk-java/pull/388 We need to pick this up for hadoop-tools/hadoop-aws. Error: java.io.IOException: File copy failed: hdfs://path-redacted -- s3a://path-redacted at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:284) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:252) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.io.IOException: Couldn't run retriable-command: Copying hdfs://path-redacted to s3a://path-redacted at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101) at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:280) ... 10 more Caused by: com.amazonaws.AmazonClientException: Unable to complete transfer: timeout value is negative at com.amazonaws.services.s3.transfer.internal.AbstractTransfer.unwrapExecutionException(AbstractTransfer.java:300) at com.amazonaws.services.s3.transfer.internal.AbstractTransfer.rethrowExecutionException(AbstractTransfer.java:284) at com.amazonaws.services.s3.transfer.internal.CopyImpl.waitForCopyResult(CopyImpl.java:67) at org.apache.hadoop.fs.s3a.S3AFileSystem.copyFile(S3AFileSystem.java:943) at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:357) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.promoteTmpToTarget(RetriableFileCopyCommand.java:220) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:137) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:100) at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87) ... 11 more Caused by: java.lang.IllegalArgumentException: timeout value is negative at java.lang.Thread.sleep(Native Method) at com.amazonaws.http.AmazonHttpClient.pauseBeforeNextRetry(AmazonHttpClient.java:864) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:353) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) at com.amazonaws.services.s3.AmazonS3Client.copyObject(AmazonS3Client.java:1507) at com.amazonaws.services.s3.transfer.internal.CopyCallable.copyInOneChunk(CopyCallable.java:143) at com.amazonaws.services.s3.transfer.internal.CopyCallable.call(CopyCallable.java:131) at com.amazonaws.services.s3.transfer.internal.CopyMonitor.copy(CopyMonitor.java:189) at com.amazonaws.services.s3.transfer.internal.CopyMonitor.call(CopyMonitor.java:134) at com.amazonaws.services.s3.transfer.internal.CopyMonitor.call(CopyMonitor.java:46) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
