[
https://issues.apache.org/jira/browse/HADOOP-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974467#comment-15974467
]
Steve Loughran commented on HADOOP-14305:
-----------------------------------------
I'm trying to think of what would make some good extra tests. There's already
one for rename, but it's for a file.
How about
* recursive rename of a 2-level directory tree, verify ls of directories find
dirs, loading files still works. listFiles(path, recursive=true) should also be
used for listing, on account of it being a different list mechanism.
* open , seek to end + read to -1, then seek to (0) and read there. Why? Helps
verify that filelengths match data, and that the re-opens done by seek() stay
valid.
Maybe we could also subclass a Huge test and add encryption, again using SSE-C
as the worst-case example. That will test multipart operations and large file
copies. With that there'd be coverage of the set of S3 verbs: PUT, mulipart
POST, COPY, LIST, GET, HEAD.
IT might also be fun to do a getFileStatus() and open() on the stock landsat
CSV, expecting failures. We'll be able to make an assert about the error
type/text. verify anything I do w.r.t diagnostics text in HADOOP-14291 doesn't
break the exception class type. (and provides details on encryption type)
Note that I've added something to the release notes of HADOOP-13075, made clear
this must be considered experimental for now. I'm confident you think it's
ready, but everything related to S3 usually takes one iteration to stabilise.
The phase II stuff all went through the HDCloud 4-6 week dev cycle before
shipping in longer-lived releases, and yes, we found things. I'll expect to see
things here too, now that you are shipping it. If they go to you internally,
don't be shy of matching them with things related to the ASF code, test and
docs.
> S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
> ---------------------------------------------------------------------------
>
> Key: HADOOP-14305
> URL: https://issues.apache.org/jira/browse/HADOOP-14305
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
> Affects Versions: 2.9.0
> Reporter: Steve Loughran
> Assignee: Steve Moist
> Priority: Minor
> Attachments: HADOOP-14305-001.patch
>
>
> The S3a encryption tests all run in parallel (they were interfering with each
> other, apparently). This adds ~1 min to the test runs.
> They should run in serial. That they fail when this is attempted due to Bad
> Auth problems must be considered a serious problem, as is indicates issues
> related to working with SSE encryption from hadoop
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
