[
https://issues.apache.org/jira/browse/HADOOP-13998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16035077#comment-16035077
]
Steve Loughran commented on HADOOP-13998:
-----------------------------------------
bq. We've run our standard downstream Hive, Spark, MR, Impala, scale, and
performance tests
If these tests were working *before* you turned s3guard on then they weren't
catching inconsistencies & so were lucky (as mine were). I'm running my spark
committer tests with the inconsistent client turned on, and it is repeatedly
failing the classic & magic committers without s3guard enabled: both depend on
consistent listing. Also found a brittleness in path cleanup for the magic
committer too; cleanup code *must* handle an FNFE if there's a file returned in
the listing but which isn't there in the GET. This is why I'd like the factory
for the inconsistent client be in src/main: it lets anyone turn on
inconsistency for their test runs
bq. This is a good point. Do you prefer timing-based microbenchmarks, or S3
request statistics (counts)?
the instrumentation ones are way less brittle; Ming has been fixing some
nanotimer-assertion in WASB which was failing intermittently. I have some tests
somewhere which call listFiles(recursive) against the amazon landsat store:
that's the reference example of a deep and wide directory tree.
> initial s3guard preview
> -----------------------
>
> Key: HADOOP-13998
> URL: https://issues.apache.org/jira/browse/HADOOP-13998
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Steve Loughran
>
> JIRA to link in all the things we think are needed for a preview/merge into
> trunk
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
