[
https://issues.apache.org/jira/browse/HADOOP-15890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16746723#comment-16746723
]
Steve Loughran commented on HADOOP-15890:
-----------------------------------------
{code}
bin/hadoop fs -cat
s3a://hwdev-steve-ireland-new/terasort-ITestMagicTerasort/results.csv
"Operation" "Duration"
"Generate" "0:22.854s"
"Terasort" "0:30.228s"
"Validate" "0:27.682s"
"Completed" "1:27.840s"
{code}
Directory staging committer:
{code}
bin/hadoop fs -cat
s3a://hwdev-steve-ireland-new/terasort-ITestDirectoryTerasort/results.csv
"Operation" "Duration"
"Generate" "0:22.111s"
"Terasort" "0:24.613s"
"Validate" "0:24.504s"
"Completed" "1:19.135s"
{code}
The client is a laptop, store S3 ireland a few hundred millis away; the latency
of S3 calls means that the magic committer, which uses S3 over a miniHDFS
cluster, suffers a lot from the latency. You'd need to be creating larger files
for the incremental write to become relevant
> Some S3A committer tests don't match ITest* pattern; don't run in maven
> -----------------------------------------------------------------------
>
> Key: HADOOP-15890
> URL: https://issues.apache.org/jira/browse/HADOOP-15890
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
> Affects Versions: 3.2.0, 3.1.1
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
>
> some of the s3A committer tests don't have the right prefix for the maven IT
> test runs to pick up
> {code}
> ITMagicCommitMRJob.java
> ITStagingCommitMRJobBad
> ITDirectoryCommitMRJob
> ITStagingCommitMRJob
> {code}
> They all work when run by name or in the IDE (which is where I developed
> them), but they don't run in maven builds.
> Fix: rename. There are some new tests in branch-3.2 from HADOOP-15107 which
> aren't in 3.1; need patches for both.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
