[
https://issues.apache.org/jira/browse/HADOOP-13028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15274562#comment-15274562
]
Chris Nauroth commented on HADOOP-13028:
----------------------------------------
Patch v010 addresses all of my prior feedback.
It's fine to keep {{S3AInstrumentation#gauge}}.
I agree that "must be private and have accessor methods" is pedantic. I think
we can control this with the Checkstyle
[HiddenField|http://checkstyle.sourceforge.net/config_coding.html#HiddenField]
rule settings. Let's not bother for this patch though. You've already done a
lot to clean up Checkstyle warnings.
I agree with updating the JIRA title and closing down the other ones that were
folded into this patch.
I'm now seeing a failure in {{TestS3ABlocksize#testBlockSize}}:
{code}
testBlockSize(org.apache.hadoop.fs.s3a.TestS3ABlocksize) Time elapsed: 3.17
sec <<< FAILURE!
java.lang.AssertionError: Double default block size in stat():
S3AFileStatus{path=s3a://cnauroth-test-aws-s3a/test/testBlockSize/file;
isDirectory=false; length=1024; replication=1; blocksize=33554432;
modification_time=1462559933000; access_time=0; owner=; group=;
permission=rw-rw-rw-; isSymlink=false} expected:<67108864> but was:<33554432>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.hadoop.fs.s3a.TestS3ABlocksize.testBlockSize(TestS3ABlocksize.java:65)
{code}
It looks like the test has an expectation that you can update block size in the
{{Configuration}} of an already-initialized {{S3AFileSystem}}, and it will
start using the new value. After this patch, the block size is read from
{{Configuration}} once and cached. I think the expectation of the test is a
little dubious, but that's the behavior that shipped in 2.7.0. It's probably
safest to remove this part of the patch for now.
I just have one more request. Please remove the unused import of
{{Configuration}} in {{S3AInstrumentation}}.
I'll be +1 after that. We'll also need a separate trunk patch.
> add low level counter metrics for S3A; use in read performance tests
> --------------------------------------------------------------------
>
> Key: HADOOP-13028
> URL: https://issues.apache.org/jira/browse/HADOOP-13028
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, metrics
> Affects Versions: 2.8.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: HADOOP-13028-001.patch, HADOOP-13028-002.patch,
> HADOOP-13028-004.patch, HADOOP-13028-005.patch, HADOOP-13028-006.patch,
> HADOOP-13028-007.patch, HADOOP-13028-008.patch, HADOOP-13028-009.patch,
> HADOOP-13028-branch-2-008.patch, HADOOP-13028-branch-2-009.patch,
> HADOOP-13028-branch-2-010.patch,
> org.apache.hadoop.fs.s3a.scale.TestS3AInputStreamPerformance-output.txt,
> org.apache.hadoop.fs.s3a.scale.TestS3AInputStreamPerformance-output.txt
>
>
> against S3 (and other object stores), opening connections can be expensive,
> closing connections may be expensive (a sign of a regression).
> S3A FS and individual input streams should have counters of the # of
> open/close/failure+reconnect operations, timers of how long things take. This
> can be used downstream to measure efficiency of the code (how often
> connections are being made), connection reliability, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
