[
https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16038746#comment-16038746
]
Steve Loughran commented on HADOOP-14475:
-----------------------------------------
Seems a good start, though it'll need some iterations to get in.
# Can you follow the naming conventions of JIRA-001.patch, eg.
HADOOP-14475-001.patch (a) the patch viewer for chrome depends on the
extension and (b) Yetus expects that naming scheme, and depends on it for
applying to different branches. Thanks.
# There's some hadoop style guides we expect, see
[https://wiki.apache.org/hadoop/HowToContribute]; with my proposed update at
[https://github.com/steveloughran/formality/blob/master/styleguide/styleguide.md]
As it is, we'd pretty much reject {{S3AFileSystemMetricsSystem}} on style,
including: CRLF endings, indentation & spacing, javadoc & doc comments, etc.
elsewhere for your IDE coalescing imports into a .* import. We don't allow them
for anything other than static imports, as it makes maintenance extra hard.
Now, regarding the code
# Why is the context of S3AInstrumentation changed to "s3a"? This isn't a
criticism, it's just me (who doesn't understand the metrics system) trying to
understand what I've got wrong when I coped that code over from WASB. We may
have a similar problem in Azure for that reason.
# why move things about in that class? We need to keep diffs to a minimum to
stop them conflicting with other patches which come in.
# what are we going to do if there is >1 S3A FS registered? Is it last one
wins, or is there a way for us to register multiple filesystems. Or, if the
attempt to register a second fails, the second one needs to know that and not
attempt to unregister itself on close.
We're going to need a test a for this. This is often the hard part, but it's
how we stop things breaking in future.
It'll have to be in the hadoop-aws package and presumably, an integration IT
test, That's where the next complication related to patch submission surfaces:
no patches in s3 get reviewed unless the submitter declares which endpoint
they've tested against, which means you need to get set up to run the existing
tests as wall as any new ones, [as
documented|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md].
Yetus can't run the tests (no credentials), we rely on the submitter to do it
before we ourselves run a final check before committing. Yes, it's a chore —but
it is how we verify that no changes break stuff.
Anyway, sorry for all that, but I hope recognise that its to keep everything
working, and to stop us all being lazy about testing and test coverage...
> Metrics of S3A don't print out when enable it in Hadoop metrics property file
> ------------------------------------------------------------------------------
>
> Key: HADOOP-14475
> URL: https://issues.apache.org/jira/browse/HADOOP-14475
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 2.8.0
> Environment: uname -a
> Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017
> x86_64 x86_64 x86_64 GNU/Linux
> cat /etc/issue
> Ubuntu 16.04.2 LTS \n \l
> Reporter: Yonger
> Attachments: s3a-metrics.patch1
>
>
> *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink
> #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #*.sink.influxdb.url=http:/xxxxxxxxxx
> #*.sink.influxdb.influxdb_port=8086
> #*.sink.influxdb.database=hadoop
> #*.sink.influxdb.influxdb_username=hadoop
> #*.sink.influxdb.influxdb_password=hadoop
> #*.sink.ingluxdb.cluster=c1
> *.period=10
> #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out
> I can't find the out put file even i run a MR job which should be used s3.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
