[
https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16256303#comment-16256303
]
Sean Mackrory edited comment on HADOOP-14475 at 11/17/17 1:51 AM:
------------------------------------------------------------------
Attaching a patch trying out the idea of a separate MetricsImpl instance
instead of using the singleton. Each instance of S3AFileSystem is still
associated with its own source. This follows what Azure does (and being a cloud
storage client I think they should be on a similar model) and I also took a few
other things from the way Azure shuts down by doing a final flush of metrics
and only calling stop if it's the last metrics source in the s3a-file-system
metrics system. Other than that, this is pretty much still 99% Yonger's work. I
didn't run this through Yetus yet and I did a *little* refactoring after doing
a full test on a cluster - I'll revisit this in the morning for sure but wanted
to let discussion on the idea continue... I also haven't incorporated all of
Fabbri's feedback, just the trivial style issues.
[~iyonger] What do you think of this change to use a seperate instance of
MetricsImpl for "s3a-file-system"? I'm a lot more confident I understand the
difference between systems and contexts now (although contexts and record names
still seem like they could serve a similar purpose?), and this seems like the
right way to go for a cloud connector as opposed to a daemon.
was (Author: mackrorysd):
Attaching a patch trying out the idea of a separate MetricsImpl instance
instead of using the singleton. Each instance of S3AFileSystem is still
associated with its own source. This follows what Azure does (and being a cloud
storage client I think they should be on a similar model) and I also took a few
other things from the way Azure shuts down by doing a final flush of metrics
and only calling stop if it's the last metrics source in the s3a-file-system
metrics system. Other than that, this is pretty much still 99% Yonger's work. I
didn't run this through Yetus yet and I did a *little* refactoring after doing
a full test on a cluster - I'll revisit this in the morning for sure but wanted
to let discussion on the idea continue... I also haven't incorporated all of
Fabbri's feedback, just the trivial style issues. I'm also thinking the source
should probably be named after the bucket (bucket + number in case of
subsequent uses of the same bucket after calling close(), not enabling FS
instance caching, or anything else that causes multiple instances for the same
bucket) as opposed to just S3AFileSystemMetrics + a number. Any thoughts?
[~iyonger] What do you think of this change to use a seperate instance of
MetricsImpl for "s3a-file-system"? I'm a lot more confident I understand the
difference between systems and contexts now (although contexts and record names
still seem like they could serve a similar purpose?), and this seems like the
right way to go for a cloud connector as opposed to a daemon.
> Metrics of S3A don't print out when enable it in Hadoop metrics property file
> ------------------------------------------------------------------------------
>
> Key: HADOOP-14475
> URL: https://issues.apache.org/jira/browse/HADOOP-14475
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 2.8.0
> Environment: uname -a
> Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017
> x86_64 x86_64 x86_64 GNU/Linux
> cat /etc/issue
> Ubuntu 16.04.2 LTS \n \l
> Reporter: Yonger
> Assignee: Yonger
> Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch,
> HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch,
> HADOOP-14475.009.patch, HADOOP-14475.010.patch, HADOOP-14775.007.patch,
> failsafe-report-s3a-it.html, failsafe-report-s3a-scale.html,
> failsafe-report-scale.html, failsafe-report-scale.zip, s3a-metrics.patch1,
> stdout.zip
>
>
> *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink
> #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #*.sink.influxdb.url=http:/xxxxxxxxxx
> #*.sink.influxdb.influxdb_port=8086
> #*.sink.influxdb.database=hadoop
> #*.sink.influxdb.influxdb_username=hadoop
> #*.sink.influxdb.influxdb_password=hadoop
> #*.sink.ingluxdb.cluster=c1
> *.period=10
> #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out
> I can't find the out put file even i run a MR job which should be used s3.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
