[
https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16254647#comment-16254647
]
Aaron Fabbri commented on HADOOP-14475:
---------------------------------------
There is a learning curve for this Metrics2 stuff. I have some basic questions
that are important, and then a couple of minor nits on the patch.
*Basic Questions*:
{{DefaultMetricsSystem}} lifecycle API is confusing to me. It appears to have
a singleton, but we are calling {{stop()}} on it from
{{S3AFileSystem#close()}}. What if other stuff in the JVM is still using the
default metrics system when we close the S3A FS? We also call {{shutdown()}}
on the {{DefaultMetricsSystem}}, but that appears to use refcounts internally
to only really shutdown when the number of {{shutdown()}} calls reaches the
number of {{init()}} calls. {{stop()}} does not do this.
The {{MetricsSystemImpl}} interface exposes {{init()}} which calls {{start()}}
internally, but also exposes {{start()}} publicly. Humm.
Anyways.. I'm guessing we shouldn't be calling {{stop()}} from S3A's
{{close()}}. Seems like the metrics' {{shutdown()}} calls {{stop()}} when
refcount goes to zero?
*Patch feedback*:
{noformat}
+ // Init Metrics systemgti
+ DefaultMetricsSystem.initialize("s3afilesystem");
{noformat}
Typo in comment?
{noformat}
+ String msName=newMetricsSourceName();
...
+ if(null!=ms){
...
+ this.recordName=rName;
{noformat}
I guess checkstyle doesn't enforce assignment spacing, but most of the code
puts spaces around '='.
{noformat}
+ @VisibleForTesting
+ public static String newMetricsSourceName() {
+ int number = metricsSourceNameCounter.incrementAndGet();
+ final String baseName = "S3AFileSystemMetrics";
+ return baseName + number;
{noformat}
This name is getting long, with the bucket being appended later. Maybe remove
"Metrics" (should be obvious from context?) and use "S3AFileSystem"?
> Metrics of S3A don't print out when enable it in Hadoop metrics property file
> ------------------------------------------------------------------------------
>
> Key: HADOOP-14475
> URL: https://issues.apache.org/jira/browse/HADOOP-14475
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 2.8.0
> Environment: uname -a
> Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017
> x86_64 x86_64 x86_64 GNU/Linux
> cat /etc/issue
> Ubuntu 16.04.2 LTS \n \l
> Reporter: Yonger
> Assignee: Yonger
> Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch,
> HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch,
> HADOOP-14475.009.patch, HADOOP-14775.007.patch, failsafe-report-s3a-it.html,
> failsafe-report-s3a-scale.html, failsafe-report-scale.html,
> failsafe-report-scale.zip, s3a-metrics.patch1, stdout.zip
>
>
> *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink
> #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #*.sink.influxdb.url=http:/xxxxxxxxxx
> #*.sink.influxdb.influxdb_port=8086
> #*.sink.influxdb.database=hadoop
> #*.sink.influxdb.influxdb_username=hadoop
> #*.sink.influxdb.influxdb_password=hadoop
> #*.sink.ingluxdb.cluster=c1
> *.period=10
> #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out
> I can't find the out put file even i run a MR job which should be used s3.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
