[jira] [Comment Edited] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269021#comment-16269021 ] Sean Mackrory edited comment on HADOOP-14475 at 11/28/17 4:44 PM: -- Yeah FileSink seems to be too primitive for real-world use: I'm looking at RollingFileSystemSink for all but basic testing of this patch. I'm supportive of adding other sinks, and of using one for testing (I gave FileSink a quick try for that, but since it truncates the file everytime you have to tail -f it, or you lose most of the data), but let's maybe handle that as a distinct issue from hooking up S3A to metrics2 as a source? was (Author: mackrorysd): Yeah FileSink seems to be too primitive for real-world use: I'm looking at RollingFileSystemSink for all but basic testing of this patch. I'm supportive of adding other sinks, especially for testing (I gave FileSink a quick try for that, but since it truncates the file everytime you have to tail -f it, or you lose most of the data), but let's maybe handle that as a distinct issue from hooking up S3A to metrics2 as a source? > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch, > HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch, > HADOOP-14475.009.patch, HADOOP-14475.010.patch, HADOOP-14475.011.patch, > HADOOP-14475.012.patch, HADOOP-14475.013.patch, HADOOP-14475.014.patch, > HADOOP-14475.015.patch, HADOOP-14775.007.patch, failsafe-report-s3a-it.html, > failsafe-report-s3a-scale.html, failsafe-report-scale.html, > failsafe-report-scale.zip, s3a-metrics.patch1, stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268073#comment-16268073 ] Sean Mackrory edited comment on HADOOP-14475 at 11/28/17 4:07 AM: -- [~iyonger] I don't believe I've changed anything that would affect whether or not the metrics source name shows up in the sink's output - I've only changed what the source name would be. I think the format even when I tested your original patch was pretty much as it is above, and did not include the source name, only the record name (which happened to be similar). If you're wanting to aggregate based on the bucket, I would use the bucket field itself. There's no guarantee that metrics source names would even be consistent among all JVMs for a given bucket, since they're assigned numbers in the order that they're created - that would only be true if every JVM had accessed the exact same buckets in the exact same order - the assumption would break down as soon as a job didn't utilize the entire cluster or a node was down during a job, etc. was (Author: mackrorysd): [~iyonger] I don't believe I've changed anything that would affect whether or not the metrics source name shows up in the sink's output - I've only changed what the source name would be. I think the format even when I tested your original patch was pretty much as it is above, and did not include the source name, only the record name (which happened to be similar). If you're wanting to aggregate based on the bucket, I would use the bucket field itself. There's no guarantee that metrics source names would even be consistent among all JVMs for a given bucket - that would only be true if every JVM had accessed the exact same buckets in the exact same order - the assumption would break down as soon as a job didn't utilize the entire cluster or a node was down during a job, etc. > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch, > HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch, > HADOOP-14475.009.patch, HADOOP-14475.010.patch, HADOOP-14475.011.patch, > HADOOP-14475.012.patch, HADOOP-14475.013.patch, HADOOP-14475.014.patch, > HADOOP-14475.015.patch, HADOOP-14775.007.patch, failsafe-report-s3a-it.html, > failsafe-report-s3a-scale.html, failsafe-report-scale.html, > failsafe-report-scale.zip, s3a-metrics.patch1, stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16256303#comment-16256303 ] Sean Mackrory edited comment on HADOOP-14475 at 11/17/17 1:51 AM: -- Attaching a patch trying out the idea of a separate MetricsImpl instance instead of using the singleton. Each instance of S3AFileSystem is still associated with its own source. This follows what Azure does (and being a cloud storage client I think they should be on a similar model) and I also took a few other things from the way Azure shuts down by doing a final flush of metrics and only calling stop if it's the last metrics source in the s3a-file-system metrics system. Other than that, this is pretty much still 99% Yonger's work. I didn't run this through Yetus yet and I did a *little* refactoring after doing a full test on a cluster - I'll revisit this in the morning for sure but wanted to let discussion on the idea continue... I also haven't incorporated all of Fabbri's feedback, just the trivial style issues. [~iyonger] What do you think of this change to use a seperate instance of MetricsImpl for "s3a-file-system"? I'm a lot more confident I understand the difference between systems and contexts now (although contexts and record names still seem like they could serve a similar purpose?), and this seems like the right way to go for a cloud connector as opposed to a daemon. was (Author: mackrorysd): Attaching a patch trying out the idea of a separate MetricsImpl instance instead of using the singleton. Each instance of S3AFileSystem is still associated with its own source. This follows what Azure does (and being a cloud storage client I think they should be on a similar model) and I also took a few other things from the way Azure shuts down by doing a final flush of metrics and only calling stop if it's the last metrics source in the s3a-file-system metrics system. Other than that, this is pretty much still 99% Yonger's work. I didn't run this through Yetus yet and I did a *little* refactoring after doing a full test on a cluster - I'll revisit this in the morning for sure but wanted to let discussion on the idea continue... I also haven't incorporated all of Fabbri's feedback, just the trivial style issues. I'm also thinking the source should probably be named after the bucket (bucket + number in case of subsequent uses of the same bucket after calling close(), not enabling FS instance caching, or anything else that causes multiple instances for the same bucket) as opposed to just S3AFileSystemMetrics + a number. Any thoughts? [~iyonger] What do you think of this change to use a seperate instance of MetricsImpl for "s3a-file-system"? I'm a lot more confident I understand the difference between systems and contexts now (although contexts and record names still seem like they could serve a similar purpose?), and this seems like the right way to go for a cloud connector as opposed to a daemon. > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch, > HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch, > HADOOP-14475.009.patch, HADOOP-14475.010.patch, HADOOP-14775.007.patch, > failsafe-report-s3a-it.html, failsafe-report-s3a-scale.html, > failsafe-report-scale.html, failsafe-report-scale.zip, s3a-metrics.patch1, > stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail:
[jira] [Comment Edited] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[
https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16254662#comment-16254662
]
Sean Mackrory edited comment on HADOOP-14475 at 11/16/17 2:37 AM:
--
{quote}DefaultMetricsSystem lifecycle API is confusing to me.{quote}
Thanks for bringing this up. One point of confusion for me has been the
difference between a system and a context (see my above comments), but one
important thing to note is that most other "systems" represent an entire JVM.
That's not true in this case when the metrics system may exist in the
NodeManager JVM that has its own metrics system(s) already and there may be
multiple instances of S3AFileSystem. I notice that Azure appears to be creating
it's own instance instead of using the singleton. I suspect we should be doing
the same.
I have tested this on an actual YARN cluster prior to the addition of the
close() stuff, and it seemed to not be interfering at all with other YARN
metrics, but what we're doing in close() does make that more serious.
was (Author: mackrorysd):
{quote}DefaultMetricsSystem lifecycle API is confusing to me.{quote}
Thanks for bringing this up. One point of confusion for me has been the
difference between a system and a context (see my above comments), but one
important thing to note is that most other "systems" represent an entire JVM.
That's not true in this case when the metrics system may exist in the
NodeManager JVM that has its own metrics system(s) already and there may be
multiple instances of S3AFileSystem. I notice that Azure appears to be creating
it's own instance instead of using the singleton. I suspect we should be doing
the same.
> Metrics of S3A don't print out when enable it in Hadoop metrics property file
> --
>
> Key: HADOOP-14475
> URL: https://issues.apache.org/jira/browse/HADOOP-14475
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.8.0
> Environment: uname -a
> Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017
> x86_64 x86_64 x86_64 GNU/Linux
> cat /etc/issue
> Ubuntu 16.04.2 LTS \n \l
>Reporter: Yonger
>Assignee: Yonger
> Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch,
> HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch,
> HADOOP-14475.009.patch, HADOOP-14775.007.patch, failsafe-report-s3a-it.html,
> failsafe-report-s3a-scale.html, failsafe-report-scale.html,
> failsafe-report-scale.zip, s3a-metrics.patch1, stdout.zip
>
>
> *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink
> #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #*.sink.influxdb.url=http:/xx
> #*.sink.influxdb.influxdb_port=8086
> #*.sink.influxdb.database=hadoop
> #*.sink.influxdb.influxdb_username=hadoop
> #*.sink.influxdb.influxdb_password=hadoop
> #*.sink.ingluxdb.cluster=c1
> *.period=10
> #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out
> I can't find the out put file even i run a MR job which should be used s3.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[
https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252804#comment-16252804
]
Yonger edited comment on HADOOP-14475 at 11/15/17 1:44 AM:
---
For the # 2 point of my update
--
without this update,the metrics be flush into file like this(each record with
same URI mark the same "context+registry name" as begin):
{code:java}
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
Note, these three records written at the same time instead of different
intervals, but they should be consider three different kinds of
metrics(different fsid,different values in real logs).
(Additional, I don't know why the metrics be registered many times within a
process,like why we need to method "newMetricsSourceName")
with this update, each record will be easy to distinguish:
{code:java}
1510631110244 s3afs.S3aFileSystemMetrics: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics2: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics3: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
was (Author: iyonger):
For the # 2 point of my update
--
without this update,the metrics be flush into file like this(each record with
same URI mark the same "context+registry name" as begin):
{code:java}
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
Note, these three records written at the same time instead of different
intervals, so they should be consider three different kinds of metrics.
(Additional, I don't know why the metrics be registered many times within a
process,like why we need to method "newMetricsSourceName")
with this update, each record will be easy to distinguish:
{code:java}
1510631110244 s3afs.S3aFileSystemMetrics: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics2: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics3: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
> Metrics of S3A don't print out when enable it in Hadoop metrics property file
> --
>
> Key: HADOOP-14475
> URL: https://issues.apache.org/jira/browse/HADOOP-14475
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.8.0
> Environment: uname -a
> Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017
> x86_64 x86_64 x86_64 GNU/Linux
> cat /etc/issue
> Ubuntu 16.04.2 LTS \n \l
>Reporter: Yonger
>Assignee: Yonger
> Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch,
> HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch,
> HADOOP-14775.007.patch, failsafe-report-s3a-it.html,
> failsafe-report-s3a-scale.html, failsafe-report-scale.html,
> failsafe-report-scale.zip, s3a-metrics.patch1, stdout.zip
>
>
> *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink
> #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #*.sink.influxdb.url=http:/xx
> #*.sink.influxdb.influxdb_port=8086
> #*.sink.influxdb.database=hadoop
> #*.sink.influxdb.influxdb_username=hadoop
> #*.sink.influxdb.influxdb_password=hadoop
> #*.sink.ingluxdb.cluster=c1
> *.period=10
>
[jira] [Comment Edited] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[
https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252804#comment-16252804
]
Yonger edited comment on HADOOP-14475 at 11/15/17 1:42 AM:
---
For the # 2 point of my update
--
without this update,the metrics be flush into file like this(each record with
same URI mark the same "context+registry name" as begin):
{code:java}
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
Note, these three records written at the same time instead of different
intervals, so they should be consider three different kinds of metrics.
(Additional, I don't know why the metrics be registered many times within a
process,like why we need to method "newMetricsSourceName")
with this update, each record will be easy to distinguish:
{code:java}
1510631110244 s3afs.S3aFileSystemMetrics: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics2: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics3: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
was (Author: iyonger):
For the # 2 point of my update
--
without this update,the metrics be flush into file like this(each record with
same URI mark the same "context+registry name" as begin):
{code:java}
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
with this update, each record will be easy to distinguish:
{code:java}
1510631110244 s3afs.S3aFileSystemMetrics: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics2: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics3: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
> Metrics of S3A don't print out when enable it in Hadoop metrics property file
> --
>
> Key: HADOOP-14475
> URL: https://issues.apache.org/jira/browse/HADOOP-14475
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.8.0
> Environment: uname -a
> Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017
> x86_64 x86_64 x86_64 GNU/Linux
> cat /etc/issue
> Ubuntu 16.04.2 LTS \n \l
>Reporter: Yonger
>Assignee: Yonger
> Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch,
> HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch,
> HADOOP-14775.007.patch, failsafe-report-s3a-it.html,
> failsafe-report-s3a-scale.html, failsafe-report-scale.html,
> failsafe-report-scale.zip, s3a-metrics.patch1, stdout.zip
>
>
> *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink
> #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #*.sink.influxdb.url=http:/xx
> #*.sink.influxdb.influxdb_port=8086
> #*.sink.influxdb.database=hadoop
> #*.sink.influxdb.influxdb_username=hadoop
> #*.sink.influxdb.influxdb_password=hadoop
> #*.sink.ingluxdb.cluster=c1
> *.period=10
> #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out
> I can't find the out put file even i run a MR job which should be used s3.
--
This message was sent by Atlassian
[jira] [Comment Edited] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[
https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252804#comment-16252804
]
Yonger edited comment on HADOOP-14475 at 11/15/17 1:31 AM:
---
For the # 2 point of my update
--
without this update,the metrics be flush into file like this(each record with
same URI mark the same "context+registry name" as begin):
{code:java}
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
with this update, each record will be easy to distinguish:
{code:java}
1510631110244 s3afs.S3aFileSystemMetrics: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics2: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics3: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
was (Author: iyonger):
For the # 2 point of my update
--
without this update,the metrics be flush into file like this(each record mark
the same "context+registry name" as begin):
{code:java}
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3AFileSystem: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
with this update, each record will be easy to distinguish:
{code:java}
1510631110244 s3afs.S3aFileSystemMetrics: Context=s3afs,
FileSystemId=dd721794-8269-44eb-911f-a39caa2e6b34-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics2: Context=s3afs,
FileSystemId=8fe27f60-8b3d-41c1-aa84-8bb812315d55-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
1510631110244 s3afs.S3aFileSystemMetrics3: Context=s3afs,
FileSystemId=6bcf9a67-2dcd-4533-aa5b-2ded7a8ba038-bdaas-demo-dfs,
fsURI=s3a://bdaas-demo-dfs/, Hostname=client01, xxx
{code}
> Metrics of S3A don't print out when enable it in Hadoop metrics property file
> --
>
> Key: HADOOP-14475
> URL: https://issues.apache.org/jira/browse/HADOOP-14475
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.8.0
> Environment: uname -a
> Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017
> x86_64 x86_64 x86_64 GNU/Linux
> cat /etc/issue
> Ubuntu 16.04.2 LTS \n \l
>Reporter: Yonger
>Assignee: Yonger
> Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch,
> HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch,
> HADOOP-14775.007.patch, failsafe-report-s3a-it.html,
> failsafe-report-s3a-scale.html, failsafe-report-scale.html,
> failsafe-report-scale.zip, s3a-metrics.patch1, stdout.zip
>
>
> *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink
> #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #*.sink.influxdb.url=http:/xx
> #*.sink.influxdb.influxdb_port=8086
> #*.sink.influxdb.database=hadoop
> #*.sink.influxdb.influxdb_username=hadoop
> #*.sink.influxdb.influxdb_password=hadoop
> #*.sink.ingluxdb.cluster=c1
> *.period=10
> #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink
> S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out
> I can't find the out put file even i run a MR job which should be used s3.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16076511#comment-16076511 ] Yonger edited comment on HADOOP-14475 at 7/6/17 1:42 PM: - @steve the method you mentioned give an empty url to skip the landsat-pds tests is not work, also I upload the gz file into my bucket according to the guide, but it failed too. when giving the empty string, error message: Tests run: 9, Failures: 0, Errors: 6, Skipped: 0, Time elapsed: 0.325 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider testInstantiationChain(org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider) Time elapsed: 0.018 sec <<< ERROR! java.lang.IllegalArgumentException: Can not create a Path from an empty string at org.apache.hadoop.fs.Path.checkPathArg(Path.java:163) at org.apache.hadoop.fs.Path.(Path.java:175) at org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider.testInstantiationChain(TestS3AAWSCredentialsProvider.java:92) and if i use default value and upload the gz file, which give me a error message with code 403. was (Author: iyonger): [~stevea] the method you mentioned give an empty url to skip the landsat-pds tests is not work, also I upload the gz file into my bucket according to the guide, but it failed too. when giving the empty string, error message: Tests run: 9, Failures: 0, Errors: 6, Skipped: 0, Time elapsed: 0.325 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider testInstantiationChain(org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider) Time elapsed: 0.018 sec <<< ERROR! java.lang.IllegalArgumentException: Can not create a Path from an empty string at org.apache.hadoop.fs.Path.checkPathArg(Path.java:163) at org.apache.hadoop.fs.Path.(Path.java:175) at org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider.testInstantiationChain(TestS3AAWSCredentialsProvider.java:92) and if i use default value and upload the gz file, which give me a error message with code 403. > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: failsafe-report-s3a-it.html, > failsafe-report-s3a-scale.html, HADOOP-14475.002.patch, s3a-metrics.patch1, > stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16058815#comment-16058815 ] Yonger edited comment on HADOOP-14475 at 6/22/17 6:18 AM: -- Fixed this issue by initializing metrics system and registering s3a metrics source implementation was (Author: iyonger): Fixed this issue by initializing metrics system and registering s3a metrcis source implementation > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: HADOOP-14475.002.patch, s3a-metrics.patch1, stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16037032#comment-16037032 ] Yonger edited comment on HADOOP-14475 at 6/5/17 2:28 PM: - This patch can fix the issue that s3a metrics don't print out to sink(file).But there also introduce an issue that the metrics info only print out once to file instead of printing out periodically. was (Author: iyonger): This patch can fix the issue that s3a metrics don't print out to sink(file).But there also introduce an issue that the metrics info only print out once in file instead of printing our periodically. > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger > Attachments: s3a-metrics.patch1 > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
