Re: [kubernetes-users] Re: Filter the cadvisor metrics

2017-07-26 Thread Ben Kochie 
On Thu, Jul 27, 2017 at 8:35 AM, wangpeng007  wrote:

> Thanks for all your guys answers!
> I have taken your advise and grabe all metrics from cadvisor and filter it
> by service!
> Thanks again!
> I still have a small question:
> As containers has many status like:
>
>- Up 1 day (paused)
>- Restarting (123) 1 day ago
>- Up 1 day
>- Removal in Progress
>- Dead
>- Created
>- Exited (123) 1 day ago
>- (empty string)
>
> Which metrics in cadvisor show this?
>

Those look like event logs, not metrics.  They also look like Kubernetes
specifc state changes, which will not be in cAdvisor, but in the API or
Kubelet.  cAdvisor is for tracking the container use metrics.

-- 
You received this message because you are subscribed to the Google Groups 
"Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to kubernetes-users+unsubscr...@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

Re: [kubernetes-users] Re: Filter the cadvisor metrics

2017-07-26 Thread wangpeng007 
Thanks for all your guys answers!
I have taken your advise and grabe all metrics from cadvisor and filter it 
by service!
Thanks again!
I still have a small question:
As containers has many status like:

   - Up 1 day (paused)
   - Restarting (123) 1 day ago
   - Up 1 day
   - Removal in Progress
   - Dead
   - Created
   - Exited (123) 1 day ago
   - (empty string)
   
Which metrics in cadvisor show this?

On Thursday, July 27, 2017 at 1:22:42 AM UTC+8, Ben Kochie wrote:
>
> Yes, that is correct.
>
> Similar techniques apply for Prometheus.  There is a set of rewrite rules 
> that can be used to filter metrics on ingestion, and recording rules to 
> aggregate data.
>
> As Matthias says, this kind of policy stuff generally better left to the 
> service reading the data, because it knows what is necessary or not.
>
> There are some use case for the metrics request to include a 
> filter/collector pattern, to help target the data returned.  But there 
> isn't a good universal pattern for this yet.  
>
> On Jul 26, 2017 18:43, "Justin W. Flory" > 
> wrote:
>
>> On 07/26/2017 02:19 AM, Matthias Rampke wrote:
>> > What issues agree these causing? They are returned pretty quickly, can
>> > you just fetch them all and filter in whatever collects them?
>> >
>> > A server side filter API could become quite complex trying to cover all
>> > possible ways to filter, while it's rather easy to filter for any
>> > particular use case on the client side.
>> >
>> > /MR
>> >
>> >
>> > On Wed, Jul 26, 2017, 05:39 wangpeng007 > 
>> > > wrote:
>> >
>> > Is there any suggestion?
>> >
>> >
>> > On Tuesday, July 18, 2017 at 4:06:57 PM UTC+8, wangpeng007 wrote:
>> >
>> > cAdvisor returns metrics for a lot of the running processes on
>> > the system its running on. This results in a lot of extra
>> > metrics that cAdvisor exposes, and for the most these are
>> > metrics we do not care about.
>> > Is there any API or configuration to filter the cadvisor 
>> metrics?
>> >
>>
>> Admittedly, I'm pretty novice to all of this, but if you're taking these
>> metrics and putting them into a time-series database (depending on what
>> you're using), you could set up opt-in filters to only collect the
>> measurements you actually want or need.
>>
>> If you're using InfluxDB / Kapacitor specifically, you could also write
>> a script that aggregates some of the measurements together into smaller
>> ones, and then write those ones out to the database for storing.
>>
>> Not sure if this really answers your question, but if the reason you're
>> wanting to filter out some of the metrics is disk space, you probably
>> want to look into a way to only collect the measurements you want / need
>> on an opt-in basis, so the rest are discarded.
>>
>> --
>> Cheers,
>> Justin W. Flory
>> jfl...@gmail.com 
>>
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "Kubernetes user discussion and Q&A" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to kubernetes-use...@googlegroups.com .
>> To post to this group, send email to kubernet...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/kubernetes-users.
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to kubernetes-users+unsubscr...@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

Re: [kubernetes-users] Re: Filter the cadvisor metrics

2017-07-26 Thread Ben Kochie 
Yes, that is correct.

Similar techniques apply for Prometheus.  There is a set of rewrite rules
that can be used to filter metrics on ingestion, and recording rules to
aggregate data.

As Matthias says, this kind of policy stuff generally better left to the
service reading the data, because it knows what is necessary or not.

There are some use case for the metrics request to include a
filter/collector pattern, to help target the data returned.  But there
isn't a good universal pattern for this yet.

On Jul 26, 2017 18:43, "Justin W. Flory"  wrote:

> On 07/26/2017 02:19 AM, Matthias Rampke wrote:
> > What issues agree these causing? They are returned pretty quickly, can
> > you just fetch them all and filter in whatever collects them?
> >
> > A server side filter API could become quite complex trying to cover all
> > possible ways to filter, while it's rather easy to filter for any
> > particular use case on the client side.
> >
> > /MR
> >
> >
> > On Wed, Jul 26, 2017, 05:39 wangpeng007  > > wrote:
> >
> > Is there any suggestion?
> >
> >
> > On Tuesday, July 18, 2017 at 4:06:57 PM UTC+8, wangpeng007 wrote:
> >
> > cAdvisor returns metrics for a lot of the running processes on
> > the system its running on. This results in a lot of extra
> > metrics that cAdvisor exposes, and for the most these are
> > metrics we do not care about.
> > Is there any API or configuration to filter the cadvisor metrics?
> >
>
> Admittedly, I'm pretty novice to all of this, but if you're taking these
> metrics and putting them into a time-series database (depending on what
> you're using), you could set up opt-in filters to only collect the
> measurements you actually want or need.
>
> If you're using InfluxDB / Kapacitor specifically, you could also write
> a script that aggregates some of the measurements together into smaller
> ones, and then write those ones out to the database for storing.
>
> Not sure if this really answers your question, but if the reason you're
> wanting to filter out some of the metrics is disk space, you probably
> want to look into a way to only collect the measurements you want / need
> on an opt-in basis, so the rest are discarded.
>
> --
> Cheers,
> Justin W. Flory
> jflo...@gmail.com
>
> --
> You received this message because you are subscribed to the Google Groups
> "Kubernetes user discussion and Q&A" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kubernetes-users+unsubscr...@googlegroups.com.
> To post to this group, send email to kubernetes-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/kubernetes-users.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to kubernetes-users+unsubscr...@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

Re: [kubernetes-users] Re: Filter the cadvisor metrics

2017-07-26 Thread Justin W. Flory 
On 07/26/2017 02:19 AM, Matthias Rampke wrote:
> What issues agree these causing? They are returned pretty quickly, can
> you just fetch them all and filter in whatever collects them?
> 
> A server side filter API could become quite complex trying to cover all
> possible ways to filter, while it's rather easy to filter for any
> particular use case on the client side.
> 
> /MR
> 
> 
> On Wed, Jul 26, 2017, 05:39 wangpeng007  > wrote:
> 
> Is there any suggestion?
> 
> 
> On Tuesday, July 18, 2017 at 4:06:57 PM UTC+8, wangpeng007 wrote:
> 
> cAdvisor returns metrics for a lot of the running processes on
> the system its running on. This results in a lot of extra
> metrics that cAdvisor exposes, and for the most these are
> metrics we do not care about. 
> Is there any API or configuration to filter the cadvisor metrics?
> 

Admittedly, I'm pretty novice to all of this, but if you're taking these
metrics and putting them into a time-series database (depending on what
you're using), you could set up opt-in filters to only collect the
measurements you actually want or need.

If you're using InfluxDB / Kapacitor specifically, you could also write
a script that aggregates some of the measurements together into smaller
ones, and then write those ones out to the database for storing.

Not sure if this really answers your question, but if the reason you're
wanting to filter out some of the metrics is disk space, you probably
want to look into a way to only collect the measurements you want / need
on an opt-in basis, so the rest are discarded.

-- 
Cheers,
Justin W. Flory
jflo...@gmail.com

-- 
You received this message because you are subscribed to the Google Groups 
"Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to kubernetes-users+unsubscr...@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature

Re: [kubernetes-users] Re: Filter the cadvisor metrics

2017-07-26 Thread Matthias Rampke 
What issues agree these causing? They are returned pretty quickly, can you
just fetch them all and filter in whatever collects them?

A server side filter API could become quite complex trying to cover all
possible ways to filter, while it's rather easy to filter for any
particular use case on the client side.

/MR

On Wed, Jul 26, 2017, 05:39 wangpeng007  wrote:

> Is there any suggestion?
>
>
> On Tuesday, July 18, 2017 at 4:06:57 PM UTC+8, wangpeng007 wrote:
>>
>> cAdvisor returns metrics for a lot of the running processes on the system
>> its running on. This results in a lot of extra metrics that cAdvisor
>> exposes, and for the most these are metrics we do not care about.
>> Is there any API or configuration to filter the cadvisor metrics?
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Kubernetes user discussion and Q&A" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kubernetes-users+unsubscr...@googlegroups.com.
> To post to this group, send email to kubernetes-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/kubernetes-users.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to kubernetes-users+unsubscr...@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.