Re: Coordinator Write Metrics per CF

[Sumanth Pasupuleti]

I have submitted a patch for
https://issues.apache.org/jira/browse/CASSANDRA-14232. Looking for
reviewers/feedback.

Thanks,
Sumanth

On Tue, Feb 13, 2018 at 2:06 PM, Benedict Elliott Smith  wrote:

> Sorry, I guess I'm tired.  I thought this was discussing local write
> latency.
>
> I'm surprised we have that and not coordinator write latency.
>
> Please do ignore me, I'm not sure why I got involved!
>
> On 13 February 2018 at 21:48, Benedict Elliott Smith 
> wrote:
>
> > For the record, I'm not certain there *is* a great deal of value in this.
> >
> > The read latency metrics are expected to vary a great deal, since the
> > entire IO subsystem is involved.
> >
> > Writes, however, go straight to a memtable.  They only block on IO if we
> > exceed our commit log flush bandwidth for an extended period of time.  We
> > already have a metric for tracking this: CommitLog.WaitingOnCommit.
> >
> > I'm not saying there won't be any latency distribution, but it is
> unlikely
> > to be terribly interesting in very many situations.  I can't off the top
> of
> > my head think of a good reason to consult this metric, that couldn't
> better
> > be answered elsewhere.
> >
> >
> >
> > On 13 February 2018 at 19:18, Sumanth Pasupuleti <
> spasupul...@netflix.com.
> > invalid> wrote:
> >
> >> Thanks Kurt and Chris for your valuable inputs. Created
> >> https://issues.apache.org/jira/browse/CASSANDRA-14232; I shall start
> >> working on this.
> >>
> >> Thanks,
> >> Sumanth
> >>
> >> On Mon, Feb 12, 2018 at 7:43 AM, Chris Lohfink 
> >> wrote:
> >>
> >> > It would be good to have it. Its not that its not there because its
> >> > difficult or anything. I think its more that the read latency metric
> was
> >> > needed for speculative retry so it was added but the write side wasn't
> >> > needed for that feature so wasn't added at same time. It would be very
> >> > useful in determining the table that the coordinator writes are slow
> to.
> >> >
> >> > Chris
> >> >
> >> > > On Feb 11, 2018, at 10:33 PM, kurt greaves 
> >> wrote:
> >> > >
> >> > > I've tried to search around for a reason for this in the past and
> >> haven't
> >> > > found one. I don't think it's a bad idea. Would be a helpful metric
> to
> >> > > diagnose internode networking issues - although I'll note that the
> >> read
> >> > > metric will also achieve this assuming you have enough reads to get
> >> some
> >> > > useful information out of it.
> >> > >
> >> > > On 9 February 2018 at 17:43, Sumanth Pasupuleti <
> >> > > sumanth.pasupuleti...@gmail.com> wrote:
> >> > >
> >> > >> There is an existing CoordinatorReadLatency table metric. I am
> >> looking
> >> > to
> >> > >> add CoordinatorWriteLatency table metric - however, before I
> attempt
> >> a
> >> > shot
> >> > >> at it, would like to know if anyone has context around why we
> >> currently
> >> > do
> >> > >> not have such metric (while we have the read metric) - if someone
> has
> >> > >> already attempted and realized its a bad idea for some reason.
> >> > >>
> >> > >> Thanks,
> >> > >> Sumanth
> >> > >>
> >> >
> >> >
> >> > -
> >> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >> >
> >> >
> >>
> >
> >
>

Re: Coordinator Write Metrics per CF

[Benedict Elliott Smith]

Sorry, I guess I'm tired.  I thought this was discussing local write
latency.

I'm surprised we have that and not coordinator write latency.

Please do ignore me, I'm not sure why I got involved!

On 13 February 2018 at 21:48, Benedict Elliott Smith 
wrote:

> For the record, I'm not certain there *is* a great deal of value in this.
>
> The read latency metrics are expected to vary a great deal, since the
> entire IO subsystem is involved.
>
> Writes, however, go straight to a memtable.  They only block on IO if we
> exceed our commit log flush bandwidth for an extended period of time.  We
> already have a metric for tracking this: CommitLog.WaitingOnCommit.
>
> I'm not saying there won't be any latency distribution, but it is unlikely
> to be terribly interesting in very many situations.  I can't off the top of
> my head think of a good reason to consult this metric, that couldn't better
> be answered elsewhere.
>
>
>
> On 13 February 2018 at 19:18, Sumanth Pasupuleti  invalid> wrote:
>
>> Thanks Kurt and Chris for your valuable inputs. Created
>> https://issues.apache.org/jira/browse/CASSANDRA-14232; I shall start
>> working on this.
>>
>> Thanks,
>> Sumanth
>>
>> On Mon, Feb 12, 2018 at 7:43 AM, Chris Lohfink 
>> wrote:
>>
>> > It would be good to have it. Its not that its not there because its
>> > difficult or anything. I think its more that the read latency metric was
>> > needed for speculative retry so it was added but the write side wasn't
>> > needed for that feature so wasn't added at same time. It would be very
>> > useful in determining the table that the coordinator writes are slow to.
>> >
>> > Chris
>> >
>> > > On Feb 11, 2018, at 10:33 PM, kurt greaves 
>> wrote:
>> > >
>> > > I've tried to search around for a reason for this in the past and
>> haven't
>> > > found one. I don't think it's a bad idea. Would be a helpful metric to
>> > > diagnose internode networking issues - although I'll note that the
>> read
>> > > metric will also achieve this assuming you have enough reads to get
>> some
>> > > useful information out of it.
>> > >
>> > > On 9 February 2018 at 17:43, Sumanth Pasupuleti <
>> > > sumanth.pasupuleti...@gmail.com> wrote:
>> > >
>> > >> There is an existing CoordinatorReadLatency table metric. I am
>> looking
>> > to
>> > >> add CoordinatorWriteLatency table metric - however, before I attempt
>> a
>> > shot
>> > >> at it, would like to know if anyone has context around why we
>> currently
>> > do
>> > >> not have such metric (while we have the read metric) - if someone has
>> > >> already attempted and realized its a bad idea for some reason.
>> > >>
>> > >> Thanks,
>> > >> Sumanth
>> > >>
>> >
>> >
>> > -
>> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> > For additional commands, e-mail: dev-h...@cassandra.apache.org
>> >
>> >
>>
>
>

Re: Coordinator Write Metrics per CF

[Benedict Elliott Smith]

For the record, I'm not certain there *is* a great deal of value in this.

The read latency metrics are expected to vary a great deal, since the
entire IO subsystem is involved.

Writes, however, go straight to a memtable.  They only block on IO if we
exceed our commit log flush bandwidth for an extended period of time.  We
already have a metric for tracking this: CommitLog.WaitingOnCommit.

I'm not saying there won't be any latency distribution, but it is unlikely
to be terribly interesting in very many situations.  I can't off the top of
my head think of a good reason to consult this metric, that couldn't better
be answered elsewhere.



On 13 February 2018 at 19:18, Sumanth Pasupuleti <
spasupul...@netflix.com.invalid> wrote:

> Thanks Kurt and Chris for your valuable inputs. Created
> https://issues.apache.org/jira/browse/CASSANDRA-14232; I shall start
> working on this.
>
> Thanks,
> Sumanth
>
> On Mon, Feb 12, 2018 at 7:43 AM, Chris Lohfink  wrote:
>
> > It would be good to have it. Its not that its not there because its
> > difficult or anything. I think its more that the read latency metric was
> > needed for speculative retry so it was added but the write side wasn't
> > needed for that feature so wasn't added at same time. It would be very
> > useful in determining the table that the coordinator writes are slow to.
> >
> > Chris
> >
> > > On Feb 11, 2018, at 10:33 PM, kurt greaves 
> wrote:
> > >
> > > I've tried to search around for a reason for this in the past and
> haven't
> > > found one. I don't think it's a bad idea. Would be a helpful metric to
> > > diagnose internode networking issues - although I'll note that the read
> > > metric will also achieve this assuming you have enough reads to get
> some
> > > useful information out of it.
> > >
> > > On 9 February 2018 at 17:43, Sumanth Pasupuleti <
> > > sumanth.pasupuleti...@gmail.com> wrote:
> > >
> > >> There is an existing CoordinatorReadLatency table metric. I am looking
> > to
> > >> add CoordinatorWriteLatency table metric - however, before I attempt a
> > shot
> > >> at it, would like to know if anyone has context around why we
> currently
> > do
> > >> not have such metric (while we have the read metric) - if someone has
> > >> already attempted and realized its a bad idea for some reason.
> > >>
> > >> Thanks,
> > >> Sumanth
> > >>
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> >
>

Re: Coordinator Write Metrics per CF

[Sumanth Pasupuleti]

Thanks Kurt and Chris for your valuable inputs. Created
https://issues.apache.org/jira/browse/CASSANDRA-14232; I shall start
working on this.

Thanks,
Sumanth

On Mon, Feb 12, 2018 at 7:43 AM, Chris Lohfink  wrote:

> It would be good to have it. Its not that its not there because its
> difficult or anything. I think its more that the read latency metric was
> needed for speculative retry so it was added but the write side wasn't
> needed for that feature so wasn't added at same time. It would be very
> useful in determining the table that the coordinator writes are slow to.
>
> Chris
>
> > On Feb 11, 2018, at 10:33 PM, kurt greaves  wrote:
> >
> > I've tried to search around for a reason for this in the past and haven't
> > found one. I don't think it's a bad idea. Would be a helpful metric to
> > diagnose internode networking issues - although I'll note that the read
> > metric will also achieve this assuming you have enough reads to get some
> > useful information out of it.
> >
> > On 9 February 2018 at 17:43, Sumanth Pasupuleti <
> > sumanth.pasupuleti...@gmail.com> wrote:
> >
> >> There is an existing CoordinatorReadLatency table metric. I am looking
> to
> >> add CoordinatorWriteLatency table metric - however, before I attempt a
> shot
> >> at it, would like to know if anyone has context around why we currently
> do
> >> not have such metric (while we have the read metric) - if someone has
> >> already attempted and realized its a bad idea for some reason.
> >>
> >> Thanks,
> >> Sumanth
> >>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: Coordinator Write Metrics per CF

[Chris Lohfink]

It would be good to have it. Its not that its not there because its difficult 
or anything. I think its more that the read latency metric was needed for 
speculative retry so it was added but the write side wasn't needed for that 
feature so wasn't added at same time. It would be very useful in determining 
the table that the coordinator writes are slow to.

Chris

> On Feb 11, 2018, at 10:33 PM, kurt greaves  wrote:
> 
> I've tried to search around for a reason for this in the past and haven't
> found one. I don't think it's a bad idea. Would be a helpful metric to
> diagnose internode networking issues - although I'll note that the read
> metric will also achieve this assuming you have enough reads to get some
> useful information out of it.
> 
> On 9 February 2018 at 17:43, Sumanth Pasupuleti <
> sumanth.pasupuleti...@gmail.com> wrote:
> 
>> There is an existing CoordinatorReadLatency table metric. I am looking to
>> add CoordinatorWriteLatency table metric - however, before I attempt a shot
>> at it, would like to know if anyone has context around why we currently do
>> not have such metric (while we have the read metric) - if someone has
>> already attempted and realized its a bad idea for some reason.
>> 
>> Thanks,
>> Sumanth
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org