Re: Coordinator Write Metrics per CF
I have submitted a patch for https://issues.apache.org/jira/browse/CASSANDRA-14232. Looking for reviewers/feedback. Thanks, Sumanth On Tue, Feb 13, 2018 at 2:06 PM, Benedict Elliott Smith wrote: > Sorry, I guess I'm tired. I thought this was discussing local write > latency. > > I'm surprised we have that and not coordinator write latency. > > Please do ignore me, I'm not sure why I got involved! > > On 13 February 2018 at 21:48, Benedict Elliott Smith > wrote: > > > For the record, I'm not certain there *is* a great deal of value in this. > > > > The read latency metrics are expected to vary a great deal, since the > > entire IO subsystem is involved. > > > > Writes, however, go straight to a memtable. They only block on IO if we > > exceed our commit log flush bandwidth for an extended period of time. We > > already have a metric for tracking this: CommitLog.WaitingOnCommit. > > > > I'm not saying there won't be any latency distribution, but it is > unlikely > > to be terribly interesting in very many situations. I can't off the top > of > > my head think of a good reason to consult this metric, that couldn't > better > > be answered elsewhere. > > > > > > > > On 13 February 2018 at 19:18, Sumanth Pasupuleti < > spasupul...@netflix.com. > > invalid> wrote: > > > >> Thanks Kurt and Chris for your valuable inputs. Created > >> https://issues.apache.org/jira/browse/CASSANDRA-14232; I shall start > >> working on this. > >> > >> Thanks, > >> Sumanth > >> > >> On Mon, Feb 12, 2018 at 7:43 AM, Chris Lohfink > >> wrote: > >> > >> > It would be good to have it. Its not that its not there because its > >> > difficult or anything. I think its more that the read latency metric > was > >> > needed for speculative retry so it was added but the write side wasn't > >> > needed for that feature so wasn't added at same time. It would be very > >> > useful in determining the table that the coordinator writes are slow > to. > >> > > >> > Chris > >> > > >> > > On Feb 11, 2018, at 10:33 PM, kurt greaves > >> wrote: > >> > > > >> > > I've tried to search around for a reason for this in the past and > >> haven't > >> > > found one. I don't think it's a bad idea. Would be a helpful metric > to > >> > > diagnose internode networking issues - although I'll note that the > >> read > >> > > metric will also achieve this assuming you have enough reads to get > >> some > >> > > useful information out of it. > >> > > > >> > > On 9 February 2018 at 17:43, Sumanth Pasupuleti < > >> > > sumanth.pasupuleti...@gmail.com> wrote: > >> > > > >> > >> There is an existing CoordinatorReadLatency table metric. I am > >> looking > >> > to > >> > >> add CoordinatorWriteLatency table metric - however, before I > attempt > >> a > >> > shot > >> > >> at it, would like to know if anyone has context around why we > >> currently > >> > do > >> > >> not have such metric (while we have the read metric) - if someone > has > >> > >> already attempted and realized its a bad idea for some reason. > >> > >> > >> > >> Thanks, > >> > >> Sumanth > >> > >> > >> > > >> > > >> > - > >> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > >> > For additional commands, e-mail: dev-h...@cassandra.apache.org > >> > > >> > > >> > > > > >
Re: Coordinator Write Metrics per CF
Sorry, I guess I'm tired. I thought this was discussing local write latency. I'm surprised we have that and not coordinator write latency. Please do ignore me, I'm not sure why I got involved! On 13 February 2018 at 21:48, Benedict Elliott Smith wrote: > For the record, I'm not certain there *is* a great deal of value in this. > > The read latency metrics are expected to vary a great deal, since the > entire IO subsystem is involved. > > Writes, however, go straight to a memtable. They only block on IO if we > exceed our commit log flush bandwidth for an extended period of time. We > already have a metric for tracking this: CommitLog.WaitingOnCommit. > > I'm not saying there won't be any latency distribution, but it is unlikely > to be terribly interesting in very many situations. I can't off the top of > my head think of a good reason to consult this metric, that couldn't better > be answered elsewhere. > > > > On 13 February 2018 at 19:18, Sumanth Pasupuleti invalid> wrote: > >> Thanks Kurt and Chris for your valuable inputs. Created >> https://issues.apache.org/jira/browse/CASSANDRA-14232; I shall start >> working on this. >> >> Thanks, >> Sumanth >> >> On Mon, Feb 12, 2018 at 7:43 AM, Chris Lohfink >> wrote: >> >> > It would be good to have it. Its not that its not there because its >> > difficult or anything. I think its more that the read latency metric was >> > needed for speculative retry so it was added but the write side wasn't >> > needed for that feature so wasn't added at same time. It would be very >> > useful in determining the table that the coordinator writes are slow to. >> > >> > Chris >> > >> > > On Feb 11, 2018, at 10:33 PM, kurt greaves >> wrote: >> > > >> > > I've tried to search around for a reason for this in the past and >> haven't >> > > found one. I don't think it's a bad idea. Would be a helpful metric to >> > > diagnose internode networking issues - although I'll note that the >> read >> > > metric will also achieve this assuming you have enough reads to get >> some >> > > useful information out of it. >> > > >> > > On 9 February 2018 at 17:43, Sumanth Pasupuleti < >> > > sumanth.pasupuleti...@gmail.com> wrote: >> > > >> > >> There is an existing CoordinatorReadLatency table metric. I am >> looking >> > to >> > >> add CoordinatorWriteLatency table metric - however, before I attempt >> a >> > shot >> > >> at it, would like to know if anyone has context around why we >> currently >> > do >> > >> not have such metric (while we have the read metric) - if someone has >> > >> already attempted and realized its a bad idea for some reason. >> > >> >> > >> Thanks, >> > >> Sumanth >> > >> >> > >> > >> > - >> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org >> > For additional commands, e-mail: dev-h...@cassandra.apache.org >> > >> > >> > >
Re: Coordinator Write Metrics per CF
For the record, I'm not certain there *is* a great deal of value in this. The read latency metrics are expected to vary a great deal, since the entire IO subsystem is involved. Writes, however, go straight to a memtable. They only block on IO if we exceed our commit log flush bandwidth for an extended period of time. We already have a metric for tracking this: CommitLog.WaitingOnCommit. I'm not saying there won't be any latency distribution, but it is unlikely to be terribly interesting in very many situations. I can't off the top of my head think of a good reason to consult this metric, that couldn't better be answered elsewhere. On 13 February 2018 at 19:18, Sumanth Pasupuleti < spasupul...@netflix.com.invalid> wrote: > Thanks Kurt and Chris for your valuable inputs. Created > https://issues.apache.org/jira/browse/CASSANDRA-14232; I shall start > working on this. > > Thanks, > Sumanth > > On Mon, Feb 12, 2018 at 7:43 AM, Chris Lohfink wrote: > > > It would be good to have it. Its not that its not there because its > > difficult or anything. I think its more that the read latency metric was > > needed for speculative retry so it was added but the write side wasn't > > needed for that feature so wasn't added at same time. It would be very > > useful in determining the table that the coordinator writes are slow to. > > > > Chris > > > > > On Feb 11, 2018, at 10:33 PM, kurt greaves > wrote: > > > > > > I've tried to search around for a reason for this in the past and > haven't > > > found one. I don't think it's a bad idea. Would be a helpful metric to > > > diagnose internode networking issues - although I'll note that the read > > > metric will also achieve this assuming you have enough reads to get > some > > > useful information out of it. > > > > > > On 9 February 2018 at 17:43, Sumanth Pasupuleti < > > > sumanth.pasupuleti...@gmail.com> wrote: > > > > > >> There is an existing CoordinatorReadLatency table metric. I am looking > > to > > >> add CoordinatorWriteLatency table metric - however, before I attempt a > > shot > > >> at it, would like to know if anyone has context around why we > currently > > do > > >> not have such metric (while we have the read metric) - if someone has > > >> already attempted and realized its a bad idea for some reason. > > >> > > >> Thanks, > > >> Sumanth > > >> > > > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > For additional commands, e-mail: dev-h...@cassandra.apache.org > > > > >
Re: Coordinator Write Metrics per CF
Thanks Kurt and Chris for your valuable inputs. Created https://issues.apache.org/jira/browse/CASSANDRA-14232; I shall start working on this. Thanks, Sumanth On Mon, Feb 12, 2018 at 7:43 AM, Chris Lohfink wrote: > It would be good to have it. Its not that its not there because its > difficult or anything. I think its more that the read latency metric was > needed for speculative retry so it was added but the write side wasn't > needed for that feature so wasn't added at same time. It would be very > useful in determining the table that the coordinator writes are slow to. > > Chris > > > On Feb 11, 2018, at 10:33 PM, kurt greaves wrote: > > > > I've tried to search around for a reason for this in the past and haven't > > found one. I don't think it's a bad idea. Would be a helpful metric to > > diagnose internode networking issues - although I'll note that the read > > metric will also achieve this assuming you have enough reads to get some > > useful information out of it. > > > > On 9 February 2018 at 17:43, Sumanth Pasupuleti < > > sumanth.pasupuleti...@gmail.com> wrote: > > > >> There is an existing CoordinatorReadLatency table metric. I am looking > to > >> add CoordinatorWriteLatency table metric - however, before I attempt a > shot > >> at it, would like to know if anyone has context around why we currently > do > >> not have such metric (while we have the read metric) - if someone has > >> already attempted and realized its a bad idea for some reason. > >> > >> Thanks, > >> Sumanth > >> > > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > >
Re: Coordinator Write Metrics per CF
It would be good to have it. Its not that its not there because its difficult or anything. I think its more that the read latency metric was needed for speculative retry so it was added but the write side wasn't needed for that feature so wasn't added at same time. It would be very useful in determining the table that the coordinator writes are slow to. Chris > On Feb 11, 2018, at 10:33 PM, kurt greaves wrote: > > I've tried to search around for a reason for this in the past and haven't > found one. I don't think it's a bad idea. Would be a helpful metric to > diagnose internode networking issues - although I'll note that the read > metric will also achieve this assuming you have enough reads to get some > useful information out of it. > > On 9 February 2018 at 17:43, Sumanth Pasupuleti < > sumanth.pasupuleti...@gmail.com> wrote: > >> There is an existing CoordinatorReadLatency table metric. I am looking to >> add CoordinatorWriteLatency table metric - however, before I attempt a shot >> at it, would like to know if anyone has context around why we currently do >> not have such metric (while we have the read metric) - if someone has >> already attempted and realized its a bad idea for some reason. >> >> Thanks, >> Sumanth >> - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
Re: Coordinator Write Metrics per CF
I've tried to search around for a reason for this in the past and haven't found one. I don't think it's a bad idea. Would be a helpful metric to diagnose internode networking issues - although I'll note that the read metric will also achieve this assuming you have enough reads to get some useful information out of it. On 9 February 2018 at 17:43, Sumanth Pasupuleti < sumanth.pasupuleti...@gmail.com> wrote: > There is an existing CoordinatorReadLatency table metric. I am looking to > add CoordinatorWriteLatency table metric - however, before I attempt a shot > at it, would like to know if anyone has context around why we currently do > not have such metric (while we have the read metric) - if someone has > already attempted and realized its a bad idea for some reason. > > Thanks, > Sumanth >