Re: Some Total and Rate metrics are not consistent

[Sam Lendle]

Yep that’s the change I saw. It does look like the issue should be fixed since 
the implementation of update for Count always adds 1 regardless of the value.

Thanks for the followup.

On 6/21/18, 1:15 PM, "John Roesler" <j...@confluent.io> wrote:

    Hi Sam,
    
    This sounds like a condition I fixed in
    
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_kafka_commit_ed51b2cdf5bdac210a6904bead1a2ca6e8411406-23diff-2D8b364ed2d0abd8e8ae21f5d322db6564R221&d=DwIFaQ&c=gFTBenQ7Vj71sUi1A4CkFnmPzqwDo07QsHw-JRepxyw&r=BNCekDhngyXB6C2Ag7PIfHotiuqjAVwLOZLQHB7fyOM&m=q-lWaxuP-yHvVqew6oPVJiQPM_xal1kSHyqFZKeXFe4&s=fHXuRWDvMLDx75JEh_h4Y50zooJ9Z03d77TnX4tnxzE&e=
    . I realized that the prior code creates a new Meter, which uses a Total
    metric instead of a Count. But that would total all the values of the
    metric, when instead what we want is to "total" the number of measurements
    (aka count them).
    
    I just peeked at the 1.1 branch, and it seems this change made it in after
    the 1.1 branch cut, so it would only be fixed in 2.0.
    
    Thanks,
    -John
    
    On Wed, Jun 20, 2018 at 8:44 PM Guozhang Wang <wangg...@gmail.com> wrote:
    
    > Thanks for reporting this Sam, could you check and confirm if this issue 
is
    > fixed in trunk? If not, we should file a JIRA.
    >
    >
    > Guozhang
    >
    > On Wed, Jun 20, 2018 at 6:41 PM, Sam Lendle <slen...@pandora.com> wrote:
    >
    > > It looks like there is indeed a bug in kafka-streams 1.1.0. I think what
    > > was happening was the time spent processing each record in ns was being
    > > added to the total metric instead of incrementing by 1 for each record.
    > > Looks like the implementation has been changed in trunk. I don't see any
    > > commit messages mentioning this particular issue, but hopefully the
    > change
    > > fixes it.
    > > ------------------------------
    > > *From:* Sam Lendle
    > > *Sent:* Wednesday, June 20, 2018 6:10:03 PM
    > > *To:* users@kafka.apache.org
    > > *Subject:* Some Total and Rate metrics are not consistent
    > >
    > >
    > > I’m trying to use the total metrics introduced in KIP-187 (
    > > 
https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D&d=DwIFaQ&c=gFTBenQ7Vj71sUi1A4CkFnmPzqwDo07QsHw-JRepxyw&r=BNCekDhngyXB6C2Ag7PIfHotiuqjAVwLOZLQHB7fyOM&m=q-lWaxuP-yHvVqew6oPVJiQPM_xal1kSHyqFZKeXFe4&s=jYa-lA5HwT7eOoFDIhz215N6PbZerIuO9AQDqLpNYlQ&e=
    > > 187+-+Add+cumulative+count+metric+for+all+Kafka+rate+metrics)
    > >
    > >
    > >
    > > For some metrics, the total and rates are not consistent. In particular,
    > > for stream-processor-node-metrics, I’m seeing about 500-800 operations
    > per
    > > second in a particular streams thread/processor node as reported by the
    > > process-rate metric, but the process-total metric is increasing by about
    > > 100 million per second. See attached screenshot from VisualVM.
    > >
    > >
    > >
    > > Other metrics seem fine, for example forward-rate and forward-total
    > > metrics under stream-processor-node-metrics are consistent.
    > >
    > >
    > >
    > > Am I misunderstanding the interpretation of the –total metrics? If this
    > is
    > > a bug, can I do anything in addition to this email to report it? File a
    > > JIRA?
    > >
    > >
    > > Best,
    > > Sam
    > >
    > >
    > >
    > >
    > >
    > >
    > >
    > >
    > >
    >
    >
    > --
    > -- Guozhang
    >