[Bug 64181] Kafkatee zero files having 10% less requests than udp2log zero files

[bugzilla-daemon]

https://bugzilla.wikimedia.org/show_bug.cgi?id=64181

--- Comment #5 from christ...@quelltextlich.at ---
Since it was constantly missing 10% and we have 10 partitions, the
assumption is that some partition did not get properly fed into
varnishkafka.

Ottomata restarted varnishkafka, and the problem disappeared:
(...-$DATE.gz is filename,
  2014-04-20T$HOUR:xx:xx is the extracted hour, and
  difference is NUMBER_OF_LINES_IN_KAFKATEE_FILE * 100 / NUMBER_OF_LINES_IN_
UDP2LOG_FILE - 100
    (we want difference to be 0)
)


...-20140421.gz: 2014-04-20T07:xx:xx   -11% difference
...-20140421.gz: 2014-04-20T08:xx:xx   -11% difference
...-20140421.gz: 2014-04-20T09:xx:xx   -11% difference
...-20140421.gz: 2014-04-20T10:xx:xx   -11% difference
...-20140421.gz: 2014-04-20T11:xx:xx   -11% difference
...-20140421.gz: 2014-04-20T12:xx:xx   -11% difference
...-20140421.gz: 2014-04-20T13:xx:xx   -10% difference
...-20140421.gz: 2014-04-20T14:xx:xx   -11% difference
...-20140421.gz: 2014-04-20T15:xx:xx   -10% difference
...-20140421.gz: 2014-04-20T16:xx:xx   -11% difference
...-20140421.gz: 2014-04-20T17:xx:xx   -11% difference
...-20140421.gz: 2014-04-20T18:xx:xx   -11% difference
...-20140421.gz: 2014-04-20T19:xx:xx   -10% difference
...-20140421.gz: 2014-04-20T20:xx:xx   -10% difference
...-20140421.gz: 2014-04-20T21:xx:xx   -10% difference
...-20140421.gz: 2014-04-20T22:xx:xx   -11% difference
...-20140421.gz: 2014-04-20T23:xx:xx   -11% difference
...-20140421.gz: 2014-04-21T00:xx:xx   -11% difference
...-20140421.gz: 2014-04-21T01:xx:xx   -11% difference
...-20140421.gz: 2014-04-21T02:xx:xx   -10% difference
...-20140421.gz: 2014-04-21T03:xx:xx   -10% difference
...-20140421.gz: 2014-04-21T04:xx:xx   -11% difference
...-20140421.gz: 2014-04-21T05.xx:xx   -11% difference
...-20140421.gz: 2014-04-21T06:xx:xx   -25% difference <-- log rotation
...-20140422.gz: 2014-04-21T06:xx:xx    -4% difference <-- log rotation
...-20140422.gz: 2014-04-21T07:xx:xx   -11% difference
...-20140422.gz: 2014-04-21T08:xx:xx   -10% difference
...-20140422.gz: 2014-04-21T09:xx:xx   -11% difference
...-20140422.gz: 2014-04-21T10:xx:xx   -11% difference
...-20140422.gz: 2014-04-21T11:xx:xx    -6% difference <-- ?
...-20140422.gz: 2014-04-21T12:xx:xx   -11% difference
...-20140422.gz: 2014-04-21T13:xx:xx   -10% difference
...-20140422.gz: 2014-04-21T14:xx:xx   -10% difference
...-20140422.gz: 2014-04-21T15:xx:xx   -10% difference
...-20140422.gz: 2014-04-21T16:xx:xx   -10% difference
...-20140422.gz: 2014-04-21T17:xx:xx   -10% difference
...-20140422.gz: 2014-04-21T18:xx:xx    -9% difference <-- ottomata restarting
kafkatee ~18:30
...-20140422.gz: 2014-04-21T19:xx:xx   -48% difference
...-20140422.gz: 2014-04-21T20:xx:xx     0% difference <-- back to normal.
...-20140422.gz: 2014-04-21T21:xx:xx     0% difference
...-20140422.gz: 2014-04-21T22:xx:xx    -1% difference
...-20140422.gz: 2014-04-21T23:xx:xx    -1% difference
...-20140422.gz: 2014-04-22T00:xx:xx    -1% difference
...-20140422.gz: 2014-04-22T01:xx:xx    -1% difference
...-20140422.gz: 2014-04-22T02:xx:xx    -1% difference
...-20140422.gz: 2014-04-22T03:xx:xx    -1% difference
...-20140422.gz: 2014-04-22T04:xx:xx    -1% difference
...-20140422.gz: 2014-04-22T05:xx:xx    -1% difference


How can we guard against partitions not being fed into varnishkafka in
future?

-- 
You are receiving this mail because:
You are the assignee for the bug.
You are on the CC list for the bug.
_______________________________________________
Wikibugs-l mailing list
Wikibugs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikibugs-l