jon-wei commented on issue #8276: KIS tasks in 0.15.1 RC2 sometimes duplicate rows with the same dimension values URL: https://github.com/apache/incubator-druid/issues/8276#issuecomment-520048197 With Kafka ingestion, it's possible that rollup is not 100%, due to the incremental publishing, e.g.: - Kafka task gets a row like `{"timestamp":"2019-08-09 14:00:00", "dim1":"hello", "metric":1}` - Kafka task hits a memory/row/time limit and does an incremental publish - If after the incremental publish, another row `{"timestamp":"2019-08-09 14:00:00", "dim1":"hello", "metric":2}` arrives, it won't be rolled up into the first already published segment, and a non-aggregating query would return both of those rows If you run an exact aggregating query like: ``` SELECT accountid, SUM(cpu) FROM "kube-metrics-minute" WHERE accountid='52c7e28eebb26a86fb1505b5' AND "__time" >= TIMESTAMP '2019-08-09 14:00:00' AND "__time" < TIMESTAMP '2019-08-09 16:30:00' AND cluster='ds' AND etype='container' AND namespace='default' AND name='app-alertproc' AND pod='alertproc-deployment-65f59b6b8-gc57m' GROUP BY accountid ``` does the sum on CPU return the expected value? You could compare the results of that query across the minute/second/30m datasources and see if they match, to rule out imperfect rollup from incremental publishes.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
