sascha-coenen edited a comment on issue #9321: Performance degradation in topN queries when SQL-compatible null handling is enabled URL: https://github.com/apache/druid/issues/9321#issuecomment-591691480 Hi. we tried the vectorization engine extensively and it works VERY well for us. Most of the times throughout the years when you guys reported significant performance gains, we could usually not reproduce them in our clusters very well. But with this thing we are right in the center: we are using local filter conditions a LOT and prior to vectorization, to my understanding, Druid had to scan over those, not applying bitmap filters whereas vectorization would already work with bitmaps. I don't know whether this assumption is true, but if feels to me that this is where the speedups might be the biggest. We do see speedups to 3x for timeseries across the board and 2x for groupby queries. The only exception is that we have this very often used class of queries, namely groupby's with an order by clause on a measure which don't profit from vectorization at all because they seem to be bound by the data volume streamed back from the historicals. To speed up this important class of queries we very much depend on the "forceLimitPushDown' feature which unfortunately hasn't been working for SQL queries for a while (since 0.11.0). This in combination with the vectorization engine would be blow away amazing. wrt the above perf issue: what you describe sounds reasonable. Let me add that it is very easy for us to reproduce this performance degradation. It does not matter which kind of groupbyV2 query we use. We basically saw it happen with pretty much any query. With topn it seems that the query needs to have a certain weight, like a couple of measures and I more than one sequential pass. You should be able to reproduce this on your end easily. Do you not experience any performance penalty in topNs at all when using SQL compatible null handling?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
