bsyk commented on code in PR #15340: URL: https://github.com/apache/druid/pull/15340#discussion_r1445530223
########## docs/development/extensions-contrib/spectator-histogram.md: ########## @@ -0,0 +1,453 @@ +--- +id: spectator-histogram +title: "Spectator Histogram module" +--- + +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +## Summary +This module provides Apache Druid approximate histogram aggregators and percentile +post-aggregators based on Spectator fixed-bucket histograms. + +Consider using this extension if you need percentile approximations and: +* want fast and accurate queries +* at a lower storage cost +* and have a large dataset +* using only positive measurements + +> The main benefit of this extension over data-sketches is the reduced storage +footprint. Which leads to smaller segment sizes, faster loading from deep storage +and lower memory usage. + +In the Druid instance shown below, the example Wikipedia dataset is loaded 3 times. +* As-is, no rollup applied +* With a single extra metric column of type `spectatorHistogram` ingesting the `added` column +* With a single extra metric column of type `quantilesDoublesSketch` ingesting the `added` column + +Spectator histograms average just 6 extra bytes per row, while the data-sketch +adds 48 bytes per row. This is an 8 x reduction in additional storage size. + + +As rollup improves, so does the size saving. For example, ingesting the wikipedia data +with day-grain query granularity and removing all dimensions except `countryName`, +we get to a segment that has just 106 rows. The base segment is 87 bytes per row, +adding a single `spectatorHistogram` column adds just 27 bytes per row on average vs +`quantilesDoublesSketch` adding 255 bytes per row. This is a 9.4 x reduction in additional storage size. +Storage gains will differ per dataset depending on the variance and rollup of the data. + +## Background +[Spectator](https://netflix.github.io/atlas-docs/spectator/) is a simple library +for instrumenting code to record dimensional time series data. +It was built, primarily, to work with [Atlas](https://netflix.github.io/atlas-docs/). +Atlas was developed by Netflix to manage dimensional time series data for near +real-time operational insight. + +With the [Atlas-Druid](https://github.com/Netflix-Skunkworks/iep-apps/tree/main/atlas-druid) +service, it's possible to use the power of Atlas queries, backed by Druid as a +data store to benefit from high-dimensionality and high-cardinality data. + +SpectatorHistogram is designed for efficient parallel aggregations while still +allowing for filtering and grouping by dimensions. +It provides similar functionality to the built-in data-sketch aggregator, but is +opinionated and optimized for typical measurements of cloud services and web-apps. Review Comment: There are opinions built into the implementation, the main one being that smaller values should be more accurately recorded. The opinion that we can afford to be off by a few million once we're into the many-millions range, but don't want to be off by more than 1 at the very small numbers. Happy to try to rephrase. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
