techdocsmith commented on code in PR #13524: URL: https://github.com/apache/druid/pull/13524#discussion_r1051165775
########## docs/development/extensions-core/datasketches-hll.md: ########## @@ -23,29 +23,32 @@ title: "DataSketches HLL Sketch module" --> -This module provides Apache Druid aggregators for distinct counting based on HLL sketch from [Apache DataSketches](https://datasketches.apache.org/) library. At ingestion time, this aggregator creates the HLL sketch objects to be stored in Druid segments. At query time, sketches are read and merged together. In the end, by default, you receive the estimate of the number of distinct values presented to the sketch. Also, you can use post aggregator to produce a union of sketch columns in the same row. -You can use the HLL sketch aggregator on columns of any identifiers. It will return estimated cardinality of the column. +This module provides Apache Druid aggregators for distinct counting based on HLL sketch from [Apache DataSketches](https://datasketches.apache.org/) library. At ingestion time, this aggregator creates the HLL sketch objects to be stored in Druid segments. At query time, sketches are read and merged together. In the end, by default, you receive the estimate of the number of distinct values presented to the sketch. Also, you can use post aggregators to produce a union of sketch columns in the same row. Review Comment: ```suggestion This module provides Apache Druid aggregators for distinct counting based on HLL sketch from [Apache DataSketches](https://datasketches.apache.org/) library. At ingestion time, this aggregator creates the HLL sketch objects to store in Druid segments. By default, Druid reads and merges sketches at query time. The default result is the estimate of the number of distinct values presented to the sketch. You can also use post aggregators to produce a union of sketch columns in the same row. ``` ########## docs/development/extensions-core/datasketches-hll.md: ########## @@ -23,29 +23,32 @@ title: "DataSketches HLL Sketch module" --> -This module provides Apache Druid aggregators for distinct counting based on HLL sketch from [Apache DataSketches](https://datasketches.apache.org/) library. At ingestion time, this aggregator creates the HLL sketch objects to be stored in Druid segments. At query time, sketches are read and merged together. In the end, by default, you receive the estimate of the number of distinct values presented to the sketch. Also, you can use post aggregator to produce a union of sketch columns in the same row. -You can use the HLL sketch aggregator on columns of any identifiers. It will return estimated cardinality of the column. +This module provides Apache Druid aggregators for distinct counting based on HLL sketch from [Apache DataSketches](https://datasketches.apache.org/) library. At ingestion time, this aggregator creates the HLL sketch objects to be stored in Druid segments. At query time, sketches are read and merged together. In the end, by default, you receive the estimate of the number of distinct values presented to the sketch. Also, you can use post aggregators to produce a union of sketch columns in the same row. +You can use the HLL sketch aggregator on any column to estimate its cardinality. To use this aggregator, make sure you [include](../../development/extensions.md#loading-extensions) the extension in your config file: ``` druid.extensions.loadList=["druid-datasketches"] ``` -### Aggregators +For additional sketch types supported in Druid, see [DataSketches extension](datasketches-extension.md). -|property|description|required?| +## Aggregators + +|Property|Description|Required?| |--------|-----------|---------| -|`type`|This String should be [`HLLSketchBuild`](#hllsketchbuild-aggregator) or [`HLLSketchMerge`](#hllsketchmerge-aggregator)|yes| -|`name`|A String for the output (result) name of the calculation.|yes| -|`fieldName`|A String for the name of the input field.|yes| +|`type`|This string should be [`HLLSketchBuild`](#hllsketchbuild-aggregator) or [`HLLSketchMerge`](#hllsketchmerge-aggregator).|yes| +|`name`|A string for the output (result) name of the calculation.|yes| +|`fieldName`|A string for the name of the input field.|yes| Review Comment: ```suggestion |`fieldName`|The name of the input field.|yes| ``` ########## docs/development/extensions-core/datasketches-hll.md: ########## @@ -23,29 +23,32 @@ title: "DataSketches HLL Sketch module" --> -This module provides Apache Druid aggregators for distinct counting based on HLL sketch from [Apache DataSketches](https://datasketches.apache.org/) library. At ingestion time, this aggregator creates the HLL sketch objects to be stored in Druid segments. At query time, sketches are read and merged together. In the end, by default, you receive the estimate of the number of distinct values presented to the sketch. Also, you can use post aggregator to produce a union of sketch columns in the same row. -You can use the HLL sketch aggregator on columns of any identifiers. It will return estimated cardinality of the column. +This module provides Apache Druid aggregators for distinct counting based on HLL sketch from [Apache DataSketches](https://datasketches.apache.org/) library. At ingestion time, this aggregator creates the HLL sketch objects to be stored in Druid segments. At query time, sketches are read and merged together. In the end, by default, you receive the estimate of the number of distinct values presented to the sketch. Also, you can use post aggregators to produce a union of sketch columns in the same row. +You can use the HLL sketch aggregator on any column to estimate its cardinality. To use this aggregator, make sure you [include](../../development/extensions.md#loading-extensions) the extension in your config file: ``` druid.extensions.loadList=["druid-datasketches"] ``` -### Aggregators +For additional sketch types supported in Druid, see [DataSketches extension](datasketches-extension.md). -|property|description|required?| +## Aggregators + +|Property|Description|Required?| |--------|-----------|---------| -|`type`|This String should be [`HLLSketchBuild`](#hllsketchbuild-aggregator) or [`HLLSketchMerge`](#hllsketchmerge-aggregator)|yes| -|`name`|A String for the output (result) name of the calculation.|yes| -|`fieldName`|A String for the name of the input field.|yes| +|`type`|This string should be [`HLLSketchBuild`](#hllsketchbuild-aggregator) or [`HLLSketchMerge`](#hllsketchmerge-aggregator).|yes| Review Comment: ```suggestion |`type`|Either [`HLLSketchBuild`](#hllsketchbuild-aggregator) or [`HLLSketchMerge`](#hllsketchmerge-aggregator).|yes| ``` ########## docs/development/extensions-core/datasketches-theta.md: ########## @@ -45,17 +50,18 @@ druid.extensions.loadList=["druid-datasketches"] } ``` -|property|description|required?| +|Property|Description|Required?| |--------|-----------|---------| -|type|This String should always be "thetaSketch"|yes| -|name|A String for the output (result) name of the calculation.|yes| -|fieldName|A String for the name of the aggregator used at ingestion time.|yes| -|isInputThetaSketch|This should only be used at indexing time if your input data contains theta sketch objects. This would be the case if you use datasketches library outside of Druid, say with Pig/Hive, to produce the data that you are ingesting into Druid |no, defaults to false| -|size|Must be a power of 2. Internally, size refers to the maximum number of entries sketch object will retain. Higher size means higher accuracy but more space to store sketches. Note that after you index with a particular size, druid will persist sketch in segments and you will use size greater or equal to that at query time. See the [DataSketches site](https://datasketches.apache.org/docs/Theta/ThetaSize) for details. In general, We recommend just sticking to default size. |no, defaults to 16384| +|`type`|This string should always be "thetaSketch"|yes| +|`name`|A string for the output (result) name of the calculation.|yes| Review Comment: ```suggestion `name`|String representing the output column to store sketch values.|yes| ``` ########## docs/development/extensions-core/datasketches-theta.md: ########## @@ -45,17 +50,18 @@ druid.extensions.loadList=["druid-datasketches"] } ``` -|property|description|required?| +|Property|Description|Required?| |--------|-----------|---------| -|type|This String should always be "thetaSketch"|yes| -|name|A String for the output (result) name of the calculation.|yes| -|fieldName|A String for the name of the aggregator used at ingestion time.|yes| -|isInputThetaSketch|This should only be used at indexing time if your input data contains theta sketch objects. This would be the case if you use datasketches library outside of Druid, say with Pig/Hive, to produce the data that you are ingesting into Druid |no, defaults to false| -|size|Must be a power of 2. Internally, size refers to the maximum number of entries sketch object will retain. Higher size means higher accuracy but more space to store sketches. Note that after you index with a particular size, druid will persist sketch in segments and you will use size greater or equal to that at query time. See the [DataSketches site](https://datasketches.apache.org/docs/Theta/ThetaSize) for details. In general, We recommend just sticking to default size. |no, defaults to 16384| +|`type`|This string should always be "thetaSketch"|yes| +|`name`|A string for the output (result) name of the calculation.|yes| +|`fieldName`|A string for the name of the aggregator used at ingestion time.|yes| +|`isInputThetaSketch`|This should only be used at indexing time if your input data contains theta sketch objects. This would be the case if you use datasketches library outside of Druid, say with Pig/Hive, to produce the data that you are ingesting into Druid |no, defaults to false| +|`size`|Must be a power of 2. Internally, size refers to the maximum number of entries sketch object will retain. Higher size means higher accuracy but more space to store sketches. Note that after you index with a particular size, druid will persist sketch in segments and you will use size greater or equal to that at query time. See the [DataSketches site](https://datasketches.apache.org/docs/Theta/ThetaSize) for details. In general, We recommend just sticking to default size. |no, defaults to 16384| Review Comment: ```suggestion |`size`|Must be a power of 2. Internally, size refers to the maximum number of entries sketch object retains. Higher size means higher accuracy but more space to store sketches. After you index with a particular size, Druid persists the sketch in segments. At query time you must use a size greater or equal to the ingested size. See the [DataSketches site](https://datasketches.apache.org/docs/Theta/ThetaSize) for details. The default is recommended for the majority of use cases. |no, defaults to 16384| ``` ########## docs/development/extensions-core/datasketches-hll.md: ########## @@ -23,29 +23,32 @@ title: "DataSketches HLL Sketch module" --> -This module provides Apache Druid aggregators for distinct counting based on HLL sketch from [Apache DataSketches](https://datasketches.apache.org/) library. At ingestion time, this aggregator creates the HLL sketch objects to be stored in Druid segments. At query time, sketches are read and merged together. In the end, by default, you receive the estimate of the number of distinct values presented to the sketch. Also, you can use post aggregator to produce a union of sketch columns in the same row. -You can use the HLL sketch aggregator on columns of any identifiers. It will return estimated cardinality of the column. +This module provides Apache Druid aggregators for distinct counting based on HLL sketch from [Apache DataSketches](https://datasketches.apache.org/) library. At ingestion time, this aggregator creates the HLL sketch objects to be stored in Druid segments. At query time, sketches are read and merged together. In the end, by default, you receive the estimate of the number of distinct values presented to the sketch. Also, you can use post aggregators to produce a union of sketch columns in the same row. +You can use the HLL sketch aggregator on any column to estimate its cardinality. To use this aggregator, make sure you [include](../../development/extensions.md#loading-extensions) the extension in your config file: ``` druid.extensions.loadList=["druid-datasketches"] ``` -### Aggregators +For additional sketch types supported in Druid, see [DataSketches extension](datasketches-extension.md). -|property|description|required?| +## Aggregators + +|Property|Description|Required?| |--------|-----------|---------| -|`type`|This String should be [`HLLSketchBuild`](#hllsketchbuild-aggregator) or [`HLLSketchMerge`](#hllsketchmerge-aggregator)|yes| -|`name`|A String for the output (result) name of the calculation.|yes| -|`fieldName`|A String for the name of the input field.|yes| +|`type`|This string should be [`HLLSketchBuild`](#hllsketchbuild-aggregator) or [`HLLSketchMerge`](#hllsketchmerge-aggregator).|yes| +|`name`|A string for the output (result) name of the calculation.|yes| Review Comment: ```suggestion `name`|String representing the output column to store sketch values.|yes| ``` ########## docs/development/extensions-core/datasketches-kll.md: ########## @@ -50,17 +52,17 @@ The result of the aggregation is a KllFloatsSketch or KllDoublesSketch that is t } ``` -|property|description|required?| +|Property|Description|Required?| |--------|-----------|---------| -|type|This String should be "KllFloatsSketch" or "KllDoublesSketch"|yes| -|name|A String for the output (result) name of the calculation.|yes| -|fieldName|A String for the name of the input field (can contain sketches or raw numeric values).|yes| -|k|Parameter that determines the accuracy and size of the sketch. Higher k means higher accuracy but more space to store sketches. Must be from 8 to 65535. See [KLL Sketch Accuracy and Size](https://datasketches.apache.org/docs/KLL/KLLAccuracyAndSize.html).|no, defaults to 200| -|maxStreamLength|This parameter defines the number of items that can be presented to each sketch before it may need to move from off-heap to on-heap memory. This is relevant to query types that use off-heap memory, including [TopN](../../querying/topnquery.md) and [GroupBy](../../querying/groupbyquery.md). Ideally, should be set high enough such that most sketches can stay off-heap.|no, defaults to 1000000000| +|`type`|This String should be "KllFloatsSketch" or "KllDoublesSketch"|yes| Review Comment: ```suggestion |`type`|Either "KllFloatsSketch" or "KllDoublesSketch"|yes| ``` ########## docs/development/extensions-core/datasketches-kll.md: ########## @@ -50,17 +52,17 @@ The result of the aggregation is a KllFloatsSketch or KllDoublesSketch that is t } ``` -|property|description|required?| +|Property|Description|Required?| |--------|-----------|---------| -|type|This String should be "KllFloatsSketch" or "KllDoublesSketch"|yes| -|name|A String for the output (result) name of the calculation.|yes| -|fieldName|A String for the name of the input field (can contain sketches or raw numeric values).|yes| -|k|Parameter that determines the accuracy and size of the sketch. Higher k means higher accuracy but more space to store sketches. Must be from 8 to 65535. See [KLL Sketch Accuracy and Size](https://datasketches.apache.org/docs/KLL/KLLAccuracyAndSize.html).|no, defaults to 200| -|maxStreamLength|This parameter defines the number of items that can be presented to each sketch before it may need to move from off-heap to on-heap memory. This is relevant to query types that use off-heap memory, including [TopN](../../querying/topnquery.md) and [GroupBy](../../querying/groupbyquery.md). Ideally, should be set high enough such that most sketches can stay off-heap.|no, defaults to 1000000000| +|`type`|This String should be "KllFloatsSketch" or "KllDoublesSketch"|yes| +|`name`|A String for the output (result) name of the calculation.|yes| +|`fieldName`|A String for the name of the input field (can contain sketches or raw numeric values).|yes| Review Comment: ```suggestion `name`|String representing the output column to store sketch values.|yes| ``` ########## docs/development/extensions-core/datasketches-theta.md: ########## @@ -45,17 +50,18 @@ druid.extensions.loadList=["druid-datasketches"] } ``` -|property|description|required?| +|Property|Description|Required?| |--------|-----------|---------| -|type|This String should always be "thetaSketch"|yes| -|name|A String for the output (result) name of the calculation.|yes| -|fieldName|A String for the name of the aggregator used at ingestion time.|yes| -|isInputThetaSketch|This should only be used at indexing time if your input data contains theta sketch objects. This would be the case if you use datasketches library outside of Druid, say with Pig/Hive, to produce the data that you are ingesting into Druid |no, defaults to false| -|size|Must be a power of 2. Internally, size refers to the maximum number of entries sketch object will retain. Higher size means higher accuracy but more space to store sketches. Note that after you index with a particular size, druid will persist sketch in segments and you will use size greater or equal to that at query time. See the [DataSketches site](https://datasketches.apache.org/docs/Theta/ThetaSize) for details. In general, We recommend just sticking to default size. |no, defaults to 16384| +|`type`|This string should always be "thetaSketch"|yes| +|`name`|A string for the output (result) name of the calculation.|yes| +|`fieldName`|A string for the name of the aggregator used at ingestion time.|yes| +|`isInputThetaSketch`|This should only be used at indexing time if your input data contains theta sketch objects. This would be the case if you use datasketches library outside of Druid, say with Pig/Hive, to produce the data that you are ingesting into Druid |no, defaults to false| Review Comment: ```suggestion |`isInputThetaSketch`|Only set this to true at indexing time if your input data contains theta sketch objects. This applies to cases when you use datasketches library outside of Druid, Pig/Hive for hexample, to produce the data to ingest into Druid |no, defaults to false| ``` ########## docs/development/extensions-core/datasketches-quantiles.md: ########## @@ -50,17 +52,18 @@ The result of the aggregation is a DoublesSketch that is the union of all sketch } ``` -|property|description|required?| +|Property|Description|Required?| |--------|-----------|---------| -|type|This String should always be "quantilesDoublesSketch"|yes| -|name|A String for the output (result) name of the calculation.|yes| -|fieldName|A String for the name of the input field (can contain sketches or raw numeric values).|yes| -|k|Parameter that determines the accuracy and size of the sketch. Higher k means higher accuracy but more space to store sketches. Must be a power of 2 from 2 to 32768. See [accuracy information](https://datasketches.apache.org/docs/Quantiles/OrigQuantilesSketch) in the DataSketches documentation for details.|no, defaults to 128| -|maxStreamLength|This parameter defines the number of items that can be presented to each sketch before it may need to move from off-heap to on-heap memory. This is relevant to query types that use off-heap memory, including [TopN](../../querying/topnquery.md) and [GroupBy](../../querying/groupbyquery.md). Ideally, should be set high enough such that most sketches can stay off-heap.|no, defaults to 1000000000| +|`type`|This string should always be "quantilesDoublesSketch"|yes| +|`name`|A string for the output (result) name of the calculation.|yes| Review Comment: ```suggestion `name`|String representing the output column to store sketch values.|yes| ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
