jill-imply commented on code in PR #12547: URL: https://github.com/apache/druid/pull/12547#discussion_r880294370
########## druid-0.23-release-notes.md: ########## @@ -5,34 +5,183 @@ Apache Druid 0.23.0 contains over 450 new features, bug fixes, performance enhan ## Query engine ### Grouping on arrays without exploding the arrays -[12078](https://github.com/apache/druid/pull/12078) +You can now group on a multi-value dimension as an array. For a datasource named "test": +``` +{"timestamp": "2011-01-12T00:00:00.000Z", "tags": ["t1","t2","t3"]} #row1 +{"timestamp": "2011-01-13T00:00:00.000Z", "tags": ["t3","t4","t5"]} #row2 +{"timestamp": "2011-01-14T00:00:00.000Z", "tags": ["t5","t6","t7"]} #row3 +{"timestamp": "2011-01-14T00:00:00.000Z", "tags": []} #row4 +``` +The following query: +```json +{ + "queryType": "groupBy", + "dataSource": "test", + "intervals": [ + "1970-01-01T00:00:00.000Z/3000-01-01T00:00:00.000Z" + ], + "granularity": { + "type": "all" + }, + "virtualColumns" : [ { + "type" : "expression", + "name" : "v0", + "expression" : "mv_to_array(\"tags\")", + "outputType" : "ARRAY<STRING>" + } ], + "dimensions": [ + { + "type": "default", + "dimension": "v0", + "outputName": "tags" + "outputType":"ARRAY<STRING>" + } + ], + "aggregations": [ + { + "type": "count", + "name": "count" + } + ] +} +``` +Returns the following: + +``` +[ + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "[]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "["t1","t2","t3"]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "[t3","t4","t5"]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 2, + "tags": "["t5","t6","t7"]" + } + } +] +``` +(#12078) +(#12253) ### Specify a column other than __time column for row comparison in first/last aggregators -[11949](https://github.com/apache/druid/pull/11949) + +You can pass time column in `*first`/`*last` aggregators and LATEST / EARLIEST SQL functions. This provides support for cases where the time is stored as a part of a column different than "__time". You can also specify another logical time column. +(#11949) +(#12145) + +### Improvements to querying user experience + +This release includes several improvements for querying: +- Added the SQL query id to response header for failed SQL query to aid in location the error messages (#11756) +- Added input type validation for DataSketches HLL (#12131) +- Improved JDBC logging (#11676) +- Added SQL functions MV_FILTER_ONLY and MV_FILTER_NONE to filter rows of multi-value string dimensions to include only the supplied list of values or none of them respectively (#11650) +- Added ARRAY_CONCAT_AGG to aggregate array inputs together into a single array(#12226) +- Added the ability to authorize the usage of query context parameters (#12396) +- Improved query IDs to make it easier to link queryies and sub-queries for end-to-end visibility of a query (#11809) Review Comment: ```suggestion - Improved query IDs to make it easier to link queries and sub-queries for end-to-end query visibility (#11809) ``` ########## druid-0.23-release-notes.md: ########## @@ -5,34 +5,183 @@ Apache Druid 0.23.0 contains over 450 new features, bug fixes, performance enhan ## Query engine ### Grouping on arrays without exploding the arrays -[12078](https://github.com/apache/druid/pull/12078) +You can now group on a multi-value dimension as an array. For a datasource named "test": +``` +{"timestamp": "2011-01-12T00:00:00.000Z", "tags": ["t1","t2","t3"]} #row1 +{"timestamp": "2011-01-13T00:00:00.000Z", "tags": ["t3","t4","t5"]} #row2 +{"timestamp": "2011-01-14T00:00:00.000Z", "tags": ["t5","t6","t7"]} #row3 +{"timestamp": "2011-01-14T00:00:00.000Z", "tags": []} #row4 +``` +The following query: +```json +{ + "queryType": "groupBy", + "dataSource": "test", + "intervals": [ + "1970-01-01T00:00:00.000Z/3000-01-01T00:00:00.000Z" + ], + "granularity": { + "type": "all" + }, + "virtualColumns" : [ { + "type" : "expression", + "name" : "v0", + "expression" : "mv_to_array(\"tags\")", + "outputType" : "ARRAY<STRING>" + } ], + "dimensions": [ + { + "type": "default", + "dimension": "v0", + "outputName": "tags" + "outputType":"ARRAY<STRING>" + } + ], + "aggregations": [ + { + "type": "count", + "name": "count" + } + ] +} +``` +Returns the following: + +``` +[ + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "[]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "["t1","t2","t3"]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "[t3","t4","t5"]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 2, + "tags": "["t5","t6","t7"]" + } + } +] +``` +(#12078) +(#12253) ### Specify a column other than __time column for row comparison in first/last aggregators -[11949](https://github.com/apache/druid/pull/11949) + +You can pass time column in `*first`/`*last` aggregators and LATEST / EARLIEST SQL functions. This provides support for cases where the time is stored as a part of a column different than "__time". You can also specify another logical time column. +(#11949) +(#12145) + +### Improvements to querying user experience + +This release includes several improvements for querying: +- Added the SQL query id to response header for failed SQL query to aid in location the error messages (#11756) Review Comment: ```suggestion - Added the SQL query ID to response header for failed SQL query to aid in locating the error messages (#11756) ``` ########## druid-0.23-release-notes.md: ########## @@ -5,34 +5,183 @@ Apache Druid 0.23.0 contains over 450 new features, bug fixes, performance enhan ## Query engine ### Grouping on arrays without exploding the arrays -[12078](https://github.com/apache/druid/pull/12078) +You can now group on a multi-value dimension as an array. For a datasource named "test": +``` +{"timestamp": "2011-01-12T00:00:00.000Z", "tags": ["t1","t2","t3"]} #row1 +{"timestamp": "2011-01-13T00:00:00.000Z", "tags": ["t3","t4","t5"]} #row2 +{"timestamp": "2011-01-14T00:00:00.000Z", "tags": ["t5","t6","t7"]} #row3 +{"timestamp": "2011-01-14T00:00:00.000Z", "tags": []} #row4 +``` +The following query: +```json +{ + "queryType": "groupBy", + "dataSource": "test", + "intervals": [ + "1970-01-01T00:00:00.000Z/3000-01-01T00:00:00.000Z" + ], + "granularity": { + "type": "all" + }, + "virtualColumns" : [ { + "type" : "expression", + "name" : "v0", + "expression" : "mv_to_array(\"tags\")", + "outputType" : "ARRAY<STRING>" + } ], + "dimensions": [ + { + "type": "default", + "dimension": "v0", + "outputName": "tags" + "outputType":"ARRAY<STRING>" + } + ], + "aggregations": [ + { + "type": "count", + "name": "count" + } + ] +} +``` +Returns the following: + +``` +[ + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "[]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "["t1","t2","t3"]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "[t3","t4","t5"]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 2, + "tags": "["t5","t6","t7"]" + } + } +] +``` +(#12078) +(#12253) ### Specify a column other than __time column for row comparison in first/last aggregators -[11949](https://github.com/apache/druid/pull/11949) + +You can pass time column in `*first`/`*last` aggregators and LATEST / EARLIEST SQL functions. This provides support for cases where the time is stored as a part of a column different than "__time". You can also specify another logical time column. +(#11949) +(#12145) + +### Improvements to querying user experience + +This release includes several improvements for querying: +- Added the SQL query id to response header for failed SQL query to aid in location the error messages (#11756) +- Added input type validation for DataSketches HLL (#12131) +- Improved JDBC logging (#11676) +- Added SQL functions MV_FILTER_ONLY and MV_FILTER_NONE to filter rows of multi-value string dimensions to include only the supplied list of values or none of them respectively (#11650) +- Added ARRAY_CONCAT_AGG to aggregate array inputs together into a single array(#12226) +- Added the ability to authorize the usage of query context parameters (#12396) +- Improved query IDs to make it easier to link queryies and sub-queries for end-to-end visibility of a query (#11809) +- Added a safe divide function to protect against division by 0 (#11904) +- You can now add a query context to internally generated `SegmentMetadata` Query (#11429) +- Added support for Druid complex types to the native expression processing system to make all Druid data usable within expressions (#11853, #12016) ## Streaming Ingestion -### Kafka Input format for parsing headers and key -[11630](https://github.com/apache/druid/pull/11630) +### Kafka input format for parsing headers and key +We've introduced a Kafka input format so you can ingest header data in addition to the message contents. For example: +- the event key field +- event headers +- the Kafka event timestamp +- the Kafka event value that stores the payload. + +(#11630) ## Native Batch Ingestion -### Multi-dimensional range partitioning -Multi-dimensional range partitioning allows users to partition their data on the ranges of any number of dimensions. It develops further on the concepts behind "single-dim" partitioning and is now arguably the most preferable secondary partitioning both for query performance and storage efficiency. -[11848](https://github.com/apache/druid/pull/11848) -[11973](https://github.com/apache/druid/pull/11973) +### Multi-dimension range partitioning +Multi-dimension range partitioning allows users to partition their data on the ranges of any number of dimensions. It develops further on the concepts behind "single-dim" partitioning and is now arguably the most preferable secondary partitioning both for query performance and storage efficiency. +(#11848) +(#11973) + +### Improved replace data behavior +In previous versions of Druid, if ingested data with `dropExisting` flag to replace data, Druid would retain the existing data for a time chunk if there was no new data to replace it. Now, if you set `dropExisting` to `true` in your `ioSpec` and ingest data for a time range that includes a time chunk with no data, Druid uses a tombstone to overshadow the existing data in the empty time chunk. +(#12137) + +This release includes several improvements for native batch ingestion: +- Druid now emits a new metric when a batch task finishes waiting for segment availability. (#11090) +- Added `segmentAvailabilityWaitTimeMs`, the time in milliseconds the milliseconds that a task waited for its segments to be handed off to Historical nodes, to `IngestionStatsAndErrorsTaskReportData` (#11090) +- Added functionality to preserve existing metrics during ingestion (#12185) +- Parallel native batch task can now provide task reports for the sequential and single phase mode (e.g., used with dynamic partitioning) as well as single phase mode subtasks (#11688) +- Added support for `RowStats` in `druid/indexer/v1/task/{task_id}/reports` API for multi-phase parallel indexing task (#12280) +- Fixed the OOM failures in dimension distribution phase of parallel indexing (#12331) +- Added support to handle null dimension values while creating partition boundaries (#11973) + +## Improvements to ingestion in general + +This release includes several improvements for ingestion in general: +- Removed the template modifier from `IncrementalIndex<AggregatorType>` because it is no longer required +- You can now use `JsonPath` functions in `JsonPath` expressions during ingestion (#11722) +- Druid no longer creates a materialized list of segment files and elimited looping over the files to reduce OOM issues (#11903) +- Added an intermediate-persist `IndexSpec` to the main "merge" method in `IndexMerger` (#11940) +- `Granularity.granularitiesFinerThan` now returns ALL if you pass in ALL (#12003) +- Added a configuation parameter for appending tasks to allow them to use a SHARED lock (#12041) +- `SchemaRegistryBasedAvroBytesDecoder` now throws a `ParseException` instead of RE when it fails to retrieve a schema (#12080) +- Added `includeAllDimensions` to `dimensionsSpec` to put all explicit dimensions first in `InputRow` and subsequently any other dimensions found in input data (#12276) +- Added the ability to store null columns in the Segments (#12279) + +## Compaction + +This release includes several improvements for compaction: +- Automatic compaction now supports complex dimensions (#11924) +- Automatic compaction now supports overlapping segment intervals (#12062) +- You can now configure automatic compaction to calculate ratio of slots available for compaction tasks from maximum slots including autoscaler maximum worker nodes (#12263) Review Comment: ```suggestion - You can now configure automatic compaction to calculate the ratio of slots available for compaction tasks from maximum slots, including autoscaler maximum worker nodes (#12263) ``` ########## druid-0.23-release-notes.md: ########## @@ -5,34 +5,183 @@ Apache Druid 0.23.0 contains over 450 new features, bug fixes, performance enhan ## Query engine ### Grouping on arrays without exploding the arrays -[12078](https://github.com/apache/druid/pull/12078) +You can now group on a multi-value dimension as an array. For a datasource named "test": +``` +{"timestamp": "2011-01-12T00:00:00.000Z", "tags": ["t1","t2","t3"]} #row1 +{"timestamp": "2011-01-13T00:00:00.000Z", "tags": ["t3","t4","t5"]} #row2 +{"timestamp": "2011-01-14T00:00:00.000Z", "tags": ["t5","t6","t7"]} #row3 +{"timestamp": "2011-01-14T00:00:00.000Z", "tags": []} #row4 +``` +The following query: +```json +{ + "queryType": "groupBy", + "dataSource": "test", + "intervals": [ + "1970-01-01T00:00:00.000Z/3000-01-01T00:00:00.000Z" + ], + "granularity": { + "type": "all" + }, + "virtualColumns" : [ { + "type" : "expression", + "name" : "v0", + "expression" : "mv_to_array(\"tags\")", + "outputType" : "ARRAY<STRING>" + } ], + "dimensions": [ + { + "type": "default", + "dimension": "v0", + "outputName": "tags" + "outputType":"ARRAY<STRING>" + } + ], + "aggregations": [ + { + "type": "count", + "name": "count" + } + ] +} +``` +Returns the following: + +``` +[ + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "[]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "["t1","t2","t3"]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "[t3","t4","t5"]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 2, + "tags": "["t5","t6","t7"]" + } + } +] +``` +(#12078) +(#12253) ### Specify a column other than __time column for row comparison in first/last aggregators -[11949](https://github.com/apache/druid/pull/11949) + +You can pass time column in `*first`/`*last` aggregators and LATEST / EARLIEST SQL functions. This provides support for cases where the time is stored as a part of a column different than "__time". You can also specify another logical time column. +(#11949) +(#12145) + +### Improvements to querying user experience + +This release includes several improvements for querying: +- Added the SQL query id to response header for failed SQL query to aid in location the error messages (#11756) +- Added input type validation for DataSketches HLL (#12131) +- Improved JDBC logging (#11676) +- Added SQL functions MV_FILTER_ONLY and MV_FILTER_NONE to filter rows of multi-value string dimensions to include only the supplied list of values or none of them respectively (#11650) +- Added ARRAY_CONCAT_AGG to aggregate array inputs together into a single array(#12226) +- Added the ability to authorize the usage of query context parameters (#12396) +- Improved query IDs to make it easier to link queryies and sub-queries for end-to-end visibility of a query (#11809) +- Added a safe divide function to protect against division by 0 (#11904) +- You can now add a query context to internally generated `SegmentMetadata` Query (#11429) +- Added support for Druid complex types to the native expression processing system to make all Druid data usable within expressions (#11853, #12016) ## Streaming Ingestion -### Kafka Input format for parsing headers and key -[11630](https://github.com/apache/druid/pull/11630) +### Kafka input format for parsing headers and key +We've introduced a Kafka input format so you can ingest header data in addition to the message contents. For example: +- the event key field +- event headers +- the Kafka event timestamp +- the Kafka event value that stores the payload. + +(#11630) ## Native Batch Ingestion -### Multi-dimensional range partitioning -Multi-dimensional range partitioning allows users to partition their data on the ranges of any number of dimensions. It develops further on the concepts behind "single-dim" partitioning and is now arguably the most preferable secondary partitioning both for query performance and storage efficiency. -[11848](https://github.com/apache/druid/pull/11848) -[11973](https://github.com/apache/druid/pull/11973) +### Multi-dimension range partitioning +Multi-dimension range partitioning allows users to partition their data on the ranges of any number of dimensions. It develops further on the concepts behind "single-dim" partitioning and is now arguably the most preferable secondary partitioning both for query performance and storage efficiency. +(#11848) +(#11973) + +### Improved replace data behavior +In previous versions of Druid, if ingested data with `dropExisting` flag to replace data, Druid would retain the existing data for a time chunk if there was no new data to replace it. Now, if you set `dropExisting` to `true` in your `ioSpec` and ingest data for a time range that includes a time chunk with no data, Druid uses a tombstone to overshadow the existing data in the empty time chunk. +(#12137) + +This release includes several improvements for native batch ingestion: +- Druid now emits a new metric when a batch task finishes waiting for segment availability. (#11090) +- Added `segmentAvailabilityWaitTimeMs`, the time in milliseconds the milliseconds that a task waited for its segments to be handed off to Historical nodes, to `IngestionStatsAndErrorsTaskReportData` (#11090) +- Added functionality to preserve existing metrics during ingestion (#12185) +- Parallel native batch task can now provide task reports for the sequential and single phase mode (e.g., used with dynamic partitioning) as well as single phase mode subtasks (#11688) +- Added support for `RowStats` in `druid/indexer/v1/task/{task_id}/reports` API for multi-phase parallel indexing task (#12280) +- Fixed the OOM failures in dimension distribution phase of parallel indexing (#12331) Review Comment: ```suggestion - Fixed the OOM failures in the dimension distribution phase of parallel indexing (#12331) ``` ########## druid-0.23-release-notes.md: ########## @@ -5,34 +5,183 @@ Apache Druid 0.23.0 contains over 450 new features, bug fixes, performance enhan ## Query engine ### Grouping on arrays without exploding the arrays -[12078](https://github.com/apache/druid/pull/12078) +You can now group on a multi-value dimension as an array. For a datasource named "test": +``` +{"timestamp": "2011-01-12T00:00:00.000Z", "tags": ["t1","t2","t3"]} #row1 +{"timestamp": "2011-01-13T00:00:00.000Z", "tags": ["t3","t4","t5"]} #row2 +{"timestamp": "2011-01-14T00:00:00.000Z", "tags": ["t5","t6","t7"]} #row3 +{"timestamp": "2011-01-14T00:00:00.000Z", "tags": []} #row4 +``` +The following query: +```json +{ + "queryType": "groupBy", + "dataSource": "test", + "intervals": [ + "1970-01-01T00:00:00.000Z/3000-01-01T00:00:00.000Z" + ], + "granularity": { + "type": "all" + }, + "virtualColumns" : [ { + "type" : "expression", + "name" : "v0", + "expression" : "mv_to_array(\"tags\")", + "outputType" : "ARRAY<STRING>" + } ], + "dimensions": [ + { + "type": "default", + "dimension": "v0", + "outputName": "tags" + "outputType":"ARRAY<STRING>" + } + ], + "aggregations": [ + { + "type": "count", + "name": "count" + } + ] +} +``` +Returns the following: + +``` +[ + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "[]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "["t1","t2","t3"]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "[t3","t4","t5"]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 2, + "tags": "["t5","t6","t7"]" + } + } +] +``` +(#12078) +(#12253) ### Specify a column other than __time column for row comparison in first/last aggregators -[11949](https://github.com/apache/druid/pull/11949) + +You can pass time column in `*first`/`*last` aggregators and LATEST / EARLIEST SQL functions. This provides support for cases where the time is stored as a part of a column different than "__time". You can also specify another logical time column. +(#11949) +(#12145) + +### Improvements to querying user experience + +This release includes several improvements for querying: +- Added the SQL query id to response header for failed SQL query to aid in location the error messages (#11756) +- Added input type validation for DataSketches HLL (#12131) +- Improved JDBC logging (#11676) +- Added SQL functions MV_FILTER_ONLY and MV_FILTER_NONE to filter rows of multi-value string dimensions to include only the supplied list of values or none of them respectively (#11650) +- Added ARRAY_CONCAT_AGG to aggregate array inputs together into a single array(#12226) +- Added the ability to authorize the usage of query context parameters (#12396) +- Improved query IDs to make it easier to link queryies and sub-queries for end-to-end visibility of a query (#11809) +- Added a safe divide function to protect against division by 0 (#11904) +- You can now add a query context to internally generated `SegmentMetadata` Query (#11429) +- Added support for Druid complex types to the native expression processing system to make all Druid data usable within expressions (#11853, #12016) ## Streaming Ingestion -### Kafka Input format for parsing headers and key -[11630](https://github.com/apache/druid/pull/11630) +### Kafka input format for parsing headers and key +We've introduced a Kafka input format so you can ingest header data in addition to the message contents. For example: +- the event key field +- event headers +- the Kafka event timestamp +- the Kafka event value that stores the payload. + +(#11630) ## Native Batch Ingestion -### Multi-dimensional range partitioning -Multi-dimensional range partitioning allows users to partition their data on the ranges of any number of dimensions. It develops further on the concepts behind "single-dim" partitioning and is now arguably the most preferable secondary partitioning both for query performance and storage efficiency. -[11848](https://github.com/apache/druid/pull/11848) -[11973](https://github.com/apache/druid/pull/11973) +### Multi-dimension range partitioning +Multi-dimension range partitioning allows users to partition their data on the ranges of any number of dimensions. It develops further on the concepts behind "single-dim" partitioning and is now arguably the most preferable secondary partitioning both for query performance and storage efficiency. +(#11848) +(#11973) + +### Improved replace data behavior +In previous versions of Druid, if ingested data with `dropExisting` flag to replace data, Druid would retain the existing data for a time chunk if there was no new data to replace it. Now, if you set `dropExisting` to `true` in your `ioSpec` and ingest data for a time range that includes a time chunk with no data, Druid uses a tombstone to overshadow the existing data in the empty time chunk. +(#12137) + +This release includes several improvements for native batch ingestion: +- Druid now emits a new metric when a batch task finishes waiting for segment availability. (#11090) +- Added `segmentAvailabilityWaitTimeMs`, the time in milliseconds the milliseconds that a task waited for its segments to be handed off to Historical nodes, to `IngestionStatsAndErrorsTaskReportData` (#11090) +- Added functionality to preserve existing metrics during ingestion (#12185) +- Parallel native batch task can now provide task reports for the sequential and single phase mode (e.g., used with dynamic partitioning) as well as single phase mode subtasks (#11688) +- Added support for `RowStats` in `druid/indexer/v1/task/{task_id}/reports` API for multi-phase parallel indexing task (#12280) +- Fixed the OOM failures in dimension distribution phase of parallel indexing (#12331) +- Added support to handle null dimension values while creating partition boundaries (#11973) + +## Improvements to ingestion in general + +This release includes several improvements for ingestion in general: +- Removed the template modifier from `IncrementalIndex<AggregatorType>` because it is no longer required +- You can now use `JsonPath` functions in `JsonPath` expressions during ingestion (#11722) +- Druid no longer creates a materialized list of segment files and elimited looping over the files to reduce OOM issues (#11903) +- Added an intermediate-persist `IndexSpec` to the main "merge" method in `IndexMerger` (#11940) +- `Granularity.granularitiesFinerThan` now returns ALL if you pass in ALL (#12003) +- Added a configuation parameter for appending tasks to allow them to use a SHARED lock (#12041) +- `SchemaRegistryBasedAvroBytesDecoder` now throws a `ParseException` instead of RE when it fails to retrieve a schema (#12080) +- Added `includeAllDimensions` to `dimensionsSpec` to put all explicit dimensions first in `InputRow` and subsequently any other dimensions found in input data (#12276) +- Added the ability to store null columns in the Segments (#12279) + +## Compaction + +This release includes several improvements for compaction: +- Automatic compaction now supports complex dimensions (#11924) +- Automatic compaction now supports overlapping segment intervals (#12062) +- You can now configure automatic compaction to calculate ratio of slots available for compaction tasks from maximum slots including autoscaler maximum worker nodes (#12263) +- You can configure the Coordinator auto compaction duty period separately from other indexing duties (#12263) Review Comment: ```suggestion - You can now configure the Coordinator auto compaction duty period separately from other indexing duties (#12263) ``` ########## druid-0.23-release-notes.md: ########## @@ -5,34 +5,183 @@ Apache Druid 0.23.0 contains over 450 new features, bug fixes, performance enhan ## Query engine ### Grouping on arrays without exploding the arrays -[12078](https://github.com/apache/druid/pull/12078) +You can now group on a multi-value dimension as an array. For a datasource named "test": +``` +{"timestamp": "2011-01-12T00:00:00.000Z", "tags": ["t1","t2","t3"]} #row1 +{"timestamp": "2011-01-13T00:00:00.000Z", "tags": ["t3","t4","t5"]} #row2 +{"timestamp": "2011-01-14T00:00:00.000Z", "tags": ["t5","t6","t7"]} #row3 +{"timestamp": "2011-01-14T00:00:00.000Z", "tags": []} #row4 +``` +The following query: +```json +{ + "queryType": "groupBy", + "dataSource": "test", + "intervals": [ + "1970-01-01T00:00:00.000Z/3000-01-01T00:00:00.000Z" + ], + "granularity": { + "type": "all" + }, + "virtualColumns" : [ { + "type" : "expression", + "name" : "v0", + "expression" : "mv_to_array(\"tags\")", + "outputType" : "ARRAY<STRING>" + } ], + "dimensions": [ + { + "type": "default", + "dimension": "v0", + "outputName": "tags" + "outputType":"ARRAY<STRING>" + } + ], + "aggregations": [ + { + "type": "count", + "name": "count" + } + ] +} +``` +Returns the following: + +``` +[ + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "[]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "["t1","t2","t3"]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "[t3","t4","t5"]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 2, + "tags": "["t5","t6","t7"]" + } + } +] +``` +(#12078) +(#12253) ### Specify a column other than __time column for row comparison in first/last aggregators -[11949](https://github.com/apache/druid/pull/11949) + +You can pass time column in `*first`/`*last` aggregators and LATEST / EARLIEST SQL functions. This provides support for cases where the time is stored as a part of a column different than "__time". You can also specify another logical time column. +(#11949) +(#12145) + +### Improvements to querying user experience + +This release includes several improvements for querying: +- Added the SQL query id to response header for failed SQL query to aid in location the error messages (#11756) +- Added input type validation for DataSketches HLL (#12131) +- Improved JDBC logging (#11676) +- Added SQL functions MV_FILTER_ONLY and MV_FILTER_NONE to filter rows of multi-value string dimensions to include only the supplied list of values or none of them respectively (#11650) +- Added ARRAY_CONCAT_AGG to aggregate array inputs together into a single array(#12226) Review Comment: ```suggestion - Added ARRAY_CONCAT_AGG to aggregate array inputs together into a single array (#12226) ``` ########## druid-0.23-release-notes.md: ########## @@ -5,34 +5,183 @@ Apache Druid 0.23.0 contains over 450 new features, bug fixes, performance enhan ## Query engine ### Grouping on arrays without exploding the arrays -[12078](https://github.com/apache/druid/pull/12078) +You can now group on a multi-value dimension as an array. For a datasource named "test": +``` +{"timestamp": "2011-01-12T00:00:00.000Z", "tags": ["t1","t2","t3"]} #row1 +{"timestamp": "2011-01-13T00:00:00.000Z", "tags": ["t3","t4","t5"]} #row2 +{"timestamp": "2011-01-14T00:00:00.000Z", "tags": ["t5","t6","t7"]} #row3 +{"timestamp": "2011-01-14T00:00:00.000Z", "tags": []} #row4 +``` +The following query: +```json +{ + "queryType": "groupBy", + "dataSource": "test", + "intervals": [ + "1970-01-01T00:00:00.000Z/3000-01-01T00:00:00.000Z" + ], + "granularity": { + "type": "all" + }, + "virtualColumns" : [ { + "type" : "expression", + "name" : "v0", + "expression" : "mv_to_array(\"tags\")", + "outputType" : "ARRAY<STRING>" + } ], + "dimensions": [ + { + "type": "default", + "dimension": "v0", + "outputName": "tags" + "outputType":"ARRAY<STRING>" + } + ], + "aggregations": [ + { + "type": "count", + "name": "count" + } + ] +} +``` +Returns the following: + +``` +[ + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "[]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "["t1","t2","t3"]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "[t3","t4","t5"]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 2, + "tags": "["t5","t6","t7"]" + } + } +] +``` +(#12078) +(#12253) ### Specify a column other than __time column for row comparison in first/last aggregators -[11949](https://github.com/apache/druid/pull/11949) + +You can pass time column in `*first`/`*last` aggregators and LATEST / EARLIEST SQL functions. This provides support for cases where the time is stored as a part of a column different than "__time". You can also specify another logical time column. +(#11949) +(#12145) + +### Improvements to querying user experience + +This release includes several improvements for querying: +- Added the SQL query id to response header for failed SQL query to aid in location the error messages (#11756) +- Added input type validation for DataSketches HLL (#12131) +- Improved JDBC logging (#11676) +- Added SQL functions MV_FILTER_ONLY and MV_FILTER_NONE to filter rows of multi-value string dimensions to include only the supplied list of values or none of them respectively (#11650) +- Added ARRAY_CONCAT_AGG to aggregate array inputs together into a single array(#12226) +- Added the ability to authorize the usage of query context parameters (#12396) +- Improved query IDs to make it easier to link queryies and sub-queries for end-to-end visibility of a query (#11809) +- Added a safe divide function to protect against division by 0 (#11904) +- You can now add a query context to internally generated `SegmentMetadata` Query (#11429) +- Added support for Druid complex types to the native expression processing system to make all Druid data usable within expressions (#11853, #12016) ## Streaming Ingestion -### Kafka Input format for parsing headers and key -[11630](https://github.com/apache/druid/pull/11630) +### Kafka input format for parsing headers and key +We've introduced a Kafka input format so you can ingest header data in addition to the message contents. For example: +- the event key field +- event headers +- the Kafka event timestamp +- the Kafka event value that stores the payload. + +(#11630) ## Native Batch Ingestion -### Multi-dimensional range partitioning -Multi-dimensional range partitioning allows users to partition their data on the ranges of any number of dimensions. It develops further on the concepts behind "single-dim" partitioning and is now arguably the most preferable secondary partitioning both for query performance and storage efficiency. -[11848](https://github.com/apache/druid/pull/11848) -[11973](https://github.com/apache/druid/pull/11973) +### Multi-dimension range partitioning +Multi-dimension range partitioning allows users to partition their data on the ranges of any number of dimensions. It develops further on the concepts behind "single-dim" partitioning and is now arguably the most preferable secondary partitioning both for query performance and storage efficiency. +(#11848) +(#11973) + +### Improved replace data behavior +In previous versions of Druid, if ingested data with `dropExisting` flag to replace data, Druid would retain the existing data for a time chunk if there was no new data to replace it. Now, if you set `dropExisting` to `true` in your `ioSpec` and ingest data for a time range that includes a time chunk with no data, Druid uses a tombstone to overshadow the existing data in the empty time chunk. +(#12137) + +This release includes several improvements for native batch ingestion: +- Druid now emits a new metric when a batch task finishes waiting for segment availability. (#11090) +- Added `segmentAvailabilityWaitTimeMs`, the time in milliseconds the milliseconds that a task waited for its segments to be handed off to Historical nodes, to `IngestionStatsAndErrorsTaskReportData` (#11090) +- Added functionality to preserve existing metrics during ingestion (#12185) +- Parallel native batch task can now provide task reports for the sequential and single phase mode (e.g., used with dynamic partitioning) as well as single phase mode subtasks (#11688) +- Added support for `RowStats` in `druid/indexer/v1/task/{task_id}/reports` API for multi-phase parallel indexing task (#12280) +- Fixed the OOM failures in dimension distribution phase of parallel indexing (#12331) +- Added support to handle null dimension values while creating partition boundaries (#11973) + +## Improvements to ingestion in general + +This release includes several improvements for ingestion in general: +- Removed the template modifier from `IncrementalIndex<AggregatorType>` because it is no longer required +- You can now use `JsonPath` functions in `JsonPath` expressions during ingestion (#11722) +- Druid no longer creates a materialized list of segment files and elimited looping over the files to reduce OOM issues (#11903) +- Added an intermediate-persist `IndexSpec` to the main "merge" method in `IndexMerger` (#11940) +- `Granularity.granularitiesFinerThan` now returns ALL if you pass in ALL (#12003) +- Added a configuation parameter for appending tasks to allow them to use a SHARED lock (#12041) +- `SchemaRegistryBasedAvroBytesDecoder` now throws a `ParseException` instead of RE when it fails to retrieve a schema (#12080) +- Added `includeAllDimensions` to `dimensionsSpec` to put all explicit dimensions first in `InputRow` and subsequently any other dimensions found in input data (#12276) +- Added the ability to store null columns in the Segments (#12279) + +## Compaction + +This release includes several improvements for compaction: +- Automatic compaction now supports complex dimensions (#11924) +- Automatic compaction now supports overlapping segment intervals (#12062) +- You can now configure automatic compaction to calculate ratio of slots available for compaction tasks from maximum slots including autoscaler maximum worker nodes (#12263) +- You can configure the Coordinator auto compaction duty period separately from other indexing duties (#12263) ## SQL ### Human-readable and actionable SQL error messages -Till 0.22.1, if a SQL query is not supported, users get a very cryptic and unhelpful error message. With this change, error message will include exactly what part of their SQL query is not supported by druid. One such example is executing a scan query that is ordered on a dimension other than time column. -[11911](https://github.com/apache/druid/pull/11911) +Until version 0.22.1, if you issued an unsupported SQL query, Druid would throw very cryptic and unhelpful error messages. With this change, error messages include exactly the part of the SQL query that is not supported in Druid. For example, if you run a scan query that is ordered on a dimension other than time column. + +(#11911) ### Cancel API for SQL queries -Users can now cancel SQL queries just like native queries can be cancelled. A new API has been added for cancelling SQL queries. Web-console now users this API to cancel SQL queries. Earlier, it only used to close the client connection while sql query keeps running in druid. +We've added a new API to cancel SQL queries, so you can now cancel SQL queries just like you can cancel native queries. You can use the API from the Web-console. In previous versions, cancellation from the console only closed the client connection while the SQL query kept running on Druid. + +(#11643) +(#11738) +(#11710) + +### Improvements to SQL user experience + +This release includes several additional improvements for SQL: + +- You no longer need to include a trailing slash `/` for JDBC connections to druid (#11737) +- You can now use scans to as outer queries (#11831) +- Added a class to sanitize JDBC exceptions and to log them (#11843) +- Added type headers to response format to make it easier for clients to interpret the results of SQL queries (#11914) +- Improved the way the `DruidRexExecutor` handles numeric arrays (#11968) +- Druid now returns an empty result after optimizing a group by query to a time series query (#12065) +- As an administrator, you can now configure the implementation for Review Comment: ```suggestion - As an administrator, you can now configure the implementation for APPROX_COUNT_DISTINCT and COUNT(DISTINCT expr) in approximate mode (#11181) ``` ########## druid-0.23-release-notes.md: ########## @@ -5,34 +5,183 @@ Apache Druid 0.23.0 contains over 450 new features, bug fixes, performance enhan ## Query engine ### Grouping on arrays without exploding the arrays -[12078](https://github.com/apache/druid/pull/12078) +You can now group on a multi-value dimension as an array. For a datasource named "test": +``` +{"timestamp": "2011-01-12T00:00:00.000Z", "tags": ["t1","t2","t3"]} #row1 +{"timestamp": "2011-01-13T00:00:00.000Z", "tags": ["t3","t4","t5"]} #row2 +{"timestamp": "2011-01-14T00:00:00.000Z", "tags": ["t5","t6","t7"]} #row3 +{"timestamp": "2011-01-14T00:00:00.000Z", "tags": []} #row4 +``` +The following query: +```json +{ + "queryType": "groupBy", + "dataSource": "test", + "intervals": [ + "1970-01-01T00:00:00.000Z/3000-01-01T00:00:00.000Z" + ], + "granularity": { + "type": "all" + }, + "virtualColumns" : [ { + "type" : "expression", + "name" : "v0", + "expression" : "mv_to_array(\"tags\")", + "outputType" : "ARRAY<STRING>" + } ], + "dimensions": [ + { + "type": "default", + "dimension": "v0", + "outputName": "tags" + "outputType":"ARRAY<STRING>" + } + ], + "aggregations": [ + { + "type": "count", + "name": "count" + } + ] +} +``` +Returns the following: + +``` +[ + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "[]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "["t1","t2","t3"]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "[t3","t4","t5"]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 2, + "tags": "["t5","t6","t7"]" + } + } +] +``` +(#12078) +(#12253) ### Specify a column other than __time column for row comparison in first/last aggregators -[11949](https://github.com/apache/druid/pull/11949) + +You can pass time column in `*first`/`*last` aggregators and LATEST / EARLIEST SQL functions. This provides support for cases where the time is stored as a part of a column different than "__time". You can also specify another logical time column. +(#11949) +(#12145) + +### Improvements to querying user experience + +This release includes several improvements for querying: +- Added the SQL query id to response header for failed SQL query to aid in location the error messages (#11756) +- Added input type validation for DataSketches HLL (#12131) +- Improved JDBC logging (#11676) +- Added SQL functions MV_FILTER_ONLY and MV_FILTER_NONE to filter rows of multi-value string dimensions to include only the supplied list of values or none of them respectively (#11650) +- Added ARRAY_CONCAT_AGG to aggregate array inputs together into a single array(#12226) +- Added the ability to authorize the usage of query context parameters (#12396) +- Improved query IDs to make it easier to link queryies and sub-queries for end-to-end visibility of a query (#11809) +- Added a safe divide function to protect against division by 0 (#11904) +- You can now add a query context to internally generated `SegmentMetadata` Query (#11429) +- Added support for Druid complex types to the native expression processing system to make all Druid data usable within expressions (#11853, #12016) ## Streaming Ingestion -### Kafka Input format for parsing headers and key -[11630](https://github.com/apache/druid/pull/11630) +### Kafka input format for parsing headers and key +We've introduced a Kafka input format so you can ingest header data in addition to the message contents. For example: +- the event key field +- event headers +- the Kafka event timestamp +- the Kafka event value that stores the payload. + +(#11630) ## Native Batch Ingestion -### Multi-dimensional range partitioning -Multi-dimensional range partitioning allows users to partition their data on the ranges of any number of dimensions. It develops further on the concepts behind "single-dim" partitioning and is now arguably the most preferable secondary partitioning both for query performance and storage efficiency. -[11848](https://github.com/apache/druid/pull/11848) -[11973](https://github.com/apache/druid/pull/11973) +### Multi-dimension range partitioning +Multi-dimension range partitioning allows users to partition their data on the ranges of any number of dimensions. It develops further on the concepts behind "single-dim" partitioning and is now arguably the most preferable secondary partitioning both for query performance and storage efficiency. +(#11848) +(#11973) + +### Improved replace data behavior +In previous versions of Druid, if ingested data with `dropExisting` flag to replace data, Druid would retain the existing data for a time chunk if there was no new data to replace it. Now, if you set `dropExisting` to `true` in your `ioSpec` and ingest data for a time range that includes a time chunk with no data, Druid uses a tombstone to overshadow the existing data in the empty time chunk. +(#12137) + +This release includes several improvements for native batch ingestion: +- Druid now emits a new metric when a batch task finishes waiting for segment availability. (#11090) +- Added `segmentAvailabilityWaitTimeMs`, the time in milliseconds the milliseconds that a task waited for its segments to be handed off to Historical nodes, to `IngestionStatsAndErrorsTaskReportData` (#11090) +- Added functionality to preserve existing metrics during ingestion (#12185) +- Parallel native batch task can now provide task reports for the sequential and single phase mode (e.g., used with dynamic partitioning) as well as single phase mode subtasks (#11688) +- Added support for `RowStats` in `druid/indexer/v1/task/{task_id}/reports` API for multi-phase parallel indexing task (#12280) +- Fixed the OOM failures in dimension distribution phase of parallel indexing (#12331) +- Added support to handle null dimension values while creating partition boundaries (#11973) + +## Improvements to ingestion in general + +This release includes several improvements for ingestion in general: +- Removed the template modifier from `IncrementalIndex<AggregatorType>` because it is no longer required +- You can now use `JsonPath` functions in `JsonPath` expressions during ingestion (#11722) +- Druid no longer creates a materialized list of segment files and elimited looping over the files to reduce OOM issues (#11903) +- Added an intermediate-persist `IndexSpec` to the main "merge" method in `IndexMerger` (#11940) +- `Granularity.granularitiesFinerThan` now returns ALL if you pass in ALL (#12003) +- Added a configuation parameter for appending tasks to allow them to use a SHARED lock (#12041) +- `SchemaRegistryBasedAvroBytesDecoder` now throws a `ParseException` instead of RE when it fails to retrieve a schema (#12080) +- Added `includeAllDimensions` to `dimensionsSpec` to put all explicit dimensions first in `InputRow` and subsequently any other dimensions found in input data (#12276) +- Added the ability to store null columns in the Segments (#12279) + +## Compaction + +This release includes several improvements for compaction: +- Automatic compaction now supports complex dimensions (#11924) +- Automatic compaction now supports overlapping segment intervals (#12062) +- You can now configure automatic compaction to calculate ratio of slots available for compaction tasks from maximum slots including autoscaler maximum worker nodes (#12263) +- You can configure the Coordinator auto compaction duty period separately from other indexing duties (#12263) ## SQL ### Human-readable and actionable SQL error messages -Till 0.22.1, if a SQL query is not supported, users get a very cryptic and unhelpful error message. With this change, error message will include exactly what part of their SQL query is not supported by druid. One such example is executing a scan query that is ordered on a dimension other than time column. -[11911](https://github.com/apache/druid/pull/11911) +Until version 0.22.1, if you issued an unsupported SQL query, Druid would throw very cryptic and unhelpful error messages. With this change, error messages include exactly the part of the SQL query that is not supported in Druid. For example, if you run a scan query that is ordered on a dimension other than time column. + +(#11911) ### Cancel API for SQL queries -Users can now cancel SQL queries just like native queries can be cancelled. A new API has been added for cancelling SQL queries. Web-console now users this API to cancel SQL queries. Earlier, it only used to close the client connection while sql query keeps running in druid. +We've added a new API to cancel SQL queries, so you can now cancel SQL queries just like you can cancel native queries. You can use the API from the Web-console. In previous versions, cancellation from the console only closed the client connection while the SQL query kept running on Druid. + +(#11643) +(#11738) +(#11710) + +### Improvements to SQL user experience + +This release includes several additional improvements for SQL: + +- You no longer need to include a trailing slash `/` for JDBC connections to druid (#11737) Review Comment: ```suggestion - You no longer need to include a trailing slash `/` for JDBC connections to Druid (#11737) ``` ########## druid-0.23-release-notes.md: ########## @@ -5,34 +5,183 @@ Apache Druid 0.23.0 contains over 450 new features, bug fixes, performance enhan ## Query engine ### Grouping on arrays without exploding the arrays -[12078](https://github.com/apache/druid/pull/12078) +You can now group on a multi-value dimension as an array. For a datasource named "test": +``` +{"timestamp": "2011-01-12T00:00:00.000Z", "tags": ["t1","t2","t3"]} #row1 +{"timestamp": "2011-01-13T00:00:00.000Z", "tags": ["t3","t4","t5"]} #row2 +{"timestamp": "2011-01-14T00:00:00.000Z", "tags": ["t5","t6","t7"]} #row3 +{"timestamp": "2011-01-14T00:00:00.000Z", "tags": []} #row4 +``` +The following query: +```json +{ + "queryType": "groupBy", + "dataSource": "test", + "intervals": [ + "1970-01-01T00:00:00.000Z/3000-01-01T00:00:00.000Z" + ], + "granularity": { + "type": "all" + }, + "virtualColumns" : [ { + "type" : "expression", + "name" : "v0", + "expression" : "mv_to_array(\"tags\")", + "outputType" : "ARRAY<STRING>" + } ], + "dimensions": [ + { + "type": "default", + "dimension": "v0", + "outputName": "tags" + "outputType":"ARRAY<STRING>" + } + ], + "aggregations": [ + { + "type": "count", + "name": "count" + } + ] +} +``` +Returns the following: + +``` +[ + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "[]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "["t1","t2","t3"]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "[t3","t4","t5"]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 2, + "tags": "["t5","t6","t7"]" + } + } +] +``` +(#12078) +(#12253) ### Specify a column other than __time column for row comparison in first/last aggregators -[11949](https://github.com/apache/druid/pull/11949) + +You can pass time column in `*first`/`*last` aggregators and LATEST / EARLIEST SQL functions. This provides support for cases where the time is stored as a part of a column different than "__time". You can also specify another logical time column. +(#11949) +(#12145) + +### Improvements to querying user experience + +This release includes several improvements for querying: +- Added the SQL query id to response header for failed SQL query to aid in location the error messages (#11756) +- Added input type validation for DataSketches HLL (#12131) +- Improved JDBC logging (#11676) +- Added SQL functions MV_FILTER_ONLY and MV_FILTER_NONE to filter rows of multi-value string dimensions to include only the supplied list of values or none of them respectively (#11650) +- Added ARRAY_CONCAT_AGG to aggregate array inputs together into a single array(#12226) +- Added the ability to authorize the usage of query context parameters (#12396) +- Improved query IDs to make it easier to link queryies and sub-queries for end-to-end visibility of a query (#11809) +- Added a safe divide function to protect against division by 0 (#11904) +- You can now add a query context to internally generated `SegmentMetadata` Query (#11429) +- Added support for Druid complex types to the native expression processing system to make all Druid data usable within expressions (#11853, #12016) ## Streaming Ingestion -### Kafka Input format for parsing headers and key -[11630](https://github.com/apache/druid/pull/11630) +### Kafka input format for parsing headers and key +We've introduced a Kafka input format so you can ingest header data in addition to the message contents. For example: +- the event key field +- event headers +- the Kafka event timestamp +- the Kafka event value that stores the payload. + +(#11630) ## Native Batch Ingestion -### Multi-dimensional range partitioning -Multi-dimensional range partitioning allows users to partition their data on the ranges of any number of dimensions. It develops further on the concepts behind "single-dim" partitioning and is now arguably the most preferable secondary partitioning both for query performance and storage efficiency. -[11848](https://github.com/apache/druid/pull/11848) -[11973](https://github.com/apache/druid/pull/11973) +### Multi-dimension range partitioning +Multi-dimension range partitioning allows users to partition their data on the ranges of any number of dimensions. It develops further on the concepts behind "single-dim" partitioning and is now arguably the most preferable secondary partitioning both for query performance and storage efficiency. +(#11848) +(#11973) + +### Improved replace data behavior +In previous versions of Druid, if ingested data with `dropExisting` flag to replace data, Druid would retain the existing data for a time chunk if there was no new data to replace it. Now, if you set `dropExisting` to `true` in your `ioSpec` and ingest data for a time range that includes a time chunk with no data, Druid uses a tombstone to overshadow the existing data in the empty time chunk. +(#12137) + +This release includes several improvements for native batch ingestion: +- Druid now emits a new metric when a batch task finishes waiting for segment availability. (#11090) +- Added `segmentAvailabilityWaitTimeMs`, the time in milliseconds the milliseconds that a task waited for its segments to be handed off to Historical nodes, to `IngestionStatsAndErrorsTaskReportData` (#11090) +- Added functionality to preserve existing metrics during ingestion (#12185) +- Parallel native batch task can now provide task reports for the sequential and single phase mode (e.g., used with dynamic partitioning) as well as single phase mode subtasks (#11688) +- Added support for `RowStats` in `druid/indexer/v1/task/{task_id}/reports` API for multi-phase parallel indexing task (#12280) +- Fixed the OOM failures in dimension distribution phase of parallel indexing (#12331) +- Added support to handle null dimension values while creating partition boundaries (#11973) + +## Improvements to ingestion in general + +This release includes several improvements for ingestion in general: +- Removed the template modifier from `IncrementalIndex<AggregatorType>` because it is no longer required +- You can now use `JsonPath` functions in `JsonPath` expressions during ingestion (#11722) +- Druid no longer creates a materialized list of segment files and elimited looping over the files to reduce OOM issues (#11903) +- Added an intermediate-persist `IndexSpec` to the main "merge" method in `IndexMerger` (#11940) +- `Granularity.granularitiesFinerThan` now returns ALL if you pass in ALL (#12003) +- Added a configuation parameter for appending tasks to allow them to use a SHARED lock (#12041) +- `SchemaRegistryBasedAvroBytesDecoder` now throws a `ParseException` instead of RE when it fails to retrieve a schema (#12080) +- Added `includeAllDimensions` to `dimensionsSpec` to put all explicit dimensions first in `InputRow` and subsequently any other dimensions found in input data (#12276) +- Added the ability to store null columns in the Segments (#12279) + +## Compaction + +This release includes several improvements for compaction: +- Automatic compaction now supports complex dimensions (#11924) +- Automatic compaction now supports overlapping segment intervals (#12062) +- You can now configure automatic compaction to calculate ratio of slots available for compaction tasks from maximum slots including autoscaler maximum worker nodes (#12263) +- You can configure the Coordinator auto compaction duty period separately from other indexing duties (#12263) ## SQL ### Human-readable and actionable SQL error messages -Till 0.22.1, if a SQL query is not supported, users get a very cryptic and unhelpful error message. With this change, error message will include exactly what part of their SQL query is not supported by druid. One such example is executing a scan query that is ordered on a dimension other than time column. -[11911](https://github.com/apache/druid/pull/11911) +Until version 0.22.1, if you issued an unsupported SQL query, Druid would throw very cryptic and unhelpful error messages. With this change, error messages include exactly the part of the SQL query that is not supported in Druid. For example, if you run a scan query that is ordered on a dimension other than time column. + +(#11911) ### Cancel API for SQL queries -Users can now cancel SQL queries just like native queries can be cancelled. A new API has been added for cancelling SQL queries. Web-console now users this API to cancel SQL queries. Earlier, it only used to close the client connection while sql query keeps running in druid. +We've added a new API to cancel SQL queries, so you can now cancel SQL queries just like you can cancel native queries. You can use the API from the Web-console. In previous versions, cancellation from the console only closed the client connection while the SQL query kept running on Druid. + +(#11643) +(#11738) +(#11710) + +### Improvements to SQL user experience + +This release includes several additional improvements for SQL: + +- You no longer need to include a trailing slash `/` for JDBC connections to druid (#11737) +- You can now use scans to as outer queries (#11831) +- Added a class to sanitize JDBC exceptions and to log them (#11843) +- Added type headers to response format to make it easier for clients to interpret the results of SQL queries (#11914) +- Improved the way the `DruidRexExecutor` handles numeric arrays (#11968) +- Druid now returns an empty result after optimizing a group by query to a time series query (#12065) Review Comment: ```suggestion - Druid now returns an empty result after optimizing a GROUP BY query to a time series query (#12065) ``` ########## druid-0.23-release-notes.md: ########## @@ -5,34 +5,183 @@ Apache Druid 0.23.0 contains over 450 new features, bug fixes, performance enhan ## Query engine ### Grouping on arrays without exploding the arrays -[12078](https://github.com/apache/druid/pull/12078) +You can now group on a multi-value dimension as an array. For a datasource named "test": +``` +{"timestamp": "2011-01-12T00:00:00.000Z", "tags": ["t1","t2","t3"]} #row1 +{"timestamp": "2011-01-13T00:00:00.000Z", "tags": ["t3","t4","t5"]} #row2 +{"timestamp": "2011-01-14T00:00:00.000Z", "tags": ["t5","t6","t7"]} #row3 +{"timestamp": "2011-01-14T00:00:00.000Z", "tags": []} #row4 +``` +The following query: +```json +{ + "queryType": "groupBy", + "dataSource": "test", + "intervals": [ + "1970-01-01T00:00:00.000Z/3000-01-01T00:00:00.000Z" + ], + "granularity": { + "type": "all" + }, + "virtualColumns" : [ { + "type" : "expression", + "name" : "v0", + "expression" : "mv_to_array(\"tags\")", + "outputType" : "ARRAY<STRING>" + } ], + "dimensions": [ + { + "type": "default", + "dimension": "v0", + "outputName": "tags" + "outputType":"ARRAY<STRING>" + } + ], + "aggregations": [ + { + "type": "count", + "name": "count" + } + ] +} +``` +Returns the following: + +``` +[ + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "[]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "["t1","t2","t3"]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "[t3","t4","t5"]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 2, + "tags": "["t5","t6","t7"]" + } + } +] +``` +(#12078) +(#12253) ### Specify a column other than __time column for row comparison in first/last aggregators -[11949](https://github.com/apache/druid/pull/11949) + +You can pass time column in `*first`/`*last` aggregators and LATEST / EARLIEST SQL functions. This provides support for cases where the time is stored as a part of a column different than "__time". You can also specify another logical time column. +(#11949) +(#12145) + +### Improvements to querying user experience + +This release includes several improvements for querying: +- Added the SQL query id to response header for failed SQL query to aid in location the error messages (#11756) +- Added input type validation for DataSketches HLL (#12131) +- Improved JDBC logging (#11676) +- Added SQL functions MV_FILTER_ONLY and MV_FILTER_NONE to filter rows of multi-value string dimensions to include only the supplied list of values or none of them respectively (#11650) +- Added ARRAY_CONCAT_AGG to aggregate array inputs together into a single array(#12226) +- Added the ability to authorize the usage of query context parameters (#12396) +- Improved query IDs to make it easier to link queryies and sub-queries for end-to-end visibility of a query (#11809) +- Added a safe divide function to protect against division by 0 (#11904) +- You can now add a query context to internally generated `SegmentMetadata` Query (#11429) +- Added support for Druid complex types to the native expression processing system to make all Druid data usable within expressions (#11853, #12016) ## Streaming Ingestion -### Kafka Input format for parsing headers and key -[11630](https://github.com/apache/druid/pull/11630) +### Kafka input format for parsing headers and key +We've introduced a Kafka input format so you can ingest header data in addition to the message contents. For example: +- the event key field +- event headers +- the Kafka event timestamp +- the Kafka event value that stores the payload. + +(#11630) ## Native Batch Ingestion -### Multi-dimensional range partitioning -Multi-dimensional range partitioning allows users to partition their data on the ranges of any number of dimensions. It develops further on the concepts behind "single-dim" partitioning and is now arguably the most preferable secondary partitioning both for query performance and storage efficiency. -[11848](https://github.com/apache/druid/pull/11848) -[11973](https://github.com/apache/druid/pull/11973) +### Multi-dimension range partitioning +Multi-dimension range partitioning allows users to partition their data on the ranges of any number of dimensions. It develops further on the concepts behind "single-dim" partitioning and is now arguably the most preferable secondary partitioning both for query performance and storage efficiency. +(#11848) +(#11973) + +### Improved replace data behavior +In previous versions of Druid, if ingested data with `dropExisting` flag to replace data, Druid would retain the existing data for a time chunk if there was no new data to replace it. Now, if you set `dropExisting` to `true` in your `ioSpec` and ingest data for a time range that includes a time chunk with no data, Druid uses a tombstone to overshadow the existing data in the empty time chunk. +(#12137) + +This release includes several improvements for native batch ingestion: +- Druid now emits a new metric when a batch task finishes waiting for segment availability. (#11090) +- Added `segmentAvailabilityWaitTimeMs`, the time in milliseconds the milliseconds that a task waited for its segments to be handed off to Historical nodes, to `IngestionStatsAndErrorsTaskReportData` (#11090) +- Added functionality to preserve existing metrics during ingestion (#12185) +- Parallel native batch task can now provide task reports for the sequential and single phase mode (e.g., used with dynamic partitioning) as well as single phase mode subtasks (#11688) +- Added support for `RowStats` in `druid/indexer/v1/task/{task_id}/reports` API for multi-phase parallel indexing task (#12280) +- Fixed the OOM failures in dimension distribution phase of parallel indexing (#12331) +- Added support to handle null dimension values while creating partition boundaries (#11973) + +## Improvements to ingestion in general + +This release includes several improvements for ingestion in general: +- Removed the template modifier from `IncrementalIndex<AggregatorType>` because it is no longer required +- You can now use `JsonPath` functions in `JsonPath` expressions during ingestion (#11722) +- Druid no longer creates a materialized list of segment files and elimited looping over the files to reduce OOM issues (#11903) +- Added an intermediate-persist `IndexSpec` to the main "merge" method in `IndexMerger` (#11940) +- `Granularity.granularitiesFinerThan` now returns ALL if you pass in ALL (#12003) +- Added a configuation parameter for appending tasks to allow them to use a SHARED lock (#12041) +- `SchemaRegistryBasedAvroBytesDecoder` now throws a `ParseException` instead of RE when it fails to retrieve a schema (#12080) +- Added `includeAllDimensions` to `dimensionsSpec` to put all explicit dimensions first in `InputRow` and subsequently any other dimensions found in input data (#12276) +- Added the ability to store null columns in the Segments (#12279) + +## Compaction + +This release includes several improvements for compaction: +- Automatic compaction now supports complex dimensions (#11924) +- Automatic compaction now supports overlapping segment intervals (#12062) +- You can now configure automatic compaction to calculate ratio of slots available for compaction tasks from maximum slots including autoscaler maximum worker nodes (#12263) +- You can configure the Coordinator auto compaction duty period separately from other indexing duties (#12263) ## SQL ### Human-readable and actionable SQL error messages -Till 0.22.1, if a SQL query is not supported, users get a very cryptic and unhelpful error message. With this change, error message will include exactly what part of their SQL query is not supported by druid. One such example is executing a scan query that is ordered on a dimension other than time column. -[11911](https://github.com/apache/druid/pull/11911) +Until version 0.22.1, if you issued an unsupported SQL query, Druid would throw very cryptic and unhelpful error messages. With this change, error messages include exactly the part of the SQL query that is not supported in Druid. For example, if you run a scan query that is ordered on a dimension other than time column. + +(#11911) ### Cancel API for SQL queries -Users can now cancel SQL queries just like native queries can be cancelled. A new API has been added for cancelling SQL queries. Web-console now users this API to cancel SQL queries. Earlier, it only used to close the client connection while sql query keeps running in druid. +We've added a new API to cancel SQL queries, so you can now cancel SQL queries just like you can cancel native queries. You can use the API from the Web-console. In previous versions, cancellation from the console only closed the client connection while the SQL query kept running on Druid. + +(#11643) +(#11738) +(#11710) + +### Improvements to SQL user experience + +This release includes several additional improvements for SQL: + +- You no longer need to include a trailing slash `/` for JDBC connections to druid (#11737) +- You can now use scans to as outer queries (#11831) Review Comment: ```suggestion - You can now use scans as outer queries (#11831) ``` ########## druid-0.23-release-notes.md: ########## @@ -5,34 +5,183 @@ Apache Druid 0.23.0 contains over 450 new features, bug fixes, performance enhan ## Query engine ### Grouping on arrays without exploding the arrays -[12078](https://github.com/apache/druid/pull/12078) +You can now group on a multi-value dimension as an array. For a datasource named "test": +``` +{"timestamp": "2011-01-12T00:00:00.000Z", "tags": ["t1","t2","t3"]} #row1 +{"timestamp": "2011-01-13T00:00:00.000Z", "tags": ["t3","t4","t5"]} #row2 +{"timestamp": "2011-01-14T00:00:00.000Z", "tags": ["t5","t6","t7"]} #row3 +{"timestamp": "2011-01-14T00:00:00.000Z", "tags": []} #row4 +``` +The following query: +```json +{ + "queryType": "groupBy", + "dataSource": "test", + "intervals": [ + "1970-01-01T00:00:00.000Z/3000-01-01T00:00:00.000Z" + ], + "granularity": { + "type": "all" + }, + "virtualColumns" : [ { + "type" : "expression", + "name" : "v0", + "expression" : "mv_to_array(\"tags\")", + "outputType" : "ARRAY<STRING>" + } ], + "dimensions": [ + { + "type": "default", + "dimension": "v0", + "outputName": "tags" + "outputType":"ARRAY<STRING>" + } + ], + "aggregations": [ + { + "type": "count", + "name": "count" + } + ] +} +``` +Returns the following: + +``` +[ + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "[]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "["t1","t2","t3"]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "[t3","t4","t5"]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 2, + "tags": "["t5","t6","t7"]" + } + } +] +``` +(#12078) +(#12253) ### Specify a column other than __time column for row comparison in first/last aggregators -[11949](https://github.com/apache/druid/pull/11949) + +You can pass time column in `*first`/`*last` aggregators and LATEST / EARLIEST SQL functions. This provides support for cases where the time is stored as a part of a column different than "__time". You can also specify another logical time column. +(#11949) +(#12145) + +### Improvements to querying user experience + +This release includes several improvements for querying: +- Added the SQL query id to response header for failed SQL query to aid in location the error messages (#11756) +- Added input type validation for DataSketches HLL (#12131) +- Improved JDBC logging (#11676) +- Added SQL functions MV_FILTER_ONLY and MV_FILTER_NONE to filter rows of multi-value string dimensions to include only the supplied list of values or none of them respectively (#11650) +- Added ARRAY_CONCAT_AGG to aggregate array inputs together into a single array(#12226) +- Added the ability to authorize the usage of query context parameters (#12396) +- Improved query IDs to make it easier to link queryies and sub-queries for end-to-end visibility of a query (#11809) +- Added a safe divide function to protect against division by 0 (#11904) +- You can now add a query context to internally generated `SegmentMetadata` Query (#11429) +- Added support for Druid complex types to the native expression processing system to make all Druid data usable within expressions (#11853, #12016) ## Streaming Ingestion -### Kafka Input format for parsing headers and key -[11630](https://github.com/apache/druid/pull/11630) +### Kafka input format for parsing headers and key +We've introduced a Kafka input format so you can ingest header data in addition to the message contents. For example: +- the event key field +- event headers +- the Kafka event timestamp +- the Kafka event value that stores the payload. + +(#11630) ## Native Batch Ingestion -### Multi-dimensional range partitioning -Multi-dimensional range partitioning allows users to partition their data on the ranges of any number of dimensions. It develops further on the concepts behind "single-dim" partitioning and is now arguably the most preferable secondary partitioning both for query performance and storage efficiency. -[11848](https://github.com/apache/druid/pull/11848) -[11973](https://github.com/apache/druid/pull/11973) +### Multi-dimension range partitioning +Multi-dimension range partitioning allows users to partition their data on the ranges of any number of dimensions. It develops further on the concepts behind "single-dim" partitioning and is now arguably the most preferable secondary partitioning both for query performance and storage efficiency. +(#11848) +(#11973) + +### Improved replace data behavior +In previous versions of Druid, if ingested data with `dropExisting` flag to replace data, Druid would retain the existing data for a time chunk if there was no new data to replace it. Now, if you set `dropExisting` to `true` in your `ioSpec` and ingest data for a time range that includes a time chunk with no data, Druid uses a tombstone to overshadow the existing data in the empty time chunk. +(#12137) + +This release includes several improvements for native batch ingestion: +- Druid now emits a new metric when a batch task finishes waiting for segment availability. (#11090) +- Added `segmentAvailabilityWaitTimeMs`, the time in milliseconds the milliseconds that a task waited for its segments to be handed off to Historical nodes, to `IngestionStatsAndErrorsTaskReportData` (#11090) +- Added functionality to preserve existing metrics during ingestion (#12185) +- Parallel native batch task can now provide task reports for the sequential and single phase mode (e.g., used with dynamic partitioning) as well as single phase mode subtasks (#11688) +- Added support for `RowStats` in `druid/indexer/v1/task/{task_id}/reports` API for multi-phase parallel indexing task (#12280) +- Fixed the OOM failures in dimension distribution phase of parallel indexing (#12331) +- Added support to handle null dimension values while creating partition boundaries (#11973) + +## Improvements to ingestion in general + +This release includes several improvements for ingestion in general: +- Removed the template modifier from `IncrementalIndex<AggregatorType>` because it is no longer required +- You can now use `JsonPath` functions in `JsonPath` expressions during ingestion (#11722) +- Druid no longer creates a materialized list of segment files and elimited looping over the files to reduce OOM issues (#11903) +- Added an intermediate-persist `IndexSpec` to the main "merge" method in `IndexMerger` (#11940) +- `Granularity.granularitiesFinerThan` now returns ALL if you pass in ALL (#12003) +- Added a configuation parameter for appending tasks to allow them to use a SHARED lock (#12041) +- `SchemaRegistryBasedAvroBytesDecoder` now throws a `ParseException` instead of RE when it fails to retrieve a schema (#12080) +- Added `includeAllDimensions` to `dimensionsSpec` to put all explicit dimensions first in `InputRow` and subsequently any other dimensions found in input data (#12276) +- Added the ability to store null columns in the Segments (#12279) + +## Compaction + +This release includes several improvements for compaction: +- Automatic compaction now supports complex dimensions (#11924) +- Automatic compaction now supports overlapping segment intervals (#12062) +- You can now configure automatic compaction to calculate ratio of slots available for compaction tasks from maximum slots including autoscaler maximum worker nodes (#12263) +- You can configure the Coordinator auto compaction duty period separately from other indexing duties (#12263) ## SQL ### Human-readable and actionable SQL error messages -Till 0.22.1, if a SQL query is not supported, users get a very cryptic and unhelpful error message. With this change, error message will include exactly what part of their SQL query is not supported by druid. One such example is executing a scan query that is ordered on a dimension other than time column. -[11911](https://github.com/apache/druid/pull/11911) +Until version 0.22.1, if you issued an unsupported SQL query, Druid would throw very cryptic and unhelpful error messages. With this change, error messages include exactly the part of the SQL query that is not supported in Druid. For example, if you run a scan query that is ordered on a dimension other than time column. + +(#11911) ### Cancel API for SQL queries -Users can now cancel SQL queries just like native queries can be cancelled. A new API has been added for cancelling SQL queries. Web-console now users this API to cancel SQL queries. Earlier, it only used to close the client connection while sql query keeps running in druid. +We've added a new API to cancel SQL queries, so you can now cancel SQL queries just like you can cancel native queries. You can use the API from the Web-console. In previous versions, cancellation from the console only closed the client connection while the SQL query kept running on Druid. + +(#11643) +(#11738) +(#11710) + +### Improvements to SQL user experience + +This release includes several additional improvements for SQL: + +- You no longer need to include a trailing slash `/` for JDBC connections to druid (#11737) +- You can now use scans to as outer queries (#11831) +- Added a class to sanitize JDBC exceptions and to log them (#11843) +- Added type headers to response format to make it easier for clients to interpret the results of SQL queries (#11914) +- Improved the way the `DruidRexExecutor` handles numeric arrays (#11968) +- Druid now returns an empty result after optimizing a group by query to a time series query (#12065) +- As an administrator, you can now configure the implementation for +APPROX_COUNT_DISTINCT and COUNT(DISTINCT expr) in approximate mode (#11181) Review Comment: ```suggestion ``` ########## druid-0.23-release-notes.md: ########## @@ -5,34 +5,183 @@ Apache Druid 0.23.0 contains over 450 new features, bug fixes, performance enhan ## Query engine ### Grouping on arrays without exploding the arrays -[12078](https://github.com/apache/druid/pull/12078) +You can now group on a multi-value dimension as an array. For a datasource named "test": +``` +{"timestamp": "2011-01-12T00:00:00.000Z", "tags": ["t1","t2","t3"]} #row1 +{"timestamp": "2011-01-13T00:00:00.000Z", "tags": ["t3","t4","t5"]} #row2 +{"timestamp": "2011-01-14T00:00:00.000Z", "tags": ["t5","t6","t7"]} #row3 +{"timestamp": "2011-01-14T00:00:00.000Z", "tags": []} #row4 +``` +The following query: +```json +{ + "queryType": "groupBy", + "dataSource": "test", + "intervals": [ + "1970-01-01T00:00:00.000Z/3000-01-01T00:00:00.000Z" + ], + "granularity": { + "type": "all" + }, + "virtualColumns" : [ { + "type" : "expression", + "name" : "v0", + "expression" : "mv_to_array(\"tags\")", + "outputType" : "ARRAY<STRING>" + } ], + "dimensions": [ + { + "type": "default", + "dimension": "v0", + "outputName": "tags" + "outputType":"ARRAY<STRING>" + } + ], + "aggregations": [ + { + "type": "count", + "name": "count" + } + ] +} +``` +Returns the following: + +``` +[ + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "[]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "["t1","t2","t3"]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "[t3","t4","t5"]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 2, + "tags": "["t5","t6","t7"]" + } + } +] +``` +(#12078) +(#12253) ### Specify a column other than __time column for row comparison in first/last aggregators -[11949](https://github.com/apache/druid/pull/11949) + +You can pass time column in `*first`/`*last` aggregators and LATEST / EARLIEST SQL functions. This provides support for cases where the time is stored as a part of a column different than "__time". You can also specify another logical time column. +(#11949) +(#12145) + +### Improvements to querying user experience + +This release includes several improvements for querying: +- Added the SQL query id to response header for failed SQL query to aid in location the error messages (#11756) +- Added input type validation for DataSketches HLL (#12131) +- Improved JDBC logging (#11676) +- Added SQL functions MV_FILTER_ONLY and MV_FILTER_NONE to filter rows of multi-value string dimensions to include only the supplied list of values or none of them respectively (#11650) +- Added ARRAY_CONCAT_AGG to aggregate array inputs together into a single array(#12226) +- Added the ability to authorize the usage of query context parameters (#12396) +- Improved query IDs to make it easier to link queryies and sub-queries for end-to-end visibility of a query (#11809) +- Added a safe divide function to protect against division by 0 (#11904) +- You can now add a query context to internally generated `SegmentMetadata` Query (#11429) Review Comment: ```suggestion - You can now add a query context to internally generated `SegmentMetadata` query (#11429) ``` ########## druid-0.23-release-notes.md: ########## @@ -5,34 +5,183 @@ Apache Druid 0.23.0 contains over 450 new features, bug fixes, performance enhan ## Query engine ### Grouping on arrays without exploding the arrays -[12078](https://github.com/apache/druid/pull/12078) +You can now group on a multi-value dimension as an array. For a datasource named "test": +``` +{"timestamp": "2011-01-12T00:00:00.000Z", "tags": ["t1","t2","t3"]} #row1 +{"timestamp": "2011-01-13T00:00:00.000Z", "tags": ["t3","t4","t5"]} #row2 +{"timestamp": "2011-01-14T00:00:00.000Z", "tags": ["t5","t6","t7"]} #row3 +{"timestamp": "2011-01-14T00:00:00.000Z", "tags": []} #row4 +``` +The following query: +```json +{ + "queryType": "groupBy", + "dataSource": "test", + "intervals": [ + "1970-01-01T00:00:00.000Z/3000-01-01T00:00:00.000Z" + ], + "granularity": { + "type": "all" + }, + "virtualColumns" : [ { + "type" : "expression", + "name" : "v0", + "expression" : "mv_to_array(\"tags\")", + "outputType" : "ARRAY<STRING>" + } ], + "dimensions": [ + { + "type": "default", + "dimension": "v0", + "outputName": "tags" + "outputType":"ARRAY<STRING>" + } + ], + "aggregations": [ + { + "type": "count", + "name": "count" + } + ] +} +``` +Returns the following: + +``` +[ + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "[]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "["t1","t2","t3"]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "[t3","t4","t5"]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 2, + "tags": "["t5","t6","t7"]" + } + } +] +``` +(#12078) +(#12253) ### Specify a column other than __time column for row comparison in first/last aggregators -[11949](https://github.com/apache/druid/pull/11949) + +You can pass time column in `*first`/`*last` aggregators and LATEST / EARLIEST SQL functions. This provides support for cases where the time is stored as a part of a column different than "__time". You can also specify another logical time column. +(#11949) +(#12145) + +### Improvements to querying user experience + +This release includes several improvements for querying: +- Added the SQL query id to response header for failed SQL query to aid in location the error messages (#11756) +- Added input type validation for DataSketches HLL (#12131) +- Improved JDBC logging (#11676) +- Added SQL functions MV_FILTER_ONLY and MV_FILTER_NONE to filter rows of multi-value string dimensions to include only the supplied list of values or none of them respectively (#11650) +- Added ARRAY_CONCAT_AGG to aggregate array inputs together into a single array(#12226) +- Added the ability to authorize the usage of query context parameters (#12396) +- Improved query IDs to make it easier to link queryies and sub-queries for end-to-end visibility of a query (#11809) +- Added a safe divide function to protect against division by 0 (#11904) +- You can now add a query context to internally generated `SegmentMetadata` Query (#11429) +- Added support for Druid complex types to the native expression processing system to make all Druid data usable within expressions (#11853, #12016) ## Streaming Ingestion -### Kafka Input format for parsing headers and key -[11630](https://github.com/apache/druid/pull/11630) +### Kafka input format for parsing headers and key +We've introduced a Kafka input format so you can ingest header data in addition to the message contents. For example: +- the event key field +- event headers +- the Kafka event timestamp +- the Kafka event value that stores the payload. + +(#11630) ## Native Batch Ingestion -### Multi-dimensional range partitioning -Multi-dimensional range partitioning allows users to partition their data on the ranges of any number of dimensions. It develops further on the concepts behind "single-dim" partitioning and is now arguably the most preferable secondary partitioning both for query performance and storage efficiency. -[11848](https://github.com/apache/druid/pull/11848) -[11973](https://github.com/apache/druid/pull/11973) +### Multi-dimension range partitioning +Multi-dimension range partitioning allows users to partition their data on the ranges of any number of dimensions. It develops further on the concepts behind "single-dim" partitioning and is now arguably the most preferable secondary partitioning both for query performance and storage efficiency. +(#11848) +(#11973) + +### Improved replace data behavior +In previous versions of Druid, if ingested data with `dropExisting` flag to replace data, Druid would retain the existing data for a time chunk if there was no new data to replace it. Now, if you set `dropExisting` to `true` in your `ioSpec` and ingest data for a time range that includes a time chunk with no data, Druid uses a tombstone to overshadow the existing data in the empty time chunk. +(#12137) + +This release includes several improvements for native batch ingestion: +- Druid now emits a new metric when a batch task finishes waiting for segment availability. (#11090) +- Added `segmentAvailabilityWaitTimeMs`, the time in milliseconds the milliseconds that a task waited for its segments to be handed off to Historical nodes, to `IngestionStatsAndErrorsTaskReportData` (#11090) Review Comment: ```suggestion - Added `segmentAvailabilityWaitTimeMs`, the duration in milliseconds that a task waited for its segments to be handed off to Historical nodes, to `IngestionStatsAndErrorsTaskReportData` (#11090) ``` ########## druid-0.23-release-notes.md: ########## @@ -5,34 +5,183 @@ Apache Druid 0.23.0 contains over 450 new features, bug fixes, performance enhan ## Query engine ### Grouping on arrays without exploding the arrays -[12078](https://github.com/apache/druid/pull/12078) +You can now group on a multi-value dimension as an array. For a datasource named "test": +``` +{"timestamp": "2011-01-12T00:00:00.000Z", "tags": ["t1","t2","t3"]} #row1 +{"timestamp": "2011-01-13T00:00:00.000Z", "tags": ["t3","t4","t5"]} #row2 +{"timestamp": "2011-01-14T00:00:00.000Z", "tags": ["t5","t6","t7"]} #row3 +{"timestamp": "2011-01-14T00:00:00.000Z", "tags": []} #row4 +``` +The following query: +```json +{ + "queryType": "groupBy", + "dataSource": "test", + "intervals": [ + "1970-01-01T00:00:00.000Z/3000-01-01T00:00:00.000Z" + ], + "granularity": { + "type": "all" + }, + "virtualColumns" : [ { + "type" : "expression", + "name" : "v0", + "expression" : "mv_to_array(\"tags\")", + "outputType" : "ARRAY<STRING>" + } ], + "dimensions": [ + { + "type": "default", + "dimension": "v0", + "outputName": "tags" + "outputType":"ARRAY<STRING>" + } + ], + "aggregations": [ + { + "type": "count", + "name": "count" + } + ] +} +``` +Returns the following: + +``` +[ + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "[]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "["t1","t2","t3"]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "[t3","t4","t5"]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 2, + "tags": "["t5","t6","t7"]" + } + } +] +``` +(#12078) +(#12253) ### Specify a column other than __time column for row comparison in first/last aggregators -[11949](https://github.com/apache/druid/pull/11949) + +You can pass time column in `*first`/`*last` aggregators and LATEST / EARLIEST SQL functions. This provides support for cases where the time is stored as a part of a column different than "__time". You can also specify another logical time column. +(#11949) +(#12145) + +### Improvements to querying user experience + +This release includes several improvements for querying: +- Added the SQL query id to response header for failed SQL query to aid in location the error messages (#11756) +- Added input type validation for DataSketches HLL (#12131) +- Improved JDBC logging (#11676) +- Added SQL functions MV_FILTER_ONLY and MV_FILTER_NONE to filter rows of multi-value string dimensions to include only the supplied list of values or none of them respectively (#11650) +- Added ARRAY_CONCAT_AGG to aggregate array inputs together into a single array(#12226) +- Added the ability to authorize the usage of query context parameters (#12396) +- Improved query IDs to make it easier to link queryies and sub-queries for end-to-end visibility of a query (#11809) +- Added a safe divide function to protect against division by 0 (#11904) +- You can now add a query context to internally generated `SegmentMetadata` Query (#11429) +- Added support for Druid complex types to the native expression processing system to make all Druid data usable within expressions (#11853, #12016) ## Streaming Ingestion -### Kafka Input format for parsing headers and key -[11630](https://github.com/apache/druid/pull/11630) +### Kafka input format for parsing headers and key +We've introduced a Kafka input format so you can ingest header data in addition to the message contents. For example: +- the event key field +- event headers +- the Kafka event timestamp +- the Kafka event value that stores the payload. + +(#11630) ## Native Batch Ingestion -### Multi-dimensional range partitioning -Multi-dimensional range partitioning allows users to partition their data on the ranges of any number of dimensions. It develops further on the concepts behind "single-dim" partitioning and is now arguably the most preferable secondary partitioning both for query performance and storage efficiency. -[11848](https://github.com/apache/druid/pull/11848) -[11973](https://github.com/apache/druid/pull/11973) +### Multi-dimension range partitioning +Multi-dimension range partitioning allows users to partition their data on the ranges of any number of dimensions. It develops further on the concepts behind "single-dim" partitioning and is now arguably the most preferable secondary partitioning both for query performance and storage efficiency. +(#11848) +(#11973) + +### Improved replace data behavior +In previous versions of Druid, if ingested data with `dropExisting` flag to replace data, Druid would retain the existing data for a time chunk if there was no new data to replace it. Now, if you set `dropExisting` to `true` in your `ioSpec` and ingest data for a time range that includes a time chunk with no data, Druid uses a tombstone to overshadow the existing data in the empty time chunk. +(#12137) + +This release includes several improvements for native batch ingestion: +- Druid now emits a new metric when a batch task finishes waiting for segment availability. (#11090) +- Added `segmentAvailabilityWaitTimeMs`, the time in milliseconds the milliseconds that a task waited for its segments to be handed off to Historical nodes, to `IngestionStatsAndErrorsTaskReportData` (#11090) +- Added functionality to preserve existing metrics during ingestion (#12185) +- Parallel native batch task can now provide task reports for the sequential and single phase mode (e.g., used with dynamic partitioning) as well as single phase mode subtasks (#11688) +- Added support for `RowStats` in `druid/indexer/v1/task/{task_id}/reports` API for multi-phase parallel indexing task (#12280) +- Fixed the OOM failures in dimension distribution phase of parallel indexing (#12331) +- Added support to handle null dimension values while creating partition boundaries (#11973) + +## Improvements to ingestion in general + +This release includes several improvements for ingestion in general: +- Removed the template modifier from `IncrementalIndex<AggregatorType>` because it is no longer required +- You can now use `JsonPath` functions in `JsonPath` expressions during ingestion (#11722) +- Druid no longer creates a materialized list of segment files and elimited looping over the files to reduce OOM issues (#11903) +- Added an intermediate-persist `IndexSpec` to the main "merge" method in `IndexMerger` (#11940) +- `Granularity.granularitiesFinerThan` now returns ALL if you pass in ALL (#12003) +- Added a configuation parameter for appending tasks to allow them to use a SHARED lock (#12041) +- `SchemaRegistryBasedAvroBytesDecoder` now throws a `ParseException` instead of RE when it fails to retrieve a schema (#12080) +- Added `includeAllDimensions` to `dimensionsSpec` to put all explicit dimensions first in `InputRow` and subsequently any other dimensions found in input data (#12276) +- Added the ability to store null columns in the Segments (#12279) + +## Compaction + +This release includes several improvements for compaction: +- Automatic compaction now supports complex dimensions (#11924) +- Automatic compaction now supports overlapping segment intervals (#12062) +- You can now configure automatic compaction to calculate ratio of slots available for compaction tasks from maximum slots including autoscaler maximum worker nodes (#12263) +- You can configure the Coordinator auto compaction duty period separately from other indexing duties (#12263) ## SQL ### Human-readable and actionable SQL error messages -Till 0.22.1, if a SQL query is not supported, users get a very cryptic and unhelpful error message. With this change, error message will include exactly what part of their SQL query is not supported by druid. One such example is executing a scan query that is ordered on a dimension other than time column. -[11911](https://github.com/apache/druid/pull/11911) +Until version 0.22.1, if you issued an unsupported SQL query, Druid would throw very cryptic and unhelpful error messages. With this change, error messages include exactly the part of the SQL query that is not supported in Druid. For example, if you run a scan query that is ordered on a dimension other than time column. + +(#11911) ### Cancel API for SQL queries -Users can now cancel SQL queries just like native queries can be cancelled. A new API has been added for cancelling SQL queries. Web-console now users this API to cancel SQL queries. Earlier, it only used to close the client connection while sql query keeps running in druid. +We've added a new API to cancel SQL queries, so you can now cancel SQL queries just like you can cancel native queries. You can use the API from the Web-console. In previous versions, cancellation from the console only closed the client connection while the SQL query kept running on Druid. Review Comment: ```suggestion We've added a new API to cancel SQL queries, so you can now cancel SQL queries just like you can cancel native queries. You can use the API from the web console. In previous versions, cancellation from the console only closed the client connection while the SQL query kept running on Druid. ``` ########## druid-0.23-release-notes.md: ########## @@ -5,34 +5,183 @@ Apache Druid 0.23.0 contains over 450 new features, bug fixes, performance enhan ## Query engine ### Grouping on arrays without exploding the arrays -[12078](https://github.com/apache/druid/pull/12078) +You can now group on a multi-value dimension as an array. For a datasource named "test": +``` +{"timestamp": "2011-01-12T00:00:00.000Z", "tags": ["t1","t2","t3"]} #row1 +{"timestamp": "2011-01-13T00:00:00.000Z", "tags": ["t3","t4","t5"]} #row2 +{"timestamp": "2011-01-14T00:00:00.000Z", "tags": ["t5","t6","t7"]} #row3 +{"timestamp": "2011-01-14T00:00:00.000Z", "tags": []} #row4 +``` +The following query: +```json +{ + "queryType": "groupBy", + "dataSource": "test", + "intervals": [ + "1970-01-01T00:00:00.000Z/3000-01-01T00:00:00.000Z" + ], + "granularity": { + "type": "all" + }, + "virtualColumns" : [ { + "type" : "expression", + "name" : "v0", + "expression" : "mv_to_array(\"tags\")", + "outputType" : "ARRAY<STRING>" + } ], + "dimensions": [ + { + "type": "default", + "dimension": "v0", + "outputName": "tags" + "outputType":"ARRAY<STRING>" + } + ], + "aggregations": [ + { + "type": "count", + "name": "count" + } + ] +} +``` +Returns the following: + +``` +[ + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "[]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "["t1","t2","t3"]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "[t3","t4","t5"]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 2, + "tags": "["t5","t6","t7"]" + } + } +] +``` +(#12078) +(#12253) ### Specify a column other than __time column for row comparison in first/last aggregators -[11949](https://github.com/apache/druid/pull/11949) + +You can pass time column in `*first`/`*last` aggregators and LATEST / EARLIEST SQL functions. This provides support for cases where the time is stored as a part of a column different than "__time". You can also specify another logical time column. +(#11949) +(#12145) + +### Improvements to querying user experience + +This release includes several improvements for querying: +- Added the SQL query id to response header for failed SQL query to aid in location the error messages (#11756) +- Added input type validation for DataSketches HLL (#12131) +- Improved JDBC logging (#11676) +- Added SQL functions MV_FILTER_ONLY and MV_FILTER_NONE to filter rows of multi-value string dimensions to include only the supplied list of values or none of them respectively (#11650) +- Added ARRAY_CONCAT_AGG to aggregate array inputs together into a single array(#12226) +- Added the ability to authorize the usage of query context parameters (#12396) +- Improved query IDs to make it easier to link queryies and sub-queries for end-to-end visibility of a query (#11809) +- Added a safe divide function to protect against division by 0 (#11904) +- You can now add a query context to internally generated `SegmentMetadata` Query (#11429) +- Added support for Druid complex types to the native expression processing system to make all Druid data usable within expressions (#11853, #12016) ## Streaming Ingestion -### Kafka Input format for parsing headers and key -[11630](https://github.com/apache/druid/pull/11630) +### Kafka input format for parsing headers and key +We've introduced a Kafka input format so you can ingest header data in addition to the message contents. For example: +- the event key field +- event headers +- the Kafka event timestamp +- the Kafka event value that stores the payload. + +(#11630) ## Native Batch Ingestion -### Multi-dimensional range partitioning -Multi-dimensional range partitioning allows users to partition their data on the ranges of any number of dimensions. It develops further on the concepts behind "single-dim" partitioning and is now arguably the most preferable secondary partitioning both for query performance and storage efficiency. -[11848](https://github.com/apache/druid/pull/11848) -[11973](https://github.com/apache/druid/pull/11973) +### Multi-dimension range partitioning +Multi-dimension range partitioning allows users to partition their data on the ranges of any number of dimensions. It develops further on the concepts behind "single-dim" partitioning and is now arguably the most preferable secondary partitioning both for query performance and storage efficiency. +(#11848) +(#11973) + +### Improved replace data behavior +In previous versions of Druid, if ingested data with `dropExisting` flag to replace data, Druid would retain the existing data for a time chunk if there was no new data to replace it. Now, if you set `dropExisting` to `true` in your `ioSpec` and ingest data for a time range that includes a time chunk with no data, Druid uses a tombstone to overshadow the existing data in the empty time chunk. +(#12137) + +This release includes several improvements for native batch ingestion: +- Druid now emits a new metric when a batch task finishes waiting for segment availability. (#11090) +- Added `segmentAvailabilityWaitTimeMs`, the time in milliseconds the milliseconds that a task waited for its segments to be handed off to Historical nodes, to `IngestionStatsAndErrorsTaskReportData` (#11090) +- Added functionality to preserve existing metrics during ingestion (#12185) +- Parallel native batch task can now provide task reports for the sequential and single phase mode (e.g., used with dynamic partitioning) as well as single phase mode subtasks (#11688) +- Added support for `RowStats` in `druid/indexer/v1/task/{task_id}/reports` API for multi-phase parallel indexing task (#12280) +- Fixed the OOM failures in dimension distribution phase of parallel indexing (#12331) +- Added support to handle null dimension values while creating partition boundaries (#11973) + +## Improvements to ingestion in general + +This release includes several improvements for ingestion in general: +- Removed the template modifier from `IncrementalIndex<AggregatorType>` because it is no longer required +- You can now use `JsonPath` functions in `JsonPath` expressions during ingestion (#11722) +- Druid no longer creates a materialized list of segment files and elimited looping over the files to reduce OOM issues (#11903) +- Added an intermediate-persist `IndexSpec` to the main "merge" method in `IndexMerger` (#11940) +- `Granularity.granularitiesFinerThan` now returns ALL if you pass in ALL (#12003) +- Added a configuation parameter for appending tasks to allow them to use a SHARED lock (#12041) +- `SchemaRegistryBasedAvroBytesDecoder` now throws a `ParseException` instead of RE when it fails to retrieve a schema (#12080) +- Added `includeAllDimensions` to `dimensionsSpec` to put all explicit dimensions first in `InputRow` and subsequently any other dimensions found in input data (#12276) +- Added the ability to store null columns in the Segments (#12279) + +## Compaction + +This release includes several improvements for compaction: +- Automatic compaction now supports complex dimensions (#11924) +- Automatic compaction now supports overlapping segment intervals (#12062) +- You can now configure automatic compaction to calculate ratio of slots available for compaction tasks from maximum slots including autoscaler maximum worker nodes (#12263) +- You can configure the Coordinator auto compaction duty period separately from other indexing duties (#12263) ## SQL ### Human-readable and actionable SQL error messages -Till 0.22.1, if a SQL query is not supported, users get a very cryptic and unhelpful error message. With this change, error message will include exactly what part of their SQL query is not supported by druid. One such example is executing a scan query that is ordered on a dimension other than time column. -[11911](https://github.com/apache/druid/pull/11911) +Until version 0.22.1, if you issued an unsupported SQL query, Druid would throw very cryptic and unhelpful error messages. With this change, error messages include exactly the part of the SQL query that is not supported in Druid. For example, if you run a scan query that is ordered on a dimension other than time column. Review Comment: ```suggestion Until version 0.22.1, if you issued an unsupported SQL query, Druid would throw very cryptic and unhelpful error messages. With this change, error messages include exactly the part of the SQL query that is not supported in Druid. For example, if you run a scan query that is ordered on a dimension other than the time column. ``` ########## druid-0.23-release-notes.md: ########## @@ -5,34 +5,183 @@ Apache Druid 0.23.0 contains over 450 new features, bug fixes, performance enhan ## Query engine ### Grouping on arrays without exploding the arrays -[12078](https://github.com/apache/druid/pull/12078) +You can now group on a multi-value dimension as an array. For a datasource named "test": +``` +{"timestamp": "2011-01-12T00:00:00.000Z", "tags": ["t1","t2","t3"]} #row1 +{"timestamp": "2011-01-13T00:00:00.000Z", "tags": ["t3","t4","t5"]} #row2 +{"timestamp": "2011-01-14T00:00:00.000Z", "tags": ["t5","t6","t7"]} #row3 +{"timestamp": "2011-01-14T00:00:00.000Z", "tags": []} #row4 +``` +The following query: +```json +{ + "queryType": "groupBy", + "dataSource": "test", + "intervals": [ + "1970-01-01T00:00:00.000Z/3000-01-01T00:00:00.000Z" + ], + "granularity": { + "type": "all" + }, + "virtualColumns" : [ { + "type" : "expression", + "name" : "v0", + "expression" : "mv_to_array(\"tags\")", + "outputType" : "ARRAY<STRING>" + } ], + "dimensions": [ + { + "type": "default", + "dimension": "v0", + "outputName": "tags" + "outputType":"ARRAY<STRING>" + } + ], + "aggregations": [ + { + "type": "count", + "name": "count" + } + ] +} +``` +Returns the following: + +``` +[ + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "[]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "["t1","t2","t3"]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "[t3","t4","t5"]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 2, + "tags": "["t5","t6","t7"]" + } + } +] +``` +(#12078) +(#12253) ### Specify a column other than __time column for row comparison in first/last aggregators -[11949](https://github.com/apache/druid/pull/11949) + +You can pass time column in `*first`/`*last` aggregators and LATEST / EARLIEST SQL functions. This provides support for cases where the time is stored as a part of a column different than "__time". You can also specify another logical time column. +(#11949) +(#12145) + +### Improvements to querying user experience + +This release includes several improvements for querying: +- Added the SQL query id to response header for failed SQL query to aid in location the error messages (#11756) +- Added input type validation for DataSketches HLL (#12131) +- Improved JDBC logging (#11676) +- Added SQL functions MV_FILTER_ONLY and MV_FILTER_NONE to filter rows of multi-value string dimensions to include only the supplied list of values or none of them respectively (#11650) +- Added ARRAY_CONCAT_AGG to aggregate array inputs together into a single array(#12226) +- Added the ability to authorize the usage of query context parameters (#12396) +- Improved query IDs to make it easier to link queryies and sub-queries for end-to-end visibility of a query (#11809) +- Added a safe divide function to protect against division by 0 (#11904) +- You can now add a query context to internally generated `SegmentMetadata` Query (#11429) +- Added support for Druid complex types to the native expression processing system to make all Druid data usable within expressions (#11853, #12016) ## Streaming Ingestion -### Kafka Input format for parsing headers and key -[11630](https://github.com/apache/druid/pull/11630) +### Kafka input format for parsing headers and key +We've introduced a Kafka input format so you can ingest header data in addition to the message contents. For example: +- the event key field +- event headers +- the Kafka event timestamp +- the Kafka event value that stores the payload. + +(#11630) ## Native Batch Ingestion -### Multi-dimensional range partitioning -Multi-dimensional range partitioning allows users to partition their data on the ranges of any number of dimensions. It develops further on the concepts behind "single-dim" partitioning and is now arguably the most preferable secondary partitioning both for query performance and storage efficiency. -[11848](https://github.com/apache/druid/pull/11848) -[11973](https://github.com/apache/druid/pull/11973) +### Multi-dimension range partitioning +Multi-dimension range partitioning allows users to partition their data on the ranges of any number of dimensions. It develops further on the concepts behind "single-dim" partitioning and is now arguably the most preferable secondary partitioning both for query performance and storage efficiency. Review Comment: ```suggestion Multi-dimension range partitioning allows users to partition their data on the ranges of any number of dimensions. It develops further on the concepts behind "single-dim" partitioning and is now arguably the most preferable secondary partitioning, both for query performance and storage efficiency. ``` ########## druid-0.23-release-notes.md: ########## @@ -71,6 +220,16 @@ Users can now cancel SQL queries just like native queries can be cancelled. A ne [12352](https://github.com/apache/druid/pull/12352) +This release includes several additional improvements for metrics: +- Druid includes the Prometheus emitter by defult (#11812) +- Fixed the missing `conversionFactor` in Prometheus emitter (12338) +- Fixed an issue with the `ingest/events/messageGap` metric (#12337) +- Added metrics for Shenandoah GC (#12369) +- Added metrics as follows: `Cpu` and `CpuSet` to `java.util.metrics.cgroups`, `ProcFsUtil` for `procfs` info, and `CgroupCpuMonitor` and `CgroupCpuSetMonitor` (#11763) +- Added support to route data through an HTTP proxy (#11891) +- Added more metrics for Jetty server thread pool usage (#11113) +- Added worker category as a dimension TaskSlot metrics of the indexing service (#11554) Review Comment: ```suggestion - Added worker category as a dimension TaskSlot metric of the indexing service (#11554) ``` ########## druid-0.23-release-notes.md: ########## @@ -5,34 +5,183 @@ Apache Druid 0.23.0 contains over 450 new features, bug fixes, performance enhan ## Query engine ### Grouping on arrays without exploding the arrays -[12078](https://github.com/apache/druid/pull/12078) +You can now group on a multi-value dimension as an array. For a datasource named "test": +``` +{"timestamp": "2011-01-12T00:00:00.000Z", "tags": ["t1","t2","t3"]} #row1 +{"timestamp": "2011-01-13T00:00:00.000Z", "tags": ["t3","t4","t5"]} #row2 +{"timestamp": "2011-01-14T00:00:00.000Z", "tags": ["t5","t6","t7"]} #row3 +{"timestamp": "2011-01-14T00:00:00.000Z", "tags": []} #row4 +``` +The following query: +```json +{ + "queryType": "groupBy", + "dataSource": "test", + "intervals": [ + "1970-01-01T00:00:00.000Z/3000-01-01T00:00:00.000Z" + ], + "granularity": { + "type": "all" + }, + "virtualColumns" : [ { + "type" : "expression", + "name" : "v0", + "expression" : "mv_to_array(\"tags\")", + "outputType" : "ARRAY<STRING>" + } ], + "dimensions": [ + { + "type": "default", + "dimension": "v0", + "outputName": "tags" + "outputType":"ARRAY<STRING>" + } + ], + "aggregations": [ + { + "type": "count", + "name": "count" + } + ] +} +``` +Returns the following: + +``` +[ + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "[]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "["t1","t2","t3"]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 1, + "tags": "[t3","t4","t5"]" + } + }, + { + "timestamp": "1970-01-01T00:00:00.000Z", + "event": { + "count": 2, + "tags": "["t5","t6","t7"]" + } + } +] +``` +(#12078) +(#12253) ### Specify a column other than __time column for row comparison in first/last aggregators -[11949](https://github.com/apache/druid/pull/11949) + +You can pass time column in `*first`/`*last` aggregators and LATEST / EARLIEST SQL functions. This provides support for cases where the time is stored as a part of a column different than "__time". You can also specify another logical time column. +(#11949) +(#12145) + +### Improvements to querying user experience + +This release includes several improvements for querying: +- Added the SQL query id to response header for failed SQL query to aid in location the error messages (#11756) +- Added input type validation for DataSketches HLL (#12131) +- Improved JDBC logging (#11676) +- Added SQL functions MV_FILTER_ONLY and MV_FILTER_NONE to filter rows of multi-value string dimensions to include only the supplied list of values or none of them respectively (#11650) +- Added ARRAY_CONCAT_AGG to aggregate array inputs together into a single array(#12226) +- Added the ability to authorize the usage of query context parameters (#12396) +- Improved query IDs to make it easier to link queryies and sub-queries for end-to-end visibility of a query (#11809) +- Added a safe divide function to protect against division by 0 (#11904) +- You can now add a query context to internally generated `SegmentMetadata` Query (#11429) +- Added support for Druid complex types to the native expression processing system to make all Druid data usable within expressions (#11853, #12016) ## Streaming Ingestion -### Kafka Input format for parsing headers and key -[11630](https://github.com/apache/druid/pull/11630) +### Kafka input format for parsing headers and key +We've introduced a Kafka input format so you can ingest header data in addition to the message contents. For example: +- the event key field +- event headers +- the Kafka event timestamp +- the Kafka event value that stores the payload. + +(#11630) ## Native Batch Ingestion -### Multi-dimensional range partitioning -Multi-dimensional range partitioning allows users to partition their data on the ranges of any number of dimensions. It develops further on the concepts behind "single-dim" partitioning and is now arguably the most preferable secondary partitioning both for query performance and storage efficiency. -[11848](https://github.com/apache/druid/pull/11848) -[11973](https://github.com/apache/druid/pull/11973) +### Multi-dimension range partitioning +Multi-dimension range partitioning allows users to partition their data on the ranges of any number of dimensions. It develops further on the concepts behind "single-dim" partitioning and is now arguably the most preferable secondary partitioning both for query performance and storage efficiency. +(#11848) +(#11973) + +### Improved replace data behavior +In previous versions of Druid, if ingested data with `dropExisting` flag to replace data, Druid would retain the existing data for a time chunk if there was no new data to replace it. Now, if you set `dropExisting` to `true` in your `ioSpec` and ingest data for a time range that includes a time chunk with no data, Druid uses a tombstone to overshadow the existing data in the empty time chunk. +(#12137) + +This release includes several improvements for native batch ingestion: +- Druid now emits a new metric when a batch task finishes waiting for segment availability. (#11090) +- Added `segmentAvailabilityWaitTimeMs`, the time in milliseconds the milliseconds that a task waited for its segments to be handed off to Historical nodes, to `IngestionStatsAndErrorsTaskReportData` (#11090) +- Added functionality to preserve existing metrics during ingestion (#12185) +- Parallel native batch task can now provide task reports for the sequential and single phase mode (e.g., used with dynamic partitioning) as well as single phase mode subtasks (#11688) +- Added support for `RowStats` in `druid/indexer/v1/task/{task_id}/reports` API for multi-phase parallel indexing task (#12280) +- Fixed the OOM failures in dimension distribution phase of parallel indexing (#12331) +- Added support to handle null dimension values while creating partition boundaries (#11973) + +## Improvements to ingestion in general + +This release includes several improvements for ingestion in general: +- Removed the template modifier from `IncrementalIndex<AggregatorType>` because it is no longer required +- You can now use `JsonPath` functions in `JsonPath` expressions during ingestion (#11722) +- Druid no longer creates a materialized list of segment files and elimited looping over the files to reduce OOM issues (#11903) +- Added an intermediate-persist `IndexSpec` to the main "merge" method in `IndexMerger` (#11940) +- `Granularity.granularitiesFinerThan` now returns ALL if you pass in ALL (#12003) +- Added a configuation parameter for appending tasks to allow them to use a SHARED lock (#12041) +- `SchemaRegistryBasedAvroBytesDecoder` now throws a `ParseException` instead of RE when it fails to retrieve a schema (#12080) +- Added `includeAllDimensions` to `dimensionsSpec` to put all explicit dimensions first in `InputRow` and subsequently any other dimensions found in input data (#12276) +- Added the ability to store null columns in the Segments (#12279) Review Comment: ```suggestion - Added the ability to store null columns in segments (#12279) ``` ########## druid-0.23-release-notes.md: ########## @@ -79,25 +238,34 @@ Users can now cancel SQL queries just like native queries can be cancelled. A ne ## Other changes +- Druid now processes lookup load failures more quickly (#12397) +- `BalanceSegments#balanceServers` now exits early when there is no balancing work to do (#11768) +- `DimensionHandler` now allows you to define a `DimensionSpec` appropriate for the type of dimension to handle (#11873) +- Added an interface for external schema providers to Druid SQL (#12043) +- Added support for a SQL INSERT planner (#11959) + # Security fixes ## Support for access control on setting query contexts Today, any context params are allowed to users. This can cause 1) a bad UX if the context param is not matured yet or 2) even query failure or system fault in the worst case if a sensitive param is abused, ex) maxSubqueryRows. Druid now has an ability to limit context params per user role. That means, a query will fail if you have a context param set in the query that is not allowed to you. -The context param authorization can be enabled using druid.auth.authorizeQueryContextParams. This is disabled by default to avoid any hassle when performing an upgrade. +The context parameter authorization can be enabled using druid.`auth.authorizeQueryContextParam`s. This is disabled by default to avoid any hassle when performing an upgrade. -[12396](https://github.com/apache/druid/pull/12396) +(#12396) -## +## Other security improvements -# Performance improvements +This release includes several additional improvements for security: +- You can now optionally enable auhorization on Druid system tables (#11720) + +## Performance improvements ### General performance ### Ingestion -- More accurate memory estimations while building an on-heap incremental index. Rather than using the max possible size of an aggregated row, Druid can now use (based on a task context flag) a closer estimate of the actual heap footprint of an aggregated row. This enables the indexer to fit more rows in memory before doing an intermediate persist. [12073](https://github.com/apache/druid/pull/12073) +- More accurate memory estimations while building an on-heap incremental index. Rather than using the max possible size of an aggregated row, Druid can now use (based on a task context flag) a closer estimate of the actual heap footprint of an aggregated row. This enables the indexer to fit more rows in memory before doing an intermediate persist. (#12073) Review Comment: ```suggestion - More accurate memory estimations while building an on-heap incremental index. Rather than using the maximum possible aggregated row size, Druid can now use (based on a task context flag) a closer estimate of the actual heap footprint of an aggregated row. This enables the indexer to fit more rows in memory before performing an intermediate persist. (#12073) ``` ########## druid-0.23-release-notes.md: ########## @@ -79,25 +238,34 @@ Users can now cancel SQL queries just like native queries can be cancelled. A ne ## Other changes +- Druid now processes lookup load failures more quickly (#12397) +- `BalanceSegments#balanceServers` now exits early when there is no balancing work to do (#11768) +- `DimensionHandler` now allows you to define a `DimensionSpec` appropriate for the type of dimension to handle (#11873) +- Added an interface for external schema providers to Druid SQL (#12043) +- Added support for a SQL INSERT planner (#11959) + # Security fixes ## Support for access control on setting query contexts Today, any context params are allowed to users. This can cause 1) a bad UX if the context param is not matured yet or 2) even query failure or system fault in the worst case if a sensitive param is abused, ex) maxSubqueryRows. Druid now has an ability to limit context params per user role. That means, a query will fail if you have a context param set in the query that is not allowed to you. -The context param authorization can be enabled using druid.auth.authorizeQueryContextParams. This is disabled by default to avoid any hassle when performing an upgrade. +The context parameter authorization can be enabled using druid.`auth.authorizeQueryContextParam`s. This is disabled by default to avoid any hassle when performing an upgrade. Review Comment: ```suggestion The context parameter authorization can be enabled using Druid.`auth.authorizeQueryContextParam`s. This is disabled by default to enable a smoother upgrade experience. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
