kfaraz commented on code in PR #13429: URL: https://github.com/apache/druid/pull/13429#discussion_r1050802128
########## docs/next-release-notes.md: ########## @@ -21,19 +21,88 @@ title: "WIP release notes for 25.0" ~ under the License. --> -## Query engine +## Highlights -### BIG_SUM SQL function +### Multi-stage query -Added SQL function `BIG_SUM` that uses the [Compressed Big Decimal](https://github.com/apache/druid/pull/10705) Druid extension. +The multi-stage query (MSQ) task engine used for SQL-based ingestion is now production ready. Use it for any supported workloads. For more information, see the following pages: -https://github.com/apache/druid/pull/13102 +- [Ingestion](https://druid.apache.org/docs/latest/ingestion/index.html) +- [SQL-based ingestion](https://druid.apache.org/docs/latest/multi-stage-query/index.html) -### Added Compressed Big Decimal min and max functions +### String dictionary compression (experimental) -Added min and max functions for Compressed Big Decimal and exposed these functions via SQL: BIG_MIN and BIG_MAX. +> Any segment written using string dictionary compression is not readable by older versions of Druid. -https://github.com/apache/druid/pull/13141 +Added support for front coded string dictionaries for smaller string columns, leading to reduced segment sizes with only minor performance penalties for most Druid queries. + +This functionality can be utilized by a new property to `IndexSpec.stringDictionaryEncoding`, which can be set to {"type":"frontCoded", "bucketSize": 4}, {"type":"frontCoded", "bucketSize": 16}, or any power of 2 that is 128 or lower. This property instructs indexing tasks to write segments with the compressed dictionaries with the specific bucket size specified. (`{"type":"utf8"}` is the default). + +For more information, see [Front coding](https://druid.apache.org/docs/latest/ingestion/ingestion-spec.html#front-coding). + +https://github.com/apache/druid/pull/12277 + +### Kubernetes native tasks + +Druid can now use Kubernetes to launch and manage tasks, eliminating the need for MiddleManagers. + +To use this feature, enable the [`druid-kubernetes-overlord-extensions`]((../extensions.md#loading-extensions) in the extensions load list for your Overlord process. + +https://github.com/apache/druid/pull/13156 + +## Behavior changes + +### Memory estimates + +The task context flag `useMaxMemoryEstimates` is now set to false by default to improve memory usage estimation. + +https://github.com/apache/druid/pull/13178 + +### HLL and quantiles sketches + +The aggregation functions for HLL and quantiles sketches returned sketches or numbers when they are finalized depending on where they were in the native query plan. + +Druid no longer finalizes aggregators in the following two cases: + + - aggregators appear in the outer level of a query + - aggregators are used as input to an expression or finalizing-field-access post-aggregator + +This change aligns the behavior of HLL and quantiles sketches with theta sketches. + +To provide backwards compatibility, you can use the `sqlFinalizeOuterSketches` query context parameter that restores the old behavior. + +https://github.com/apache/druid/pull/13247 + +### Kill tasks do not include markAsUnuseddone + +When you kill a task, Druid no longer automatically marks segments as unused. You must explicitly mark them as unused with `POST /druid/coordinator/v1/datasources/{dataSourceName}/markUnused`. +For more information, see the [API reference](https://druid.apache.org/docs/latest/operations/api-reference.html#coordinator) + +https://github.com/apache/druid/pull/13104 + +### Segment discovery + +The default segment discovery method now uses HTTP instead of ZooKeeper. + +This update changes the defaults for the following properties: + +| Property | New default | Previous default | +| - | - | - | +| `druid.serverview.type` for segment management | http | batch | +| `druid.coordinator.loadqueuepeon.type` for segment management | http | curator | +| `druid.indexer.runner.type` for the Overlord | httpRemote | local | + +To use ZooKeeper instead of HTTP, change the values for the properties back to the previous defaults. Review Comment: We should also mention that the Zookeeper based implementations for the above items are now deprecated and will be removed in subsequent releases. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
