317brian commented on code in PR #18919: URL: https://github.com/apache/druid/pull/18919#discussion_r2770468134
########## docs/release-info/release-notes.md: ########## @@ -61,59 +61,282 @@ This section contains important information about new and existing features. This section contains detailed release notes separated by areas. +#### Druid operator + +Druid Operator is a Kubernetes controller that manages the lifecycle of your Druid clusters. The operator simplifies the management of Druid clusters with its custom logic that is configurable through +Kubernetes CRDs. + +[#18435](https://github.com/apache/druid/pull/18435) + +#### Cost-based autoscaling for streaming ingestion + +Druid now supports cost-based autoscaling for streaming ingestion that optimizes task count by balancing lag reduction against resource efficiency.. This autoscaling strategy uses the following formula: + +``` +totalCost = lagWeight × lagRecoveryTime + idleWeight × idlenessCost +``` + +which accounts for the time to clear the backlog and compute time: + +``` +lagRecoveryTime = aggregateLag / (taskCount × avgProcessingRate) — time to clear backlog +idlenessCost = taskCount × taskDuration × predictedIdleRatio — wasted compute time +``` + +[#18819](https://github.com/apache/druid/pull/18819) + +#### Kubernetes client mode (experimental) + +In `kubernetes-overlord-extensions` an experimental Kubernetes client mode was added. The new mode uses the `fabric8` `SharedInformers` to cache k8s metadata. This greatly reduces API traffic between the Overlord and k8s control plane. You can try out this feature using the following config: + +``` +druid.indexer.runner.useK8sSharedInformers=true +``` + +[#18599](https://github.com/apache/druid/pull/18599) + +#### cgroup v2 support + +cgroup v2 is now supported, and all cgroup metrics now emit `cgroupversion` to identify which version is being used. + +The following metrics automatically switch to v2 if v2 is detected: `CgroupCpuMonitor` , `CgroupCpuSetMonitor`, `CgroupDiskMonitor`,`MemoryMonitor`. `CpuAcctDeltaMonitor` fails gracefully if v2 is detected. + +Additionally, `CgroupV2CpuMonitor` now also emits `cgroup/cpu/shares` and `cgroup/cpu/cores_quota`. + +[#18705](https://github.com/apache/druid/pull/18705) + +#### Query reports for Dart + +Dart now supports query reports for running and recently completed queries. The reports can be fetched from the `/druid/v2/sql/queries/<sqlQueryId>/reports` endpoint. + +The format of the response is a JSON object with two keys, "query" and "report". The "query" key is the same info that is available from the existing `/druid/v2/sql/queries` endpoint. The "report" key is a report map including an MSQ report. + +You can control the retention behavior for reports using the following configs: + +* `druid.msq.dart.controller.maxRetainedReportCount`: Max number of reports that are retained. The default is 0, meaning no reports are retained +* `druid.msq.dart.controller.maxRetainedReportDuration`: How long reports are retained in ISO 8601 duration format. The default is `PT0S`, meaning time-based expiration is turned off + +[#18886](https://github.com/apache/druid/pull/18886) + +#### New segment format + +The new version 10 segment format improves upon version 9. Version 10 supports partial segment downloads, a feature provided by the experimental virtual storage fabric feature. To streamline partial fetches, the base segment contents get combined into a single file called `druid.segment`. + +Set `druid.indexer.task.buildV10=true` to make segments in the new format. + +Note that previous versions of Druid don't support version 10. If you downgrade, you must reindex your data with a supported segment format version. Review Comment: ```suggestion The new version 10 segment format improves upon version 9. It is off by default and not compatible with older segment format versions. Set `druid.indexer.task.buildV10=true` to make segments in the new format. If you downgrade, you must reindex your data with a supported segment format version. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
