clintropolis commented on code in PR #18919: URL: https://github.com/apache/druid/pull/18919#discussion_r2739168113
########## docs/release-info/release-notes.md: ########## @@ -61,59 +61,282 @@ This section contains important information about new and existing features. This section contains detailed release notes separated by areas. +#### Druid operator + +Druid Operator is a Kubernetes controller that manages the lifecycle of your Druid clusters. The operator simplifies the management of Druid clusters with its custom logic that is configurable through +Kubernetes CRDs. + +[#18435](https://github.com/apache/druid/pull/18435) + +#### Cost-based autoscaling for streaming ingestion + +Druid now supports cost-based autoscaling for streaming ingestion that optimizes task count by balancing lag reduction against resource efficiency.. This autoscaling strategy uses the following formula: + +``` +totalCost = lagWeight × lagRecoveryTime + idleWeight × idlenessCost +``` + +which accounts for the time to clear the backlog and compute time: + +``` +lagRecoveryTime = aggregateLag / (taskCount × avgProcessingRate) — time to clear backlog +idlenessCost = taskCount × taskDuration × predictedIdleRatio — wasted compute time +``` + +[#18819](https://github.com/apache/druid/pull/18819) + +#### Kubernetes client mode (experimental) + +THe new experimental Kubernetes client mode uses the `fabric8` `SharedInformers` to cache k8s metadata. This greatly reduces API traffic between the Overlord and k8s control plane. You can try out this feature using the following config: + +``` +druid.indexer.runner.useK8sSharedInformers=true +``` + +[#18599](https://github.com/apache/druid/pull/18599) + +#### cgroup v2 support + +cgroup v2 is now supported, and all cgroup metrics now emit `cgroupversion` to identify which version is being used. + +The following metrics automatically switch to v2 if v2 is detected: `CgroupCpuMonitor` , `CgroupCpuSetMonitor`, `CgroupDiskMonitor`,`MemoryMonitor`. `CpuAcctDeltaMonitor` fails gracefully if v2 is detected. + +Additionally, `CgroupV2CpuMonitor` now also emits `cgroup/cpu/shares` and `cgroup/cpu/cores_quota`. + +[#18705](https://github.com/apache/druid/pull/18705) + +#### Query reports for Dart + +Dart now supports query reports for running and recently completed queries. The reports can be fetched from the `/druid/v2/sql/queries/<sqlQueryId>/reports` endpoint. + +The format of the response is a JSON object with two keys, "query" and "report". The "query" key is the same info that is available from the existing `/druid/v2/sql/queries` endpoint. The "report" key is a report map including an MSQ report. + +You can control the retention behavior for reports using the following configs: + +* `druid.msq.dart.controller.maxRetainedReportCount`: Max number of reports that are retained. The default is 0, meaning no reports are retained +* `druid.msq.dart.controller.maxRetainedReportDuration`: How long reports are retained in ISO 8601 duration format. The default is `PT0S`, meaning time-based expiration is turned off + +[#18886](https://github.com/apache/druid/pull/18886) + +#### New segment format + +The new version 10 segment format improves upon version 9. Version 10 supports partial segment downloads, a feature provided by the experimental virtual storage fabric feature. To streamline partial fetches, the base segment contents get combined into a single file called `druid.segment`. Review Comment: >Version 10 supports partial segment downloads, a feature provided by the experimental virtual storage fabric feature. To streamline partial fetches, the base segment contents get combined into a single file called `druid.segment`. this isn't really true yet, while partial downloads is why it was developed but this functionality is not yet implemented. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
