This is an automated email from the ASF dual-hosted git repository.
kgyrtkirk pushed a commit to branch 36.0.0
in repository https://gitbox.apache.org/repos/asf/druid.git
The following commit(s) were added to refs/heads/36.0.0 by this push:
new 6a88a829460 docs 36.0.0 release notes (#18919)
6a88a829460 is described below
commit 6a88a829460664ec85cbc1425f148c1eb0849935
Author: 317brian <[email protected]>
AuthorDate: Mon Feb 9 06:20:49 2026 -0800
docs 36.0.0 release notes (#18919)
---------
Co-authored-by: Frank Chen <[email protected]>
Co-authored-by: Lucas Capistrant <[email protected]>
Co-authored-by: Zoltan Haindrich <[email protected]>
---
docs/release-info/release-notes.md | 247 +++++++++++++++++++++++++++++++++++--
docs/release-info/upgrade-notes.md | 64 ++++++++++
website/.spelling | 2 +
3 files changed, 301 insertions(+), 12 deletions(-)
diff --git a/docs/release-info/release-notes.md
b/docs/release-info/release-notes.md
index d0573c5cbcc..7e3624df20e 100644
--- a/docs/release-info/release-notes.md
+++ b/docs/release-info/release-notes.md
@@ -24,15 +24,15 @@ title: "Release notes"
<!--Replace {{DRUIDVERSION}} with the correct Druid version.-->
-Apache Druid \{\{DRUIDVERSION}} contains over $NUMBER_FEATURES new features,
bug fixes, performance enhancements, documentation improvements, and additional
test coverage from $NUMBER_OF_CONTRIBUTORS contributors.
+Apache Druid 36.0.0 contains over 189 new features, bug fixes, performance
enhancements, documentation improvements, and additional test coverage from 34
contributors.
<!--
Replace {{MILESTONE}} with the correct milestone number. For example:
https://github.com/apache/druid/issues?q=is%3Aclosed+milestone%3A28.0+sort%3Aupdated-desc+
-->
-See the [complete set of
changes](https://github.com/apache/druid/issues?q=is%3Aclosed+milestone%3A{{MILESTONE}}+sort%3Aupdated-desc+)
for additional details, including bug fixes.
+See the [complete set of
changes](https://github.com/apache/druid/issues?q=is%3Aclosed+milestone%3A36.0.0+sort%3Aupdated-desc+)
for additional details, including bug fixes.
-Review the [upgrade notes](#upgrade-notes) and [incompatible
changes](#incompatible-changes) before you upgrade to Druid \{\{DRUIDVERSION}}.
+Review the [upgrade notes](#upgrade-notes) before you upgrade to Druid 36.0.0.
If you are upgrading across multiple versions, see the [Upgrade
notes](upgrade-notes.md) page, which lists upgrade notes for the most recent
Druid versions.
<!--
@@ -61,59 +61,282 @@ This section contains important information about new and
existing features.
This section contains detailed release notes separated by areas.
+#### Druid operator
+
+Druid Operator is a Kubernetes controller that manages the lifecycle of your
Druid clusters. The operator simplifies the management of Druid clusters with
its custom logic that is configurable through
+Kubernetes CRDs.
+
+[#18435](https://github.com/apache/druid/pull/18435)
+
+#### Cost-based autoscaling for streaming ingestion
+
+Druid now supports cost-based autoscaling for streaming ingestion that
optimizes task count by balancing lag reduction against resource efficiency..
This autoscaling strategy uses the following formula:
+
+```
+totalCost = lagWeight × lagRecoveryTime + idleWeight × idlenessCost
+```
+
+which accounts for the time to clear the backlog and compute time:
+
+```
+lagRecoveryTime = aggregateLag / (taskCount × avgProcessingRate) — time to
clear backlog
+idlenessCost = taskCount × taskDuration × predictedIdleRatio — wasted compute
time
+```
+
+[#18819](https://github.com/apache/druid/pull/18819)
+
+#### Kubernetes client mode (experimental)
+
+In `kubernetes-overlord-extensions` an experimental Kubernetes client mode was
added. The new mode uses the `fabric8` `SharedInformers` to cache k8s metadata.
This greatly reduces API traffic between the Overlord and k8s control plane.
You can try out this feature using the following config:
+
+```
+druid.indexer.runner.useK8sSharedInformers=true
+```
+
+[#18599](https://github.com/apache/druid/pull/18599)
+
+#### cgroup v2 support
+
+cgroup v2 is now supported, and all cgroup metrics now emit `cgroupversion` to
identify which version is being used.
+
+The following metrics automatically switch to v2 if v2 is detected:
`CgroupCpuMonitor` , `CgroupCpuSetMonitor`,
`CgroupDiskMonitor`,`MemoryMonitor`. `CpuAcctDeltaMonitor` fails gracefully if
v2 is detected.
+
+Additionally, `CgroupV2CpuMonitor` now also emits `cgroup/cpu/shares` and
`cgroup/cpu/cores_quota`.
+
+[#18705](https://github.com/apache/druid/pull/18705)
+
+#### Query reports for Dart
+
+Dart now supports query reports for running and recently completed queries.
The reports can be fetched from the
`/druid/v2/sql/queries/<sqlQueryId>/reports` endpoint.
+
+The format of the response is a JSON object with two keys, "query" and
"report". The "query" key is the same info that is available from the existing
`/druid/v2/sql/queries` endpoint. The "report" key is a report map including an
MSQ report.
+
+You can control the retention behavior for reports using the following configs:
+
+* `druid.msq.dart.controller.maxRetainedReportCount`: Max number of reports
that are retained. The default is 0, meaning no reports are retained
+* `druid.msq.dart.controller.maxRetainedReportDuration`: How long reports are
retained in ISO 8601 duration format. The default is `PT0S`, meaning time-based
expiration is turned off
+
+[#18886](https://github.com/apache/druid/pull/18886)
+
+#### New segment format
+
+The new version 10 segment format improves upon version 9. It is off by
default and not compatible with older segment format versions.
+
+Set `druid.indexer.task.buildV10=true` to make segments in the new format.
+
+If you downgrade, you must reindex your data with a supported segment format
version.
+
+You can use the `bin/dump-segment` tool to view segment metadata. The tool
outputs serialized JSON.
+
+[#18880](https://github.com/apache/druid/pull/18880)
[#18901](https://github.com/apache/druid/pull/18901)
+
### Web console
+#### New info available in the web console
+
+The web console now includes information about the number of available
processors and the total memory (in binary bytes).
+
+This information is also available through the `sys.servers` table.
+
+[#18613](https://github.com/apache/druid/pull/18613)
+
#### Other web console improvements
+* Added tracking for inactive workers for MSQ execution stages
[#18768](https://github.com/apache/druid/pull/18768)
+* Added a refresh button for JSON views and stage viewers
[#18768](https://github.com/apache/druid/pull/18768)
+* You can now define `ARRAY` type parameters in the query view
[#18586](https://github.com/apache/druid/pull/18586)
+* Changed system table queries to now automatically use the native engine
[#18857](https://github.com/apache/druid/pull/18857)
+* Improved time charts to support multiple measures
[#18701](https://github.com/apache/druid/pull/18701)
+
### Ingestion
+* Added support for AWS `InternalError` code retries
[#18720](https://github.com/apache/druid/pull/18720)
+* Improved ingestion to be more resilient. Ingestion tasks no longer fail if
the task log upload fails with an exception
[#18748](https://github.com/apache/druid/pull/18748)
+* Improved how Druid handles situations where data doesn't match the expected
type [#18878](https://github.com/apache/druid/pull/18878)
+* Improved JSON ingestion so that Druid can compute JSON values directly from
dictionary or index structures, allowing ingestion to skip persisting raw JSON
data entirely. This reduces on-disk storage size
[#18589](https://github.com/apache/druid/pull/18589)
+* You can now choose between full dictionary-based indexing and nulls-only
indexing for long/double fields in a nested column
[#18722](https://github.com/apache/druid/pull/18722)
+
#### SQL-based ingestion
+##### Additional ingestion configurations
+
+You can now use the following configs to control how your data gets ingested
and stored:
+
+* `maxInputFilesPerWorker`: Controls the maximum number of input files or
segments per worker.
+* `maxPartitions`: Controls the maximum number of output partitions for any
single stage, which affects how many segments are generated during ingestion.
+
+[#18826](https://github.com/apache/druid/pull/18826)
+
##### Other SQL-based ingestion improvements
+* Added `maxRowsInMemory` to replace `rowsInMemory`. `rowsInMemory` now
functions as an alternate way to provide that config and is ignored if
`maxRowsInMemory` is specified. Previously, only `rowsInMemory` existed
[#18832](https://github.com/apache/druid/pull/18832)
+
#### Streaming ingestion
+##### Record offset and partition
+
+You can now ingest the record offset (`offsetColumnName`) and partition
(`partitionColumnName`) using the `KafkaInputFormat`. Their default names are
`kafka.offset` and `kafka.partition` respectively .
+
+[#18757](https://github.com/apache/druid/pull/18757)
+
##### Other streaming ingestion improvements
+* Improved supervisors so that they can't kill tasks while the supervisor is
stopping [#18767](https://github.com/apache/druid/pull/18767)
+* Improved the lag-based autoscaler for streaming ingestion
[#18745](https://github.com/apache/druid/pull/18745)
+* Improved the `SeekableStream` supervisor autoscaler to wait for tasks to
complete before attempting subsequent scale operations. This helps prevent
duplicate supervisor history entries
[#18715](https://github.com/apache/druid/pull/18715)
+
### Querying
#### Other querying improvements
+* Improved the user experience for invalid `regex_exp` queries. An error gets
returned now [#18762](https://github.com/apache/druid/pull/18762)
+
### Cluster management
+#### Dynamic capacity for Kubernetes-based deployments
+
+Druid can now dynamically tune the task runner capacity.
+
+Include the `capacity` field in a POST API call to
`/druid/indexer/v1/k8s/taskrunner/executionconfig`. Setting a value this way
overrides `druid.indexer.runner.capacity`.
+
+[#18591](https://github.com/apache/druid/pull/18591)
+
+#### Server properties table
+
+The `sys.server_properties` table exposes the runtime properties configured
for each Druid server. Each row represents a single property key-value pair
associated with a specific server.
+
+[#18692](https://github.com/apache/druid/pull/18692)
+
#### Other cluster management improvements
+* Added quality of service filtering for the Overlord so that health check
threads don't get blocked [#18033](https://github.com/apache/druid/pull/18033)
+
### Data management
#### Other data management improvements
+* Added the `mostFragmentedFirst` compaction policy that prioritizes intervals
with the most small uncompacted segments
[#18802](https://github.com/apache/druid/pull/18802)
+* Improved how segment files get deleted to prevent partial segment files from
remaining in the event of a failure during a delete operation
[#18696](https://github.com/apache/druid/pull/18696)
+* Improved compaction so that it identifies multi-value dimensions for
dimension schemas that can produce them
[#18760](https://github.com/apache/druid/pull/18760)
+* Improved the reliability when pulling segments
[#18821](https://github.com/apache/druid/pull/18821)
+
### Metrics and monitoring
+#### Task metrics
+
+All task metrics now emit the following dimensions: `taskId`, `dataSource`,
`taskType`, `groupId`, and `id`. Note that `id` is emitted for backwards
compatibility. It will be removed in favor of the `taskId` dimension in a
future release.
+
+[#18876](https://github.com/apache/druid/pull/18876)
+
+#### Ingestion metrics
+
+The following metrics for streaming and batch tasks now emit the actual values
instead of 0: `ingest/merge/time`, `ingest/merge/cpu`, and
`ingest/persists/cpu`.
+
+[#18866](https://github.com/apache/druid/pull/18866)
+
+##### `statsd` metrics
+
+The following metrics have been added to the default list for `statsd`:
+
+* `task/action/run/time`
+* `task/status/queue/count`
+* `task/status/updated/count`
+* `ingest/handoff/time`
+
+[#18846](https://github.com/apache/druid/pull/18846)
+
+#### Other metrics and monitoring improvements
+
+* Added `reason` dimension to `ingest/events/thrownAway` metric. This allows
for increased observability on why certain events are being logically excluded
from ingest [#188855](https://github.com/apache/druid/pull/18855)
+* Added logging for all handlers for a stage before they start or stop, which
can help you understand execution order
[#18662](https://github.com/apache/druid/pull/18662)
+* Added new Jetty thread pool metrics to capture request-serving thread
statistics: `jetty/threadPool/utilized`, `jetty/threadPool/ready` and
`jetty/threadPool/utilizationRate`
[#18883](https://github.com/apache/druid/pull/18883)
+* Added `tier` and `priority` dimensions to the `segments/max` metric
[#18890](https://github.com/apache/druid/pull/18890)
+* Added `GroupByStatsMonitor`, which includes `dataSource` and `taskId`
dimensions for metrics emitted on peons
[#18711](https://github.com/apache/druid/pull/18711)
+* Added `task/waiting/time` metric, which measures the time it takes for a
task to be placed onto the task runner for scheduling and running
[#18735](https://github.com/apache/druid/pull/18735)
+* Added the `supervisorId` to streaming task metrics to help clarify
situations where multiple supervisors ingest data into a single datasource
[#18803](https://github.com/apache/druid/pull/18803)
+* Added `StorageMonitor` to `druid.monitoring.monitors` to measure storage and
virtual storage usage by segment cache
[#18742](https://github.com/apache/druid/pull/18742)
+* Druid now logs the following:
+ * total bytes gathered when the max scatter-gather bytes limit is reached
[#18841](https://github.com/apache/druid/pull/18841)
+ * `query/bytes` metric for even for failed requests
[#18842](https://github.com/apache/druid/pull/18842)
+* Changed Prometheus emitter TTL tracking to consider all label value
combinations instead of just the metric name. Labels aren't tracked when the
TTL isn't set [#18718](https://github.com/apache/druid/pull/18718)
[#18689](https://github.com/apache/druid/pull/18689)
+* Changed lifecycle `stop()` to be logged at the `info` level to match
`start()` [#18640](https://github.com/apache/druid/pull/18640)
+* Changed the trigger for metrics emission so that metrics get emitted any
time a task completes [#18766](https://github.com/apache/druid/pull/18766)
+
+
+* Improved the metrics emitter so that it emits metrics for all task
completions [#18766](https://github.com/apache/druid/pull/18766)
+
### Extensions
-### Documentation improvements
+#### SpectatorHistogram extension
+
+* Added `SPECTATOR_COUNT` and `SPECTATOR_PERCENTILE` SQL functions
[#18885](https://github.com/apache/druid/pull/18885)
+* Improved the performance of the SpectatorHistogram extension through
vectorization [#18813](https://github.com/apache/druid/pull/18813)
## Upgrade notes and incompatible changes
### Upgrade notes
-#### Front-coded dictionaries
+#### MSQ controller tasks
+
+When upgrading from Druid 30 or earlier, MSQ `query_controller` tasks can fail
during a rolling update due to the addition of new counters that are not
backwards compatible with these older versions. You can either retry any failed
queries after the update completes; or you can set `includeAllCounters` to
`false` in the query context for any MSQ jobs that need to run during the
rolling update; or you can upgrade to Druid 31–35 first before upgrading to
Druid 36.
+
+[#18761](https://github.com/apache/druid/pull/18761)
-<!--Carry this forward until 32. Then move it to incompatible changes -->
+#### Segment format
-In Druid 32.0.0, the front coded dictionaries feature will be turned on by
default. Front-coded dictionaries reduce storage and improve performance by
optimizing for strings where the front part looks similar.
+Druid 36.0.0 supports segment format version 10. Previous versions of Druid
don't support version 10. If you downgrade, you must reindex your data with a
supported segment format version.
-Once this feature is on, you cannot easily downgrade to an earlier version
that does not support the feature.
+[#18880](https://github.com/apache/druid/pull/18880)
-For more information, see [Migration guide: front-coded
dictionaries](./migr-front-coded-dict.md).
+#### Deprecated metrics
-If you're already using this feature, you don't need to take any action.
+Monitors on peons that previously emitted the `id` dimension from
`JettyMonitor`, `OshiSysMonitor`, `JvmMonitor`, `JvmCpuMonitor`,
`JvmThreadsMonitor` and `SysMonitor` to represent the task ID are deprecated
and will be removed in a future release. Use the `taskId` dimension instead.
+[#18709](https://github.com/apache/druid/pull/18709)
-### Incompatible changes
+#### Removed metrics
+
+The following obsolete metrics have been removed:
+
+* `segment/cost/raw` [#18846](https://github.com/apache/druid/pull/18846)
+* `segment/cost/normalized`
[#18846](https://github.com/apache/druid/pull/18846)
+* `segment/cost/normalization`
[#18846](https://github.com/apache/druid/pull/18846)
+* `task/action/log/time` [#18649](https://github.com/apache/druid/pull/18649)
### Developer notes
+#### Segment file interfaces
+
+New `SegmentFileBuilder` and `SegmentFileMapper` interfaces have been defined
to replace direct usages of `FileSmoosher` and `SmooshedFileMapper` to abstract
the segment building and reading process.
+
+The main developer visible changes for extension writers with custom column
implementations is that the `Serializer` interface has changed the `writeTo`
method:
+
+* It now accepts a `SegmentFileBuilder` instead of a `FileSmoosher`
+* The `ColumnBuilder` method `getFileMapper` now returns a `SegmentFileMapper`
instead of `SmooshedFileMapper`.
+
+Extensions which do not provide custom column implementations should not be
impacted by these changes.
+
+[#18608](https://github.com/apache/druid/pull/18608)
+
+#### Other developer improvements
+
+* Added the ability to override the default Kafka image for testing
[#18739](https://github.com/apache/druid/pull/18739)
+* Changed `fastDecompressor` to `safeDecompressor`
[#18930](https://github.com/apache/druid/pull/18930)
+* Extensions can now provide query kit implementations
[#18875](https://github.com/apache/druid/pull/18875)
+* Removed version overrides in individual `pom` files. For a full list, see
the pull request [#18708](https://github.com/apache/druid/pull/18708)
+
#### Dependency updates
-The following dependencies have had their versions bumped:
\ No newline at end of file
+The following dependencies have had their versions bumped:
+
+* `org.apache.logging.log4j:log4j-core` from `2.22.1` to `2.25.3`
[#18874](https://github.com/apache/druid/pull/18874)
+* `org.mozilla:rhino` from `1.7.14` to `1.7.14.1`
[#18868](https://github.com/apache/druid/pull/18868)
+* `net.java.dev.jna` and `net.java.dev.jna` versions from `5.13.0` to `5.18.1`
for Oshi monitor [#18848](https://github.com/apache/druid/pull/18848)
+* `com.github.oshi:oshi-core from` from `6.4.4` to `6.9.1`
[#18839](https://github.com/apache/druid/pull/18839)
+* `bcpkix-jdk18on` from `1.78.1` to `1.79`
[#18834](https://github.com/apache/druid/pull/18834)
+* `org.eclipse.jetty` from `12.0.25`to `12.0.30`
[#18773](https://github.com/apache/druid/pull/18773)
+* `hamcrest` from `1.3` to `2.2`
[#18708](https://github.com/apache/druid/pull/18708)
+* `org.apache.commons:commons-lang3` from `3.18.0` to `3.19.0`
[#18695](https://github.com/apache/druid/pull/18695)
+* `org.apache.maven.plugins:maven-shade-plugin` from `3.5.0` to `3.6.1`
+* `com.netflix.spectator` from `1.7.0` to `1.9.0`
[#18887](https://github.com/apache/druid/pull/18887)
+* `org.bouncycastle:bcpkix-jdk18on` from `1.79` to `1.81` to resolve
`SONATYPE-2025-001911`
diff --git a/docs/release-info/upgrade-notes.md
b/docs/release-info/upgrade-notes.md
index 440eb7d77ef..d492da377e9 100644
--- a/docs/release-info/upgrade-notes.md
+++ b/docs/release-info/upgrade-notes.md
@@ -38,6 +38,70 @@ For more information, see [Migration guide: front-coded
dictionaries](./migr-fro
If you're already using this feature, you don't need to take any action.
+## 36.0.0
+
+### Upgrade notes
+
+#### Deprecated metrics
+
+Monitors on peons that previously emitted the `id` dimension from
`JettyMonitor`, `OshiSysMonitor`, `JvmMonitor`, `JvmCpuMonitor`,
`JvmThreadsMonitor` and `SysMonitor` to represent the task ID are deprecated
and will be removed in a future release. Use the `taskId` dimension instead.
+
+[#18709](https://github.com/apache/druid/pull/18709)
+
+#### Some `statsd` metrics removed
+
+The following obsolete metrics have been removed:
+
+* `segment/cost/raw`
+* `segment/cost/normalized`
+* `segment/cost/normalization`
+
+[#18846](https://github.com/apache/druid/pull/18846)
+
+## 35.0.0
+
+### Upgrade notes
+
+#### Fallback vectorization on by default
+
+The `druid.expressions.allowVectorizeFallback` now defaults to `true`.
Additionally, `SAFE_DIVIDE` can now vectorize as a fallback.
+
+[#18549](https://github.com/apache/druid/pull/18549)
+
+#### Java 11 support removed
+
+Upgrade to Java 17 or 21. Note that some versions of Java 21 encountered
issues during test, specifically Java 21.05-21.07. If possible, avoid these
versions.
+
+[#18424](https://github.com/apache/druid/pull/18424)
+
+#### Jetty 12
+
+A new server configuration option has been added:
`druid.server.http.uriCompliance`. Jetty 12 by default has strict enforcement
of `RFC3986` URI format. This is a change from Jetty 9. To retain compatibility
with legacy Druid, this config defaults to `LEGACY`, which uses the more
permissive URI format enforcement that Jetty 9 used. If the cluster you operate
does not require legacy compatibility, we recommend you use the upstream Jetty
default of `RFC3986` in your Druid deployment. See t [...]
+
+[#18424](https://github.com/apache/druid/pull/18424)
+
+#### Kerberos authentication
+
+The `druid.auth.authenticator.kerberos.cookieSignatureSecret` config is now
mandatory.
+
+[#18368](https://github.com/apache/druid/pull/18368)
+
+#### Multi-stage query task engine
+
+The MSQ task engine is now a core capability of Druid rather than an
extension. It has been in the default extension load list for several releases.
+
+Remove `druid-multi-stage-query` from `druid.extensions.loadList` in
`common.runtimes.properties` before you upgrade.
+
+Druid 35.0.0 will ignore the extension if it's in the load list. Future
versions of Druid will fail to start since it cannot locate the extension.
+
+[#18394](https://github.com/apache/druid/pull/18394)
+
+#### pac4j extension
+
+Due to the upgrade from `pac4j` 4 to 5, session serialization has changed from
`pac4j`’s `JavaSerializer` to standard Java serialization. As a result, clients
of clusters using the `pac4j` extension may be logged out during rolling
upgrades and need to re‑authenticate.
+
+[#18259](https://github.com/apache/druid/pull/18259)
+
## 34.0.0
### Upgrade notes
diff --git a/website/.spelling b/website/.spelling
index 3486f8a8d88..1dfd71c68d7 100644
--- a/website/.spelling
+++ b/website/.spelling
@@ -314,6 +314,7 @@ config
configs
consumerProperties
cron
+CRDs
csv
customizable
dataset
@@ -596,6 +597,7 @@ unannouncements
unary
unassign
uncomment
+uncompacted
underutilization
unintuitive
unioned
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
