capistrant commented on code in PR #18630: URL: https://github.com/apache/druid/pull/18630#discussion_r2491912738
########## docs/release-info/release-notes.md: ########## @@ -57,63 +57,386 @@ For tips about how to write a good release note, see [Release notes](https://git This section contains important information about new and existing features. +### Jetty + +Druid 35 uses Jetty 12. This change may impact your deployment. For more information,see [the upgrade note for Jetty 12](#jetty-12) + +### Java support + +Druid now supports Java 21. Note that some versions of Java 21 encountered issues during test, specifically Java 21.05-21.07. If possible, avoid these versions. + +Additionally, support for Java 11 has been removed. Upgrade to Java 17 or 21. + +[#18424](https://github.com/apache/druid/pull/18424) [#18624](https://github.com/apache/druid/pull/18624) + +### Projections +Projections now support static filters. Additionally, there have been general improvements to performance and reliability. + +[#18342](https://github.com/apache/druid/pull/18342) [#18535](https://github.com/apache/druid/pull/18535) + +### Virtual storage (experimental) + +Virtual storage fabric mode for enables Historical servers to serve more segments than what their physical disk can hold. Instead of loading segments when they're published, segments get loaded on demand during queries. Any unneeded segments get removed from disk but get loaded again when processing a query requires that particular segment. + +To enable virtual storage, set `druid.segmentCache.virtualStorage` to `true`. + +This feature is experimental and has several limitations. For more information, see the pull request: + +[#18176](https://github.com/apache/druid/pull/18176) + +### New input format + +Druid now supports a `lines` input format. Druid reads each line from an input as UTF-8 text and creates a single column named `line` that contains the entire line as a string. Use this for reading line-oriented data in a simple form for later processing. + +[#18433](https://github.com/apache/druid/pull/18433) + +### Multi-stage query task engine + +The MSQ task engine is now a core capability of Druid rather than an extension. It has been in the default extension load list for several releases. + +Remove `druid-multi-stage-query` from `druid.extensions.loadList` in `common.runtimes.properties` before you upgrade. + +Druid 35.0.0 will ignore the extension if it's in the load list. Future versions of Druid will fail to start since it can't locate the extension. + +[#18394](https://github.com/apache/druid/pull/18394) + +### Improved monitor loading + +You can now specify monitors in `common.runtime.properties` and each monitor will be loaded only on the applicable server types. Previously, you needed to define monitors in the specific `runtime.properties` file for the service a monitor is meant for. + +[#18321](https://github.com/apache/druid/pull/18321) + +### Exact count extension + +A new contributor extension (`druid-exact-count-bitmap`) adds support for exact cardinality counting using Roaring Bitmap over a Long column. + +[#18021](https://github.com/apache/druid/pull/18021) + + +### Improved `indexSpec` + + +[#17762](https://github.com/apache/druid/pull/17762) + ## Functional area and related changes This section contains detailed release notes separated by areas. ### Web console +#### Time zones + +You can now configure whether the web console displays local time or UTC. This setting is stored locally in your browser and doesn't impact other users. + +Note that the URL maintains the query parameters in UTC time, but the Druid console automatically converts the filter to local time. + +[#18455](https://github.com/apache/druid/pull/18455) + #### Other web console improvements +- Added better support for MSQ task engine-based compaction tasks. They now use the stages pane to render the compaction report instead of showing the JSON [#18545](https://github.com/apache/druid/pull/18545) +- Added a version column to the **Services** tab so that you can see what version a service is running. This is helpful during rolling upgrades to verify the state of the cluster and upgrade [#18542](https://github.com/apache/druid/pull/18542) +- Improved the resiliency of the web console when the supervisor history is extensive [#18416](https://github.com/apache/druid/pull/18416) + + ### Ingestion +#### Dimension schemas + +At ingestion time, dimension schemas in `dimensionsSpec` are now strictly validated against allowed types. Previously an invalid type would fall back to string dimension. Now, such values are rejected. Users must specify a type that's one of the allowed types. Omitting type still defaults to string, preserving backward compatibility. + +[#18565](https://github.com/apache/druid/pull/18565) + +#### Other ingestion improvements + +- Added support for session tokens (`sessionToken`) to the S3 input source [#18609](https://github.com/apache/druid/pull/18609) +- Improved performance of task APIs served by the Overlord. Druid now reads the in-memory state of the Overlord before fetching task information from the metadata store [#18448](https://github.com/apache/druid/pull/18448) +- Improved task execution so that they can successfully complete even if there are problems pushing logs and reports to deep storage [#18210](https://github.com/apache/druid/pull/18210) + #### SQL-based ingestion ##### Other SQL-based ingestion improvements +- Added the ability to configure the maximum frame size. Generally, you don't need to change this unless you have very large rows [#18442](https://github.com/apache/druid/pull/18442) +- Added logging for when segment processing fails [#18378](https://github.com/apache/druid/pull/18378) +- Improved logging to store the cause of invalid field exceptions [#18517](https://github.com/apache/druid/pull/18517) [#18517](https://github.com/apache/druid/pull/18517) + + #### Streaming ingestion ##### Other streaming ingestion improvements +- Added a count parameter to the supervisor history API [#18416](https://github.com/apache/druid/pull/18416) + ### Querying +#### Caching scan query results + +Druid now supports result-level caching for scan queries. + +By default, this behavior is turned off. To enable, override the `druid.*.cache.unCacheable` property. + +[#18568](https://github.com/apache/druid/pull/18568) + +#### New expressions for sketches + +Druid now supports the following expressions: + +- `HLL_SKETCH_ESTIMATE_WITH_ERROR_BOUNDS` +- `THETA_SKETCH_ESTIMATE_WITH_ERROR_BOUNDS` + +These estimates work on a sketch column and have the same behavior as the post aggregators. + +[#18426](https://github.com/apache/druid/pull/18426) + +#### New multi-value SQL functions + +Druid now supports the following multi-value functions: + +- `MV_FILTER_REGEX`: filters a multi-value expression to include only values matching the specified regular expression pattern +- `MV_FILTER_PREFIX`: filters a multi-value expression to include only values that start with the specified prefix + +[#18281](https://github.com/apache/druid/pull/18281) + +#### Exception handling + +You can now write exceptions that Druid encounters as a row and then verify whether the exception is expected. To use this feature, set the `writeExceptionBodyAsResponseRow` query context parameter to `true`. + +[#18571](https://github.com/apache/druid/pull/18571) + #### Other querying improvements +- Added stricter validation for `GREATEST` and `LEAST` [#18562](https://github.com/apache/druid/pull/18562) +- Added a query context option called `realtimeSegmentsOnly` that returns results from realtime segments only when set to `true` [#18329](https://github.com/apache/druid/pull/18329) +- Improved the performance of the following query types through vectorization: + - `CASE_SEARCHED` and `CASE_SIMPLE` queries [#18512](https://github.com/apache/druid/pull/18512) + - `timestamp_ceil` and `timestamp_extract` queries [#18517](https://github.com/apache/druid/pull/18517) + - `IF` expressions where the `THEN` and `ELSE` expressions [#18507](https://github.com/apache/druid/pull/18507) +- Improved projections so that the granularity in queries can match UTC time zones [#18403](https://github.com/apache/druid/pull/18403) +- Improved the deserialization time for intervals by more than 40% [#18477](https://github.com/apache/druid/pull/18477) +- Improved the performance of `AND`/`OR` as well as `NOT`/`IS TRUE`/`IS FALSE` [#18491](https://github.com/apache/druid/pull/18491) [#18488](https://github.com/apache/druid/pull/18488) +- Improved the performance of scan queries [#18441](https://github.com/apache/druid/pull/18441) +- Improved the performance of metadata queries by parallelizing them at the data-node level [#18592](https://github.com/apache/druid/pull/18592) + ### Cluster management +#### Labels for the metadata table + +You can now configure labels for Druid services. Use the `druid.labels` field in the configuration options for the service and provide a JSON object of key-value pairs, such as `druid.labels={"location":"Airtrunk"}` or `druid.labels.location=Airtrunk`. + +These labels can be queried using the `sys.servers` system table and are also displayed on the **Services** tab in the web console. + +[#18547](https://github.com/apache/druid/pull/18547) + +#### TLS server configs + +Add a `druid.server.https.forceApplyConfig` config flag to indicate that the `TLSServerConfig` values should be applied irrespective of whether a preexisting `SslContextFactory.Server` binding exists. + +[#18610](https://github.com/apache/druid/pull/18610) + +#### Error message strategy + +You can now configure Druid to log an error message and return an error ID for non-user targeted messages. This can be configured by setting `druid.server.http.errorResponseTransform.strategy` to `persona`. + +[#18487](https://github.com/apache/druid/pull/18487) + +#### Kerberos authentication + +The `druid.auth.authenticator.kerberos.cookieSignatureSecret` config is now mandatory. + +[#18368](https://github.com/apache/druid/pull/18368) + #### Other cluster management improvements +- Added a `version` column to `sys.server`. This is useful during rolling upgrades to verify the state of the cluster [#18542](https://github.com/apache/druid/pull/18542) +- Added support for short, unique index names in metadata stores [#18515](https://github.com/apache/druid/pull/18515) +- Added config `stopTaskCountPercent` to lag-based auto-scaler to allow task count to be scaled as a percentage of the current task count. [#18480](https://github.com/apache/druid/pull/18480) +- Added headers to the user agent for services, improving traceability between services [#18505](https://github.com/apache/druid/pull/18505) +- Added a new Kafka emitter config: `druid.emitter.kafka.producer.shutdownTimeout`. It controls how long the Kafka producer waits for pending requests to finish before shutting down [#18427](https://github.com/apache/druid/pull/18427) +- Changed how Netty worker threads get calculated, lowering the number of threads used at small processor counts [#18493](https://github.com/apache/druid/pull/18493) +- Changed the response for the `/status/properties` to be sorted lexicographically on the property names. The response is more readable now and related keys are grouped together [#18506](https://github.com/apache/druid/pull/18506). +- Improved segment loading during startup in certain situations by optimizing concurrent writing [#18470](https://github.com/apache/druid/pull/18470) + ### Data management +#### Segment timeout + +You can now configure timeout for a query's segment processing. + +To set a timeout, set `druid.processing.numTimeoutThreads` to a low number, such as 3, on all data nodes where you want this feature enabled. + +Then, include the `perSegmentTimeout` query context parameter in your query and set it to a value in milliseconds. + #### Other data management improvements +- Improved S3 storage to support storing segments in S3 without zip compression [#18544](https://github.com/apache/druid/pull/18544) + ### Metrics and monitoring +#### Improved Prometheus emitter + +You can now specify the TTL for a metric. If a metric value has not been updated within the TTL period then the value will stop being emitted. + +Set `flushPeriod` to the `exporter` strategy in the `prometheus-emitter`. + +[#18598](https://github.com/apache/druid/pull/18598) + +#### Other metrics and monitoring improvements + +- Added a `statusCode` dimension to `query/time` and `sqlQuery/time` metrics containing the query result status code [#18633](https://github.com/apache/druid/pull/18633) [#18631](https://github.com/apache/druid/pull/18631) +- Added `segment/schemaCache/refreshSkipped/count` metric that shows the number of segments that skipped schema refresh [#18561](https://github.com/apache/druid/pull/18561) +- Added a metric for when auto-scaling /gets skipped when the publishing task set is not empty [#18536](https://github.com/apache/druid/pull/18536) +- Added Kafka consumer metrics for indexing tasks in the `prometheus-emitter` default metrics config [#18458](https://github.com/apache/druid/pull/18458) +- Added an event when a datasource gets dropped from the Coordinator: `segment/schemaCache/datasource/dropped/count` [#18497](https://github.com/apache/druid/pull/18497) +- Added the `taskType` dimension to the following Prometheus metrics to distinguish between realtime and compaction tasks: `ingest/count`, `ingest/segments/count`, and `ingest/tombstones/count` [#18452](https://github.com/apache/druid/pull/18452) +- Added a server dimension to segment assignment skip metrics like `segment/assignSkipped/count` [#18313](https://github.com/apache/druid/pull/18313) +- Changed `CgroupUtil` to be less noisy [#18472](https://github.com/apache/druid/pull/18472) +- Reduced the clutter of output logs when there's a mismatch in Kafka and task partition set [#18234](https://github.com/apache/druid/pull/18234) + ### Extensions -### Documentation improvements +#### Iceberg + +You can now filter Iceberg data based on a configurable time window. + +[#18531](https://github.com/apache/druid/pull/18531) + +#### Kubernetes + +#### HTTP client + +You can now configure what HTTP client library the `Fabric8` `KubernetesClient` uses under the hood to communicate with the k8s server tasks run on. The default remains the same client and config that Druid 34 uses. The additional options that are now available are `okhttp` and a native JDK HttpClient. The default client and config should suffice for most use cases. + +Set the `druid.indexer.runner.k8sAndWorker.http.httpClientType` config to `vertx` (default), `okhttp`, or `jdk`. + +[#18540](https://github.com/apache/druid/pull/18540) + +#### Pod logs + +You can now set a timeout for asynchronous operations that interact with k8s pod logs: `druid.indexer.runner.podLogOperationTimeout`. + +[#18587](https://github.com/apache/druid/pull/18587) + +##### Task logs + +For Middle Manager-less ingestion, persisting task logs no longer have the potential to block indefinitely. Instead, there is a time limit for persisting logs, that if breached, results in giving up on the persist. The default timeout is 5 minutes, but can be configured by overriding `druid.indexer.runner.logSaveTimeout` with a duration, such as `PT60S`. + +Additionally, task log storage for the Kubernetes runner is now optional. + +[#18444](https://github.com/apache/druid/pull/18444) [#18341](https://github.com/apache/druid/pull/18341) + +##### Improved fault tolerance + +The Kubernetes extension is now more resilient against certain transient faults. + +[#18406](https://github.com/apache/druid/pull/18406) + +##### Faster task failure + +Task pods will now output Kubernetes job events when failing during the pod creation phase, and K8s tasks will fail fast under unrecoverable conditions. + +[#18159](https://github.com/apache/druid/pull/18159) + +#### Mount additional configs + +Previously, you could specify additional configuration to be mounted to `_common` via an environment variable, but this only applied to the Druid process. You can now also apply the same configuration to ingestion tasks submitted as Kubernetes jobs (MiddleManager-less mode), ensuring consistency across all task types. + +[#18447](https://github.com/apache/druid/pull/18447) ## Upgrade notes and incompatible changes ### Upgrade notes -#### Front-coded dictionaries +#### Fallback vectorization on by default + +The `druid.expressions.allowVectorizeFallback` now defaults to `true`. Additionally, `SAFE_DIVIDE` can now vectorize as a fallback. + +[#18549](https://github.com/apache/druid/pull/18549) + +#### Jetty 12 + +Druid now uses Jetty 12. Your deployment may be impacted depending, specifically with regards to URI compliance and SNI host checks. -<!--Carry this forward until 32. Then move it to incompatible changes --> +A new server configuration option has been added: `druid.server.http.uriCompliance`. Jetty 12 by default has strict enforcement of `RFC3986` URI format. This is a change from Jetty 9. To retain compatibility with legacy Druid, this config defaults to `LEGACY`, which uses the more permissive URI format enforcement that Jetty 9 used. If the cluster you operate does not require legacy compatibility, we recommend you use the upstream Jetty default of `RFC3986` in your Druid deployment. See the jetty documentation for more info. Review Comment: nit change to wording about why the default is the default for this new config. ```suggestion A new server configuration option has been added: `druid.server.http.uriCompliance`. Jetty 12 by default has strict enforcement of `RFC3986` URI format. This is a change from Jetty 9. To avoid potential breaking changes in existing Druid deployments, this config defaults to `LEGACY`, which uses the more permissive URI format enforcement that Jetty 9 used. If the cluster you operate does not require legacy compatibility, we recommend you use the upstream Jetty default of `RFC3986` in your Druid deployment. See the jetty documentation for more info. ``` ########## docs/release-info/release-notes.md: ########## @@ -57,63 +57,386 @@ For tips about how to write a good release note, see [Release notes](https://git This section contains important information about new and existing features. +### Jetty + +Druid 35 uses Jetty 12. This change may impact your deployment. For more information,see [the upgrade note for Jetty 12](#jetty-12) + +### Java support + +Druid now supports Java 21. Note that some versions of Java 21 encountered issues during test, specifically Java 21.05-21.07. If possible, avoid these versions. + +Additionally, support for Java 11 has been removed. Upgrade to Java 17 or 21. + +[#18424](https://github.com/apache/druid/pull/18424) [#18624](https://github.com/apache/druid/pull/18624) + +### Projections +Projections now support static filters. Additionally, there have been general improvements to performance and reliability. + +[#18342](https://github.com/apache/druid/pull/18342) [#18535](https://github.com/apache/druid/pull/18535) + +### Virtual storage (experimental) + +Virtual storage fabric mode for enables Historical servers to serve more segments than what their physical disk can hold. Instead of loading segments when they're published, segments get loaded on demand during queries. Any unneeded segments get removed from disk but get loaded again when processing a query requires that particular segment. + +To enable virtual storage, set `druid.segmentCache.virtualStorage` to `true`. + +This feature is experimental and has several limitations. For more information, see the pull request: + +[#18176](https://github.com/apache/druid/pull/18176) + +### New input format + +Druid now supports a `lines` input format. Druid reads each line from an input as UTF-8 text and creates a single column named `line` that contains the entire line as a string. Use this for reading line-oriented data in a simple form for later processing. + +[#18433](https://github.com/apache/druid/pull/18433) + +### Multi-stage query task engine + +The MSQ task engine is now a core capability of Druid rather than an extension. It has been in the default extension load list for several releases. + +Remove `druid-multi-stage-query` from `druid.extensions.loadList` in `common.runtimes.properties` before you upgrade. + +Druid 35.0.0 will ignore the extension if it's in the load list. Future versions of Druid will fail to start since it can't locate the extension. + +[#18394](https://github.com/apache/druid/pull/18394) + +### Improved monitor loading + +You can now specify monitors in `common.runtime.properties` and each monitor will be loaded only on the applicable server types. Previously, you needed to define monitors in the specific `runtime.properties` file for the service a monitor is meant for. + +[#18321](https://github.com/apache/druid/pull/18321) + +### Exact count extension + +A new contributor extension (`druid-exact-count-bitmap`) adds support for exact cardinality counting using Roaring Bitmap over a Long column. + +[#18021](https://github.com/apache/druid/pull/18021) + + +### Improved `indexSpec` + + +[#17762](https://github.com/apache/druid/pull/17762) + ## Functional area and related changes This section contains detailed release notes separated by areas. ### Web console +#### Time zones + +You can now configure whether the web console displays local time or UTC. This setting is stored locally in your browser and doesn't impact other users. + +Note that the URL maintains the query parameters in UTC time, but the Druid console automatically converts the filter to local time. + +[#18455](https://github.com/apache/druid/pull/18455) + #### Other web console improvements +- Added better support for MSQ task engine-based compaction tasks. They now use the stages pane to render the compaction report instead of showing the JSON [#18545](https://github.com/apache/druid/pull/18545) +- Added a version column to the **Services** tab so that you can see what version a service is running. This is helpful during rolling upgrades to verify the state of the cluster and upgrade [#18542](https://github.com/apache/druid/pull/18542) +- Improved the resiliency of the web console when the supervisor history is extensive [#18416](https://github.com/apache/druid/pull/18416) + + ### Ingestion +#### Dimension schemas + +At ingestion time, dimension schemas in `dimensionsSpec` are now strictly validated against allowed types. Previously an invalid type would fall back to string dimension. Now, such values are rejected. Users must specify a type that's one of the allowed types. Omitting type still defaults to string, preserving backward compatibility. + +[#18565](https://github.com/apache/druid/pull/18565) + +#### Other ingestion improvements + +- Added support for session tokens (`sessionToken`) to the S3 input source [#18609](https://github.com/apache/druid/pull/18609) +- Improved performance of task APIs served by the Overlord. Druid now reads the in-memory state of the Overlord before fetching task information from the metadata store [#18448](https://github.com/apache/druid/pull/18448) +- Improved task execution so that they can successfully complete even if there are problems pushing logs and reports to deep storage [#18210](https://github.com/apache/druid/pull/18210) + #### SQL-based ingestion ##### Other SQL-based ingestion improvements +- Added the ability to configure the maximum frame size. Generally, you don't need to change this unless you have very large rows [#18442](https://github.com/apache/druid/pull/18442) +- Added logging for when segment processing fails [#18378](https://github.com/apache/druid/pull/18378) +- Improved logging to store the cause of invalid field exceptions [#18517](https://github.com/apache/druid/pull/18517) [#18517](https://github.com/apache/druid/pull/18517) + + #### Streaming ingestion ##### Other streaming ingestion improvements +- Added a count parameter to the supervisor history API [#18416](https://github.com/apache/druid/pull/18416) + ### Querying +#### Caching scan query results + +Druid now supports result-level caching for scan queries. + +By default, this behavior is turned off. To enable, override the `druid.*.cache.unCacheable` property. + +[#18568](https://github.com/apache/druid/pull/18568) + +#### New expressions for sketches + +Druid now supports the following expressions: + +- `HLL_SKETCH_ESTIMATE_WITH_ERROR_BOUNDS` +- `THETA_SKETCH_ESTIMATE_WITH_ERROR_BOUNDS` + +These estimates work on a sketch column and have the same behavior as the post aggregators. + +[#18426](https://github.com/apache/druid/pull/18426) + +#### New multi-value SQL functions + +Druid now supports the following multi-value functions: + +- `MV_FILTER_REGEX`: filters a multi-value expression to include only values matching the specified regular expression pattern +- `MV_FILTER_PREFIX`: filters a multi-value expression to include only values that start with the specified prefix + +[#18281](https://github.com/apache/druid/pull/18281) + +#### Exception handling + +You can now write exceptions that Druid encounters as a row and then verify whether the exception is expected. To use this feature, set the `writeExceptionBodyAsResponseRow` query context parameter to `true`. + +[#18571](https://github.com/apache/druid/pull/18571) + #### Other querying improvements +- Added stricter validation for `GREATEST` and `LEAST` [#18562](https://github.com/apache/druid/pull/18562) +- Added a query context option called `realtimeSegmentsOnly` that returns results from realtime segments only when set to `true` [#18329](https://github.com/apache/druid/pull/18329) +- Improved the performance of the following query types through vectorization: + - `CASE_SEARCHED` and `CASE_SIMPLE` queries [#18512](https://github.com/apache/druid/pull/18512) + - `timestamp_ceil` and `timestamp_extract` queries [#18517](https://github.com/apache/druid/pull/18517) + - `IF` expressions where the `THEN` and `ELSE` expressions [#18507](https://github.com/apache/druid/pull/18507) +- Improved projections so that the granularity in queries can match UTC time zones [#18403](https://github.com/apache/druid/pull/18403) +- Improved the deserialization time for intervals by more than 40% [#18477](https://github.com/apache/druid/pull/18477) +- Improved the performance of `AND`/`OR` as well as `NOT`/`IS TRUE`/`IS FALSE` [#18491](https://github.com/apache/druid/pull/18491) [#18488](https://github.com/apache/druid/pull/18488) +- Improved the performance of scan queries [#18441](https://github.com/apache/druid/pull/18441) +- Improved the performance of metadata queries by parallelizing them at the data-node level [#18592](https://github.com/apache/druid/pull/18592) + ### Cluster management +#### Labels for the metadata table + +You can now configure labels for Druid services. Use the `druid.labels` field in the configuration options for the service and provide a JSON object of key-value pairs, such as `druid.labels={"location":"Airtrunk"}` or `druid.labels.location=Airtrunk`. + +These labels can be queried using the `sys.servers` system table and are also displayed on the **Services** tab in the web console. + +[#18547](https://github.com/apache/druid/pull/18547) + +#### TLS server configs + +Add a `druid.server.https.forceApplyConfig` config flag to indicate that the `TLSServerConfig` values should be applied irrespective of whether a preexisting `SslContextFactory.Server` binding exists. + +[#18610](https://github.com/apache/druid/pull/18610) + +#### Error message strategy + +You can now configure Druid to log an error message and return an error ID for non-user targeted messages. This can be configured by setting `druid.server.http.errorResponseTransform.strategy` to `persona`. + +[#18487](https://github.com/apache/druid/pull/18487) + +#### Kerberos authentication + +The `druid.auth.authenticator.kerberos.cookieSignatureSecret` config is now mandatory. + +[#18368](https://github.com/apache/druid/pull/18368) + #### Other cluster management improvements +- Added a `version` column to `sys.server`. This is useful during rolling upgrades to verify the state of the cluster [#18542](https://github.com/apache/druid/pull/18542) +- Added support for short, unique index names in metadata stores [#18515](https://github.com/apache/druid/pull/18515) +- Added config `stopTaskCountPercent` to lag-based auto-scaler to allow task count to be scaled as a percentage of the current task count. [#18480](https://github.com/apache/druid/pull/18480) +- Added headers to the user agent for services, improving traceability between services [#18505](https://github.com/apache/druid/pull/18505) +- Added a new Kafka emitter config: `druid.emitter.kafka.producer.shutdownTimeout`. It controls how long the Kafka producer waits for pending requests to finish before shutting down [#18427](https://github.com/apache/druid/pull/18427) +- Changed how Netty worker threads get calculated, lowering the number of threads used at small processor counts [#18493](https://github.com/apache/druid/pull/18493) +- Changed the response for the `/status/properties` to be sorted lexicographically on the property names. The response is more readable now and related keys are grouped together [#18506](https://github.com/apache/druid/pull/18506). +- Improved segment loading during startup in certain situations by optimizing concurrent writing [#18470](https://github.com/apache/druid/pull/18470) + ### Data management +#### Segment timeout + +You can now configure timeout for a query's segment processing. + +To set a timeout, set `druid.processing.numTimeoutThreads` to a low number, such as 3, on all data nodes where you want this feature enabled. + +Then, include the `perSegmentTimeout` query context parameter in your query and set it to a value in milliseconds. + #### Other data management improvements +- Improved S3 storage to support storing segments in S3 without zip compression [#18544](https://github.com/apache/druid/pull/18544) + ### Metrics and monitoring +#### Improved Prometheus emitter + +You can now specify the TTL for a metric. If a metric value has not been updated within the TTL period then the value will stop being emitted. + +Set `flushPeriod` to the `exporter` strategy in the `prometheus-emitter`. + +[#18598](https://github.com/apache/druid/pull/18598) + +#### Other metrics and monitoring improvements + +- Added a `statusCode` dimension to `query/time` and `sqlQuery/time` metrics containing the query result status code [#18633](https://github.com/apache/druid/pull/18633) [#18631](https://github.com/apache/druid/pull/18631) +- Added `segment/schemaCache/refreshSkipped/count` metric that shows the number of segments that skipped schema refresh [#18561](https://github.com/apache/druid/pull/18561) +- Added a metric for when auto-scaling /gets skipped when the publishing task set is not empty [#18536](https://github.com/apache/druid/pull/18536) +- Added Kafka consumer metrics for indexing tasks in the `prometheus-emitter` default metrics config [#18458](https://github.com/apache/druid/pull/18458) +- Added an event when a datasource gets dropped from the Coordinator: `segment/schemaCache/datasource/dropped/count` [#18497](https://github.com/apache/druid/pull/18497) +- Added the `taskType` dimension to the following Prometheus metrics to distinguish between realtime and compaction tasks: `ingest/count`, `ingest/segments/count`, and `ingest/tombstones/count` [#18452](https://github.com/apache/druid/pull/18452) +- Added a server dimension to segment assignment skip metrics like `segment/assignSkipped/count` [#18313](https://github.com/apache/druid/pull/18313) +- Changed `CgroupUtil` to be less noisy [#18472](https://github.com/apache/druid/pull/18472) +- Reduced the clutter of output logs when there's a mismatch in Kafka and task partition set [#18234](https://github.com/apache/druid/pull/18234) + ### Extensions -### Documentation improvements +#### Iceberg + +You can now filter Iceberg data based on a configurable time window. + +[#18531](https://github.com/apache/druid/pull/18531) + +#### Kubernetes + +#### HTTP client + +You can now configure what HTTP client library the `Fabric8` `KubernetesClient` uses under the hood to communicate with the k8s server tasks run on. The default remains the same client and config that Druid 34 uses. The additional options that are now available are `okhttp` and a native JDK HttpClient. The default client and config should suffice for most use cases. + +Set the `druid.indexer.runner.k8sAndWorker.http.httpClientType` config to `vertx` (default), `okhttp`, or `jdk`. + +[#18540](https://github.com/apache/druid/pull/18540) + +#### Pod logs + +You can now set a timeout for asynchronous operations that interact with k8s pod logs: `druid.indexer.runner.podLogOperationTimeout`. + +[#18587](https://github.com/apache/druid/pull/18587) + +##### Task logs + +For Middle Manager-less ingestion, persisting task logs no longer have the potential to block indefinitely. Instead, there is a time limit for persisting logs, that if breached, results in giving up on the persist. The default timeout is 5 minutes, but can be configured by overriding `druid.indexer.runner.logSaveTimeout` with a duration, such as `PT60S`. + +Additionally, task log storage for the Kubernetes runner is now optional. + +[#18444](https://github.com/apache/druid/pull/18444) [#18341](https://github.com/apache/druid/pull/18341) + +##### Improved fault tolerance + +The Kubernetes extension is now more resilient against certain transient faults. Review Comment: ```suggestion The `druid-kubernetes-overlord-extensions` that uses Kubernetes for launching and managing tasks instead of the Middle Managers is now more resilient against certain transient faults. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
