317brian commented on code in PR #18630: URL: https://github.com/apache/druid/pull/18630#discussion_r2487682107
########## docs/release-info/release-notes.md: ########## @@ -57,63 +57,373 @@ For tips about how to write a good release note, see [Release notes](https://git This section contains important information about new and existing features. +### Jetty + +Druid 35 uses Jetty 12. This change may impact your deployment. For more information,see [the upgrade note for Jetty 12](#jetty-12) + +### Java support + +Druid now supports Java 21. Note that some versions of Java 21 encountered issues during test, specifically Java 21.05-21.07. If possible, avoid these versions. + +Additionally, support for Java 11 has been removed. Upgrade to Java 17 or 21. + +[#18424](https://github.com/apache/druid/pull/18424) [#18624](https://github.com/apache/druid/pull/18624) + +### Projections (beta) + +Projections now support static filters. Additionally, there have been general improvements to performance and reliability. + +[#18342](https://github.com/apache/druid/pull/18342) [#18535](https://github.com/apache/druid/pull/18535) + +### Virtual storage (experimental) + +[#18176](https://github.com/apache/druid/pull/18176) + +### New input format + +Druid now supports a `lines` input format. Druid reads each line from an input as UTF-8 text and creates a single column named `line` that contains the entire line as a string. Use this for reading line-oriented data in a simple form for later processing. + +[#18433](https://github.com/apache/druid/pull/18433) + +### Multi-stage query task engine + +The MSQ task engine is now a core capability of Druid rather than an extension. It has been in the default extension load list for several releases. + +Remove `druid-multi-stage-query` from `druid.extensions.loadList` in `common.runtimes.properties` before you upgrade. + +Druid 35.0.0 will ignore the extension if it's in the load list. Future versions of Druid will fail to start since it can't locate the extension. + +[#18394](https://github.com/apache/druid/pull/18394) + +### Improved monitor loading + +You can now specify monitors in `common.runtime.properties`. Previously, you needed to define monitors in the specific `runtime.properties` file for the service a monitor is meant for. + +[#18321](https://github.com/apache/druid/pull/18321) + +### Exact count extension + +A new contributor extension (`druid-exact-count-bitmap`) adds support for exact cardinality counting using Roaring Bitmap over a Long column. + +[#18021](https://github.com/apache/druid/pull/18021) + + +### Improved `indexSpec` + + +[#17762](https://github.com/apache/druid/pull/17762) + ## Functional area and related changes This section contains detailed release notes separated by areas. ### Web console +#### Time zones + +You can now configure whether the web console displays local time or UTC. This setting is stored locally in your browser and doesn't impact other users. + +Note that the URL maintains the query parameters in UTC time, but the Druid console automatically converts the filter to local time. + +[#18455](https://github.com/apache/druid/pull/18455) + #### Other web console improvements +- Added better support for MSQ task engine-based compaction tasks. They now use the stages pane to render the compaction report instead of showing the JSON [#18545](https://github.com/apache/druid/pull/18545) +- Added a version column to the **Services** tab so that you can see what version a service is running. This is helpful during rolling upgrades to verify the state of the cluster and upgrade [#18542](https://github.com/apache/druid/pull/18542) +- Improved the resiliency of the web console when the supervisor history is extensive [#18416](https://github.com/apache/druid/pull/18416) + + ### Ingestion +#### Dimension schemas + +At ingestion time, dimension schemas in `dimensionsSpec` are now strictly validated against allowed types. Previously an invalid type would fall back to string dimension. Now, such values are rejected. Users must specify a type that's one of the allowed types. Omitting type still defaults to string, preserving backward compatibility. + +[#18565](https://github.com/apache/druid/pull/18565) + +#### Other ingestion improvements + +- Added support for session tokens (`sessionToken`) to the S3 input source [#18609](https://github.com/apache/druid/pull/18609) +- Improved task performance. Druid now polls from memory before fetching task information from the metadata database [#18448](https://github.com/apache/druid/pull/18448) +- Improved task execution so that they can successfully complete even if there are problems pushing logs and reports to deep storage [#18210](https://github.com/apache/druid/pull/18210) + #### SQL-based ingestion ##### Other SQL-based ingestion improvements +- Added the ability to configure the maximum frame size. Generally, you don't need to change this unless you have very large rows [#18442](https://github.com/apache/druid/pull/18442) +- Added logging for when segment processing fails [#18378](https://github.com/apache/druid/pull/18378) +- Improved logging to store the cause of invalid field exceptions [#18517](https://github.com/apache/druid/pull/18517) [#18517](https://github.com/apache/druid/pull/18517) + + #### Streaming ingestion ##### Other streaming ingestion improvements +- Added a count parameter to the supervisor history API [#18416](https://github.com/apache/druid/pull/18416) + ### Querying +#### Caching scan query results + +Druid now supports result-level caching for scan queries. + +By default, this behavior is turned off. To enable, override the `druid.*.cache.unCacheable` property. + +[#18568](https://github.com/apache/druid/pull/18568) + +#### New expressions for sketches + +Druid now supports the following expressions: + +- `HLL_SKETCH_ESTIMATE_WITH_ERROR_BOUNDS` +- `THETA_SKETCH_ESTIMATE_WITH_ERROR_BOUNDS` + +These estimates work on a sketch column and have the same behavior as the post aggregators. + +[#18426](https://github.com/apache/druid/pull/18426) + +#### New multi-value SQL functions + +Druid now supports the following multi-value functions: + +- `MV_FILTER_REGEX`: filters a multi-value expression to include only values matching the specified regular expression pattern +- `MV_FILTER_PREFIX`: filters a multi-value expression to include only values that start with the specified prefix + +[#18281](https://github.com/apache/druid/pull/18281) + +#### Exception handling + +You can now write exceptions that Druid encounters as a row and then verify whether the exception is expected. To use this feature, set the `writeExceptionBodyAsResponseRow` query context parameter to `true`. + +[#18571](https://github.com/apache/druid/pull/18571) + #### Other querying improvements +- Added stricter validation for `GREATEST` and `LEAST` [#18562](https://github.com/apache/druid/pull/18562) +- Added a query context option called `realtimeSegmentsOnly` that returns results from realtime segments only when set to `true` [#18329](https://github.com/apache/druid/pull/18329) +- Improved the performance of the following query types through vectorization: + - `CASE_SEARCHED` and `CASE_SIMPLE` queries [#18512](https://github.com/apache/druid/pull/18512) + - `timestamp_ceil` and `timestamp_extract` queries [#18517](https://github.com/apache/druid/pull/18517) + - `IF` expressions where the `THEN` and `ELSE` expressions [#18507](https://github.com/apache/druid/pull/18507) +- Improved projections so that the granularity in queries can match UTC time zones [#18403](https://github.com/apache/druid/pull/18403) +- Improved the deserialization time for intervals by more than 40% [#18477](https://github.com/apache/druid/pull/18477) +- Improved the performance of `AND`/`OR` as well as `NOT`/`IS TRUE`/`IS FALSE` [#18491](https://github.com/apache/druid/pull/18491) [#18488](https://github.com/apache/druid/pull/18488) +- Improved the performance of scan queries [#18441](https://github.com/apache/druid/pull/18441) +- Improved the performance of metadata queries by parallelizing them at the data-node level [#18592](https://github.com/apache/druid/pull/18592) + ### Cluster management +#### Labels for the metadata table + +You can now configure labels for Druid services. Use the `druid.labels` field in the configuration options for the service and provide a JSON object of key-value pairs, such as `druid.labels={"location":"Airtrunk"}` or `druid.labels.location=Airtrunk`. + +One way to view these labels is on the **Services** tab in the web console. + +[#18547](https://github.com/apache/druid/pull/18547) + +#### TLS server configs + +Add a `druid.server.https.forceApplyConfig` config flag to indicate that the `TLSServerConfig` values should be applied irrespective of whether a preexisting `SslContextFactory.Server` binding exists. + +[#18610](https://github.com/apache/druid/pull/18610) + +#### Error message strategy + +You can now configure Druid to log an error message and return an error ID for non-user targeted messages. This can be configured by setting `druid.server.http.errorResponseTransform.strategy` to `persona`. + +[#18487](https://github.com/apache/druid/pull/18487) + +#### Kerberos authentication + +The `druid.auth.authenticator.kerberos.cookieSignatureSecret` config is now mandatory. + +[#18368](https://github.com/apache/druid/pull/18368) + #### Other cluster management improvements +- Added a `version` column `sys.server`. This is useful during rolling upgrades to verify the state of the cluster [#18542](https://github.com/apache/druid/pull/18542) Review Comment: 18547 has its own entry -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
