317brian commented on code in PR #18630: URL: https://github.com/apache/druid/pull/18630#discussion_r2505336479
########## docs/release-info/release-notes.md: ########## @@ -57,63 +57,388 @@ For tips about how to write a good release note, see [Release notes](https://git This section contains important information about new and existing features. +### Jetty + +Druid 35 uses Jetty 12. This change may impact your deployment. For more information,see [the upgrade note for Jetty 12](#jetty-12) + +### Java support + +Druid now supports Java 21. Note that some versions of Java 21 encountered issues during test, specifically Java 21.05-21.07. If possible, avoid these versions. + +Additionally, support for Java 11 has been removed. Upgrade to Java 17 or 21. + +[#18424](https://github.com/apache/druid/pull/18424) [#18624](https://github.com/apache/druid/pull/18624) + +### Projections + +Projections now support static filters. Additionally, there have been general improvements to performance and reliability. + +[#18342](https://github.com/apache/druid/pull/18342) [#18535](https://github.com/apache/druid/pull/18535) + +### Virtual storage (experimental) + +Virtual storage fabric mode for enables Historical servers to serve more segments than what their physical disk can hold. Instead of loading segments when they're published, segments get loaded on demand during queries. Any unneeded segments get removed from disk but get loaded again when processing a query requires that particular segment. + +To enable virtual storage, set `druid.segmentCache.virtualStorage` to `true`. + +This feature is experimental and has several limitations. For more information, see the pull request: + +[#18176](https://github.com/apache/druid/pull/18176) + +### New input format + +Druid now supports a `lines` input format. Druid reads each line from an input as UTF-8 text and creates a single column named `line` that contains the entire line as a string. Use this for reading line-oriented data in a simple form for later processing. + +[#18433](https://github.com/apache/druid/pull/18433) + +### Multi-stage query task engine + +The MSQ task engine is now a core capability of Druid rather than an extension. It has been in the default extension load list for several releases. + +Remove `druid-multi-stage-query` from `druid.extensions.loadList` in `common.runtimes.properties` before you upgrade. + +Druid 35.0.0 will ignore the extension if it's in the load list. Future versions of Druid will fail to start since it can't locate the extension. + +[#18394](https://github.com/apache/druid/pull/18394) + +### Improved monitor loading + +You can now specify monitors in `common.runtime.properties` and each monitor will be loaded only on the applicable server types. Previously, you needed to define monitors in the specific `runtime.properties` file for the service a monitor is meant for. + +[#18321](https://github.com/apache/druid/pull/18321) + +### Exact count extension + +A new contributor extension (`druid-exact-count-bitmap`) adds support for exact cardinality counting using Roaring Bitmap over a Long column. + +[#18021](https://github.com/apache/druid/pull/18021) + + +### Improved `indexSpec` + +Users can now specify a format specification for each JSON column individually, which will override the `IndexSpec` defined in the ingestion job. Additionally, a system-wide default `IndexSpec` can be set using the `druid.indexing.formats.indexSpec` property. + +[#17762](https://github.com/apache/druid/pull/17762) [#18638](https://github.com/apache/druid/pull/18638) + ## Functional area and related changes This section contains detailed release notes separated by areas. ### Web console +#### Time zones + +You can now configure whether the web console displays local time or UTC. This setting is stored locally in your browser and doesn't impact other users. + +Note that the URL maintains the query parameters in UTC time, but the Druid console automatically converts the filter to local time. + +[#18455](https://github.com/apache/druid/pull/18455) + #### Other web console improvements +- Added better support for MSQ task engine-based compaction tasks. They now use the stages pane to render the compaction report instead of showing the JSON [#18545](https://github.com/apache/druid/pull/18545) +- Added a version column to the **Services** tab so that you can see what version a service is running. This is helpful during rolling upgrades to verify the state of the cluster and upgrade [#18542](https://github.com/apache/druid/pull/18542) +- Improved the resiliency of the web console when the supervisor history is extensive [#18416](https://github.com/apache/druid/pull/18416) + + ### Ingestion +#### Dimension schemas + +At ingestion time, dimension schemas in `dimensionsSpec` are now strictly validated against allowed types. Previously an invalid type would fall back to string dimension. Now, such values are rejected. Users must specify a type that's one of the allowed types. Omitting type still defaults to string, preserving backward compatibility. + +[#18565](https://github.com/apache/druid/pull/18565) + +#### Other ingestion improvements + +- Added support for session tokens (`sessionToken`) to the S3 input source [#18609](https://github.com/apache/druid/pull/18609) +- Improved performance of task APIs served by the Overlord. Druid now reads the in-memory state of the Overlord before fetching task information from the metadata store [#18448](https://github.com/apache/druid/pull/18448) +- Improved task execution so that they can successfully complete even if there are problems pushing logs and reports to deep storage [#18210](https://github.com/apache/druid/pull/18210) + #### SQL-based ingestion ##### Other SQL-based ingestion improvements +- Added the ability to configure the maximum frame size. Generally, you don't need to change this unless you have very large rows [#18442](https://github.com/apache/druid/pull/18442) +- Added logging for when segment processing fails [#18378](https://github.com/apache/druid/pull/18378) +- Improved logging to store the cause of invalid field exceptions [#18517](https://github.com/apache/druid/pull/18517) [#18517](https://github.com/apache/druid/pull/18517) + + #### Streaming ingestion ##### Other streaming ingestion improvements +- Added a count parameter to the supervisor history API [#18416](https://github.com/apache/druid/pull/18416) + ### Querying +#### Caching scan query results + +Druid now supports result-level caching for scan queries. + +By default, this behavior is turned off. To enable, override the `druid.*.cache.unCacheable` property. + +[#18568](https://github.com/apache/druid/pull/18568) + +#### New expressions for sketches + +Druid now supports the following expressions: + +- `HLL_SKETCH_ESTIMATE_WITH_ERROR_BOUNDS` +- `THETA_SKETCH_ESTIMATE_WITH_ERROR_BOUNDS` + +These estimates work on a sketch column and have the same behavior as the post aggregators. + +[#18426](https://github.com/apache/druid/pull/18426) + +#### New multi-value SQL functions + +Druid now supports the following multi-value functions: + +- `MV_FILTER_REGEX`: filters a multi-value expression to include only values matching the specified regular expression pattern +- `MV_FILTER_PREFIX`: filters a multi-value expression to include only values that start with the specified prefix + +[#18281](https://github.com/apache/druid/pull/18281) + +#### Exception handling + +You can now write exceptions that Druid encounters as a row and then verify whether the exception is expected. To use this feature, set the `writeExceptionBodyAsResponseRow` query context parameter to `true`. + +[#18571](https://github.com/apache/druid/pull/18571) + #### Other querying improvements +- Added stricter validation for `GREATEST` and `LEAST` [#18562](https://github.com/apache/druid/pull/18562) +- Added a query context option called `realtimeSegmentsOnly` that returns results from realtime segments only when set to `true` [#18329](https://github.com/apache/druid/pull/18329) +- Improved the performance of the following query types through vectorization: + - `CASE_SEARCHED` and `CASE_SIMPLE` queries [#18512](https://github.com/apache/druid/pull/18512) + - `timestamp_ceil` and `timestamp_extract` queries [#18517](https://github.com/apache/druid/pull/18517) + - `IF` expressions where the `THEN` and `ELSE` expressions [#18507](https://github.com/apache/druid/pull/18507) +- Improved projections so that the granularity in queries can match UTC time zones [#18403](https://github.com/apache/druid/pull/18403) Review Comment: ```suggestion ``` ########## docs/release-info/release-notes.md: ########## @@ -57,63 +57,390 @@ For tips about how to write a good release note, see [Release notes](https://git This section contains important information about new and existing features. +### Jetty + +Druid 35 uses Jetty 12. This change may impact your deployment. For more information,see [the upgrade note for Jetty 12](#jetty-12) + +### Java support + +Druid now supports Java 21. Note that some versions of Java 21 encountered issues during test, specifically Java 21.05-21.07. If possible, avoid these versions. + +Additionally, support for Java 11 has been removed. Upgrade to Java 17 or 21. + +[#18424](https://github.com/apache/druid/pull/18424) [#18624](https://github.com/apache/druid/pull/18624) + +### Projections + +Projections now support static filters. Additionally, there have been general improvements to performance and reliability. + +[#18342](https://github.com/apache/druid/pull/18342) [#18535](https://github.com/apache/druid/pull/18535) Review Comment: ```suggestion Projections have been improved: - Static filters are now supported - The granularity in queries can match UTC time zones Additionally, there have been general improvements to performance and reliability. [#18342](https://github.com/apache/druid/pull/18342) [#18535](https://github.com/apache/druid/pull/18535) [#18403](https://github.com/apache/druid/pull/18403) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
