clintropolis commented on code in PR #19945: URL: https://github.com/apache/druid/pull/19945#discussion_r4152518288
########## docs/release-info/release-notes.md: ########## @@ -57,6 +57,80 @@ For tips about how to write a good release note, see [Release notes](https://git This section contains important information about new and existing features. +#### Java + +Druid now supports Java 25. While Java 21 is still supported, we recommend you upgrade to Java 25. + +Support for Java 17 has been dropped. + +[#19304](https://github.com/apache/druid/pull/19304) [#19336](https://github.com/apache/druid/pull/19336) + +#### Historical tier aliases + +You can map a virtual tier name to a number of real Historical tiers with the `historicalTierAliases` Coordinator dynamic configuration. This creates a single identifier for a group of Historical tiers. When a load/drop rule references the alias, the Coordinator replaces it with the actual tiers. For example, if you map the Historical tiers `hot_1` and `hot_2` to the alias `hot`, the rule `{"hot": 2}` loads 2 replicas of each onto `hot_1` and `hot_2`. + +[#19204](https://github.com/apache/druid/pull/19204) [#19667](https://github.com/apache/druid/pull/19667) + +#### New load rule types + +Adds a new family of retention rules, `loadPartialByPeriod`, `loadPartialByInterval`, `loadPartialForever`, laying the groundwork for partial loading of version 10 segment projections on Historicals. + +[#19374](https://github.com/apache/druid/pull/19374) + +#### Realtime segments mode query context + +The `realtimeSegmentsOnly` query context parameter has been deprecated and replaced with `realtimeSegmentsMode`. + +You can set `realtimeSegmentsMode` to one of the following: + +- `include` (default): query all segments, including realtime segments +- `exclude`: skip realtime segments for the query +- `exclusive`: query only realtime segments + +This is useful when performing things like blue/green deployments and you only want to query new Historical replica autoscaling groups and not touch any "live" nodes (neither realtime nor historical). + +[#19486](https://github.com/apache/druid/pull/19486) + +#### Faster segment metadata search + +You can set `druid.segment.timeline.fastIntervalSearch` to `true` so that Druid uses an index based on interval trees to store that metadata in memory for faster identification and retrieval. + +This feature is off by default. + +[#19138](https://github.com/apache/druid/issues/19138) [#19850](https://github.com/apache/druid/issues/19850) + +#### Fetching files in the background + +The MSQ task engine now supports workers fetching input files from cloud storage asynchronously. This overlaps downloading with processing, which generally improves throughput when reading larger numbers of files. Otherwise, workers stream each file directly from cloud storage while processing it. + +This feature is on by default and is controlled by the `backgroundFetchExternalFiles` setting. + +[#19539](https://github.com/apache/druid/pull/19539) + +#### Segment prefetching for Dart + +Dart now supports the runtime property `druid.msq.dart.worker.segmentLoadAheadCount`, which controls the number of segments that Dart prefetches. If set greater than 0 for a worker, this setting becomes the default `segmentLoadAheadCount` value for the worker. If a query includes the `segmentLoadAheadCount` query context parameter, the query context takes precedence. Review Comment: this is only really applicable to virtual storage mode historicals i think, though i suppose the docs for it don't mention that either ########## docs/release-info/release-notes.md: ########## @@ -57,6 +57,80 @@ For tips about how to write a good release note, see [Release notes](https://git This section contains important information about new and existing features. +#### Java + +Druid now supports Java 25. While Java 21 is still supported, we recommend you upgrade to Java 25. + +Support for Java 17 has been dropped. + +[#19304](https://github.com/apache/druid/pull/19304) [#19336](https://github.com/apache/druid/pull/19336) + +#### Historical tier aliases + +You can map a virtual tier name to a number of real Historical tiers with the `historicalTierAliases` Coordinator dynamic configuration. This creates a single identifier for a group of Historical tiers. When a load/drop rule references the alias, the Coordinator replaces it with the actual tiers. For example, if you map the Historical tiers `hot_1` and `hot_2` to the alias `hot`, the rule `{"hot": 2}` loads 2 replicas of each onto `hot_1` and `hot_2`. + +[#19204](https://github.com/apache/druid/pull/19204) [#19667](https://github.com/apache/druid/pull/19667) + +#### New load rule types + +Adds a new family of retention rules, `loadPartialByPeriod`, `loadPartialByInterval`, `loadPartialForever`, laying the groundwork for partial loading of version 10 segment projections on Historicals. Review Comment: partial load rules allow virtual storage mode historicals (also not documented i think, druid.segmentCache.virtualStorage and druid.segmentCache.virtualStoragePartialDownloadsEnabled both true) to eagerly load "parts" of v10 segments that are stored in deep storage unzipped (druid.storage.zip=false but only s3 supports that). There are matchers that must be set on the partial rule that can select the parts of the v10 segments (v10 segments are internally organized into 'bundles' which are sort of logical containers that hold all of the columns of a projection, where projection could be the base table (or individual cluster groups for clustered segments which when concatenated together form the base table projection), or individual aggregate projections). I'm hesitant to call this stuff out since like the other partial load stuff it its going to be pretty tough to use without docs. I wonder if we should consolidate all of this stuff in an 'experimental virtual storage mode improvements' section (partial load on demand and the partial load rules built on top of it) or just remove it as suggested in other thread -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
