kfaraz commented on code in PR #19287: URL: https://github.com/apache/druid/pull/19287#discussion_r3747009503
########## docs/release-info/release-notes.md: ########## @@ -57,50 +57,627 @@ For tips about how to write a good release note, see [Release notes](https://git This section contains important information about new and existing features. +### Hadoop-based ingestion + +Support for Hadoop-based ingestion has been removed. The feature was deprecated in Druid 34. + +Use one of Druid's other supported ingestion methods, such as SQL-based ingestion or MiddleManager-less ingestion using Kubernetes. + +[#19109](https://github.com/apache/druid/pull/19109) + +### Query blocklist + +You can now use the Broker API (`/druid/coordinator/v1/config/broker`) to create a query blocklist to dynamically block queries by datasource, query type, or query context. The blocklist takes effect without a restarting Druid. Block rules use `AND` logic, which means all criteria must match. + +The following example blocks all groupBy queries on the `wikipedia` datasource with a query context parameter of `priority` equal to `0`: + +``` +POST /druid/coordinator/v1/config/broker + { + "queryBlocklist": [ + { + "ruleName": "block-wikipedia-groupbys", + "dataSources": ["wikipedia"], + "queryTypes": ["groupBy"], + "contextMatches": {"priority": "0"} + } + ] + } +``` + +[#19011](https://github.com/apache/druid/pull/19011) + +### Minor compaction for Overlord-based compaction (experimental) + +You can now configure minor compaction to compact only newly ingested segments while upgrading existing compacted segments. When Druid upgrades segments, it updates the metadata instead of using resources to compact it again. You can use the native compaction engine or the MSQ task engine. + +Use the `mostFragmentedFirst` compaction policy and set either a percentage of rows-based or byte-based threshold for minor compaction. + +[#19059](https://github.com/apache/druid/pull/19059) [#19205](https://github.com/apache/druid/pull/19205) [#19016](https://github.com/apache/druid/pull/19016) + +### Cascading reindexing (experimental) + +Using cascading reindexing, you can now define age-based rules to automatically apply different compaction configurations based on the age of your data. While standard auto-compaction applies a single flat configuration across an entire datasource, cascading reindexing lets you tailor your compaction settings to the characteristics of your data. + +For example, you can keep recent data in hourly segments while automatically rolling up to daily segments after 90 days to reduce segment count. You can also layer on age-based row deletion (such as dropping bot traffic from older data), change compression settings, or shift to rollup with coarser query granularity as data ages. Rules are defined inline in the supervisor spec. + +You must use compaction supervisors with the MSQ task engine to use cascading reindexing. + +[#18939](https://github.com/apache/druid/pull/18939) [#19213](https://github.com/apache/druid/pull/19213) [#19106](https://github.com/apache/druid/pull/19106) [#19078](https://github.com/apache/druid/pull/19078) + +### Multi-supervisor ingestion + +Multi-supervisor ingestion is now generally available. You can run multiple stream supervisors that ingest into the same datasource. + +[#18983](https://github.com/apache/druid/pull/18983) + +### Read-only authorizer + +Added a `ReadOnly` authorizer to Druid. This is the first global authorizer for Druid. The authorizer enforces a global restriction on all non-READ operations, denying them regardless of individual user permissions. You can use this capability to ensure all users of a specific authorizer are limited to READ access. + +There is a known limitation where some endpoints currently require WRITE access despite being READ-only, such as `GET /druid/indexer/v1/supervisor`. These operations will fail. + +[#19243](https://github.com/apache/druid/pull/19243) + +### Thrift input format + +As part of the Thrift contributor extension, Druid now supports Thrift-encoded data for Kafka and Kinesis streaming ingestion using `InputFormat`. Previously, Druid supported this through parsers, which have been removed in Druid 37. + +[#19111](https://github.com/apache/druid/pull/19111) + +To use this feature, you must add `druid-thrift-extensions` to your extension load list. + +### Incremental cache Review Comment: @317brian , this item should also be called out in the upgrade-notes for Druid 37 release since it affects the Overlord memory usage. We would also need to provide a rough estimate for the increase in Overlord memory. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
