techdocsmith commented on code in PR #20618: URL: https://github.com/apache/druid/pull/20618#discussion_r4299811718
########## docs/release-info/release-notes.md: ########## @@ -67,40 +108,226 @@ This section contains detailed release notes separated by areas. ### Ingestion +- Added support for RE2/J regex for the `regex` input format [#19514](https://github.com/apache/druid/pull/19514) +- Changed the S3 client to use the AWS-recommended standard retry policy. You can customize the retry strategy using `druid.s3.retryMode` and `druid.s3.maxAttempts` [#19892](https://github.com/apache/druid/pull/19892) + #### SQL-based ingestion ##### Other SQL-based ingestion improvements #### Streaming ingestion +##### Ingesting from selected partitions + +Kafka supervisors now accept `ioConfig.partitionIds`, which lets you ingest only selected partitions of a topic. For example, you can use this for independent diagnostic ingestion. + +[#20474](https://github.com/apache/druid/pull/20474) + +##### Simulate cost-based autoscaler + +You can now simulate the cost-based autoscaler for Kafka to help determine the optimal task count for different amounts of lag. + +[#19687](https://github.com/apache/druid/pull/19687) + +##### Cost-based autoscaler critical lag + +You can now set a critical lag threshold for the cost-based autoscaler. When lag reaches 75% of the threshold, the autoscaler skips the lag cost factor. When lag reaches 100%, the autoscaler scales up to the maximum task count. + +[#19655](https://github.com/apache/druid/pull/19655) + ##### Other streaming ingestion improvements ### Querying +#### Query planning time + +You can now limit how long the Broker spends planning a SQL query with the new `druid.sql.planner.maxPlanningTimeMs` runtime property. You can also set it per query using the `maxPlanningTimeMs` query context parameter. When planning exceeds the configured budget, the query fails with an HTTP 504 Query timeout error instead of tying up a Broker thread. The default value is `0`, which disables the timeout. + +[#20314](https://github.com/apache/druid/pull/20314) + #### Other querying improvements +- You can now use Dart to query metadata tables such as those in the `sys` schema [#19952](https://github.com/apache/druid/pull/19952) +- `CAST(x AS LONG)` is now accepted as an alias for `CAST(x AS BIGINT)` in Druid SQL [#20402](https://github.com/apache/druid/pull/20402) +- Added framing support to the FIRST_VALUE and LAST_VALUE window functions [#19301](https://github.com/apache/druid/pull/19301) +- Improved the performance of timeseries, timeBoundary, and scan queries that run in descending order [#19947](https://github.com/apache/druid/pull/19947) +- Improved the performance of COALESCE [#18400](https://github.com/apache/druid/pull/18400) +- Improved S3 API usage efficiency for MSQ tasks using durable storage and MSQ export [#19893](https://github.com/apache/druid/pull/19893) +- Improved validation for catalog types and parsing for native type strings [#19769](https://github.com/apache/druid/pull/19769) +- Upgraded to Calcite 1.42.0 [#19976](https://github.com/apache/druid/pull/19976) [#19370](https://github.com/apache/druid/pull/19370) + ### Cluster management +#### Kubernetes + +The `druid-kubernetes-extensions` extension now support a `k8sNodeLabel` dynamic config provider, which resolves configuration values from the labels of the Kubernetes node a Druid process runs on. + +[#20404](https://github.com/apache/druid/pull/20404) + +#### Environment variables + +The `environment` dynamic config provider now omits variables that are not set in the environment instead of producing a null value that fails consumers such as the Kafka supervisor with a `NullPointerException`. A skipped variable is logged at info level. + +[#20381](https://github.com/apache/druid/pull/20381) + +#### Connection pools + +You can now configure how the Druid connection pool tracks connection demand using `druid.global.http.poolImplementation`. + +The default is `retaining`. With `retaining`, the pool keeps every connection it opens and replaces each stale or broken connection with a new one, so the pool stays at its high-water mark. + +[#20542](https://github.com/apache/druid/pull/20542) [#20273](https://github.com/apache/druid/pull/20273) + +#### HTTP response headers + +Druid can now optionally return `X-Druid-Server`, `X-Druid-Service`, and `X-Druid-Version` on HTTP responses, allowing operators to identify the process and version that served a direct or Router-proxied request. These headers expose cluster topology and version information, so only enable them if all clients are permitted to receive it. To enable them, set `druid.server.http.enableResponseIdentityHeaders` to `true`. + +[#20268](https://github.com/apache/druid/pull/20268) + +#### Improved LDAP support + +Added support for LDAP group-based authorization on servers that do not return the `memberOf` attribute (such as OpenLDAP). Configure `groupBaseDn` and `groupSearch` on the LDAP credentials validator to enable a reverse group lookup that resolves group memberships automatically. + +[#19432](https://github.com/apache/druid/pull/19432) + +#### Table visibility + +Users who query tables they don't have READ permission for now see a "not found" error instead of a "forbidden" error. + +To restore the previous behavior, set `druid.sql.planner.authorizeTableVisibility` to `false`. Review Comment: Link to upgrade note/breaking change? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
