FrankChen021 commented on code in PR #20866: URL: https://github.com/apache/druid/pull/20866#discussion_r4502072723
########## docs/release-info/release-notes.md: ########## @@ -57,50 +57,300 @@ For tips about how to write a good release note, see [Release notes](https://git This section contains important information about new and existing features. -## Functional area and related changes +### DDL statements for the catalog (experimental) -This section contains detailed release notes separated by areas. +Druid now supports CREATE and ALTER statements to manage metadata in the catalog. Use them to declare a table's columns, time partitioning, clustering, aggregate projections, and physical layout, instead of setting these in each ingestion spec. + +This feature is off by default. To enable it, set `druid.sql.planner.enableCatalogDdl` to `true`. + +For more information, see [Catalog](../development/extensions-core/catalog.md). + +[#19830](https://github.com/apache/druid/pull/19830) [#20377](https://github.com/apache/druid/pull/20377) [#20452](https://github.com/apache/druid/pull/20452) [#19770](https://github.com/apache/druid/pull/19770) [#19711](https://github.com/apache/druid/pull/19711) + +### Kafka header-based filtering + +Kafka supervisors can now filter records by their headers. This reduces ingestion CPU and memory use when you only need some of a topic's records. + +To use the filter, add `headerBasedFilterConfig` to the supervisor's `ioConfig`. Use an `in` filter where `dimension` is the header name and `values` lists the header values to ingest. Records whose header value isn't in the list are dropped, and offsets still advance. + +Filtering is permissive: records are ingested if they don't have the header, if the header value is null, empty, or can't be decoded, or if evaluation fails. + +The new `ingest/events/filtered` metric reports how many records header filtering dropped. + +[#18525](https://github.com/apache/druid/pull/18525) + +### Druid JDBC driver + +Druid now provides its own JDBC driver that uses the `/druid/v2/sql/` endpoint. We recommend that you switch from the Avatica driver to this driver. The new driver provides the following benefits: driver-side connection management that doesn't require sticky load balancing, streaming results, support for SET statements, and support for non-native SQL engines. + +[#19951](https://github.com/apache/druid/pull/19951) + +### Netty 4.2 -### Web console +Druid now uses Netty 4.2 for its HTTP client. Previously, Druid used Netty 3. This is a substantive change in memory characteristics: -#### Other web console improvements +- Direct memory: The HTTP client now consumes some direct memory. Operators running close to their `-XX:MaxDirectMemorySize` budget may need to increase it. +- Pooled buffers: Buffers are now pooled, so resident memory may appear higher than under Netty 3, even when idle. This is normal for a pooled allocator. + +There are new configs that control the behavior of the HTTP client: `druid.global.http.connectTimeout`, `druid.global.http.allocator`, `druid.broker.http.connectTimeout`, and `druid.broker.http.allocator`. You can also set some Netty configs directly. + +[#19567](https://github.com/apache/druid/pull/19567) + +### Quality of service filtering for Coordinator + +You can now enable QoS filtering for the Coordinator to prevent a burst of slow API requests from exhausting the thread pool and blocking health checks. + +Use `druid.coordinator.server.maxConcurrentRequests` to configure this feature (default: max(1, max(serverHttpNumThreads - 4, serverHttpNumThreads * 0.8))). Set this property to `0` to turn it off. + +Leader-status endpoints are excluded from the cap so load balancers/monitors stay unaffected. + +[#19271](https://github.com/apache/druid/pull/19271) + +## Functional area and related changes + +This section contains detailed release notes separated by areas. ### Ingestion +- Added support for RE2/J regex for the `regex` input format [#19514](https://github.com/apache/druid/pull/19514) +- Changed the S3 client to use the AWS-recommended standard retry policy. You can customize the retry strategy using `druid.s3.retryMode` and `druid.s3.maxAttempts` [#19892](https://github.com/apache/druid/pull/19892) + #### SQL-based ingestion ##### Other SQL-based ingestion improvements #### Streaming ingestion +##### Ingesting from selected partitions + +Kafka supervisors now accept `ioConfig.partitionIds`, which lets you ingest only selected partitions of a topic. For example, you can use this for independent diagnostic ingestion. + +[#20474](https://github.com/apache/druid/pull/20474) + +##### Simulate cost-based autoscaler + +You can now simulate the cost-based autoscaler for Kafka to help determine the optimal task count for different amounts of lag. + +[#19687](https://github.com/apache/druid/pull/19687) + +##### Cost-based autoscaler critical lag + +You can now set a critical lag threshold for the cost-based autoscaler. When lag reaches 75% of the threshold, the autoscaler skips the lag cost factor. When lag reaches 100%, the autoscaler scales up to the maximum task count. + +[#19655](https://github.com/apache/druid/pull/19655) + ##### Other streaming ingestion improvements ### Querying +#### Query planning time + +You can now limit how long the Broker spends planning a SQL query with the new `druid.sql.planner.maxPlanningTimeMs` runtime property. You can also set it per query using the `maxPlanningTimeMs` query context parameter. When planning exceeds the configured budget, the query fails with an HTTP 504 Query timeout error instead of tying up a Broker thread. The default value is `0`, which disables the timeout. + +[#20314](https://github.com/apache/druid/pull/20314) + #### Other querying improvements +- You can now use Dart to query metadata tables such as those in the `sys` schema [#19952](https://github.com/apache/druid/pull/19952) +- `CAST(x AS LONG)` is now accepted as an alias for `CAST(x AS BIGINT)` in Druid SQL [#20402](https://github.com/apache/druid/pull/20402) +- Added `datasource` filter pushdown to `sys.segments` table [#19718](https://github.com/apache/druid/pull/19718) +- Added framing support to the FIRST_VALUE and LAST_VALUE window functions [#19301](https://github.com/apache/druid/pull/19301) +- Improved the performance of timeseries, timeBoundary, and scan queries that run in descending order [#19947](https://github.com/apache/druid/pull/19947) +- Improved the performance of COALESCE [#18400](https://github.com/apache/druid/pull/18400) +- Improved S3 API usage efficiency for MSQ tasks using durable storage and MSQ export [#19893](https://github.com/apache/druid/pull/19893) +- Improved validation for catalog types and parsing for native type strings [#19769](https://github.com/apache/druid/pull/19769) +- Upgraded to Calcite 1.42.0 [#19976](https://github.com/apache/druid/pull/19976) [#19370](https://github.com/apache/druid/pull/19370) + ### Cluster management +#### Kubernetes node labels + +The `druid-kubernetes-extensions` extension now support a `k8sNodeLabel` dynamic config provider, which resolves configuration values from the labels of the Kubernetes node a Druid process runs on. + +[#20404](https://github.com/apache/druid/pull/20404) + +#### Peon pod template + +You can now retrieve the Peon pod template names currently configured on the Overlord using the new `GET` `/druid/indexer/v1/k8s/taskrunner/podTemplates` endpoint. It returns each configured template's name and full pod spec. The endpoint requires CONFIG read permission and returns 404 when the Overlord is not using the custom template pod adapter. Review Comment: [P2] Endpoint response is documented as a full pod spec **Finding:** The endpoint serializes getPodTemplateNames(), which returns a List<String>, and its test asserts that the response entity is the names list. It does not return the configured pod specs, so consumers following this note will expect fields that are absent. **Suggestion:** Describe the response as the configured template names only, or change the endpoint if full pod specs are intended. ########## docs/release-info/release-notes.md: ########## @@ -57,50 +57,300 @@ For tips about how to write a good release note, see [Release notes](https://git This section contains important information about new and existing features. -## Functional area and related changes +### DDL statements for the catalog (experimental) -This section contains detailed release notes separated by areas. +Druid now supports CREATE and ALTER statements to manage metadata in the catalog. Use them to declare a table's columns, time partitioning, clustering, aggregate projections, and physical layout, instead of setting these in each ingestion spec. + +This feature is off by default. To enable it, set `druid.sql.planner.enableCatalogDdl` to `true`. + +For more information, see [Catalog](../development/extensions-core/catalog.md). + +[#19830](https://github.com/apache/druid/pull/19830) [#20377](https://github.com/apache/druid/pull/20377) [#20452](https://github.com/apache/druid/pull/20452) [#19770](https://github.com/apache/druid/pull/19770) [#19711](https://github.com/apache/druid/pull/19711) + +### Kafka header-based filtering + +Kafka supervisors can now filter records by their headers. This reduces ingestion CPU and memory use when you only need some of a topic's records. + +To use the filter, add `headerBasedFilterConfig` to the supervisor's `ioConfig`. Use an `in` filter where `dimension` is the header name and `values` lists the header values to ingest. Records whose header value isn't in the list are dropped, and offsets still advance. + +Filtering is permissive: records are ingested if they don't have the header, if the header value is null, empty, or can't be decoded, or if evaluation fails. + +The new `ingest/events/filtered` metric reports how many records header filtering dropped. + +[#18525](https://github.com/apache/druid/pull/18525) + +### Druid JDBC driver + +Druid now provides its own JDBC driver that uses the `/druid/v2/sql/` endpoint. We recommend that you switch from the Avatica driver to this driver. The new driver provides the following benefits: driver-side connection management that doesn't require sticky load balancing, streaming results, support for SET statements, and support for non-native SQL engines. + +[#19951](https://github.com/apache/druid/pull/19951) + +### Netty 4.2 -### Web console +Druid now uses Netty 4.2 for its HTTP client. Previously, Druid used Netty 3. This is a substantive change in memory characteristics: -#### Other web console improvements +- Direct memory: The HTTP client now consumes some direct memory. Operators running close to their `-XX:MaxDirectMemorySize` budget may need to increase it. +- Pooled buffers: Buffers are now pooled, so resident memory may appear higher than under Netty 3, even when idle. This is normal for a pooled allocator. + +There are new configs that control the behavior of the HTTP client: `druid.global.http.connectTimeout`, `druid.global.http.allocator`, `druid.broker.http.connectTimeout`, and `druid.broker.http.allocator`. You can also set some Netty configs directly. + +[#19567](https://github.com/apache/druid/pull/19567) + +### Quality of service filtering for Coordinator + +You can now enable QoS filtering for the Coordinator to prevent a burst of slow API requests from exhausting the thread pool and blocking health checks. + +Use `druid.coordinator.server.maxConcurrentRequests` to configure this feature (default: max(1, max(serverHttpNumThreads - 4, serverHttpNumThreads * 0.8))). Set this property to `0` to turn it off. + +Leader-status endpoints are excluded from the cap so load balancers/monitors stay unaffected. + +[#19271](https://github.com/apache/druid/pull/19271) + +## Functional area and related changes + +This section contains detailed release notes separated by areas. ### Ingestion +- Added support for RE2/J regex for the `regex` input format [#19514](https://github.com/apache/druid/pull/19514) +- Changed the S3 client to use the AWS-recommended standard retry policy. You can customize the retry strategy using `druid.s3.retryMode` and `druid.s3.maxAttempts` [#19892](https://github.com/apache/druid/pull/19892) + #### SQL-based ingestion ##### Other SQL-based ingestion improvements #### Streaming ingestion +##### Ingesting from selected partitions + +Kafka supervisors now accept `ioConfig.partitionIds`, which lets you ingest only selected partitions of a topic. For example, you can use this for independent diagnostic ingestion. + +[#20474](https://github.com/apache/druid/pull/20474) + +##### Simulate cost-based autoscaler + +You can now simulate the cost-based autoscaler for Kafka to help determine the optimal task count for different amounts of lag. + +[#19687](https://github.com/apache/druid/pull/19687) + +##### Cost-based autoscaler critical lag + +You can now set a critical lag threshold for the cost-based autoscaler. When lag reaches 75% of the threshold, the autoscaler skips the lag cost factor. When lag reaches 100%, the autoscaler scales up to the maximum task count. + +[#19655](https://github.com/apache/druid/pull/19655) + ##### Other streaming ingestion improvements ### Querying +#### Query planning time + +You can now limit how long the Broker spends planning a SQL query with the new `druid.sql.planner.maxPlanningTimeMs` runtime property. You can also set it per query using the `maxPlanningTimeMs` query context parameter. When planning exceeds the configured budget, the query fails with an HTTP 504 Query timeout error instead of tying up a Broker thread. The default value is `0`, which disables the timeout. + +[#20314](https://github.com/apache/druid/pull/20314) + #### Other querying improvements +- You can now use Dart to query metadata tables such as those in the `sys` schema [#19952](https://github.com/apache/druid/pull/19952) +- `CAST(x AS LONG)` is now accepted as an alias for `CAST(x AS BIGINT)` in Druid SQL [#20402](https://github.com/apache/druid/pull/20402) +- Added `datasource` filter pushdown to `sys.segments` table [#19718](https://github.com/apache/druid/pull/19718) +- Added framing support to the FIRST_VALUE and LAST_VALUE window functions [#19301](https://github.com/apache/druid/pull/19301) +- Improved the performance of timeseries, timeBoundary, and scan queries that run in descending order [#19947](https://github.com/apache/druid/pull/19947) +- Improved the performance of COALESCE [#18400](https://github.com/apache/druid/pull/18400) +- Improved S3 API usage efficiency for MSQ tasks using durable storage and MSQ export [#19893](https://github.com/apache/druid/pull/19893) +- Improved validation for catalog types and parsing for native type strings [#19769](https://github.com/apache/druid/pull/19769) +- Upgraded to Calcite 1.42.0 [#19976](https://github.com/apache/druid/pull/19976) [#19370](https://github.com/apache/druid/pull/19370) + ### Cluster management +#### Kubernetes node labels + +The `druid-kubernetes-extensions` extension now support a `k8sNodeLabel` dynamic config provider, which resolves configuration values from the labels of the Kubernetes node a Druid process runs on. + +[#20404](https://github.com/apache/druid/pull/20404) + +#### Peon pod template + +You can now retrieve the Peon pod template names currently configured on the Overlord using the new `GET` `/druid/indexer/v1/k8s/taskrunner/podTemplates` endpoint. It returns each configured template's name and full pod spec. The endpoint requires CONFIG read permission and returns 404 when the Overlord is not using the custom template pod adapter. + +[#19712](https://github.com/apache/druid/pull/19712) + +#### Environment variables + +The `environment` dynamic config provider now omits variables that are not set in the environment instead of producing a null value that fails consumers such as the Kafka supervisor with a `NullPointerException`. A skipped variable is logged at info level. + +[#20381](https://github.com/apache/druid/pull/20381) + +#### Connection pools + +You can now configure how the Druid connection pool tracks connection demand using `druid.global.http.poolImplementation`. + +The default is `retaining`. With `retaining`, the pool keeps every connection it opens and replaces each stale or broken connection with a new one, so the pool stays at its high-water mark. + +[#20542](https://github.com/apache/druid/pull/20542) [#20273](https://github.com/apache/druid/pull/20273) + +#### HTTP response headers + +Druid can now optionally return `X-Druid-Server`, `X-Druid-Service`, and `X-Druid-Version` on HTTP responses, allowing operators to identify the process and version that served a direct or Router-proxied request. These headers expose cluster topology and version information, so only enable them if all clients are permitted to receive it. To enable them, set `druid.server.http.enableResponseIdentityHeaders` to `true`. + +[#20268](https://github.com/apache/druid/pull/20268) + +#### Improved LDAP support + +Added support for LDAP group-based authorization on servers that do not return the `memberOf` attribute (such as OpenLDAP). Configure `groupBaseDn` and `groupSearch` on the LDAP credentials validator to enable a reverse group lookup that resolves group memberships automatically. + +[#19432](https://github.com/apache/druid/pull/19432) + +#### Table visibility + +Users who query tables they don't have READ permission for now see a "not found" error instead of a "forbidden" error. + +To restore the previous behavior, set `druid.sql.planner.authorizeTableVisibility` to `false`. + +Additionally, when set to `true`, INSERT and REPLACE require both READ and WRITE access to the target table. If this property is set to `false`, they only require WRITE permission.Regardless of the value of this property, tables only appear in INFORMATION_SCHEMA if the user has READ access to them. + +This is a breaking change that affects the permissions you need. For more information, see [permissions changes for SQL-based ingestion](#permissions-changes-for-sql-based-ingestion). + +[#20075](https://github.com/apache/druid/pull/20075) + +#### JAVA_HOME + +Middle Managers now honor JAVA_HOME when using the `bin/java` script, which should be present in the working directory. Review Comment: [P2] JAVA_HOME launcher script name is incorrect **Finding:** The note says the launcher uses bin/java, but ForkingTaskRunner checks for <workingDir>/bin/run-java and otherwise falls back to plain java. Operators following this instruction will not activate the JAVA_HOME-aware bundled script. **Suggestion:** Change the script name in the note to bin/run-java and keep the prerequisite aligned with the launcher’s working-directory check. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
