morningman opened a new issue, #65399: URL: https://github.com/apache/doris/issues/65399
# 4.0.7 Release Notes 4.0.6 Release Notes: #64065 Apache Doris 4.0.7 is a maintenance release in the 4.0 series. It focuses on query correctness, load stability, compute-storage decoupled deployments, File Cache reliability, and external data source compatibility. All 4.0.x users are advised to upgrade. Highlights of this release include: - Cloud enhancements: add table-level event-driven cache warmup, enable Packed File and empty Rowset optimizations by default, and fix several Recycler, Schema Change, Compaction, and File Cache issues. - Object storage improvements: support S3 Storage Vault credential providers without a Role ARN, and allow deterministic S3 paths to be accessed without `ListBucket` permission. - Better observability: add FE connection-limit metrics, per-job Routine Load metrics, File Cache queue metrics, and Compute Group information for MTMV refresh tasks. - Many correctness and stability fixes covering TopN, joins, materialized view rewriting, Stream Load, Routine Load, external catalogs, File Cache crashes, and memory usage. > Before upgrading, read the Behavior Changes section below. # Behavior Changes - `INSERT OVERWRITE` now permanently removes replaced partitions instead of placing them in the Recycle Bin. This reduces Recycle Bin pressure, but these partitions can no longer be recovered from it (#62510). - Cloud mode now enables `enable_packed_file` and `skip_writing_empty_rowset_metadata` by default (#63475). - The BE `/api/file_cache?op=clear` API no longer supports synchronous clearing. Requests with `sync=true` now run asynchronously and return a warning (#64321). - Object storage clients no longer retry S3 `SlowDown` or Azure HTTP 429 responses inside the SDK retry policy, avoiding additional delays before higher-level handling takes effect (#63776). - `COM_RESET_CONNECTION` now resets connection-scoped state consistently with MySQL (#63884). - In Cloud mode, `SHOW PARTITIONS` now reports `StorageMedium` as `OBJECT_STORAGE` and `ReplicaAllocation` as `NULL` (#60871). # New Features & Improvements ## Compute-Storage Decoupling - Add table-level filtering through `ON TABLES` for event-driven cache warmup (#63832). - Add a dynamic Recycler instance filter configuration (#63822). - Cache the Compute Group ID per query and remove redundant locks from the Backend selection hot path (#63636). - Skip waiting for asynchronous Rowset warmup to reduce write latency (#63877). ## Object Storage & External Data Sources - Support using an S3 Storage Vault credential provider without specifying a Role ARN (#64766). - Use HEAD requests for deterministic S3 paths, allowing access with only `s3:GetObject` permission and without `s3:ListBucket` (#60414). - Fill the Hive metadata cache while estimating table row counts to avoid duplicate HMS access during planning (#63470). ## Observability & Operations - Add `doris_fe_connection_max` and per-user `doris_fe_user_connection_max` metrics (#64742). - Add per-job Routine Load metrics and refresh Routine Load lag more promptly (#63576, #63654). - Add partition-filter conditions to SQL Block Rules (#62196). - Show the Compute Group used by MTMV refresh tasks (#63206). - Include scan-node IDs in Nereids physical plans (#62509). - Record session variables set by per-query `SET_VAR` hints in Audit Logs (#64569). - Improve password masking for `CREATE USER` and `ALTER USER` statements in Audit Logs (#62141). - Add support for LDAP default roles (#63411). ## Memory & Performance - Treat reclaimable `inactive_file` page cache as available memory in cgroup environments, reducing unnecessary query cancellation (#64347). - Add a BE-level cache for Tablet schemas used during load (#64581). - Reduce Row Store MemTable flush memory usage (#63342). - Truncate Segment key bounds without retaining complete backing buffers (#63469). - Aggregate non-MOW Segment key bounds to reduce metadata size (#64305). - Release Packed File writer buffers immediately after flush (#63967). - Use Segment footer `raw_data_bytes` to improve first-time Compaction batch-size estimation (#62263). - Bound File Cache LRU replay queues and add queue backlog metrics (#64381). # Important Bug Fixes ## Query Result Correctness - Fix incorrect Set Operation rewriting caused by using CTE-producer output in place of the regular child output (#64908). - Remove an unsafe TopN-to-Max rewrite that could produce incorrect results (#63519). - Fix incorrect row counts when merging nested TopN operators with an outer `OFFSET` (#64306). - Preserve deterministic ordering when merging TopN operators whose order keys have a prefix relationship (#64685). - Fix TopN Runtime Filter activation (#63969). - Reject TopN expressions in correlated scalar subqueries instead of generating an invalid plan (#64251). - Fix aggregate `ORDER BY` expression pushdown for window functions and scalar subqueries (#64787). - Fix Cast Project pushdown through `UNION DISTINCT`, which could change deduplication semantics (#64080). - Fix incorrect elimination of Null-Aware predicates in subqueries (#64639). - Fix pre-aggregation context leaking between Join branches (#63357). - Fix Runtime Filter handling for Outer Joins (#64157). - Align Legacy Planner literal comparison semantics with Nereids (#63481). - Fix missing query result packets during an FE Master switch, which could leave clients waiting until timeout (#62721). - Fix MTMV rewriting for multi-hop Outer Joins and null-reject compensation (#62492, #63268). - Fix mutation of excluded trigger tables while processing MTMV refresh triggers (#62984). - Fix View queries failing after the schema of an underlying external table changes (#64007). ## Functions & Types - Fix `convert_tz` constant folding and partition pruning across Daylight Saving Time transitions (#63853, #64029). - Preserve the sign of negative sub-hour TIMESTAMPTZ offsets (#62823). - Fix `datediff` constant folding with zero dates (#64084). - Fix `json_contains` returning false for duplicate elements in candidate arrays (#63301). - Support Boolean casts in `array_first` and `array_last` (#64847). - Validate the Lambda arity accepted by `array_sort` (#64825). - Reject invalid multi-argument `COUNT(DISTINCT ...)` Window Functions during analysis (#64783). - Limit `retention()` to its supported maximum of 32 arguments, preventing out-of-bounds memory access (#64521). - Reject `COUNT(DISTINCT variant)` with a clear unsupported-type error instead of a BE internal error (#63479). - Reject Lucene-syntax search on columns without an Inverted Index (#63857). - Reject invalid IPv4 default values during `CREATE TABLE` analysis (#62906). - Fix parsing of `ISNULL` expressions by placing them under `primaryExpression` (#63619). ## Load & Transactions - Fix truncated input when inferring schemas from compressed files through `http_stream` (#64769). - Fix Stream Load failures caused by integer overflow while comparing large Backend IDs (#63565). - Replace Tablet Writer close polling with event-based wakeup to reduce waiting and scheduling overhead (#64221). - Fix a Broken Pipe risk when redirecting Stream Load requests without consuming the request body (#64303). - Fix `load_to_single_tablet` routing for Auto Partition tables (#64356). - Fix `COPY INTO ... SELECT` failing to bind file-column placeholders (#64395). - Keep shared Delta Writer state independent of the Runtime State belonging to its original sink (#64349). - Fix `enable_insert_strict` incorrectly changing the semantics of `enable_strict_cast` (#63794). - Delay `INSERT OVERWRITE` partition routing until incremental open (#63209). - Select the transaction INSERT Backend from the current Compute Group (#63634). - Avoid quorum stalls when Delete Push tasks fail (#61647). - Fix an NPE when an Auto Partition is concurrently dropped during creation (#65357). - Keep load row-count metrics monotonic when Auto Partition creates new partitions (#64109). - Fix an NPE in force-finished Publish tasks (#63069). ## Routine Load - Serialize Routine Load task renewal when task submission fails (#64731). - Improve Kafka `read_committed` zero-row diagnostics and delay retries for zero-row batches (#63664, #64046). ## Compute-Storage Decoupling - Refresh the base Tablet before retrying Cloud Schema Change V1 to avoid repeated Compaction conflicts caused by stale versions (#64312). - Fix races between Empty Cumulative Compaction and Base/Cumulative Compaction on the same Tablet (#64619). - Fix reading Packed Inverted Index files after a File Cache miss (#64383). - Validate Recycle Rowset key state during `commit_rowset` to preserve idempotency and prevent invalid commits (#63985). - Recycle empty Rowsets without requiring a Resource ID (#64630). - Prevent Tablet KV metadata leaks after a partially failed recycle operation (#63377). - Deduplicate pending one-shot Cache Warmup jobs (#62384). - Fix Azure resource persistence across FE restart and replay (#65052). - Avoid false Tablet diagnosis alarms in Cloud mode (#60805). ## File Cache & Storage - Fix a race while scanning Backend Tablet Rowset maps (#65288). - Initialize Thread Context on asynchronous IO worker threads (#64846). - Fix incorrect fallback reads after a partial File Cache hit, and move LRU updates off the query read path (#61083). - Fix crashes when delayed File Cache holders reference an already-removed cache cell (#62437). - Fix crashes when a finalized scan cannot find its local File Cache writer (#62389). - Fix File Cache percentage overflow when clearing the cache immediately after a BE restart (#63410). - Make synchronous internal File Cache removal safe (#64578). - Fix duplicate File Cache metric accumulation in `NewOlapScanner` (#61072). - Exclude Cache Warmup reads from File Cache hit-ratio metrics (#63394). - Fix a finalized Pipeline task being submitted again and crashing BE (#64946). ## Lakehouse & External Data Sources - Fix Iceberg `COUNT(*)` pushdown throwing an NPE when snapshot summary counters are absent (#64648). - Fix unsupported mapping of Iceberg `varint` types (#64331). - Fix format detection for migrated Iceberg tables (#64134). - Fix Parquet Reader failures or BE crashes when a previous Row Group used Dictionary Filtering (#63168). - Normalize HDFS default paths and OSS bucket endpoint paths across external catalogs (#63476, #64943). - Push SQL Server and Oracle Boolean predicates as `1` and `0` instead of unsupported `TRUE` and `FALSE` literals (#64760). - Correctly recognize Doris-compatible targets in MySQL JDBC Catalogs (#64389). - Fix a MaxCompute Scanner memory leak and improve large-field writes (#61245). ## Metadata, Protocol & Observability - Fix remote Arrow Flight SQL result receiver initialization (#63136). - Fix `SHOW PROCESSLIST FULL` returning unexpected results (#64631). - Fix persisted-variable output in `SHOW VARIABLES` (#63734). - Support `$` in MySQL-compatible pattern matching (#64259). - Preserve Histogram metric labels when exporting Prometheus metrics (#63485). - Skip dropped columns during Follower FE statistics synchronization (#63882). > These notes focus on user-visible and operations-visible changes. Test-only commits, CI adjustments, release version updates, and purely internal changes without external symptoms are omitted. Refer to the complete 4.0.6-rc02 to 4.0.7-rc02 comparison for the full commit history. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
