gavinchou opened a new issue, #54868: URL: https://github.com/apache/doris/issues/54868
## Behavior Changes - Adjust the permission requirements for `show frontends` and `show backends` to align with the corresponding RESTful API, i.e., requiring the `SELECT_PRIV` permission on the `information_schema` database (#50140) - Admin and root users with specified domains are no longer considered system users (#50904) - Storage: The default number of concurrent transactions per database is adjusted to 10000 (#51367, #52380) ## New Features ### Lakehouse (No specific content) ### Asynchronous Materialized Views (No specific content) ### Query Optimizer - Support MySQL's aggregate roll-up syntax `GROUP BY ... WITH ROLLUP` (#51948) ### Query Execution - `Like` statement supports `escape` syntax (#52540) ### Semi-structured Data Management - Support building non-tokenized inverted indexes and ngram bloomfilter indexes only for new data by setting the session variable `enable_add_index_for_new_data=true` (#52251, #48461) ### Storage (No new features; relevant changes see Behavior Changes) ### New Functions - Added data functions: `cot`/`sec`/`cosec` (#52872) ## Improvements ### Data Ingestion - Optimize error message prompts for `SHOW CREATE LOAD` (#53694, #53730) ### Primary Key Model - Add segment key bounds truncation capability to avoid single large import failures (#45287, #51595) ### Storage - Enhance the reliability of compaction and imported data (#51048, #51852, #51129, #51483) - Optimize balance speed (#51733, #52813, #52013, #52309) - Optimize table creation speed (#52688, #52918) - Optimize compaction default parameters and observability (#53244, #53562, #52321, #52605, #53133, #53215, #51649, #52389, #52044, #52297) - Optimize the issue of query error -230 (#52440, #54131) - Add system table `backend_tablets` (#52195) - Optimize the performance of querying `information_schema.tables` from follower nodes in cloud mode (#51240, #51405) ### Storage-Compute Decoupled - Enhance observability of Meta-service recycler (#52882, #53100, #51542, #52430, #53116, #52729, #53351, #53114, #53320, #52714, #50910, #51487, #51409, #51884, #52523, #53117) - Support cross-compute group incremental preheating during import compaction (#52370, #52514, #53406, #53860, #53861, #52339, #52426, #53525, #53530, #53339, #53523, #52991, #53555, #53666, #51594, #52291, #53752) - Optimize Storage vault connectivity check (#51175, #52319, #52541, #52602, #53075, #53227) - Support updating storage backend information via MS API (#51162, #51685) ### Lakehouse - Optimize ORC zlib decompression performance in x86 environment and fix potential issues (#51775) - Optimize the default number of concurrent threads for external table reading (#51415) - Optimize error messages for Catalogs that do not support DDL operations (#50696) ### Asynchronous Materialized Views - Optimize the performance of transparent rewriting planning (#49514) ### Query Optimizer - The `group_concat` function now allows parameters of non-string types (#52805) - The `sum` and `avg` functions allow parameters of non-numeric types (#49954) - Expand the scope of support for delayed materialization in TOP-N queries, enabling delayed materialization when querying partial columns (#52522) - When creating partitions, list partitions allow inclusion of `MAX_VALUE` (#46076) - Optimize the performance of sampling and collecting statistical information for aggregate model tables (#49918) - Optimize the accuracy of NDV values when sampling and collecting statistical information (#50574) ### Inverted Index - Unify the order of properties displayed for inverted indexes in `show create table` (#51467) - Add per-condition profile metrics (such as hit rows and execution time) for inverted index filter conditions to facilitate performance analysis (#47504) - Enhance the display of inverted index-related information in profiles (#48950, #48826, #51495) ### Permissions - Ranger supports setting permissions for storage vault and compute group (#47925) ## Bug Fixes ### Data Ingestion - Fix the correctness issue that may occur when importing CSV files with multi-character separators (#53374, #53634) - Fix the issue where the result of `ROUTINE LOAD` task display is incorrect after modifying task properties (#53038, #53098) - Fix the issue where the one-stream multi-table import plan becomes invalid after primary node restart or Leader switch (#53799, #53829) - Fix the issue where all scheduling tasks are blocked because `ROUTINE LOAD` tasks cannot find available BE nodes (#52654, #52791) - Fix the concurrent read-write conflict issue of `runningTxnIds` (#51615, #51639) ### Primary Key Model - Optimize the import performance of mow tables under high-frequency concurrent imports (#52360, #52439, #47281, #52225) - mow table full compaction releases space of deleted data (#51874, #52256) - Fix the potential import failure issue of mow tables in extreme scenarios (#52547, #52848) - Optimize the compaction performance of mow tables (#52476, #52952) - Fix the potential correctness issue of mow tables during concurrent imports and schema changes (#52582, #52974) - Fix the issue where schema change on empty mow tables may cause import stuck or schema change failure (#51780, #52166) - Fix the memory leak issue of mow delete bitmap cache (#52718, #52756, #52931) - Fix the potential correctness issue of mow tables after schema change (#51353, #51531) ### Storage - Fix the missing rowset issue in clone process caused by compaction (#53984, #54162, #52812, #53497, #53193, #53527) - Fix the issue of inaccurate size calculation and default value for autobucket (#51258, #51682, #52801, #53250) - Fix the potential correctness issue caused by bucket columns (#54037, #54024, #54072, #54109) - Fix the issue where single-column tables cannot be renamed (#47275, #52340) - Fix the potential memory leak issue of memtable (#52902, #52965) - Fix the inconsistent error reporting issue for unsupported operations in empty table transaction writes (#52133, #52635) ### Storage-Compute Decoupled - Several fixes for File cache (#51197, #51793, #53783, #53915, #51684, #51904, #51776, #52556, #51463, #51603, #51711, #52152) - Fix the issue where cumulative point may roll back during schema process (#53402, #53446) - Fix the issue where background tasks affect automatic restart (#51729, #52519) - Fix the unhandled exception issue in data recycling process in azure environment (#53042, #53224) - Fix the issue where file cache is not cleaned up in time when compacting a single rowset (#51674) ### Lakehouse - Fix the transaction commit failure issue for Iceberg table writes in Kerberos environment (#51508) - Fix the query issue for hudi in kerberos environment (#51713) - Fix the potential deadlock issue in multi-Catalog scenarios (#53626) - Fix the metadata inconsistency issue caused by concurrent Catalog refresh in some cases (#51787) - Fix the issue where ORC footer is read multiple times in some cases (#51277) - Fix the issue where Table Valued Function cannot read compressed json files (#51983) - SQL Server Catalog supports identifying IDENTITY column information (#51285) - SQL Convertor supports specifying multiple URLs for high availability (#52636) ### Asynchronous Materialized Views - Fix the issue where partition compensation may be performed incorrectly when the query is optimized to an empty result set (#51700) ### Query Optimizer - Fix the issue where factors other than `sql_select_limit` affect DML execution results (#53379) - Fix the issue where materialized CTEs may report errors in extreme cases when starting local shuffle (#52870) - Fix the issue where prepared insert statements cannot be executed on non-master nodes (#48689) - Fix the result error issue when casting `ipv4` to string (#51546) ### Permissions - When a user has multiple roles, the permissions of the multiple roles will be merged before authorization (#52948) ### Query Execution - Fix issues with some json functions (#52744, #52915, #53364) - Fix the potential BE Core issue when the asynchronous thread pool is full (#52365) - Fix the incorrect result issue of `hll_to_base64` (#51831) - Fix the result error issue when casting `decimal256` to float (#54140) - Fix two memory leak issues (#51952, #51929, #52542) - Fix the be core issue caused by `bitmap_from_base64` (#53018) - Fix the potential be core issue caused by `array_map` function (#51618) - Fix the potential error issue of `split_by_regexp` function (#51293) - Fix the potential result error issue of `bitmap_union` function under extremely large data volumes (#52033) - Fix the potential core issue of `format round` function under some boundary values (#53855) ### Inverted Index - Fix the memory leak issue of inverted indexes in abnormal situations (#53235) - Fix the error reporting issue when writing and querying empty index files (#51984, #51393) - Capture IO exceptions in inverted index string reading to avoid process crash due to exceptions (#51844) ### Complex Data Types - Fix the potential type inference error when Variant Nested data types conflict (#52696) - Fix the parameter type inference error of `map` function (#52696) - Fix the issue where data is incorrectly converted to NULL when specifying `'$.'` as the path in jsonpath (#52211) - Fix the issue where the serialization format cannot be restored when a subfield of Variant contains `.` (#51930) ### Others - Fix the insufficient length issue of the IP field in the auditlog table (#52762, #52984) - Fix the issue where the query id recorded in the audit log is that of the previous query when SQL parsing fails (#53107) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
