chenjinbao1989 opened a new pull request, #1768: URL: https://github.com/apache/cloudberry/pull/1768
Merge upstream PostgreSQL REL_16_9 into Cloudberry, upgrading the kernel from PG 14.4 to PG 16.9. This spans two major versions (PG 15 and PG 16), bringing in 5730 commits. Key features and changes, sorted by importance: MERGE command SQL-standard MERGE for INSERT/UPDATE/DELETE in a single statement. ruleutils decompilation, PlaceHolderVar support in actions Fixes for partitioned tables, cross-partition triggers, RLS, EvalPlanQual, self-modified tuples, pg_stat_statements Logical replication enhancements Built-in support for prepared transactions (2PC) Enable two-phase via replication protocol Streaming changes after speculative aborts New pg_create_subscription predefined role Parallel apply workers (PG 16) Row/column filtering, schema-level publication (PG 15) Critical fix for data loss in logical replication Slot invalidation, snapshot, and memory leak fixes SQL/JSON IS JSON predicate Standard JSON constructor functions (json_array, json_object, json_scalar, json_serialize) Auto-generated query jumbling via gen_node_support.pl Fix json_array() subquery double transformation WAL and storage subsystem LZ4 compression for full-page writes (FPW) Custom WAL resource managers Skip WAL recycling/preallocation during archive recovery XLOG_FPI_FOR_HINT honors full_page_writes setting Direct I/O support on macOS Buffer manager infrastructure refactor for faster relation extension New smgrzeroextend(), FileZero(), FileFallocate() Remove HeapBitmapScan skip_fetch optimization (correctness issue) WAL/Recovery separation: xlogrecovery.c, xlogprefetcher.c (PG 15) Backup sink architecture (PG 15) Query optimizer / executor Performance: Hash table acceleration for NOT IN(values) Datum sorts for single-column sorts Result Cache renamed to Memoize with ongoing fixes Parallel Hash Full Join Reduced planning cost for deeply-nested views Fewer pallocs when building partition bounds Outer join optimization enhancement (PG 16) Symmetric hash join optimization (PG 16) Critical fixes: Planner failure to identify multiple hashable ScalarArrayOpExprs Consistent whole-row Var construction in parsing and planning WindowAgg evaluation crash WITH RECURSIVE UNION assert failure ORDER BY / DISTINCT aggregates with FILTER setrefs.c missing expression processing on prune steps Parallel Hash Join extreme skew detection pgstat monitoring enhancements pgstat shared memory rewrite (PG 15): Stats collector process removed; stats now in shared memory Monolithic pgstat.c split into 14 files under utils/activity/ pg_stat_io (PG 16 new view): Detailed I/O statistics: shared buffer hits, I/O times, writeback pg_stat_statements: JIT counters, temp file block I/O timing Utility query string normalization, MERGE support 32-bit integer overflow fix Other: pg_stat_wal time accumulation as instr_time Macro-generated pg_stat_get*() functions for tables and databases Enhanced pg_stat_reset_single_table_counters SP-GiST index scans counted in pg_stat New test helpers: pg_stat_force_next_flush(), pg_stat_have_stats() libpq and client protocol New PQsendFlushRequest Pipeline mode state machine fix SASL code refactored to generic interface Escape function fixes for invalid encoding data (security) PQescapeLiteral()/PQescapeIdentifier() length handling fix New pg_encoding_set_invalid() Build-time check that libpq doesn't call exit()/abort() Security and privilege model CREATEROLE privilege restrictions (major security improvement) New GUCs: createrole_self_grant, reserved_connections New predefined roles: pg_maintain, pg_create_subscription Non-superuser predefined roles for vacuum/analyze Revoke PUBLIC CREATE from public schema (now owned by pg_database_owner) Security invoker views (SECURITY INVOKER) session_authorization and role interaction fixes scram_SaltedPassword() integer overflow fix pg_dump / pg_upgrade / pg_basebackup toolchain pg_dump: Generic compression API, zstd support, LZ4 frame-only format pg_basebackup: Extended compression options, server-side compression with -Fp Client-side LZ4 decompression, parallel zstd compression pg_upgrade: Fix unintentional 'NULL' string literal Fix XMLSERIALIZE(NO INDENT) cross-version upgrade pg_dumpall handling of dangling OIDs in pg_auth_members Partitioned table improvements Self-referencing FKs in partitioned tables Detach partition with top-level FK fix Reset relhassubclass on ATTACH Trigger rename consistency, preserve firing state on clone Disallow partitionwise join/grouping on collation mismatch ICU and collation Build ICU support by default initdb uses uloc_getDefault() CREATE DATABASE LOCALE applies to all collation providers Node support function auto-generation (FIXME) gen_node_support.pl auto-generates copyfuncs/equalfuncs/outfuncs/readfuncs Query jumbling code auto-generation NodeTag ABI stability check Global renames and structural changes RelFileNode -> RelFileLocator (pervasive rename across hundreds of files) GUC system split: guc.c -> guc.c + guc_tables.c + guc_funcs.c varatt.h separated from postgres.h Test framework: PostgresNode.pm -> PostgreSQL::Test::Cluster SQL types and functions Numeric scale allows negative or greater than precision New ANY_VALUE aggregate function unnest(multirange), range_agg with multirange inputs pg_size_pretty/pg_size_bytes support petabytes ALTER TABLE SET ACCESS METHOD SYSTEM_USER function numeric_mul() overflow fix psql improvements PSQL_WATCH_PAGER for \watch command New \drg command for role grants \copy from sends data in larger chunks pg_waldump --rmgr multiple specification Critical data corruption and crash fixes Fix data corruption when relation truncation fails Fail instead of corrupting page header on non-LP_NORMAL TID update Fix unsafe BufferDescriptors access GB18030 SIGSEGV from out-of-bounds read Snowball stemmer null pointer dereference after OOM Rare standby assertion failure on primary restart catcache invalidation during list entry construction Vacuum / Autovacuum Parallel VACUUM (PG 16, vacuumparallel.c new file) ON COMMIT DELETE ROWS avoids ERROR after relhassubclass=f Use WaitLatch() instead of pg_usleep() at end-of-vacuum truncation Prevent numeric overflow in parallel numeric aggregates ANALYZE preserves relhasindex for partitioned tables Co-authored-by: liushengsong [[email protected]](mailto:[email protected]) Co-authored-by: reshke [[email protected]](mailto:[email protected]) Co-authored-by: Hao Wu [[email protected]](mailto:[email protected]) Co-authored-by: Jianghua.yjh [[email protected]](mailto:[email protected]) Co-authored-by: Dianjin Wang [[email protected]](mailto:[email protected]) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
