Hi hackers,
While testing the new v19 planner extension APIs with an extension that
keeps per-rel state, I found that RelOptInfos created by flat copy
share their extension_state array with the rel they were copied from.
build_grouped_rel(), create_unique_paths() and
create_partial_unique_paths() memcpy() a RelOptInfo and reset the path
fields, and build_minmax_path() does the same for PlannerInfo, but none
of them resets extension_state / extension_state_allocated. With
joinrel_setup_hook (4020b370f2) and build_simple_rel_hook (91f33a2ae9),
the natural place for an extension to attach state is right when a rel
is built, and make_join_rel() runs joinrel_setup_hook before
make_grouped_join_rel() copies the joinrel. So:
1. When join_path_setup_hook or set_join_pathlist_hook is called for
the grouped joinrel, GetRelOptInfoExtensionState() returns the state
stored for the plain joinrel, and anything stored for the grouped
joinrel overwrites the plain joinrel's entry.
2. If the store needs a slot past the end of the array,
SetRelOptInfoExtensionState() repallocs the shared array. That
frees the array the plain joinrel still points to, and the next
lookup on the plain joinrel reads freed memory.
Eager aggregation is on by default, so a GROUP BY over a two-table join
is enough. The attached test module stores a pointer to the owning rel
in its state, so it can tell whose state it gets back. On
REL_19_STABLE (2359968b44) with --enable-cassert, using the tables from
eager_aggregate.sql:
SET test_extstate.mode = 1;
EXPLAIN (COSTS OFF)
SELECT t1.a, avg(t2.c)
FROM eager_agg_t1 t1 JOIN eager_agg_t2 t2 ON t1.b = t2.b
GROUP BY t1.a;
WARNING: extension state on grouped joinrel (b 1 2) belongs to a different
RelOptInfo
SET test_extstate.mode = 2; -- also stores into a higher slot
EXPLAIN ... same query ...
WARNING: plain joinrel (b 1 2) still points at array 0x740a72cbfc48, which
repalloc just freed; slot 1 now reads 0x7f7f7f7f7f7f7f7f
server closed the connection unexpectedly.
The crash is in the module's join_path_setup_hook, called from
add_paths_to_joinrel() <- populate_joinrel_with_paths() <- make_join_rel() for
the plain joinrel.
With enable_eager_aggregate = off there are no warnings.
This is the same pattern as 58fca6bdf2 ("Don't pass grouped relations to
FDWs"), which cleared the FDW fields in build_grouped_rel() [1];
extension_state needs the same treatment there and at the other copy
sites. For the unique-ified rel, the intent was that it stays opaque
to extensions [2], but it is passed to the join hooks as outerrel /innerrel, so
extensions do see it.
The attached patch resets the two fields at all four copy sites. The
build_minmax_path() copy is not reachable through any current hook,
because preprocess_minmax_aggregates() runs before anything can attach
state to the parent PlannerInfo, but resetting it there seems right
too. The patch also adds the "constant string" note that
GetExplainExtensionId() already has to GetPlannerExtensionId(), since
it keeps the caller's pointer rather than copying the name.
With the patch, the reproducer shows no warnings, the plan still has
the Partial HashAggregate below the join, and the core regression
tests pass (installcheck-parallel, 239 tests, cassert + ASan build).
For the eager-deduplication series [3]: its proposed
copy_rel_without_paths() would be a good single place to reset
extension_state, if that lands.
I found this while reviewing the v19 extension-facing planner APIs with
the help of an AI code-review and confirmed it on the build
above.
[1]
https://postgr.es/m/ca+tgmob7ism9ykrm44vjuduacchw-fy54mv5njptztl9uny...@mail.gmail.com
[2]
https://postgr.es/m/cambws4-uxdxmazo-ggnx7y4twodutc4uvmnhubn+8afhp52...@mail.gmail.com
[3]
https://postgr.es/m/ph0pr18mb444315cd3e17255ab4a629bea6...@ph0pr18mb4443.namprd18.prod.outlook.com
Regards,
Shubhankar K.
Member Technical Staff
Zoho Corporation
repro.sql
Description: Binary data
test_extstate.c
Description: Binary data
v1-0001-Don-t-let-flat-copied-planner-structures-share-ex.patch
Description: Binary data
==1290966==ERROR: AddressSanitizer: SEGV on unknown address (pc 0x7cef47e418ba
bp 0x7ffdbda76c10 sp 0x7ffdbda76b30 T0)
==1290966==The signal is caused by a READ memory access.
==1290966==Hint: this fault was caused by a dereference of a high value address
(see register values below). Disassemble the provided pc to learn which
register was used.
#0 0x7cef47e418ba in tes_join_path_setup test_extstate.c:82:27
#1 0x60d2366ada3b in add_paths_to_joinrel
/home/shubha-18887/zoho/postgres19/src/backend/optimizer/path/joinpath.c:178:3
#2 0x60d2366be624 in populate_joinrel_with_paths
/home/shubha-18887/zoho/postgres19/src/backend/optimizer/path/joinrels.c:1131:4
#3 0x60d2366bb2c7 in make_join_rel
/home/shubha-18887/zoho/postgres19/src/backend/optimizer/path/joinrels.c:800:2
#4 0x60d2366ba16c in make_rels_by_clause_joins
/home/shubha-18887/zoho/postgres19/src/backend/optimizer/path/joinrels.c:300:11
#5 0x60d2366b8f75 in join_search_one_level
/home/shubha-18887/zoho/postgres19/src/backend/optimizer/path/joinrels.c:123:4
#6 0x60d236649a7d in standard_join_search
/home/shubha-18887/zoho/postgres19/src/backend/optimizer/path/allpaths.c:3981:3
#7 0x60d236642a25 in make_rel_from_joinlist
/home/shubha-18887/zoho/postgres19/src/backend/optimizer/path/allpaths.c:3912:11
#8 0x60d2366419cb in make_one_rel
/home/shubha-18887/zoho/postgres19/src/backend/optimizer/path/allpaths.c:244:8
#9 0x60d23671dec3 in query_planner
/home/shubha-18887/zoho/postgres19/src/backend/optimizer/plan/planmain.c:377:14
#10 0x60d236729d72 in grouping_planner
/home/shubha-18887/zoho/postgres19/src/backend/optimizer/plan/planner.c:1962:17
#11 0x60d236726692 in subquery_planner
/home/shubha-18887/zoho/postgres19/src/backend/optimizer/plan/planner.c:1367:2
#12 0x60d23671f55b in standard_planner
/home/shubha-18887/zoho/postgres19/src/backend/optimizer/plan/planner.c:543:9
#13 0x60d23671e19e in planner
/home/shubha-18887/zoho/postgres19/src/backend/optimizer/plan/planner.c:348:12
#14 0x60d236b6d584 in pg_plan_query
/home/shubha-18887/zoho/postgres19/src/backend/tcop/postgres.c:917:9
#15 0x60d2360b0485 in standard_ExplainOneQuery
/home/shubha-18887/zoho/postgres19/src/backend/commands/explain.c:359:9
#16 0x60d2360afd33 in ExplainOneQuery
/home/shubha-18887/zoho/postgres19/src/backend/commands/explain.c:315:3
#17 0x60d2360af681 in ExplainQuery
/home/shubha-18887/zoho/postgres19/src/backend/commands/explain.c:228:4
#18 0x60d236b84d1c in standard_ProcessUtility
/home/shubha-18887/zoho/postgres19/src/backend/tcop/utility.c:868:4
#19 0x60d236b83a5e in ProcessUtility
/home/shubha-18887/zoho/postgres19/src/backend/tcop/utility.c:525:3
#20 0x60d236b82aa7 in PortalRunUtility
/home/shubha-18887/zoho/postgres19/src/backend/tcop/pquery.c:1153:2
#21 0x60d236b7f97f in FillPortalStore
/home/shubha-18887/zoho/postgres19/src/backend/tcop/pquery.c:1026:4
#22 0x60d236b7eead in PortalRun
/home/shubha-18887/zoho/postgres19/src/backend/tcop/pquery.c:760:6
#23 0x60d236b73fd5 in exec_simple_query
/home/shubha-18887/zoho/postgres19/src/backend/tcop/postgres.c:1290:10
#24 0x60d236b728d3 in PostgresMain
/home/shubha-18887/zoho/postgres19/src/backend/tcop/postgres.c:4856:7
#25 0x60d236b63041 in BackendMain
/home/shubha-18887/zoho/postgres19/src/backend/tcop/backend_startup.c:124:2
#26 0x60d2368947f5 in postmaster_child_launch
/home/shubha-18887/zoho/postgres19/src/backend/postmaster/launch_backend.c:268:3
#27 0x60d23689ff93 in BackendStartup
/home/shubha-18887/zoho/postgres19/src/backend/postmaster/postmaster.c:3635:8
#28 0x60d23689d508 in ServerLoop
/home/shubha-18887/zoho/postgres19/src/backend/postmaster/postmaster.c:1727:6
#29 0x60d23689b8ce in PostmasterMain
/home/shubha-18887/zoho/postgres19/src/backend/postmaster/postmaster.c:1414:11
#30 0x60d2364836de in main
/home/shubha-18887/zoho/postgres19/src/backend/main/main.c:227:4
#31 0x7cef47c2a1c9 in __libc_start_call_main
csu/../sysdeps/nptl/libc_start_call_main.h:58:16
#32 0x7cef47c2a28a in __libc_start_main csu/../csu/libc-start.c:360:3
#33 0x60d2359c8394 in _start
(/home/shubha-18887/zoho/rel19/bin/postgres+0x433394)
==1290966==Register values:
rax = 0x0fefefefefefefef rbx = 0x00007ffdbda76c20 rcx = 0x0000000000000008
rdx = 0x00007a7f47003fb8
rdi = 0x00007b3f4703cc68 rsi = 0x0000000000000001 rbp = 0x00007ffdbda76c10
rsp = 0x00007ffdbda76b30
r8 = 0x0000000000000000 r9 = 0x000078ef45dd1a20 r10 = 0x0000000000000f01
r11 = 0x0000100037b46dc8
r12 = 0x0000000000000003 r13 = 0x0000000000000000 r14 = 0x000060d237870570
r15 = 0x00007cef4894e000
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV test_extstate.c:82:27 in tes_join_path_setup
