On 1/28/23 13:05, Tomas Vondra wrote:
>
> FWIW I'll wait for dikkop to finish the current buildfarm run (it's
> currently chewing on HEAD) and then will try to do runs of the 'joins'
> test in a loop. That's where dikkop got stuck before.
>
So I did that - same configure options as the buildfarm client, and a
'make check' (with only tests up to the 'join' suite, because that's
where it got stuck before). And it took only ~15 runs (~1h) to hit this
again on dikkop.
As before, there are three processes - leader + 2 workers, but the query
is different - this time it's this one:
-- A couple of other hash join tests unrelated to work_mem management.
-- Check that EXPLAIN ANALYZE has data even if the leader doesn't
participate
savepoint settings;
set local max_parallel_workers_per_gather = 2;
set local work_mem = '4MB';
set local parallel_leader_participation = off;
select * from hash_join_batches(
$$
select count(*) from simple r join simple s using (id);
$$);
I managed to collect the fstat/procstat stuff Thomas asked for, and the
backtraces - attached. I still have the core files, in case we look at
something. As before, running gcore on the second worker (29081) gets
this unstuck - it sends some signal that apparently wakes it up.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL CompanyGNU gdb (GDB) 12.1 [GDB v12.1 for FreeBSD]
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-portbld-freebsd14.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./tmp_install/usr/local/pgsql/bin/postgres...
warning: exec file is newer than core file.
[New LWP 100597]
warning: Could not load shared library symbols for /mnt/data/postgres/tmp_install/usr/local/pgsql/lib/plpgsql.so.
Do you need "set solib-search-path" or "set sysroot"?
Core was generated by `postgres: freebsd regression [local] SELECT'.
#0 _poll () at _poll.S:4
4 _poll.S: No such file or directory.
(gdb) bt
#0 _poll () at _poll.S:4
#1 0x00000000891c1850 in __thr_poll (fds=0x4, nfds=1, timeout=-1) at /usr/src/lib/libthr/thread/thr_syscalls.c:338
#2 0x000000000076e930 in WaitEventSetWaitBlock (nevents=1, occurred_events=0x811ed188, cur_timeout=-1, set=0x9ba2e590) at latch.c:1171
#3 WaitEventSetWait (set=set@entry=0x9ba2e590, timeout=timeout@entry=-1, occurred_events=occurred_events@entry=0x811ed188, nevents=nevents@entry=1, wait_event_info=wait_event_info@entry=134217731) at latch.c:1000
#4 0x000000000076ee18 in WaitLatchOrSocket (latch=0x9abe49c0, wakeEvents=wakeEvents@entry=1, sock=sock@entry=-1, timeout=-1, timeout@entry=0, wait_event_info=wait_event_info@entry=134217731) at latch.c:385
#5 0x000000000076eef0 in WaitLatch (latch=<optimized out>, wakeEvents=wakeEvents@entry=1, timeout=timeout@entry=0, wait_event_info=wait_event_info@entry=134217731) at latch.c:339
#6 0x00000000006481ec in gather_readnext (gatherstate=<optimized out>) at nodeGather.c:367
#7 gather_getnext (gatherstate=0x9b824818) at nodeGather.c:256
#8 ExecGather (pstate=0x9b824818) at nodeGather.c:207
#9 0x0000000000638824 in ExecProcNodeInstr (node=0x9b824818) at execProcnode.c:462
#10 0x0000000000641708 in ExecProcNode (node=0x9b824818) at ../../../src/include/executor/executor.h:248
#11 fetch_input_tuple (aggstate=aggstate@entry=0x9b824368) at nodeAgg.c:407
#12 0x0000000000642ee8 in agg_retrieve_direct (aggstate=0x9b824368) at nodeAgg.c:1746
#13 ExecAgg (pstate=0x9b824368) at nodeAgg.c:1561
#14 0x0000000000638824 in ExecProcNodeInstr (node=0x9b824368) at execProcnode.c:462
#15 0x00000000006313a4 in ExecProcNode (node=0x9b824368) at ../../../src/include/executor/executor.h:248
#16 ExecutePlan (execute_once=<optimized out>, dest=0xbbac78 <donothingDR>, direction=<optimized out>, numberTuples=0, sendTuples=<optimized out>, operation=CMD_SELECT, use_parallel_mode=<optimized out>, planstate=0x9b824368,
estate=0x9b824118) at execMain.c:1712
#17 standard_ExecutorRun (queryDesc=0x9b93d550, direction=<optimized out>, count=0, execute_once=<optimized out>) at execMain.c:353
#18 0x00000000005ca960 in ExplainOnePlan (plannedstmt=plannedstmt@entry=0x9b93d438, into=into@entry=0x0, es=es@entry=0x905b39f8,
queryString=queryString@entry=0x905e2918 "explain (analyze, format 'json') \n select count(*) from simple r join simple s using (id);\n", params=params@entry=0x0, queryEnv=queryEnv@entry=0x0,
planduration=planduration@entry=0x811ed450) at explain.c:536
#19 0x00000000005cacac in ExplainOneQuery (query=<optimized out>, cursorOptions=256, into=0x0, es=0x905b39f8, queryString=0x905e2918 "explain (analyze, format 'json') \n select count(*) from simple r join simple s using (id);\n",
params=0x0, queryEnv=0x0) at explain.c:372
#20 0x00000000005cb228 in ExplainQuery (pstate=0x905b38e0, stmt=0x905e2b28, queryString=0x905e2918 "explain (analyze, format 'json') \n select count(*) from simple r join simple s using (id);\n", params=0x0, queryEnv=0x0,
dest=0x905b3848) at ../../../src/include/nodes/nodes.h:582
#21 0x000000000079bcc4 in standard_ProcessUtility (pstmt=0x905e2a10, queryString=0x905e2918 "explain (analyze, format 'json') \n select count(*) from simple r join simple s using (id);\n", context=PROCESS_UTILITY_QUERY, params=0x0,
queryEnv=0x0, dest=0x905b3848, completionTag=0x811ed5e0 "") at utility.c:675
#22 0x0000000000799c50 in PortalRunUtility (portal=portal@entry=0x90664230, pstmt=0x905e2a10, isTopLevel=isTopLevel@entry=false, setHoldSnapshot=setHoldSnapshot@entry=true, dest=dest@entry=0x905b3848,
completionTag=completionTag@entry=0x811ed5e0 "") at pquery.c:1174
#23 0x000000000079a150 in FillPortalStore (portal=portal@entry=0x90664230, isTopLevel=isTopLevel@entry=false) at ../../../src/include/nodes/nodes.h:582
#24 0x000000000079a734 in PortalRunFetch (portal=portal@entry=0x90664230, fdirection=fdirection@entry=FETCH_FORWARD, count=count@entry=10, dest=dest@entry=0xbbacc8 <spi_printtupDR>) at pquery.c:1463
#25 0x0000000000664054 in _SPI_cursor_operation (portal=0x90664230, direction=FETCH_FORWARD, count=10, dest=0xbbacc8 <spi_printtupDR>) at spi.c:2711
#26 0x00000000a3bd1998 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
GNU gdb (GDB) 12.1 [GDB v12.1 for FreeBSD]
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-portbld-freebsd14.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./tmp_install/usr/local/pgsql/bin/postgres...
warning: exec file is newer than core file.
[New LWP 100395]
Core was generated by `postgres: parallel worker for PID 29046 '.
#0 _poll () at _poll.S:4
4 _poll.S: No such file or directory.
(gdb) bt
#0 _poll () at _poll.S:4
#1 0x00000000891c1850 in __thr_poll (fds=0x4, nfds=2, timeout=-1) at /usr/src/lib/libthr/thread/thr_syscalls.c:338
#2 0x000000000076e930 in WaitEventSetWaitBlock (nevents=1, occurred_events=0x811edf18, cur_timeout=-1, set=0x90645eb0) at latch.c:1171
#3 WaitEventSetWait (set=0x90645eb0, timeout=timeout@entry=-1, occurred_events=occurred_events@entry=0x811edf18, nevents=nevents@entry=1, wait_event_info=wait_event_info@entry=134217738) at latch.c:1000
#4 0x000000000078c5b0 in ConditionVariableSleep (cv=cv@entry=0x9e85e308, wait_event_info=wait_event_info@entry=134217738) at condition_variable.c:157
#5 0x000000000076a778 in BarrierArriveAndWait (barrier=barrier@entry=0x9e85e2f0, wait_event_info=134217738) at barrier.c:191
#6 0x000000000064d910 in ExecHashJoinImpl (parallel=true, pstate=0x9070feb8) at nodeHashjoin.c:328
#7 ExecParallelHashJoin (pstate=0x9070feb8) at nodeHashjoin.c:600
#8 0x0000000000638824 in ExecProcNodeInstr (node=0x9070feb8) at execProcnode.c:462
#9 0x0000000000641708 in ExecProcNode (node=0x9070feb8) at ../../../src/include/executor/executor.h:248
#10 fetch_input_tuple (aggstate=aggstate@entry=0x9070fb70) at nodeAgg.c:407
#11 0x0000000000642ee8 in agg_retrieve_direct (aggstate=0x9070fb70) at nodeAgg.c:1746
#12 ExecAgg (pstate=0x9070fb70) at nodeAgg.c:1561
#13 0x0000000000638824 in ExecProcNodeInstr (node=0x9070fb70) at execProcnode.c:462
#14 0x00000000006313a4 in ExecProcNode (node=0x9070fb70) at ../../../src/include/executor/executor.h:248
#15 ExecutePlan (execute_once=<optimized out>, dest=0x90665a78, direction=<optimized out>, numberTuples=0, sendTuples=<optimized out>, operation=CMD_SELECT, use_parallel_mode=<optimized out>, planstate=0x9070fb70, estate=0x9070f118)
at execMain.c:1712
#16 standard_ExecutorRun (queryDesc=0x90666e90, direction=<optimized out>, count=0, execute_once=<optimized out>) at execMain.c:353
#17 0x0000000000636434 in ParallelQueryMain (seg=0x90564fb8, toc=0x9e85e000) at execParallel.c:1402
#18 0x0000000000509c58 in ParallelWorkerMain (main_arg=<optimized out>) at parallel.c:1433
#19 0x0000000000705b88 in StartBackgroundWorker () at bgworker.c:911
#20 0x000000000071263c in do_start_bgworker (rw=0x90585780) at postmaster.c:5831
#21 maybe_start_bgworkers () at postmaster.c:6056
#22 0x0000000000712ef8 in sigusr1_handler (postgres_signal_arg=<optimized out>) at postmaster.c:5169
#23 0x00000000891bf37c in handle_signal (actp=actp@entry=0x811ee9c0, sig=sig@entry=30, info=info@entry=0x811eea30, ucp=ucp@entry=0x811eea80) at /usr/src/lib/libthr/thread/thr_sig.c:303
#24 0x00000000891beab8 in thr_sighandler (sig=30, info=0x811eea30, _ucp=0x811eea80) at /usr/src/lib/libthr/thread/thr_sig.c:246
#25 <signal handler called>
#26 _select () at _select.S:4
#27 0x00000000891c1b0c in __thr_select (numfds=4, readfds=0x811eefc0, writefds=0x0, exceptfds=0x0, timeout=0x811ef448) at /usr/src/lib/libthr/thread/thr_syscalls.c:495
#28 0x0000000000713720 in ServerLoop () at postmaster.c:1692
#29 0x00000000007150e8 in PostmasterMain (argc=argc@entry=8, argv=argv@entry=0x811ef998) at postmaster.c:1401
#30 0x000000000047aaf8 in main (argc=8, argv=0x811ef998) at main.c:228
GNU gdb (GDB) 12.1 [GDB v12.1 for FreeBSD]
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-portbld-freebsd14.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./tmp_install/usr/local/pgsql/bin/postgres...
warning: exec file is newer than core file.
[New LWP 100091]
Core was generated by `postgres: parallel worker for PID 29046 '.
#0 _poll () at _poll.S:4
4 _poll.S: No such file or directory.
(gdb) bt
#0 _poll () at _poll.S:4
#1 0x00000000891c1850 in __thr_poll (fds=0x4, nfds=2, timeout=-1) at /usr/src/lib/libthr/thread/thr_syscalls.c:338
#2 0x000000000076e930 in WaitEventSetWaitBlock (nevents=1, occurred_events=0x811ede68, cur_timeout=-1, set=0x90645eb0) at latch.c:1171
#3 WaitEventSetWait (set=0x90645eb0, timeout=timeout@entry=-1, occurred_events=occurred_events@entry=0x811ede68, nevents=nevents@entry=1, wait_event_info=wait_event_info@entry=134217737) at latch.c:1000
#4 0x000000000078c5b0 in ConditionVariableSleep (cv=cv@entry=0x9d25e308, wait_event_info=wait_event_info@entry=134217737) at condition_variable.c:157
#5 0x000000000076a778 in BarrierArriveAndWait (barrier=barrier@entry=0x9d25e2f0, wait_event_info=134217737) at barrier.c:191
#6 0x000000000064c160 in MultiExecParallelHash (node=0x9070f908) at nodeHash.c:312
#7 MultiExecHash (node=node@entry=0x9070f908) at nodeHash.c:112
#8 0x0000000000638f0c in MultiExecProcNode (node=node@entry=0x9070f908) at execProcnode.c:502
#9 0x000000000064d410 in ExecHashJoinImpl (parallel=true, pstate=0x9070feb8) at nodeHashjoin.c:290
#10 ExecParallelHashJoin (pstate=0x9070feb8) at nodeHashjoin.c:600
#11 0x0000000000638824 in ExecProcNodeInstr (node=0x9070feb8) at execProcnode.c:462
#12 0x0000000000641708 in ExecProcNode (node=0x9070feb8) at ../../../src/include/executor/executor.h:248
#13 fetch_input_tuple (aggstate=aggstate@entry=0x9070fb70) at nodeAgg.c:407
#14 0x0000000000642ee8 in agg_retrieve_direct (aggstate=0x9070fb70) at nodeAgg.c:1746
#15 ExecAgg (pstate=0x9070fb70) at nodeAgg.c:1561
#16 0x0000000000638824 in ExecProcNodeInstr (node=0x9070fb70) at execProcnode.c:462
#17 0x00000000006313a4 in ExecProcNode (node=0x9070fb70) at ../../../src/include/executor/executor.h:248
#18 ExecutePlan (execute_once=<optimized out>, dest=0x90665a78, direction=<optimized out>, numberTuples=0, sendTuples=<optimized out>, operation=CMD_SELECT, use_parallel_mode=<optimized out>, planstate=0x9070fb70, estate=0x9070f118)
at execMain.c:1712
#19 standard_ExecutorRun (queryDesc=0x90666e90, direction=<optimized out>, count=0, execute_once=<optimized out>) at execMain.c:353
#20 0x0000000000636434 in ParallelQueryMain (seg=0x90564fb8, toc=0x9d25e000) at execParallel.c:1402
#21 0x0000000000509c58 in ParallelWorkerMain (main_arg=<optimized out>) at parallel.c:1433
#22 0x0000000000705b88 in StartBackgroundWorker () at bgworker.c:911
#23 0x000000000071263c in do_start_bgworker (rw=0x90585500) at postmaster.c:5831
#24 maybe_start_bgworkers () at postmaster.c:6056
#25 0x0000000000712ef8 in sigusr1_handler (postgres_signal_arg=<optimized out>) at postmaster.c:5169
#26 0x00000000891bf37c in handle_signal (actp=actp@entry=0x811ee9c0, sig=sig@entry=30, info=info@entry=0x811eea30, ucp=ucp@entry=0x811eea80) at /usr/src/lib/libthr/thread/thr_sig.c:303
#27 0x00000000891beab8 in thr_sighandler (sig=30, info=0x811eea30, _ucp=0x811eea80) at /usr/src/lib/libthr/thread/thr_sig.c:246
#28 <signal handler called>
#29 _select () at _select.S:4
#30 0x00000000891c1b0c in __thr_select (numfds=4, readfds=0x811eefc0, writefds=0x0, exceptfds=0x0, timeout=0x811ef448) at /usr/src/lib/libthr/thread/thr_syscalls.c:495
#31 0x0000000000713720 in ServerLoop () at postmaster.c:1692
#32 0x00000000007150e8 in PostmasterMain (argc=argc@entry=8, argv=argv@entry=0x811ef998) at postmaster.c:1401
#33 0x000000000047aaf8 in main (argc=8, argv=0x811ef998) at main.c:228
USER CMD PID FD MOUNT INUM MODE SZ|DV R/W
freebsd postgres 29046 text /mnt/data 641445 -rwxr-xr-x 34355232 r
freebsd postgres 29046 wd /mnt/data 642781 drwx------ 1024 r
freebsd postgres 29046 root / 2 drwxr-xr-x 512 r
freebsd postgres 29046 0 /dev 83 crw--w---- pts/2 rw
freebsd postgres 29046 1 /mnt/data 643123 -rw-r--r-- 627618 w
freebsd postgres 29046 2 /mnt/data 643123 -rw-r--r-- 627618 w
freebsd postgres 29046 3 /mnt/data 563678 -rw-r--r-- 4370 w
freebsd postgres 29046 4 /mnt/data 645428 -rw------- 8192 rw
freebsd postgres 29046 5* pipe ffff0000dfd9e110 <-> ffff0000dfd9e268 0 rw
freebsd postgres 29046 6 /mnt/data 646282 -rw------- 8192 rw
freebsd postgres 29046 7* internet6 dgram udp ffffa000c7b58000
freebsd postgres 29046 8* local stream ffffa0000c951800 <-> ffffa000eadae800
freebsd postgres 29046 9* pipe ffff0000e0514110 <-> ffff0000e0514268 0 rw
freebsd postgres 29046 10* pipe ffff0000e0514268 <-> ffff0000e0514110 0 rw
freebsd postgres 29046 11 /mnt/data 643769 -rw------- 24576 rw
freebsd postgres 29046 12 /mnt/data 643968 -rw------- 8192 rw
freebsd postgres 29046 13 /mnt/data 644017 -rw------- 24576 rw
freebsd postgres 29046 14 /mnt/data 643994 -rw------- 8192 rw
freebsd postgres 29046 15 /mnt/data 645469 -rw------- 24576 rw
freebsd postgres 29046 16 /mnt/data 646168 -rw------- 8192 rw
freebsd postgres 29046 17 /mnt/data 643771 -rw------- 24576 rw
freebsd postgres 29046 18 /mnt/data 644000 -rw------- 8192 rw
freebsd postgres 29046 19 /mnt/data 646033 -rw------- 16777216 rw
freebsd postgres 29046 20 /mnt/data 646285 -rw------- 8192 rw
freebsd postgres 29046 21 /mnt/data 643840 -rw------- 24576 rw
freebsd postgres 29046 22 /mnt/data 643991 -rw------- 8192 rw
freebsd postgres 29046 23 /mnt/data 644117 -rw------- 8192 rw
freebsd postgres 29046 24 /mnt/data 644717 -rw------- 2932736 rw
freebsd postgres 29046 25 /mnt/data 645001 -rw------- 245760 rw
freebsd postgres 29046 26 /mnt/data 645003 -rw------- 245760 rw
freebsd postgres 29046 27 /mnt/data 645005 -rw------- 245760 rw
freebsd postgres 29046 28 /mnt/data 645010 -rw------- 245760 rw
freebsd postgres 29046 29 /mnt/data 646284 -rw------- 8192 rw
freebsd postgres 29046 30 /mnt/data 646283 -rw------- 8192 rw
freebsd postgres 29046 31 /mnt/data 644929 -rw------- 24576 rw
freebsd postgres 29046 32 /mnt/data 646280 -rw------- 8192 rw
freebsd postgres 29046 33 /mnt/data 646281 -rw------- 8192 rw
freebsd postgres 29046 34 /mnt/data 645185 -rw------- 8192 rw
freebsd postgres 29046 35 /mnt/data 644099 -rw------- 8192 rw
freebsd postgres 29046 36 /mnt/data 645528 -rw------- 0 rw
freebsd postgres 29046 37 /mnt/data 645546 -rw------- 8192 rw
freebsd postgres 29046 38 /mnt/data 644114 -rw------- 8192 rw
freebsd postgres 29046 39 /mnt/data 643837 -rw------- 0 rw
freebsd postgres 29046 40 /mnt/data 646286 -rw------- 8192 rw
freebsd postgres 29046 41 /mnt/data 646287 -rw------- 8192 rw
freebsd postgres 29046 42 /mnt/data 646288 -rw------- 5464064 rw
freebsd postgres 29046 43 /mnt/data 644045 -rw------- 24576 rw
freebsd postgres 29046 44 /mnt/data 646292 -rw------- 245760 rw
freebsd postgres 29046 45 /mnt/data 646293 -rw------- 8192 rw
freebsd postgres 29046 46 /mnt/data 646294 -rw------- 0 rw
freebsd postgres 29046 47 /mnt/data 646295 -rw------- 8192 rw
freebsd postgres 29046 48 /mnt/data 646296 -rw------- 8192 rw
freebsd postgres 29046 49 /mnt/data 646297 -rw------- 8192 rw
freebsd postgres 29046 50 /mnt/data 646298 -rw------- 8192 rw
freebsd postgres 29046 51 /mnt/data 646299 -rw------- 8192 rw
freebsd postgres 29046 52 /mnt/data 645380 -rw------- 24576 rw
freebsd postgres 29046 53 /mnt/data 644047 -rw------- 24576 rw
freebsd postgres 29046 54 /mnt/data 643996 -rw------- 8192 rw
freebsd postgres 29046 55 /mnt/data 646300 -rw------- 16384 rw
freebsd postgres 29046 56 /mnt/data 646301 -rw------- 8192 rw
freebsd postgres 29046 57 /mnt/data 646302 -rw------- 16384 rw
freebsd postgres 29046 58 /mnt/data 646303 -rw------- 0 rw
freebsd postgres 29046 59 /mnt/data 646304 -rw------- 8192 rw
freebsd postgres 29046 60 /mnt/data 644019 -rw------- 24576 rw
freebsd postgres 29046 61 /mnt/data 643979 -rw------- 8192 rw
freebsd postgres 29046 62 /mnt/data 645894 -rw------- 24576 rw
freebsd postgres 29046 63 /mnt/data 643975 -rw------- 8192 rw
freebsd postgres 29046 64 /mnt/data 645541 -rw------- 24576 rw
freebsd postgres 29046 65 /mnt/data 646305 -rw------- 8192 rw
freebsd postgres 29046 66 /mnt/data 646306 -rw------- 16384 rw
freebsd postgres 29046 67 /mnt/data 646307 -rw------- 8192 rw
freebsd postgres 29046 68 /mnt/data 646308 -rw------- 16384 rw
freebsd postgres 29046 69 /mnt/data 646309 -rw------- 8192 rw
freebsd postgres 29046 70 /mnt/data 646310 -rw------- 16384 rw
freebsd postgres 29046 71 /mnt/data 644134 -rw------- 8192 rw
freebsd postgres 29046 72 /mnt/data 646311 -rw------- 8192 rw
freebsd postgres 29046 73 /mnt/data 646312 -rw------- 16384 rw
freebsd postgres 29046 74 /mnt/data 646313 -rw------- 8192 rw
freebsd postgres 29046 75 /mnt/data 646314 -rw------- 16384 rw
freebsd postgres 29046 76 /mnt/data 643847 -rw------- 425984 rw
freebsd postgres 29046 77 /mnt/data 646315 -rw------- 8192 rw
freebsd postgres 29046 78 /mnt/data 644064 -rw------- 8192 rw
freebsd postgres 29046 79 /mnt/data 644957 -rw------- 24576 rw
freebsd postgres 29046 80 /mnt/data 644716 -rw------- 565248 rw
freebsd postgres 29046 81 /mnt/data 646056 -rw------- 24576 rw
freebsd postgres 29046 82 /mnt/data 644992 -rw------- 40960 rw
freebsd postgres 29046 83 /mnt/data 644994 -rw------- 40960 rw
freebsd postgres 29046 84 /mnt/data 644995 -rw------- 40960 rw
freebsd postgres 29046 85 /mnt/data 644997 -rw------- 40960 rw
freebsd postgres 29046 86 /mnt/data 646099 -rw------- 8192 rw
freebsd postgres 29046 87 /mnt/data 645464 -rw------- 8192 rw
freebsd postgres 29046 88 /mnt/data 645418 -rw------- 327680 rw
freebsd postgres 29046 89 /mnt/data 643767 -rw------- 24576 rw
freebsd postgres 29046 90 /mnt/data 643980 -rw------- 8192 rw
freebsd postgres 29046 91 /mnt/data 645459 -rw------- 49152 rw
freebsd postgres 29046 92 /mnt/data 645460 -rw------- 98304 rw
freebsd postgres 29046 93 /mnt/data 645461 -rw------- 57344 rw
freebsd postgres 29046 94 /mnt/data 646143 -rw------- 1368064 rw
freebsd postgres 29046 95 /mnt/data 646357 -rw------- 16384 rw
freebsd postgres 29046 96 /mnt/data 646348 -rw------- 1368064 rw
freebsd postgres 29046 97 /mnt/data 646351 -rw------- 1368064 rw
USER CMD PID FD MOUNT INUM MODE SZ|DV R/W
freebsd postgres 29080 text /mnt/data 641445 -rwxr-xr-x 34355232 r
freebsd postgres 29080 wd /mnt/data 642781 drwx------ 1024 r
freebsd postgres 29080 root / 2 drwxr-xr-x 512 r
freebsd postgres 29080 0 /dev 83 crw--w---- pts/2 rw
freebsd postgres 29080 1 /mnt/data 643123 -rw-r--r-- 627618 w
freebsd postgres 29080 2 /mnt/data 643123 -rw-r--r-- 627618 w
freebsd postgres 29080 3 /mnt/data 563678 -rw-r--r-- 4370 w
freebsd postgres 29080 4 /mnt/data 645418 -rw------- 327680 rw
freebsd postgres 29080 5* pipe ffff0000dfd9e110 <-> ffff0000dfd9e268 0 rw
freebsd postgres 29080 6 /mnt/data 645428 -rw------- 8192 rw
freebsd postgres 29080 7* internet6 dgram udp ffffa000c7b58000
freebsd postgres 29080 8* pipe ffff0000e01f82d8 <-> ffff0000e01f8430 0 rw
freebsd postgres 29080 9* pipe ffff0000e01f8430 <-> ffff0000e01f82d8 0 rw
USER CMD PID FD MOUNT INUM MODE SZ|DV R/W
freebsd postgres 29081 text /mnt/data 641445 -rwxr-xr-x 34355232 r
freebsd postgres 29081 wd /mnt/data 642781 drwx------ 1024 r
freebsd postgres 29081 root / 2 drwxr-xr-x 512 r
freebsd postgres 29081 0 /dev 83 crw--w---- pts/2 rw
freebsd postgres 29081 1 /mnt/data 643123 -rw-r--r-- 627618 w
freebsd postgres 29081 2 /mnt/data 643123 -rw-r--r-- 627618 w
freebsd postgres 29081 3 /mnt/data 563678 -rw-r--r-- 4370 w
freebsd postgres 29081 4 /mnt/data 645418 -rw------- 327680 rw
freebsd postgres 29081 5* pipe ffff0000dfd9e110 <-> ffff0000dfd9e268 0 rw
freebsd postgres 29081 6 /mnt/data 645428 -rw------- 8192 rw
freebsd postgres 29081 7* internet6 dgram udp ffffa000c7b58000
freebsd postgres 29081 8* pipe ffff0000e00d5888 <-> ffff0000e00d59e0 0 rw
freebsd postgres 29081 9* pipe ffff0000e00d59e0 <-> ffff0000e00d5888 0 rw
PID TID COMM SIG FLAGS
29046 100597 postgres HUP --
29046 100597 postgres INT --
29046 100597 postgres QUIT --
29046 100597 postgres ILL --
29046 100597 postgres TRAP --
29046 100597 postgres ABRT --
29046 100597 postgres EMT --
29046 100597 postgres FPE --
29046 100597 postgres KILL --
29046 100597 postgres BUS --
29046 100597 postgres SEGV --
29046 100597 postgres SYS --
29046 100597 postgres PIPE --
29046 100597 postgres ALRM --
29046 100597 postgres TERM --
29046 100597 postgres URG --
29046 100597 postgres STOP --
29046 100597 postgres TSTP --
29046 100597 postgres CONT --
29046 100597 postgres CHLD --
29046 100597 postgres TTIN --
29046 100597 postgres TTOU --
29046 100597 postgres IO --
29046 100597 postgres XCPU --
29046 100597 postgres XFSZ --
29046 100597 postgres VTALRM --
29046 100597 postgres PROF --
29046 100597 postgres WINCH --
29046 100597 postgres INFO --
29046 100597 postgres USR1 --
29046 100597 postgres USR2 --
29046 100597 postgres 32 --
29046 100597 postgres 33 --
29046 100597 postgres 34 --
29046 100597 postgres 35 --
29046 100597 postgres 36 --
29046 100597 postgres 37 --
29046 100597 postgres 38 --
29046 100597 postgres 39 --
29046 100597 postgres 40 --
29046 100597 postgres 41 --
29046 100597 postgres 42 --
29046 100597 postgres 43 --
29046 100597 postgres 44 --
29046 100597 postgres 45 --
29046 100597 postgres 46 --
29046 100597 postgres 47 --
29046 100597 postgres 48 --
29046 100597 postgres 49 --
29046 100597 postgres 50 --
29046 100597 postgres 51 --
29046 100597 postgres 52 --
29046 100597 postgres 53 --
29046 100597 postgres 54 --
29046 100597 postgres 55 --
29046 100597 postgres 56 --
29046 100597 postgres 57 --
29046 100597 postgres 58 --
29046 100597 postgres 59 --
29046 100597 postgres 60 --
29046 100597 postgres 61 --
29046 100597 postgres 62 --
29046 100597 postgres 63 --
29046 100597 postgres 64 --
29046 100597 postgres 65 --
29046 100597 postgres 66 --
29046 100597 postgres 67 --
29046 100597 postgres 68 --
29046 100597 postgres 69 --
29046 100597 postgres 70 --
29046 100597 postgres 71 --
29046 100597 postgres 72 --
29046 100597 postgres 73 --
29046 100597 postgres 74 --
29046 100597 postgres 75 --
29046 100597 postgres 76 --
29046 100597 postgres 77 --
29046 100597 postgres 78 --
29046 100597 postgres 79 --
29046 100597 postgres 80 --
29046 100597 postgres 81 --
29046 100597 postgres 82 --
29046 100597 postgres 83 --
29046 100597 postgres 84 --
29046 100597 postgres 85 --
29046 100597 postgres 86 --
29046 100597 postgres 87 --
29046 100597 postgres 88 --
29046 100597 postgres 89 --
29046 100597 postgres 90 --
29046 100597 postgres 91 --
29046 100597 postgres 92 --
29046 100597 postgres 93 --
29046 100597 postgres 94 --
29046 100597 postgres 95 --
29046 100597 postgres 96 --
29046 100597 postgres 97 --
29046 100597 postgres 98 --
29046 100597 postgres 99 --
29046 100597 postgres 100 --
29046 100597 postgres 101 --
29046 100597 postgres 102 --
29046 100597 postgres 103 --
29046 100597 postgres 104 --
29046 100597 postgres 105 --
29046 100597 postgres 106 --
29046 100597 postgres 107 --
29046 100597 postgres 108 --
29046 100597 postgres 109 --
29046 100597 postgres 110 --
29046 100597 postgres 111 --
29046 100597 postgres 112 --
29046 100597 postgres 113 --
29046 100597 postgres 114 --
29046 100597 postgres 115 --
29046 100597 postgres 116 --
29046 100597 postgres 117 --
29046 100597 postgres 118 --
29046 100597 postgres 119 --
29046 100597 postgres 120 --
29046 100597 postgres 121 --
29046 100597 postgres 122 --
29046 100597 postgres 123 --
29046 100597 postgres 124 --
29046 100597 postgres 125 --
29046 100597 postgres 126 --
29046 100597 postgres 127 --
29046 100597 postgres 128 --
PID TID COMM SIG FLAGS
29080 100395 postgres HUP --
29080 100395 postgres INT --
29080 100395 postgres QUIT --
29080 100395 postgres ILL --
29080 100395 postgres TRAP --
29080 100395 postgres ABRT --
29080 100395 postgres EMT --
29080 100395 postgres FPE --
29080 100395 postgres KILL --
29080 100395 postgres BUS --
29080 100395 postgres SEGV --
29080 100395 postgres SYS --
29080 100395 postgres PIPE --
29080 100395 postgres ALRM --
29080 100395 postgres TERM --
29080 100395 postgres URG --
29080 100395 postgres STOP --
29080 100395 postgres TSTP --
29080 100395 postgres CONT --
29080 100395 postgres CHLD --
29080 100395 postgres TTIN --
29080 100395 postgres TTOU --
29080 100395 postgres IO --
29080 100395 postgres XCPU --
29080 100395 postgres XFSZ --
29080 100395 postgres VTALRM --
29080 100395 postgres PROF --
29080 100395 postgres WINCH --
29080 100395 postgres INFO --
29080 100395 postgres USR1 --
29080 100395 postgres USR2 --
29080 100395 postgres 32 --
29080 100395 postgres 33 --
29080 100395 postgres 34 --
29080 100395 postgres 35 --
29080 100395 postgres 36 --
29080 100395 postgres 37 --
29080 100395 postgres 38 --
29080 100395 postgres 39 --
29080 100395 postgres 40 --
29080 100395 postgres 41 --
29080 100395 postgres 42 --
29080 100395 postgres 43 --
29080 100395 postgres 44 --
29080 100395 postgres 45 --
29080 100395 postgres 46 --
29080 100395 postgres 47 --
29080 100395 postgres 48 --
29080 100395 postgres 49 --
29080 100395 postgres 50 --
29080 100395 postgres 51 --
29080 100395 postgres 52 --
29080 100395 postgres 53 --
29080 100395 postgres 54 --
29080 100395 postgres 55 --
29080 100395 postgres 56 --
29080 100395 postgres 57 --
29080 100395 postgres 58 --
29080 100395 postgres 59 --
29080 100395 postgres 60 --
29080 100395 postgres 61 --
29080 100395 postgres 62 --
29080 100395 postgres 63 --
29080 100395 postgres 64 --
29080 100395 postgres 65 --
29080 100395 postgres 66 --
29080 100395 postgres 67 --
29080 100395 postgres 68 --
29080 100395 postgres 69 --
29080 100395 postgres 70 --
29080 100395 postgres 71 --
29080 100395 postgres 72 --
29080 100395 postgres 73 --
29080 100395 postgres 74 --
29080 100395 postgres 75 --
29080 100395 postgres 76 --
29080 100395 postgres 77 --
29080 100395 postgres 78 --
29080 100395 postgres 79 --
29080 100395 postgres 80 --
29080 100395 postgres 81 --
29080 100395 postgres 82 --
29080 100395 postgres 83 --
29080 100395 postgres 84 --
29080 100395 postgres 85 --
29080 100395 postgres 86 --
29080 100395 postgres 87 --
29080 100395 postgres 88 --
29080 100395 postgres 89 --
29080 100395 postgres 90 --
29080 100395 postgres 91 --
29080 100395 postgres 92 --
29080 100395 postgres 93 --
29080 100395 postgres 94 --
29080 100395 postgres 95 --
29080 100395 postgres 96 --
29080 100395 postgres 97 --
29080 100395 postgres 98 --
29080 100395 postgres 99 --
29080 100395 postgres 100 --
29080 100395 postgres 101 --
29080 100395 postgres 102 --
29080 100395 postgres 103 --
29080 100395 postgres 104 --
29080 100395 postgres 105 --
29080 100395 postgres 106 --
29080 100395 postgres 107 --
29080 100395 postgres 108 --
29080 100395 postgres 109 --
29080 100395 postgres 110 --
29080 100395 postgres 111 --
29080 100395 postgres 112 --
29080 100395 postgres 113 --
29080 100395 postgres 114 --
29080 100395 postgres 115 --
29080 100395 postgres 116 --
29080 100395 postgres 117 --
29080 100395 postgres 118 --
29080 100395 postgres 119 --
29080 100395 postgres 120 --
29080 100395 postgres 121 --
29080 100395 postgres 122 --
29080 100395 postgres 123 --
29080 100395 postgres 124 --
29080 100395 postgres 125 --
29080 100395 postgres 126 --
29080 100395 postgres 127 --
29080 100395 postgres 128 --
PID TID COMM SIG FLAGS
29081 100091 postgres HUP --
29081 100091 postgres INT --
29081 100091 postgres QUIT --
29081 100091 postgres ILL --
29081 100091 postgres TRAP --
29081 100091 postgres ABRT --
29081 100091 postgres EMT --
29081 100091 postgres FPE --
29081 100091 postgres KILL --
29081 100091 postgres BUS --
29081 100091 postgres SEGV --
29081 100091 postgres SYS --
29081 100091 postgres PIPE --
29081 100091 postgres ALRM --
29081 100091 postgres TERM --
29081 100091 postgres URG --
29081 100091 postgres STOP --
29081 100091 postgres TSTP --
29081 100091 postgres CONT --
29081 100091 postgres CHLD --
29081 100091 postgres TTIN --
29081 100091 postgres TTOU --
29081 100091 postgres IO --
29081 100091 postgres XCPU --
29081 100091 postgres XFSZ --
29081 100091 postgres VTALRM --
29081 100091 postgres PROF --
29081 100091 postgres WINCH --
29081 100091 postgres INFO --
29081 100091 postgres USR1 --
29081 100091 postgres USR2 --
29081 100091 postgres 32 --
29081 100091 postgres 33 --
29081 100091 postgres 34 --
29081 100091 postgres 35 --
29081 100091 postgres 36 --
29081 100091 postgres 37 --
29081 100091 postgres 38 --
29081 100091 postgres 39 --
29081 100091 postgres 40 --
29081 100091 postgres 41 --
29081 100091 postgres 42 --
29081 100091 postgres 43 --
29081 100091 postgres 44 --
29081 100091 postgres 45 --
29081 100091 postgres 46 --
29081 100091 postgres 47 --
29081 100091 postgres 48 --
29081 100091 postgres 49 --
29081 100091 postgres 50 --
29081 100091 postgres 51 --
29081 100091 postgres 52 --
29081 100091 postgres 53 --
29081 100091 postgres 54 --
29081 100091 postgres 55 --
29081 100091 postgres 56 --
29081 100091 postgres 57 --
29081 100091 postgres 58 --
29081 100091 postgres 59 --
29081 100091 postgres 60 --
29081 100091 postgres 61 --
29081 100091 postgres 62 --
29081 100091 postgres 63 --
29081 100091 postgres 64 --
29081 100091 postgres 65 --
29081 100091 postgres 66 --
29081 100091 postgres 67 --
29081 100091 postgres 68 --
29081 100091 postgres 69 --
29081 100091 postgres 70 --
29081 100091 postgres 71 --
29081 100091 postgres 72 --
29081 100091 postgres 73 --
29081 100091 postgres 74 --
29081 100091 postgres 75 --
29081 100091 postgres 76 --
29081 100091 postgres 77 --
29081 100091 postgres 78 --
29081 100091 postgres 79 --
29081 100091 postgres 80 --
29081 100091 postgres 81 --
29081 100091 postgres 82 --
29081 100091 postgres 83 --
29081 100091 postgres 84 --
29081 100091 postgres 85 --
29081 100091 postgres 86 --
29081 100091 postgres 87 --
29081 100091 postgres 88 --
29081 100091 postgres 89 --
29081 100091 postgres 90 --
29081 100091 postgres 91 --
29081 100091 postgres 92 --
29081 100091 postgres 93 --
29081 100091 postgres 94 --
29081 100091 postgres 95 --
29081 100091 postgres 96 --
29081 100091 postgres 97 --
29081 100091 postgres 98 --
29081 100091 postgres 99 --
29081 100091 postgres 100 --
29081 100091 postgres 101 --
29081 100091 postgres 102 --
29081 100091 postgres 103 --
29081 100091 postgres 104 --
29081 100091 postgres 105 --
29081 100091 postgres 106 --
29081 100091 postgres 107 --
29081 100091 postgres 108 --
29081 100091 postgres 109 --
29081 100091 postgres 110 --
29081 100091 postgres 111 --
29081 100091 postgres 112 --
29081 100091 postgres 113 --
29081 100091 postgres 114 --
29081 100091 postgres 115 --
29081 100091 postgres 116 --
29081 100091 postgres 117 --
29081 100091 postgres 118 --
29081 100091 postgres 119 --
29081 100091 postgres 120 --
29081 100091 postgres 121 --
29081 100091 postgres 122 --
29081 100091 postgres 123 --
29081 100091 postgres 124 --
29081 100091 postgres 125 --
29081 100091 postgres 126 --
29081 100091 postgres 127 --
29081 100091 postgres 128 --
PID TT STAT TIME COMMAND
0 - DLs 1:11.79 [kernel]
1 - ILs 0:01.04 /sbin/init
2 - WL 4:06.46 [clock]
3 - DL 0:00.00 [crypto]
4 - DL 20:45.01 [cam]
5 - DL 0:00.00 [busdma]
6 - DL 1:20.46 [rand_harvestq]
7 - DL 0:16.57 [mmcsd0: mmc/sd card]
8 - DL 2:50.05 [pagedaemon]
9 - DL 0:00.00 [vmdaemon]
10 - DL 0:00.00 [audit]
11 - RNL 73786:51.82 [idle]
12 - WL 43:35.84 [intr]
13 - DL 0:12.26 [geom]
14 - DL 0:00.00 [sequencer 00]
15 - DL 89:12.00 [usb]
16 - DL 1:14.46 [bufdaemon]
17 - DL 8:50.29 [syncer]
18 - DL 0:11.40 [vnlru]
958 - Ss 0:00.86 /sbin/devd
1177 - Ss 0:05.99 /usr/sbin/syslogd -s
1211 - Ss 5:09.27 /usr/sbin/powerd
1235 - Ss 1:42.54 /usr/local/sbin/ntpd -p /var/db/ntp/ntpd.pid -c /etc/ntp.conf -f /var/db/ntp/ntpd.drift
1282 - Ss 0:04.56 sshd: /usr/sbin/sshd [listener] 0 of 10-100 startups (sshd)
1286 - Is 0:10.88 /usr/sbin/cron -s
1314 - Is 0:00.55 dhclient: system.syslog (dhclient)
1317 - Is 0:02.02 dhclient: genet0 [priv] (dhclient)
1399 - ICs 0:00.82 dhclient: genet0 (dhclient)
4512 - Is 0:00.06 screen (screen-4.9.0)
28765 - Is 0:00.85 postgres: checkpointer (postgres)
28766 - Ss 0:00.45 postgres: background writer (postgres)
28767 - Ss 0:00.54 postgres: walwriter (postgres)
28768 - Ss 0:02.33 postgres: autovacuum launcher (postgres)
28769 - Ss 0:08.92 postgres: stats collector (postgres)
28770 - Is 0:00.07 postgres: logical replication launcher (postgres)
29046 - Is 0:02.80 postgres: freebsd regression [local] SELECT (postgres)
29080 - Is 0:00.03 postgres: parallel worker for PID 29046 (postgres)
29081 - Is 0:00.03 postgres: parallel worker for PID 29046 (postgres)
32174 - Is 0:00.10 sshd: freebsd [priv] (sshd)
32176 - S 0:00.11 sshd: freebsd@pts/0 (sshd)
32209 - Ss 0:00.10 sshd: freebsd [priv] (sshd)
32211 - S 0:00.02 sshd: freebsd@pts/1 (sshd)
1300 u0 Is+ 0:00.06 /usr/libexec/getty 3wire ttyu0
32177 0 Is 0:00.10 -bash (bash)
32202 0 IC+ 0:00.01 tail -f src/test/regress/results/join.out
32205 0 I+ 0:00.00 tail: system.fileargs (tail)
32212 1 Ss 0:00.03 -bash (bash)
32218 1 R+ 0:00.01 ps ax
4513 2 Is 0:00.15 -/usr/local/bin/bash
27504 2 I+ 0:00.02 make check
27506 2 I+ 0:00.05 /usr/local/bin/gmake check
28747 2 I+ 0:00.02 /usr/local/bin/gmake -C src/test/regress check
28753 2 I+ 0:00.12 ../../../src/test/regress/pg_regress --temp-instance=./tmp_check --inputdir=. --bindir= --dlpath=. --max-concurrent-tests=20 --schedule=./parallel_schedule
28762 2 S+ 0:11.84 postgres -D /mnt/data/postgres/src/test/regress/./tmp_check/data -F -c listen_addresses= -k /tmp/pg_regress-c4bdnU
29045 2 I+ 0:00.17 psql -X -a -q -d regression
