Hello,
30.05.2024 18:00, Alexander Lakhin wrote:
While reviewing recent buildfarm failures, I came across this one:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=crake&dt=2024-05-23%2004%3A11%3A03
upgrade.crake/REL_16_STABLE/REL9_5_STABLE-ctl4.log
waiting for server to shut
down...........................................................................................................................
failed
pg_ctl: server does not shut down
I've grepped through logs of the last 167
xversion-upgrade-REL9_5_STABLE-REL_16_STABLE/*ctl4.log on crake and got
the following results:
waiting for server to shut down........ done
waiting for server to shut down............................... done
waiting for server to shut down.............. done
waiting for server to shut down........ done
waiting for server to shut down....................... done
waiting for server to shut down..... done
waiting for server to shut down...... done
waiting for server to shut down............................... done
waiting for server to shut down.... done
waiting for server to shut down....................................... done
waiting for server to shut down....................... done
waiting for server to shut down............................................ done
waiting for server to shut down..... done
waiting for server to shut down....................... done
waiting for server to shut down.... done
waiting for server to shut down..... done
waiting for server to shut down................... done
waiting for server to shut down...................................... done
waiting for server to shut down........ done
waiting for server to shut down................ done
waiting for server to shut down............ done
waiting for server to shut down........... done
waiting for server to shut down........ done
waiting for server to shut down.... done
waiting for server to shut
down.................................................. done
waiting for server to shut
down........................................................... done
waiting for server to shut down..... done
waiting for server to shut down.............. done
waiting for server to shut down......... done
waiting for server to shut
down........................................................... done
waiting for server to shut down................ done
waiting for server to shut down.................................... done
waiting for server to shut down........ done
waiting for server to shut down............ done
waiting for server to shut
down................................................................... done
waiting for server to shut down..... done
waiting for server to shut down..............................................
done
waiting for server to shut down............. done
waiting for server to shut down............. done
waiting for server to shut down..... done
waiting for server to shut down..............................................
done
waiting for server to shut down...... done
waiting for server to shut down....... done
waiting for server to shut down....... done
waiting for server to shut down.......................... done
waiting for server to shut down............ done
waiting for server to shut down.................... done
waiting for server to shut down..... done
waiting for server to shut down........ done
waiting for server to shut down....... done
waiting for server to shut down....................... done
waiting for server to shut down........... done
waiting for server to shut down.............. done
waiting for server to shut down.................... done
waiting for server to shut down............. done
waiting for server to shut down...................................... done
waiting for server to shut down...............................................
done
waiting for server to shut down........................................ done
waiting for server to shut down.......... done
waiting for server to shut down......................................... done
waiting for server to shut down......................................... done
waiting for server to shut down........ done
waiting for server to shut down..... done
waiting for server to shut
down................................................................... done
waiting for server to shut down............................. done
waiting for server to shut down.......... done
waiting for server to shut down..... done
waiting for server to shut down......................... done
waiting for server to shut down...... done
waiting for server to shut down..... done
waiting for server to shut down......... done
waiting for server to shut
down.......................................................... done
waiting for server to shut down............. done
waiting for server to shut down..... done
waiting for server to shut down........ done
waiting for server to shut down........................................... done
waiting for server to shut down.... done
waiting for server to shut down..... done
waiting for server to shut down...... done
waiting for server to shut down............. done
waiting for server to shut
down.......................................................... done
waiting for server to shut down........................... done
waiting for server to shut down..... done
waiting for server to shut down....... done
waiting for server to shut down...... done
waiting for server to shut down......................................... done
waiting for server to shut down.......................... done
waiting for server to shut down.......................... done
waiting for server to shut down................................... done
waiting for server to shut down..... done
waiting for server to shut down.................. done
waiting for server to shut down....................... done
waiting for server to shut down................. done
waiting for server to shut down........................................... done
waiting for server to shut down........ done
waiting for server to shut
down...................................................................... done
waiting for server to shut down............. done
waiting for server to shut
down.................................................................... done
waiting for server to shut down................... done
waiting for server to shut down.............. done
waiting for server to shut down..................... done
waiting for server to shut
down.................................................................................................... done
waiting for server to shut down..... done
waiting for server to shut down....................... done
waiting for server to shut down................................ done
waiting for server to shut down....... done
waiting for server to shut down....... done
waiting for server to shut down.............. done
waiting for server to shut down................................................
done
waiting for server to shut down...... done
waiting for server to shut down..................... done
waiting for server to shut down...... done
waiting for server to shut down........ done
waiting for server to shut down..... done
waiting for server to shut down.... done
waiting for server to shut down......... done
waiting for server to shut down...... done
waiting for server to shut down........... done
waiting for server to shut down...................... done
waiting for server to shut down............ done
waiting for server to shut down................. done
waiting for server to shut down.... done
waiting for server to shut down................................... done
waiting for server to shut down.............................. done
waiting for server to shut
down.......................................................................................
done
waiting for server to shut
down................................................................................................... done
waiting for server to shut down...... done
waiting for server to shut down.... done
waiting for server to shut down........... done
waiting for server to shut down..... done
waiting for server to shut down......... done
waiting for server to shut down.................... done
waiting for server to shut down..... done
waiting for server to shut down....................... done
waiting for server to shut down................. done
waiting for server to shut down........................................ done
waiting for server to shut down...... done
waiting for server to shut down.... done
waiting for server to shut down.................................... done
waiting for server to shut down.............................. done
waiting for server to shut down.................................. done
waiting for server to shut
down.............................................................. done
waiting for server to shut
down.................................................. done
waiting for server to shut
down............................................................. done
waiting for server to shut down............ done
waiting for server to shut down............ done
waiting for server to shut down...................................... done
waiting for server to shut down.......................................... done
waiting for server to shut down................................. done
waiting for server to shut down................... done
waiting for server to shut
down...........................................................................................................................
failed
waiting for server to shut
down................................................. done
waiting for server to shut down............ done
waiting for server to shut
down................................................................... done
waiting for server to shut down......... done
waiting for server to shut down......... done
waiting for server to shut down.................... done
waiting for server to shut down........ done
waiting for server to shut down............................... done
waiting for server to shut down......... done
waiting for server to shut
down......................................................... done
waiting for server to shut
down................................................... done
waiting for server to shut down............ done
waiting for server to shut down...... done
waiting for server to shut down........................................... done
waiting for server to shut
down.......................................................................................... done
waiting for server to shut
down...............................................................................
done
Thus, pg_ctl stopped waiting after 120 seconds timeout, but we can see
"allowed" duration around 100 seconds.
A similar failure have occurred today:
https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=opaleye&dt=2024-06-08%2001%3A41%3A41
waiting for server to shut
down...........................................................................................................................
failed
pg_ctl: server does not shut down
and the last StopDb-C:4 runs on opaleye show the following timings:
waiting for server to shut down............. done
waiting for server to shut down......................... done
waiting for server to shut down......................... done
waiting for server to shut down.......................... done
waiting for server to shut down.............. done
waiting for server to shut down...................... done
waiting for server to shut down................................. done
waiting for server to shut down........................... done
waiting for server to shut down............ done
waiting for server to shut down................ done
waiting for server to shut down.................. done
waiting for server to shut down............................... done
waiting for server to shut down............................ done
waiting for server to shut down.................. done
waiting for server to shut down..................... done
waiting for server to shut down........................ done
waiting for server to shut down.......................... done
waiting for server to shut down......................................... done
waiting for server to shut down.................... done
waiting for server to shut down..................................... done
waiting for server to shut down.............................................
done
waiting for server to shut down........................... done
waiting for server to shut down................... done
waiting for server to shut down............. done
waiting for server to shut down........................ done
waiting for server to shut down......................... done
waiting for server to shut down................... done
waiting for server to shut down................. done
waiting for server to shut down..................... done
waiting for server to shut down......................... done
waiting for server to shut down............................... done
waiting for server to shut
down.................................................. done
waiting for server to shut down............... done
waiting for server to shut down................. done
waiting for server to shut down..................... done
waiting for server to shut down................ done
waiting for server to shut down.............. done
waiting for server to shut down................ done
waiting for server to shut down.......................... done
waiting for server to shut down.................. done
waiting for server to shut down.................... done
waiting for server to shut down................ done
waiting for server to shut down....................... done
waiting for server to shut down................ done
waiting for server to shut down...................... done
waiting for server to shut down............... done
waiting for server to shut down.............. done
waiting for server to shut down........................... done
waiting for server to shut down............. done
waiting for server to shut down..................... done
waiting for server to shut down................. done
waiting for server to shut down........................... done
waiting for server to shut down..................... done
waiting for server to shut down................ done
waiting for server to shut down........................................... done
waiting for server to shut down....................... done
waiting for server to shut
down..............................................................................
done
waiting for server to shut
down...........................................................................................................................
failed
So maybe it would make sense to increase default PGCTLTIMEOUT for
buildfarm clients, say, to 180 seconds?
Best regards,
Alexander
