Hi Daniel,
Forgot to send my email (was hungry ;), so we crossed on wire. Thanks for looking at it. IMO, it's clearly easier to increase the DB memory limit, as
least for now.
Looking at https://montastic.com/checkpoints I confirm that all is OK now.
BTW we need to update tools\demo-backup\ofbizdocker\README.md
The VM is no longer ofbiz-vm1.apache.org, it's now ofbiz-ec2-or.apache.org
Also not sure why, I was unable to sudo as ofbizdocker using the same than last
Challenge password.
Thanks and cheers
Jacques
Le 17/10/2026 à 19:57, Daniel Watford a écrit :
Summary: Postgres database exceeded its 300 MB memory limit, causing the kernel to kill the process. Upon restart, the database failed to recover
from corruption.
Resolved by: Tearing down and restarting demos. This caused the databases to
reset.
To consider: Perhaps the 300 MB limit for the database process is too low? Or
perhaps some run-away process in object caused excessive database use?
Hi Jacques,
I took a look at the standard output and standard error logs for ofbiz-trunk
and found there was a huge amount, suggesting some sort of cycle.
Filtering the logs for the Postgres database, I used the command:
docker compose logs db | less
Extract from the logs:
db-1 | 2026-10-17 02:35:56.952 UTC [64] LOG: database system was shut down at
2026-10-17 02:35:56 UTC
db-1 | 2026-10-17 02:35:56.962 UTC [1] LOG: database system is ready to
accept connections
db-1 | 2026-10-17 11:47:24.253 UTC [1] LOG: server process (PID 1747) was
terminated by signal 9: Killed
db-1 | 2026-10-17 11:47:24.253 UTC [1] DETAIL: Failed process was running: SELECT PTYACCPREF.PARTY_ID, PTYACCPREF.BASE_CURRENCY_UOM_ID,
PTYGROUP.GROUP_NAME, PTYROLE.ROLE_TYPE_ID FROM (public.PARTY_ACCTG_PREFERENCE PTYACCPREF INNER JOIN public.PARTY_GROUP PTYGROUP ON
PTYACCPREF.PARTY_ID = PTYGROUP.PARTY_ID) INNER JOIN public.PARTY_ROLE PTYROLE ON PTYACCPREF.PARTY_ID = PTYROLE.PARTY_ID WHERE
(((PTYROLE.ROLE_TYPE_ID = $1)))
db-1 | 2026-10-17 11:47:24.253 UTC [1] LOG: terminating any other active
server processes
db-1 | 2026-10-17 11:47:24.253 UTC [2369] WARNING: terminating connection
because of crash of another server process at character 245
db-1 | 2026-10-17 11:47:24.253 UTC [2369] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit,
because another server process exited abnormally and possibly corrupted shared memory.
db-1 | 2026-10-17 11:47:24.253 UTC [2369] HINT: In a moment you should be
able to reconnect to the database and repeat your command.
db-1 | 2026-10-17 11:47:24.253 UTC [2357] WARNING: terminating connection
because of crash of another server process
db-1 | 2026-10-17 11:47:24.253 UTC [2357] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit,
because another server process exited abnormally and possibly corrupted shared memory.
It appears that the database was killed at 11:47:24.253 UTC. The question is
why....
Searching in /var/log/syslog for 2026-10-17T11:47 we find:
2026-10-17T11:47:24.235788+00:00 ofbiz-ec2-or kernel: postgres invoked
oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
2026-10-17T11:47:24.235808+00:00 ofbiz-ec2-or kernel: CPU: 2 UID: 100999 PID: 1648549 Comm: postgres Not tainted 7.0.0-1014-aws #14~24.04.1-Ubuntu
PREEMPTLAZY
2026-10-17T11:47:24.235813+00:00 ofbiz-ec2-or kernel: Hardware name: Xen HVM
domU, BIOS 4.11.amazon 08/24/2006
...
...
2026-10-17T11:47:24.239755+00:00 ofbiz-ec2-or kernel: memory: usage 307200kB,
limit 307200kB, failcnt 1879
2026-10-17T11:47:24.239771+00:00 ofbiz-ec2-or kernel: swap: usage 0kB, limit
0kB, failcnt 0
2026-10-17T11:47:24.239773+00:00 ofbiz-ec2-or kernel: Memory cgroup stats for
/system.slice/docker-59d2f448671fcb358419dfe26eced00b69cebe3e7edfbc9e16f3fb3a0f0746b9.scope:
In the syslog snippet above we see mention of the 'oom-killer'. This is the
kernel killing processes when Out Of Memory.
The log shows that the process was using 307200KB (approx 300MB) which was at
the limit, triggering the process to be killed.
I have run the /home/ofbizdocker/pull-and-restart.sh script as a way to tear down and restart the demos. The demos appear to have restarted and are
operational again.
If we see this type of problem occuring again we will need to decide if the DB memory limit is too low, or whether we should be investigating some
other aspect of OFBiz.
Thanks
Dan.
On Sat, 17 Oct 2026 at 18:34, Daniel Watford <[email protected]> wrote:
Hi Jaques,
I'm also taking a look. There appear to be a huge number of logs. So many
that it is difficult to find the relevant text.
There doesn't appear to be any issues with lack of disk space.
I'll investigate by shutting the demos down and then bringing them up one
by one, checking logs as I go.
Thanks
Dan.
On Sat, 17 Oct 2026 at 17:31, Jacques Le Roux via dev
<[email protected]> wrote:
Hi,
Between 16:45 UTC and "now", OFBiz demos are slow (or even very slow)
or down.
Trunk is by far the much affected, which is weird... I'll have a look...
Jacques
--
Daniel Watford
--
Daniel Watford
