> Check with pfctl -si if you reach a limit Thanks, will do.
Marc Peters also suggested to check pf state limit, upon digging into that I found https://serverascode.com/2011/09/12/openbsd-pf-set-limit-states.html and therefore added set limit states 200000 to pf.conf.
