Re: pf, send(2) and EACCES
On Fri, 28 Aug 2020 22:33:30 +0200, Claudio Jeker wrote: > Have a look at the pf(4) stats. especially check if the congestion > counter increases when you see the error. If pf(4) detects a network > congestion then ruleset evaluation is skipped and only state matching > happens. In that case you can get EACCESS for connections that would > normally be allowed by pf(4). Thanks, I'll take a look at `systat pf` if it happens again. Daniel
Re: pf, send(2) and EACCES
On Fri, Aug 28, 2020 at 11:40:17AM -0400, Daniel Jakots wrote: > On Fri, 28 Aug 2020 16:06:48 +0200, Sebastien Marie > wrote: > > > - generate lot of postgresql access. from postgresql thread, the > > statement seems to be a SELECT, so it would be fine to ran in loop > > (hopping no cache and real traffic generated). > > > > - run pfctl -Treplace in a loop (with a set of different files as the > > kernel code takes care if host are added, changed, deleted) > > I ran the select on one machine and the pfctl -Treplace on db1 both in > a `while :` for about two hours and it didn't happen. > > I'll try again if the problem happens genuinely again. Have a look at the pf(4) stats. especially check if the congestion counter increases when you see the error. If pf(4) detects a network congestion then ruleset evaluation is skipped and only state matching happens. In that case you can get EACCESS for connections that would normally be allowed by pf(4). -- :wq Claudio
Re: pf, send(2) and EACCES
On Fri, 28 Aug 2020 16:06:48 +0200, Sebastien Marie wrote: > - generate lot of postgresql access. from postgresql thread, the > statement seems to be a SELECT, so it would be fine to ran in loop > (hopping no cache and real traffic generated). > > - run pfctl -Treplace in a loop (with a set of different files as the > kernel code takes care if host are added, changed, deleted) I ran the select on one machine and the pfctl -Treplace on db1 both in a `while :` for about two hours and it didn't happen. I'll try again if the problem happens genuinely again. Thanks, Daniel
Re: pf, send(2) and EACCES
On Fri, Aug 28, 2020 at 09:27:10AM -0400, Daniel Jakots wrote: > On Fri, 28 Aug 2020 08:32:59 +0200, Sebastien Marie > wrote: > > > On Thu, Aug 27, 2020 at 03:27:58PM -0400, Daniel Jakots wrote: > > > Hi, > > > > > > I'm chasing a weird behavior with postgresql. Sometimes (it's very > > > infrequent) a sql request fails with "could not send data to client: > > > Permission denied". I reported the problem on pgsql-general@ [0] > > > and if I understood correctly, this happens when pgsql uses send(2) > > > and gets EACCES. > > > > > > According to send(2) this happens when "The connection was blocked > > > by pf(4)". I have a cron that modifies a table with > > > `pfctl -t TABLE_NAME -Tr -f TABLE_FILE_PATH` > > > > > > The file is large so it's not exactly immediate. Could pf > > > temporarily block new connections while it loads the file? Or am I > > > looking at the wrong thing? > > > > > > > From your pf rules, does the postgresql connection could be blocked if > > TABLE_NAME is empty/inconsistent ? > > > > Could you add (if you don't have already tested it), an explicit > > allow rule for postgresql to ensure the connection will success ? > > They are distinct rules: > # grep -e api_bans -e 5432 /etc/pf.conf > table persist file "/etc/pf.api" > block drop in quick from > pass in on vio0 proto tcp from $docker3 to (self) port 5432 > pass in on vio0 proto tcp from $web1 to (self) port 5432 > > The thing is that it happens very rarely, and I'm not sure how to > reproduce it. > if the problem is related to `pfctl -Treplace', you could try: - generate lot of postgresql access. from postgresql thread, the statement seems to be a SELECT, so it would be fine to ran in loop (hopping no cache and real traffic generated). - run pfctl -Treplace in a loop (with a set of different files as the kernel code takes care if host are added, changed, deleted) - maybe doing it at a "safe" time when not used a lot, if the host is on production :) assuming the problem is a race somewhere, it should raise the possible occurences of it. -- Sebastien Marie
Re: pf, send(2) and EACCES
On Fri, 28 Aug 2020 08:32:59 +0200, Sebastien Marie wrote: > On Thu, Aug 27, 2020 at 03:27:58PM -0400, Daniel Jakots wrote: > > Hi, > > > > I'm chasing a weird behavior with postgresql. Sometimes (it's very > > infrequent) a sql request fails with "could not send data to client: > > Permission denied". I reported the problem on pgsql-general@ [0] > > and if I understood correctly, this happens when pgsql uses send(2) > > and gets EACCES. > > > > According to send(2) this happens when "The connection was blocked > > by pf(4)". I have a cron that modifies a table with > > `pfctl -t TABLE_NAME -Tr -f TABLE_FILE_PATH` > > > > The file is large so it's not exactly immediate. Could pf > > temporarily block new connections while it loads the file? Or am I > > looking at the wrong thing? > > > > From your pf rules, does the postgresql connection could be blocked if > TABLE_NAME is empty/inconsistent ? > > Could you add (if you don't have already tested it), an explicit > allow rule for postgresql to ensure the connection will success ? They are distinct rules: # grep -e api_bans -e 5432 /etc/pf.conf table persist file "/etc/pf.api" block drop in quick from pass in on vio0 proto tcp from $docker3 to (self) port 5432 pass in on vio0 proto tcp from $web1 to (self) port 5432 The thing is that it happens very rarely, and I'm not sure how to reproduce it. > From my reading, pfctl -Treplace is using DIOCRSETADDRS ioctl. On > userland side, it tries to do it in one step (see > src/sbin/pfctl/pfctl_table.c line 228), but could iterate on > pfr_set_addrs() (I am unsure if the change is atomic or if the > iteration is to ensure the change will be atomic with large enough > buffer for result). > > The DIOCRSETADDRS ioctl on kernel side is done under PF_LOCK(). But I > didn't check if the match rule would be done under PF_LOCK() or not > (I am not familiar enough with pf(4) code to find the code which do > the check). Merci, Daniel
Re: pf, send(2) and EACCES
On Thu, Aug 27, 2020 at 03:27:58PM -0400, Daniel Jakots wrote: > Hi, > > I'm chasing a weird behavior with postgresql. Sometimes (it's very > infrequent) a sql request fails with "could not send data to client: > Permission denied". I reported the problem on pgsql-general@ [0] and if > I understood correctly, this happens when pgsql uses send(2) and gets > EACCES. > > According to send(2) this happens when "The connection was blocked by > pf(4)". I have a cron that modifies a table with > `pfctl -t TABLE_NAME -Tr -f TABLE_FILE_PATH` > > The file is large so it's not exactly immediate. Could pf temporarily > block new connections while it loads the file? Or am I looking at the > wrong thing? > >From your pf rules, does the postgresql connection could be blocked if TABLE_NAME is empty/inconsistent ? Could you add (if you don't have already tested it), an explicit allow rule for postgresql to ensure the connection will success ? >From my reading, pfctl -Treplace is using DIOCRSETADDRS ioctl. On userland >side, it tries to do it in one step (see src/sbin/pfctl/pfctl_table.c line 228), but could iterate on pfr_set_addrs() (I am unsure if the change is atomic or if the iteration is to ensure the change will be atomic with large enough buffer for result). The DIOCRSETADDRS ioctl on kernel side is done under PF_LOCK(). But I didn't check if the match rule would be done under PF_LOCK() or not (I am not familiar enough with pf(4) code to find the code which do the check). Thanks. -- Sebastien Marie
Re: pf, send(2) and EACCES
On Thu, 27 Aug 2020 16:16:17 -0400, "Sven F." wrote: > pflog0 will tell you what is block if you log it, and can tell you if > it is I would have been surprised otherwise (since normally packets pass) but I looked and there was no log about blocked packet at that time.
Re: pf, send(2) and EACCES
On Thu, Aug 27, 2020 at 3:30 PM Daniel Jakots wrote: > > Hi, > > I'm chasing a weird behavior with postgresql. Sometimes (it's very > infrequent) a sql request fails with "could not send data to client: > Permission denied". I reported the problem on pgsql-general@ [0] and if > I understood correctly, this happens when pgsql uses send(2) and gets > EACCES. > > According to send(2) this happens when "The connection was blocked by > pf(4)". I have a cron that modifies a table with > `pfctl -t TABLE_NAME -Tr -f TABLE_FILE_PATH` > > The file is large so it's not exactly immediate. Could pf temporarily > block new connections while it loads the file? Or am I looking at the > wrong thing? > > > [0]: https://www.postgresql.org/message-id/20200827111031.5ee46257%40anegada > > > Cheers, > Daniel > pflog0 will tell you what is block if you log it, and can tell you if it is -- -- - Knowing is not enough; we must apply. Willing is not enough; we must do
pf, send(2) and EACCES
Hi, I'm chasing a weird behavior with postgresql. Sometimes (it's very infrequent) a sql request fails with "could not send data to client: Permission denied". I reported the problem on pgsql-general@ [0] and if I understood correctly, this happens when pgsql uses send(2) and gets EACCES. According to send(2) this happens when "The connection was blocked by pf(4)". I have a cron that modifies a table with `pfctl -t TABLE_NAME -Tr -f TABLE_FILE_PATH` The file is large so it's not exactly immediate. Could pf temporarily block new connections while it loads the file? Or am I looking at the wrong thing? [0]: https://www.postgresql.org/message-id/20200827111031.5ee46257%40anegada Cheers, Daniel
