[pfSense] pfsync_undefer_state: unable to find deferred state

[Steve Yates]

I found thread
https://forum.pfsense.org/index.php?topic=87541.60
...and posted there but it's old and references 2.1.x and 2.2.x versions.  
After upgrading from 2.2.6 to 2.3.1_5 we get a long spew of this logged during 
a Limiter-limited rsync each night (it also shows on the console screen):

Jul 8 02:47:36  kernel  defer_state: unable to find deferred 
statepfsync_undefer_state: unable to find deferred state

Jul 8 02:47:36  kernel  _undefer_state: unable to find deferred 
statepfsync_undefer_state: unable to find deferred statepfsync_undefer_state: 
unable to find deferred statepfsync_undefer_state: unable to find deferred 
statepfsync_undefer_state: unable to find deferred statepfsync_undefer_state: 
unable to find deferred statepfsync_undefer_state: unable to find deferred 
statepfsync_undefer_state: unable to find deferred statepfsync_undefer_state: 
unable to find deferred statepfsync_undefer_state: unable to find deferred 
statepfsync_undefer_state: unable to find deferred statepfsync_undefer_state: 
unable to find deferred statepfsync_undefer_state: unable to find deferred 
statepfsync_undefer_state: unable to find deferred statepfsync_undefer_state: 
unable to find deferred statepfsync_undefer_state: unable to find deferred 
statepfsync_undefer_state: unable to find deferred statepfsync_undefer_state: 
unable to find deferred statepfsync_undefer_state: unable to find deferred 
statepfsync_und
 efer_state: unable to find deferred statepf

Jul 8 02:47:36  kernel  ync_undefer_state: unable to find deferred 
statepfsync_undefer_state: unable to find deferred statepfsync_undefer_state: 
unable to find deferred statepfsync_undefer_state: unable to find deferred 
statepfsync_undefer_state: unable to find deferred statepfsync_undefer_state: 
unable to find deferred statepfsync_undefer_state: unable to find deferred 
statepfsync_undefer_state: unable to find deferred statepfsync_undefer_state: 
unable to find deferred statepfsync_undefer_state: unable to find deferred 
statepfsync_undefer_state: unable to find deferred statepfsync_undefer_state: 
unable to find deferred statepfsync_undefer_state: unable to find deferred 
statepfsync_undefer_state: unable to find deferred statepfsync_undefer_state: 
unable to find deferred statepfsync_undefer_state: unable to find deferred 
statepfsync_undefer_state: unable to find deferred statepfsync_undefer_state: 
unable to find deferred statepfsync_undefer_state: unable to find deferred 
statepfsync_
 undefer_state: unable to find deferred stat


It continues while traffic that triggers the limiter rule is in effect and ends 
immediately upon traffic's end.

The Limiter set up is only using Firewall\Traffic Shaper\Limiters:
LimitBackupUpLAN
50Mbit/sOvernight [Mon - Sun / 0:00-6:45]
15Mbit/sDay
LimitBackupUpLAN
50Mbit/sOvernight
15Mbit/sDay

The limiter is on a rule on the LAN interface, with "In / Out pipe" set.  It 
only matches to one IP.  Neither checking "No pfSync" nor setting "State type" 
to None seem to have any effect.  I think that's the equivalent of what they 
mentioned in the forum thread... 'unchek  the flag "State Type" to "NO pfsync".'

I can duplicate this at will...in this case an "rsync --dry-run" is plenty.

It doesn't seem to have any effect on traffic since the copy works fine, it 
appears to just be a logging issue.

--

Steve Yates
ITS, Inc.

___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

Re: [pfSense] 502 Bad Gateway

[Jim Pingle]

On 07/08/2016 10:09 AM, Bill Arlofski wrote:
> I just realized something thanks to your post.  It seems that I have also
> witnessed that OpenVPN stops working when this occurs.

It would depend on the type of OpenVPN. RA or SSL/TLS using certificates
would likely fail as the scripts the verify parts of the cert and
perform the authentication are PHP. So if PHP is not functioning
properly, those can fail. The root problem is still PHP.

Jim

___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

Re: [pfSense] 502 Bad Gateway

[Bill Arlofski]

On 07/08/2016 03:21 AM, Chris Buechler wrote:
> It's worth trying at least. The case I had where that problem was
> replicable was worked around in 2.3.1_5 on this ticket.
> https://redmine.pfsense.org/issues/6318
> 
> There may be a related issue still outstanding from that, as the root
> cause is still an issue. The timeout works around it where I saw it,
> though that system does still have an occasional 502. I just pushed
> that out to 2.4.0 since we're rolling a 2.3.2 soon, but it will be
> looked at. If removing the widget does fix the issue, then you know
> it's a remaining symptom of #6318. It's not an easily replicable
> issue, most people aren't seeing it.

Hi Chris, thanks for confirming what I have seen and for locating the bug
report for me to review.

Please see J's email in this thread and my reply about this also seeming to
affect OpenVPN.

Thanks!

Bill


-- 
Bill Arlofski
Reverse Polarity, LLC
http://www.revpol.com/
-- Not responsible for anything below this line --
___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

Re: [pfSense] 502 Bad Gateway

[Bill Arlofski]

On 07/08/2016 02:44 AM, J. Echter wrote:
> Hi,
> 
> same issue here. I cannot access OpenVPN after a while and locally i get
> bad gateway when connecting to pfSense webui.
> 
> J


Yes!

I just realized something thanks to your post.  It seems that I have also
witnessed that OpenVPN stops working when this occurs.

I recently had a user complain that they could not connect using OpenVPN and
when I went to investigate, I was seeing the "Bad Gateway" in the gui.

At that time, I was in a rush, so I just rebooted the firewall and moved on.

So I guess this is more of a functionality problem than just broken access to
the GUI...

If I see the "502 Bad Gateway" problem again, I will test my OpenVPN client to
see if it is working..

Thanks for the confirmation of the "Bad Gateway" and also for the correlation
of the seemingly-related OpenVPN issue...

Best regards,

Bill


-- 
Bill Arlofski
Reverse Polarity, LLC
http://www.revpol.com/
-- Not responsible for anything below this line --
___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

Re: [pfSense] 502 Bad Gateway

[Bill Arlofski]

On 07/07/2016 02:50 PM, Vick Khera wrote:
> On Thu, Jul 7, 2016 at 2:16 PM, Bill Arlofski 
> wrote:
> 
>> I guess I will remove it the next time this happens and see if there is any
>> change.
>>
> 
> It seems to me you should remove it *before* to see if you avoid it
> happening.

Hi Vick!

hehe  Well, that would be no fun :)

Actually, it is not such a big issue now that I know I can simply ssh into the
device and restart the php-fpm process.

But yes, I could just remove it now and see if it never happens again. :)

Cheers!

Bill


-- 
Bill Arlofski
Reverse Polarity, LLC
http://www.revpol.com/
-- Not responsible for anything below this line --
___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

Re: [pfSense] 502 Bad Gateway

[Chris Buechler]

On Thu, Jul 7, 2016 at 1:16 PM, Bill Arlofski  wrote:
> On 07/07/2016 08:09 AM, Jon Gerdes wrote:
>> Bill
>>
>> I maybe off target here but the IPSEC widget used to cause php-fpm
>> daemon to die after a few days.
>>
>> I haven't looked into it since but removing that widget fixed it for me
>> on two pfSenses.
>>
>> Cheers
>> Jon
>
> Hi Jon,
>
> Hmmm, I do have the IPsec widget on my dashboard, so this is at least
> somewhere to start. :)
>
> I guess I will remove it the next time this happens and see if there is any
> change.
>
> Do you know if this is a known (and reported) issue?
>

It's worth trying at least. The case I had where that problem was
replicable was worked around in 2.3.1_5 on this ticket.
https://redmine.pfsense.org/issues/6318

There may be a related issue still outstanding from that, as the root
cause is still an issue. The timeout works around it where I saw it,
though that system does still have an occasional 502. I just pushed
that out to 2.4.0 since we're rolling a 2.3.2 soon, but it will be
looked at. If removing the widget does fix the issue, then you know
it's a remaining symptom of #6318. It's not an easily replicable
issue, most people aren't seeing it.
___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

Re: [pfSense] 502 Bad Gateway

[J. Echter]

i forgot to mention:

pfSense 2.3.1.

It works again if i restart php-fm.

J
___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold

Re: [pfSense] 502 Bad Gateway

[J. Echter]

Am 05.07.2016 um 19:19 schrieb Bill Arlofski:
> Hi everyone...
> 
> I noticed after one of the recent upgrades to the 2.2.x "RELEASE" series
> everything works perfectly fine for a while but then, I get "502 Bad Gateway"
> message when I attempt to access the web GUI.
> 
> Up until recently, I had been "fixing" this by physically powering off the
> firewall by pushing the power button which causes a clean shutdown. Then I
> would power it back up and it works fine for a random amount of time - usually
> days or weeks, sometimes even months - but this last time it only lasted about
> 23 hours.
> 
> I am currently running the 2.3.1-RELEASE-p5 (amd64) nanobsd (4g) distribution.
> 
> Today, I have enabled the ssh service and checked the nginx*.log files and I 
> see:
> 
> [2.3.1-RELEASE][usern...@vai.revpol.com]: clog -f /var/log/nginx-error.log
> 
> 8<
> 2016/07/05 12:40:01 [error] 48883#0: *257237 upstream timed out (60: Operation
> timed out) while reading response header from upstream, client: 192.168.254.4,
> server: , request: "GET /getstats.php HTTP/1.1", upstream:
> "fastcgi://unix:/var/run/php-fpm.socket", host: "vai.revpol.com:4443",
> referrer: "https://vai.revpol.com:4443/;
> clog: ERROR: could not write output (Bad address)
> 8<
> 
> 
> At that same time, system.log shows that the php-fpm.socket socket does not 
> exist:
> 8<
> Jul 5 12:47:00 vai vai.revpol.com nginx: 2016/07/05 12:47:00 [crit] 48883#0:
> *257442 connect() to unix:/var/run/php-fpm.socket failed (2: No such file or
> directory) while connecting to upstream, client: 192.168.254.4, server: ,
> request: "GET /getstats.php HTTP/1.1", upstream:
> "fastcgi://unix:/var/run/php-fpm.socket:", host: "vai.revpol.com:4443",
> referrer: "https://vai.revpol.com:4443/;
> 8<
> 
> 
> From the console menu, I can choose option 16 (Restart PHP-FPM), and then the
> web gui is accessible again.
> 
> Enter an option: 16
> 8<
 Killing php-fpm
 Starting php-fpm
> *** Welcome to pfSense 2.3.1-RELEASE-p5 (amd64 nanobsd) on vai ***
> 8<
> 
> So, I am suspecting that the php-fpm process is dying (forgot to run a ps
> command before restarting it).
> 
> Right now, /tmp/php_errors.txt is a zero byte file but I suspect that may be
> due to the restart of php-fpm due to its timestamp. I will take a look at this
> file the next time the gui dies.
> 
> 
> Is there anything I can do to increase debugging to help identify why this
> process is dying?
> 
> Additional info:  Typically I have a Firefox tab "idling" on the dashboard
> page which includes the "Traffic Graphs" widget with 4 graphs,
> Autoscale=Follow and 1 second updates.
> 
> 
> Thanks!
> 
> Bill
> 

Hi,

same issue here. I cannot access OpenVPN after a while and locally i get
bad gateway when connecting to pfSense webui.

J
___
pfSense mailing list
https://lists.pfsense.org/mailman/listinfo/list
Support the project with Gold! https://pfsense.org/gold