Re: [PacketFence-users] locationlog mysql writes periodically don't happen until packetfence service restart

Derek Wuelfrath Thu, 21 May 2015 07:28:09 -0700

John,

Just to make sure I’m trying to replicate the same thing, you are using an 
exclusive inline setup only in L2 architecture, is that right ? (Excuse me if 
I’m wrong, not that I don’t pay attention, it’s just that there’s so many 
things at the same time :))

I'll send what I get from debugging info to this thread.
Please do, that would help a lot to see what is happening.

Also, if possible, can you send a PCAP from the PacketFence server on the
inline subnet while replicating the issue. Another one when it is working.

Thanks for your time.

Cheers!
dw.

--
Derek Wuelfrath
[email protected] :: +1.514.447.4918 (x110) :: +1.866.353.6153 (x110)
Inverse inc. (www.inverse.ca) :: Leaders behind SOGo (www.sogo.nu) and
PacketFence (www.packetfence.org)

On May 21, 2015 at 09:45:23, John Baker ([email protected]) wrote:

Does pfdhcplistener write the information to the locationlog table? It does
seem connected to that but the ACK test doesn't pan out. I went through it
breaking again and watched as I registered something but the counts were the
same. I also tried specifically restarting just the pfdhcplistener process and
that didn't help. But, when I restarted all of packetfence it saw the machine I
had just registered and wrote to the locationlog again as soon as it got an ACK.

I don't know all of the internal processes her but what strikes me as important
here is that when it's broken pfdhcplistner.log has a record of new
transactions from machines ,registered or not, that already have entries in
locationlog but the table is not updated with new information. It just ignores
machines that it has not seen before.

I'll send what I get from debugging info to this thread.

On Wed, May 20, 2015 at 1:31 PM, Fabrice DURAND <[email protected]> wrote:
Hi

so it's really a pfdhcplistener issue.

Let's do that:

tshark -i eth0 -f "(port 67 or port 68) and ( udp[250:1]=0x05)"
and
tail -f /var/log/messages|grep ACK

and compare if you have the same number of ACK

If it's different then you have an issue with the pcap lib.

Regards
Fabrice

Le 2015-05-20 11:59, John Baker a écrit :
Hi,

DHCPD works fine. All of the problem client get leases and the process shows up
in the syslog. However, clients with the locationlog problem do not show up in
pfdhcplistner log or ipset. Clients that don't have this problem show up in
both.

On Tue, May 19, 2015 at 1:13 PM, Fabrice DURAND <[email protected]> wrote:
Hi John,

are you able to capture dhcp traffic on the inline interface and check if you
are able to get the dhcp request from the device that have an issue.

I remember that i got an issue with inline with a buggy lib (libpcap) and this
lib only capture 1/4 dhcp packet.
So if you can try a tail -f pfdhcplistener.log and a tcpdump on dhcp traffic in
the same time to compare.

Regards
Fabrice

Le 2015-05-19 12:23, John Baker a écrit :
Hi,

Sorry, symptomatically it's been confusing and I just figured out how to get
deeper into the program and set debugging on processes to look at specific
problems.

Here's what I have. We use inline mode exclusively and for some time we have
had sporadic problems with users registering and then getting stuck on the
"Your network should be enabled within a minute or two. If it is not reboot
your computer" screen. When I was onsite to confirm I found that their mac and
IP never got put in ipset. Just restarting the packetfence service always made
it work again and it happened seldom enough that I didn't think it was a
serious problem. But when I upgraded to 5 and now 5.02 it become more pervasive
and seemed to be happening all the time.

Then I found that it correlated with " httpd.portal(process#) WARN:
[mac-address] Can't re-evaluate access because no open locationlog entry was
found (pf::enforcement::reevaluate_access)" errors in packetfence.log. I also
found that problem mac addresses didn't show up at all in pfdhcplistener.log
though they were getting addresses from dhcpd.

So again what I found yesterday when I did more thorough testing was that only
new devices were not working. Devices registered and then unregistered were
able to register again. Previously registered devices all had entries in the
locationlog table but nothing was being updated or written to it. Once I
restarted packtefence newly registered devices were written to the table and
everything was happy.

Anyhow, I figured out that I didn't actually have full debugging for
pfdhcplistener on and got that on this morning. Should I also use debug on some
part of httpd.portal?

I have no idea when the problem will start again but I know what to look for to
know that it's happening now. I can replicate the error consistently with any
new machine when it's happening but I can't make it happen in the first place.
I'm not sure what part is getting stuck. All of the processes all appear to be
running properly when this happens.

Could I just be missing some maintenance script that needs to run?

I also have a steady stream of "WARN: Unable to perform a Fingerbank lookup for
device with MAC address" errors but I'm not sure if that has any connection to
the problem or not.

thank you

On Tue, May 19, 2015 at 10:19 AM, Derek Wuelfrath <[email protected]> wrote:
Hello John,

Can you just do a quick recap in reply describing the scenario in which it
doesn’t work, and the scenario in which it is working.
I’ll then have a look at the workflow in the code and see if we are missing
something.

Cheers!
dw.

--
Derek Wuelfrath
[email protected] :: +1.514.447.4918 (x110) :: +1.866.353.6153 (x110)
Inverse inc. (www.inverse.ca) :: Leaders behind SOGo (www.sogo.nu) and
PacketFence (www.packetfence.org)

On May 18, 2015 at 19:08:03, John Baker ([email protected]) wrote:

Ok, I have been pestering with this error. I pulled in 4 never registered
computers along with a couple that were registered and then unregistered later.
What I found was that only the previously registered ones worked after
registration. All others were registered but stuck on the success screen and
never added to IPset.

Further digging revealed that the locationlog in in the database had not been
written to for 4 days. So I unregistered all but one of the new devices and
restarted the packetfence service. After doing so the new one that I left
registered now had an entry in locationlog. I registered the others again and
they all then wrote to locationlog without any problem.

Any suggestions on why this might be happening? Do I just need to restart the
service once a day with a cron job? Being that it runs iptables on the router
this doesn't seem like a particularly safe method.

--
John Baker
Network Administrator
Marlboro College
Phone: 451-7551 Cell: 490-0066
------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y_______________________________________________
PacketFence-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/packetfence-users

--
John Baker
Network Administrator
Marlboro College
Phone: 451-7551 Cell: 490-0066

------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y

_______________________________________________
PacketFence-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/packetfence-users

--
Fabrice Durand
[email protected] :: +1.514.447.4918 (x135) :: www.inverse.ca
Inverse inc. :: Leaders behind SOGo (http://www.sogo.nu) and PacketFence
(http://packetfence.org)

--
John Baker
Network Administrator
Marlboro College
Phone: 451-7551 Cell: 490-0066

_______________________________________________
PacketFence-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/packetfence-users

--
Fabrice Durand
[email protected] :: +1.514.447.4918 (x135) :: www.inverse.ca
Inverse inc. :: Leaders behind SOGo (http://www.sogo.nu) and PacketFence
(http://packetfence.org)

PacketFence-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/packetfence-users

------------------------------------------------------------------------------
One dashboard for servers and applications across Physical-Virtual-Cloud 
Widest out-of-the-box monitoring support with 50+ applications
Performance metrics, stats and reports that give you Actionable Insights
Deep dive visibility with transaction tracing using APM Insight.
http://ad.doubleclick.net/ddm/clk/290420510;117567292;y

_______________________________________________
PacketFence-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/packetfence-users

Re: [PacketFence-users] locationlog mysql writes periodically don't happen until packetfence service restart

Reply via email to