Louis,
Thanks for your input on the issue, some responses to your request for
info below,
>> The main problem seems to be that that the haproxy service is not
starting.
>> In the syslog we just get a generic service failure with no details
>>
>> May 29 16:51:08 pf2 systemd[1]: Started PacketFence HAProxy Load
Balancer.
>> May 29 16:51:08 pf2 systemd[1]: packetfence-haproxy.service: main
process
>> exited, code=exited, status=1/FAILURE
>> May 29 16:51:08 pf2 systemd[1]: Unit packetfence-haproxy.service entered
>> failed state.
>> May 29 16:51:08 pf2 systemd[1]: packetfence-haproxy.service holdoff time
>> over, scheduling restart.
>> May 29 16:51:08 pf2 systemd[1]: Stopping PacketFence HAProxy Load
Balancer...
>>
>Let's try a few things.
>
>First, can you please post the output to these commands:
>
># systemctl status packetfence-haproxy
pf2:~# systemctl status packetfence-haproxy -l
* packetfence-haproxy.service - PacketFence HAProxy Load Balancer
Loaded: loaded (/lib/systemd/system/packetfence-haproxy.service; enabled)
Active: failed (Result: start-limit) since Mon 2017-05-29 16:51:15 EDT;
3 days ago
Process: 1031 ExecStart=/usr/sbin/haproxy-systemd-wrapper -f
/usr/local/pf/var/conf/haproxy.conf -p /usr/local/pf/var/run/haproxy.pid
(code=exited, status=1/FAILURE)
Process: 977 ExecStartPre=/usr/local/pf/bin/pfcmd service haproxy
generateconfig (code=exited, status=0/SUCCESS)
Main PID: 1031 (code=exited, status=1/FAILURE)
May 29 16:51:15 pf2 systemd[1]: packetfence-haproxy.service: main process
exited, code=exited, status=1/FAILURE
May 29 16:51:15 pf2 systemd[1]: Unit packetfence-haproxy.service entered
failed state.
May 29 16:51:15 pf2 haproxy-systemd-wrapper[1031]: haproxy-systemd-wrapper:
exit, haproxy RC=1
May 29 16:51:15 pf2 systemd[1]: packetfence-haproxy.service holdoff time
over, scheduling restart.
May 29 16:51:15 pf2 systemd[1]: Stopping PacketFence HAProxy Load
Balancer...
May 29 16:51:15 pf2 systemd[1]: Starting PacketFence HAProxy Load
Balancer...
May 29 16:51:15 pf2 systemd[1]: packetfence-haproxy.service start request
repeated too quickly, refusing to start.
May 29 16:51:15 pf2 systemd[1]: Failed to start PacketFence HAProxy Load
Balancer.
May 29 16:51:15 pf2 systemd[1]: Unit packetfence-haproxy.service entered
failed state.
># systemctl cat packetfence-haproxy
pf2:~# systemctl cat packetfence-haproxy
# /lib/systemd/system/packetfence-haproxy.service
[Unit]
Description=PacketFence HAProxy Load Balancer
Before=packetfence-httpd.portal.service packetfence-httpd.admin.service
Wants=packetfence-config.service
[Service]
StartLimitBurst=3
StartLimitInterval=60
PIDFile=/usr/local/pf/var/run/haproxy.pid
ExecStartPre=/usr/local/pf/bin/pfcmd service haproxy generateconfig
ExecStart=/usr/sbin/haproxy-systemd-wrapper -f
/usr/local/pf/var/conf/haproxy.conf -p /usr/local/pf/var/run/haproxy.pid
ExecReload=/bin/kill -USR2 $MAINPID
Restart=on-failure
[Install]
WantedBy=packetfence-base.target
> # ps -ef | grep haproxy
pf2:~# ps -ef | grep haproxy
root 11820 11782 0 10:16 pts/0 00:00:00 grep haproxy
>As to the configuration itself, look in
/usr/local/pf/var/conf/haproxy.conf to
>see the configuration that is actually generated by the conf/haproxy.conf
>template.
We did peek in here and nothing jumped out at us.
>You can try running haproxy in debug mode to see what error messages may
be
>lurking there:
>
># /usr/sbin/haproxy -f /usr/local/pf/var/conf/haproxy.conf -p
>/usr/local/pf/var/run/haproxy.pid -d
This was very helpful and immediately brought us to conclude it was related
to a change in our certs, that we opportunistically pushed out, as a root
cause of our issue. Is there a place in the docs that describes how to get
these debug outputs, to better help us help ourselves in the future?
pf2:~# /usr/sbin/haproxy -f /usr/local/pf/var/conf/haproxy.conf -p
/usr/local/pf/var/run/haproxy.pid -d
[ALERT] 152/125205 (13132) : parsing
[/usr/local/pf/var/conf/haproxy.conf:110] : 'bind 10.4.2.2:443' : unable to
load SSL private key from PEM file '/usr/local/pf/conf/ssl/server.pem'.
[ALERT] 152/125205 (13132) : parsing
[/usr/local/pf/var/conf/haproxy.conf:156] : 'bind 10.4.3.2:443' :
'/usr/local/pf/conf/ssl/server.pem'.
[ALERT] 152/125205 (13132) : parsing
[/usr/local/pf/var/conf/haproxy.conf:202] : 'bind 10.4.1.2:443' : unable to
load SSL private key from PEM file '/usr/local/pf/conf/ssl/server.pem'.
[ALERT] 152/125205 (13132) : Error(s) found in configuration file :
/usr/local/pf/var/conf/haproxy.conf
[WARNING] 152/125205 (13132) : Proxy 'stats': in multi-process mode, stats
will be limited to process assigned to the current request.
[ALERT] 152/125205 (13132) : Proxy 'portal-https-10.4.2.2': no SSL
certificate specified for bind '10.4.2.2:443' at
[/usr/local/pf/var/conf/haproxy.conf:110] (use 'crt').
[ALERT] 152/125205 (13132) : Proxy 'portal-https-10.4.3.2': no SSL
certificate specified for bind '10.4.3.2:443' at
[/usr/local/pf/var/conf/haproxy.conf:156] (use 'crt').
[ALERT] 152/125205 (13132) : Proxy 'portal-https-10.4.1.2': no SSL
certificate specified for bind '10.4.1.2:443' at
[/usr/local/pf/var/conf/haproxy.conf:202] (use 'crt').
[ALERT] 152/125205 (13132) : Fatal errors found in configuration.
The actual issue was that even though the cert, key and intermediate were
concatenated together into the .pem file, in the right order, one of the
files had different LF/CR formatting (windows vs linux), something
introduced by our ca, that was not obvious, and did not affect applying the
same files to the configuration GUI (nor any other system using the same
wildcard certs).
On a note related to upgrade in general, our team saw the release for 7.1,
which we are excited about with the inclusion of Ubiquiti devices, and I
had some comments back on the upgrade process that might help clarify
things for other users upgrading and using the UPGRADE.asciidoc as a
reference. We think it would be worthwhile to tell people to explicitly
execute the Version specific steps prior to the Distribution specific
steps. Some justification follows.
We knew from our v6.5 to 7.0 upgrade that the section for "Upgrading from a
version prior to 7.1.0" had to be executed before the section for "Debian
based systems" because it would not make sense to not upgrade the MariaDB
first. For anyone who started on v7.0.1 or later and who might
appropriately skip the "Upgrading from a version prior to 7.0.0" section,
it really is not clear which group of steps you should execute first ->
i.e. Should the user perform the Distribution specific steps before the
Version specific steps or vice-versa. It does hint in the doc that 'some
steps may be required to be done BEFORE the packages upgrades' but it
never really says clearly 'Go do all the Version specific steps further
down the document before you come back up and do your distribution-specific
steps'. Anyone that reads it all, and just executes in order, would (we
think) be doing it in the incorrect order.
cheers,
Ian
On Mon, May 29, 2017 at 5:21 PM, Ian MacDonald <i...@netstatz.com> wrote:
> We run PF on Debian Jessie with current packetfence 7.0.2 packages.
>
> pf2:~# dpkg -l | grep packetfence
> ii packetfence 7.0.2-1
> all PacketFence network registration / worm
> mitigation system
> ii packetfence-config 7.0.2-1
> all Service use to manage PacketFence
> configuration.
> ii packetfence-doc 7.0.2-1
> all documentation for packetfence
> ii packetfence-golang-daemon 7.0.2-1
> amd64 PacketFence Golang binary.
> ii packetfence-ntlm-wrapper 7.0.2-1
> amd64 C wrapper around the ntlm_auth utility to
> log authentication latency and success/failure.
> ii packetfence-pfcmd-suid 7.0.2-1
> amd64 C wrapper that replace perl-suid dependence
> ii packetfence-redis-cache 7.0.2-1
> all Init script to manage redis server.
>
>
> Running regular security updates on our PF 6.5 server we ended up with
> many PF 7.0 components, except those with MariaDB dependencies, sort of
> pushing us into the upgrade.
>
> After following https://github.com/inverse-inc/packetfence/blob/
> stable/UPGRADE.asciidoc, not quite in the specified order, we have a PF
> 7.0 system now, but a few hangover issues.
>
> The instruction *"Disable packetfence-mariadb on boot" *had us confused.
> If we follow the recommendation to execute the systemctl disable
> packetfence-mariadb we end up without a DB after boot and related
> connectivity and service startup errors. So we have left the service
> enabled.
>
> The main problem seems to be that that the haproxy service is not
> starting. We do not know why and can't seem to get any detailed logging
> from the service startup. We tried changing
>
> log %%active_active_ip%% local0
>
> to
>
> log %%active_active_ip%% local0 debug
>
> in /usr/local/pf/conf/haproxy.conf with no additional log output we could
> find.
>
> In the syslog we just get a generic service failure with no details
>
> May 29 16:51:08 pf2 systemd[1]: Started PacketFence HAProxy Load Balancer.
> May 29 16:51:08 pf2 systemd[1]: packetfence-haproxy.service: main process
> exited, code=exited, status=1/FAILURE
> May 29 16:51:08 pf2 systemd[1]: Unit packetfence-haproxy.service entered
> failed state.
> May 29 16:51:08 pf2 systemd[1]: packetfence-haproxy.service holdoff time
> over, scheduling restart.
> May 29 16:51:08 pf2 systemd[1]: Stopping PacketFence HAProxy Load
> Balancer...
>
> How do we get more debug on this startup failure?; as we have scoured the
> logs without any indication as to why this is happening.
>
> We have some other issues, that could be dependent on the haproxy startup,
> so we will leave those for now.
>
>
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
PacketFence-users mailing list
PacketFence-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/packetfence-users
