We restaged our environment
https://github.com/inverse-inc/packetfence/issues/7403 describes some
similar symptoms, so I have added some additional debug below from
cat /etc/network/interfaces
/usr/local/pf/sbin/pfperl-api get /api/v1/config/interfaces | jq
ip -br a
docker container ls
Following some of the triage steps in 7403, I additionally enabled debug on
pfperl-api, restarted the service and hit the Wizard Step 1 again, and
while I was capturing output for this email (below), the interfaces all of
a sudden appeared. The log output is also below from the process
restart, and I note there are some WARN level messages related to an ip
command exiting with a non-zero value for interfaces that do not appear in
my interface list.
packetfence.log:Jan 17 00:33:19 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(15) WARN: [mac:[undef]] Problem trying to run command: LANG=C
sudo ip -4 -o addr show veth17036df called from (eval). Child exited with
non-zero value 1 (pf::util::pf_run)
It seemed odd, so I reverted the DEBUG log to INFO, and rebooted the system
to see if I could reproduce the behavior. Sure enough, sitting on the
wizard step 1, as I had believed to have done previously, yielded no
interfaces in the Configurator and no WARN messages in the log.
It seemed that the service restart somehow resulted in the GUI being
updated. So we restart the service, and sure enough the interfaces
populated in the Configurator. No sign of the WARN messages either in the
log which seemed suppressed when set to INFO.
I re-ran the dump of the interfaces via API, and it gave the following
strange result
pf5:~# /usr/local/pf/sbin/pfperl-api get /api/v1/config/interfaces | jq
Device "veth7629818" does not exist.
parse error: Invalid numeric literal at line 1, column 19
Device "veth7629818" does not exist.
Device "veth6d6535d" does not exist.
Device "veth6d6535d" does not exist.
Unable to flush stdout: Broken pipe
cheers,
Ian
packetfence.log:Jan 17 00:31:15 pf5 pfperl-api-docker-wrapper[69046]:
Running with args --sig-proxy=true --rm --name=pfperl-api
--add-host=containers-gateway.internal:host-gateway -h pf5 -v
/var/lib/mysql:/var/lib/mysql -v /etc/sudoers:/etc/sudoers -v
/etc/sudoers.d/:/etc/sudoers.d/ -v
/usr/local/fingerbank/conf:/usr/local/fingerbank/conf -v
/usr/local/fingerbank/db:/usr/local/fingerbank/db -v
/usr/local/pf/var/run:/usr/local/pf/var/run -ePF_UID=996 -e PF_GID=996
-eFINGERBANK_UID=997 -e FINGERBANK_GID=997 -eIS_A_CLASSIC_PF_CONTAINER=yes
-v /etc/localtime:/etc/localtime:ro -v
/usr/local/pf/conf:/usr/local/pf/conf -v
/usr/local/pf/raddb/certs:/usr/local/pf/raddb/certs --privileged -v
/run/systemd/system:/run/systemd/system -v
/var/run/dbus/system_bus_socket:/var/run/dbus/system_bus_socket -v
/usr/local/fingerbank/conf:/usr/local/fingerbank/conf -v
/etc/sysconfig/:/etc/sysconfig -v /etc/network:/etc/network -v
/etc/resolv.conf:/etc/resolv.conf --network=host -e HOST_OS
-v/usr/local/pf/var/conf/:/usr/local/pf/var/conf/
-v/usr/local/pf/html/captive-portal/profile-templates:/usr/local/pf/html/captive-portal/profile-templates
packetfence.log:Jan 17 00:31:15 pf5 pfperl-api-docker-wrapper[69053]:
Error: No such container: pfperl-api
packetfence.log:Jan 17 00:31:17 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(8) DEBUG: [mac:[undef]] invalid IP: from cluster::__ANON__
(pf::util::valid_ip)
packetfence.log:Jan 17 00:31:19 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(8) DEBUG: [mac:[undef]] cache get for namespace='configfiles',
key='/usr/local/pf/conf/roles.conf', cache='Redis:l1_cache', time='0ms':
MISS (not in cache) (CHI::Driver::_log_get_result)
packetfence.log:Jan 17 00:31:19 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(8) DEBUG: [mac:[undef]] cache get for namespace='Default',
key='HASH(0x55f9c40983b0)', cache='RawMemory', time='0ms': MISS (not in
cache) (CHI::Driver::_log_get_result)
packetfence.log:Jan 17 00:31:19 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(8) DEBUG: [mac:[undef]] cache set for namespace='Default',
key='{"encoding":null,"reconnect":"60","server":"containers-gateway.internal:6379"}',
size=1, expires='never', cache='RawMemory', time='0ms'
(CHI::Driver::_log_set_result)
packetfence.log:Jan 17 00:31:19 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(8) DEBUG: [mac:[undef]] cache get for namespace='configfiles',
key='/usr/local/pf/conf/roles.conf', cache='Redis', time='1ms': HIT
(CHI::Driver::_log_get_result)
packetfence.log:Jan 17 00:31:19 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(8) DEBUG: [mac:[undef]] cache set for namespace='configfiles',
key='/usr/local/pf/conf/roles.conf', size=1, expires='never',
cache='Redis:l1_cache', time='0ms' (CHI::Driver::_log_set_result)
packetfence.log:Jan 17 00:31:20 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(8) DEBUG: [mac:[undef]] cache get for namespace='configfiles',
key='/usr/local/pf/conf/switches.conf', cache='Redis:l1_cache', time='0ms':
MISS (not in cache) (CHI::Driver::_log_get_result)
packetfence.log:Jan 17 00:31:20 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(8) DEBUG: [mac:[undef]] cache get for namespace='configfiles',
key='/usr/local/pf/conf/switches.conf', cache='Redis', time='1ms': HIT
(CHI::Driver::_log_get_result)
packetfence.log:Jan 17 00:31:20 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(8) DEBUG: [mac:[undef]] cache set for namespace='configfiles',
key='/usr/local/pf/conf/switches.conf', size=1, expires='never',
cache='Redis:l1_cache', time='0ms' (CHI::Driver::_log_set_result)
packetfence.log:Jan 17 00:31:23 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(8) INFO: [mac:[undef]] Listening at "http://127.0.0.1:22224";
(Mojo::Server::Daemon::_listen)
packetfence.log:Jan 17 00:31:23 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(8) INFO: [mac:[undef]] Listening at "http://100.64.0.1:22224";
(Mojo::Server::Daemon::_listen)
packetfence.log:Jan 17 00:31:23 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(8) INFO: [mac:[undef]] Manager 8 started
(Mojo::Server::Prefork::run)
packetfence.log:Jan 17 00:31:23 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(8) INFO: [mac:[undef]] Worker 11 started
(Mojo::Server::Prefork::_spawn)
packetfence.log:Jan 17 00:31:23 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(8) INFO: [mac:[undef]] Worker 12 started
(Mojo::Server::Prefork::_spawn)
packetfence.log:Jan 17 00:31:23 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(8) INFO: [mac:[undef]] Worker 13 started
(Mojo::Server::Prefork::_spawn)
packetfence.log:Jan 17 00:31:24 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(8) INFO: [mac:[undef]] Worker 14 started
(Mojo::Server::Prefork::_spawn)
packetfence.log:Jan 17 00:31:24 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(8) INFO: [mac:[undef]] Worker 15 started
(Mojo::Server::Prefork::_spawn)
packetfence.log:Jan 17 00:31:24 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(8) INFO: [mac:[undef]] Worker 16 started
(Mojo::Server::Prefork::_spawn)
packetfence.log:Jan 17 00:31:24 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(8) INFO: [mac:[undef]] Worker 17 started
(Mojo::Server::Prefork::_spawn)
packetfence.log:Jan 17 00:31:24 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(8) INFO: [mac:[undef]] Worker 18 started
(Mojo::Server::Prefork::_spawn)
packetfence.log:Jan 17 00:32:46 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(8) DEBUG: [mac:[undef]] GET
"/api/v1/configurator/config/interfaces" (Mojolicious::dispatch)
packetfence.log:Jan 17 00:32:46 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(8) DEBUG: [mac:[undef]] GET
"/api/v1/configurator/translation/en" (Mojolicious::dispatch)
packetfence.log:Jan 17 00:32:46 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(8) DEBUG: [mac:[undef]] Routing to controller
"pf::UnifiedApi::Controller::Configurator" and action "allowed"
(Mojolicious::Routes::_controller)
packetfence.log:Jan 17 00:32:46 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(8) DEBUG: [mac:[undef]] Routing to controller
"pf::UnifiedApi::Controller::Configurator" and action "allowed"
(Mojolicious::Routes::_controller)
packetfence.log:Jan 17 00:32:46 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(8) DEBUG: [mac:[undef]] Routing to controller
"pf::UnifiedApi::Controller::Configurator" and action "allowed"
(Mojolicious::Routes::_controller)
packetfence.log:Jan 17 00:32:46 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(8) DEBUG: [mac:[undef]] Routing to controller
"pf::UnifiedApi::Controller::Config::Interfaces" and action "list"
(Mojolicious::Routes::_controller)
packetfence.log:Jan 17 00:32:46 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(8) DEBUG: [mac:[undef]] Routing to controller
"pf::UnifiedApi::Controller::Translations" and action "resource"
(Mojolicious::Routes::_controller)
packetfence.log:Jan 17 00:32:46 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(8) DEBUG: [mac:[undef]] Routing to controller
"pf::UnifiedApi::Controller::Translations" and action "get"
(Mojolicious::Routes::_controller)
packetfence.log:Jan 17 00:32:47 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(8) DEBUG: [mac:[undef]] 200 OK (0.281034s, 3.558/s)
(Mojolicious::Controller::rendered)
packetfence.log:Jan 17 00:32:52 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(15) DEBUG: [mac:[undef]] LAST_TOUCH_CACHE is more than 5 seconds
old. Considering config::Pf() as invalid to reload it.
(pfconfig::cached::is_valid)
packetfence.log:Jan 17 00:33:08 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(15) DEBUG: [mac:[undef]] LAST_TOUCH_CACHE is more than 5 seconds
old. Considering config::Pf() as invalid to reload it.
(pfconfig::cached::is_valid)
packetfence.log:Jan 17 00:33:19 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(15) WARN: [mac:[undef]] Problem trying to run command: LANG=C
sudo ip -4 -o addr show veth17036df called from (eval). Child exited with
non-zero value 1 (pf::util::pf_run)
packetfence.log:Jan 17 00:33:30 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(15) WARN: [mac:[undef]] Problem trying to run command: LANG=C
sudo ip -6 -o addr show veth17036df called from (eval). Child exited with
non-zero value 1 (pf::util::pf_run)
packetfence.log:Jan 17 00:33:35 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(15) WARN: [mac:[undef]] Problem trying to run command: LANG=C
sudo ip -4 -o addr show veth3900ec3 called from (eval). Child exited with
non-zero value 1 (pf::util::pf_run)
packetfence.log:Jan 17 00:33:35 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(15) DEBUG: [mac:[undef]] LAST_TOUCH_CACHE is more than 5 seconds
old. Considering config::Pf() as invalid to reload it.
(pfconfig::cached::is_valid)
packetfence.log:Jan 17 00:33:35 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(15) WARN: [mac:[undef]] Problem trying to run command: LANG=C
sudo ip -6 -o addr show veth3900ec3 called from (eval). Child exited with
non-zero value 1 (pf::util::pf_run)
packetfence.log:Jan 17 00:33:35 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(15) DEBUG: [mac:[undef]] cache get for namespace='configfiles',
key='/usr/local/pf/conf/pf.conf', cache='Redis:l1_cache', time='0ms': MISS
(not in cache) (CHI::Driver::_log_get_result)
packetfence.log:Jan 17 00:33:35 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(15) DEBUG: [mac:[undef]] cache get for namespace='configfiles',
key='/usr/local/pf/conf/pf.conf', cache='Redis', time='2ms': HIT
(CHI::Driver::_log_get_result)
packetfence.log:Jan 17 00:33:35 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(15) DEBUG: [mac:[undef]] cache set for namespace='configfiles',
key='/usr/local/pf/conf/pf.conf', size=1, expires='never',
cache='Redis:l1_cache', time='0ms' (CHI::Driver::_log_set_result)
packetfence.log:Jan 17 00:33:35 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(15) DEBUG: [mac:[undef]] cache get for namespace='configfiles',
key='/usr/local/pf/conf/networks.conf', cache='Redis:l1_cache', time='0ms':
MISS (not in cache) (CHI::Driver::_log_get_result)
packetfence.log:Jan 17 00:33:35 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(15) DEBUG: [mac:[undef]] cache get for namespace='configfiles',
key='/usr/local/pf/conf/networks.conf', cache='Redis', time='0ms': HIT
(CHI::Driver::_log_get_result)
packetfence.log:Jan 17 00:33:35 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(15) DEBUG: [mac:[undef]] cache set for namespace='configfiles',
key='/usr/local/pf/conf/networks.conf', size=1, expires='never',
cache='Redis:l1_cache', time='0ms' (CHI::Driver::_log_set_result)
packetfence.log:Jan 17 00:33:35 pf5 pfperl-api-docker-wrapper[69063]:
pfperl-api(15) DEBUG: [mac:[undef]] 200 OK (48.510927s, 0.021/s)
(Mojolicious::Controller::rendered)
pf5:~# cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth0
iface eth0 inet static
address 10.2.1.2
gateway 10.2.1.1
netmask 255.255.255.0
auto eth1
iface eth1 inet static
address 10.2.2.2
netmask 255.255.255.0
auto eth2
iface eth2 inet static
address 10.2.3.2
netmask 255.255.255.0
# post-up ethtool -K eth0 tx off
#
# The commented out line above will disable TCP checksumming which
# might resolve problems for some users. It is disabled by default
#
pf5:~# /usr/local/pf/sbin/pfperl-api get /api/v1/config/interfaces | jq
{
"items": [
{
"additional_listening_daemons": [],
"address": "10.2.3.2/24",
"coa": null,
"dhcpd_enabled": null,
"dns": null,
"high_availability": 0,
"hwaddr": "00:16:3e:dc:7d:ff",
"id": "eth2",
"ifindex": "4",
"ipaddress": "10.2.3.2",
"ipv6_address": null,
"ipv6_prefix": null,
"is_running": true,
"master": null,
"name": "eth2",
"nat_enabled": null,
"netmask": "255.255.255.0",
"network": "10.2.3.0",
"network_iseditable": false,
"networks": [],
"not_editable": false,
"reg_network": null,
"split_network": null,
"type": "none",
"vip": null,
"vlan": null
},
{
"additional_listening_daemons": [],
"address": "10.2.1.2/24",
"coa": null,
"dhcpd_enabled": null,
"dns": null,
"high_availability": 0,
"hwaddr": "00:16:3e:dc:7d:fd",
"id": "eth0",
"ifindex": "2",
"ipaddress": "10.2.1.2",
"ipv6_address": null,
"ipv6_prefix": null,
"is_running": true,
"master": null,
"name": "eth0",
"nat_enabled": null,
"netmask": "255.255.255.0",
"network": "10.2.1.0",
"network_iseditable": false,
"networks": [],
"not_editable": false,
"reg_network": null,
"split_network": null,
"type": "none",
"vip": null,
"vlan": null
},
{
"additional_listening_daemons": [],
"address": "10.2.2.2/24",
"coa": null,
"dhcpd_enabled": null,
"dns": null,
"high_availability": 0,
"hwaddr": "00:16:3e:dc:7d:fe",
"id": "eth1",
"ifindex": "3",
"ipaddress": "10.2.2.2",
"ipv6_address": null,
"ipv6_prefix": null,
"is_running": true,
"master": null,
"name": "eth1",
"nat_enabled": null,
"netmask": "255.255.255.0",
"network": "10.2.2.0",
"network_iseditable": false,
"networks": [],
"not_editable": false,
"reg_network": null,
"split_network": null,
"type": "none",
"vip": null,
"vlan": null
}
],
"status": 200
}
pf5:~# ip -br a
lo UNKNOWN 127.0.0.1/8 66.70.255.147/32
eth0 UP 10.2.1.2/24
eth1 UP 10.2.2.2/24
eth2 UP 10.2.3.2/24
docker0 UP 100.64.0.1/24
vetha438e01@if7 UP
veth9a8f516@if15 UP
veth9c67be1@if17 UP
vethd507174@if19 UP
veth66ae1a3@if21 UP
veth4cf12b0@if29 UP
veth5523006@if143 UP
veth2ce3bab@if145 UP
pf5:~# docker container ls
CONTAINER ID IMAGE
COMMAND CREATED STATUS PORTS
NAMES
ad3e39f58b9e packetfence/pfpki:maintenance-12-1
"/bin/sh -c '/usr/lo…" 24 seconds ago Up 22 seconds
0.0.0.0:22225->22225/tcp
pfpki
502748cf55ea packetfence/httpd.dispatcher:maintenance-12-1
"/bin/sh -c '/usr/lo…" 24 seconds ago Up 23 seconds
0.0.0.0:5252->5252/tcp,
0.0.0.0:8888-8889->8888-8889/tcp httpd.dispatcher
c35fe36e3faa packetfence/haproxy-admin:maintenance-12-1
"/bin/sh -c '/usr/sb…" 2 minutes ago Up 2 minutes
0.0.0.0:1027->1027/tcp,
0.0.0.0:1443->1443/tcp haproxy-admin
3b16297ce6c0 packetfence/httpd.webservices:maintenance-12-1
"/bin/sh -c '/usr/lo…" 2 minutes ago Up 2 minutes
0.0.0.0:9090->9090/tcp
httpd.webservices
67785e33d97d packetfence/httpd.aaa:maintenance-12-1
"/bin/sh -c '/usr/sb…" 2 minutes ago Up 2 minutes
100.64.0.1:7070->7070/tcp,
127.0.0.1:7070->7070/tcp httpd.aaa
da945bf5a660 packetfence/haproxy-portal:maintenance-12-1
"/bin/sh -c '/usr/sb…" 2 minutes ago Up 2 minutes
haproxy-portal
7d0a2eed248e packetfence/pfconnector-server:maintenance-12-1
"/usr/local/pf/sbin/…" 2 minutes ago Up 2 minutes
pfconnector-server
5ec4b4defefd packetfence/httpd.admin_dispatcher:maintenance-12-1
"/bin/sh -c '/usr/lo…" 2 minutes ago Up 2 minutes
0.0.0.0:8890-8891->8890-8891/tcp
httpd.admin_dispatcher
76734c5d1895 packetfence/api-frontend:maintenance-12-1
"/bin/sh -c '/usr/lo…" 2 minutes ago Up 2 minutes
0.0.0.0:9999->9999/tcp
api-frontend
c54dd9e92bb5 packetfence/pfconnector-client:maintenance-12-1
"/usr/local/pf/sbin/…" 2 minutes ago Up 2 minutes
pfconnector-client
b4582f4b635c packetfence/pfperl-api:maintenance-12-1
"/bin/sh -c '/usr/lo…" 3 minutes ago Up 3 minutes
pfperl-api
2b28c406dc03 packetfence/pfconfig:maintenance-12-1
"/bin/sh -c /usr/loc…" 3 minutes ago Up 3 minutes
100.64.0.1:44444->44444/tcp,
127.0.0.1:44444->44444/tcp pfconfig
[image: PF121_WizardNoINF.png]
>
>
_______________________________________________
PacketFence-users mailing list
PacketFence-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/packetfence-users
