Hi, I had this problem with cephfs in the vm mainly, when firewall is stopped (rules are flushed - but existing connections still conntrack), then start again the firewall,
and conntrack put in invalid because it don't have tracked connection sequence when firewall was stopped. This could happen with previous proxmox release, when /etc/pve/cluster.cfg couldn't be read during restart of pve-cluster I think. (this has been fixed in last pve-firewall). But to really be sure to not have the problem anymore : add in /etc/sysctl.conf net.netfilter.nf_conntrack_tcp_be_liberal = 1 and to get it loaded at boot, add /etc/modules-load.d/nf_conntrack.conf nf_conntrack nf_conntrack_ipv4 nf_conntrack_ipv6 ----- Mail original ----- De: "Mark Schouten" <m...@tuxis.nl> À: "proxmoxve" <pve-user@pve.proxmox.com> Envoyé: Mercredi 8 Mai 2019 02:35:15 Objet: [PVE-User] Ceph and firewalling Hi, While upgrading two clusters tonight, it seems that the Ceph-cluster gets confused by the updates of tonight. I think it has something to do with the firewall and connection tracking. A restart of ceph-mon on a node seems to work. I *think* the issue is that when pve-firewall is upgraded, the conntracktable is emptied, and existing connections are captured by the 'ctstate INVALID'-rule. But it is kinda hard to reproduce. If you ask me, the rules for the 'management' ipset should be applied before the conntrack-rules, or am I setting things up incorrectly? The following packages are updated in this run: root@proxmox01:~# grep upgrade /var/log/dpkg.log 2019-05-08 02:09:46 upgrade base-files:amd64 9.9+deb9u8 9.9+deb9u9 2019-05-08 02:09:46 upgrade ceph-mds:amd64 12.2.11-pve1 12.2.12-pve1 2019-05-08 02:09:47 upgrade ceph-mgr:amd64 12.2.11-pve1 12.2.12-pve1 2019-05-08 02:09:48 upgrade ceph-mon:amd64 12.2.11-pve1 12.2.12-pve1 2019-05-08 02:09:49 upgrade ceph:amd64 12.2.11-pve1 12.2.12-pve1 2019-05-08 02:09:49 upgrade ceph-osd:amd64 12.2.11-pve1 12.2.12-pve1 2019-05-08 02:09:51 upgrade ceph-base:amd64 12.2.11-pve1 12.2.12-pve1 2019-05-08 02:09:52 upgrade ceph-common:amd64 12.2.11-pve1 12.2.12-pve1 2019-05-08 02:09:54 upgrade librbd1:amd64 12.2.11-pve1 12.2.12-pve1 2019-05-08 02:09:54 upgrade python-rados:amd64 12.2.11-pve1 12.2.12-pve1 2019-05-08 02:09:54 upgrade python-rbd:amd64 12.2.11-pve1 12.2.12-pve1 2019-05-08 02:09:54 upgrade python-rgw:amd64 12.2.11-pve1 12.2.12-pve1 2019-05-08 02:09:54 upgrade python-ceph:amd64 12.2.11-pve1 12.2.12-pve1 2019-05-08 02:09:54 upgrade python-cephfs:amd64 12.2.11-pve1 12.2.12-pve1 2019-05-08 02:09:54 upgrade libcephfs2:amd64 12.2.11-pve1 12.2.12-pve1 2019-05-08 02:09:54 upgrade librgw2:amd64 12.2.11-pve1 12.2.12-pve1 2019-05-08 02:09:55 upgrade libradosstriper1:amd64 12.2.11-pve1 12.2.12-pve1 2019-05-08 02:09:55 upgrade librados2:amd64 12.2.11-pve1 12.2.12-pve1 2019-05-08 02:09:55 upgrade ceph-fuse:amd64 12.2.11-pve1 12.2.12-pve1 2019-05-08 02:09:56 upgrade libhttp-daemon-perl:all 6.01-1 6.01-2 2019-05-08 02:09:56 upgrade libjs-jquery:all 3.1.1-2 3.1.1-2+deb9u1 2019-05-08 02:09:56 upgrade libmariadbclient18:amd64 10.1.37-0+deb9u1 10.1.38-0+deb9u1 2019-05-08 02:09:56 upgrade libpng16-16:amd64 1.6.28-1 1.6.28-1+deb9u1 2019-05-08 02:09:56 upgrade libpq5:amd64 9.6.11-0+deb9u1 9.6.12-0+deb9u1 2019-05-08 02:09:56 upgrade rsync:amd64 3.1.2-1+deb9u1 3.1.2-1+deb9u2 2019-05-08 02:09:56 upgrade pve-cluster:amd64 5.0-33 5.0-36 2019-05-08 02:09:56 upgrade libpve-storage-perl:all 5.0-39 5.0-41 2019-05-08 02:09:57 upgrade pve-firewall:amd64 3.0-18 3.0-20 2019-05-08 02:09:57 upgrade pve-ha-manager:amd64 2.0-8 2.0-9 2019-05-08 02:09:57 upgrade pve-qemu-kvm:amd64 2.12.1-2 2.12.1-3 2019-05-08 02:09:59 upgrade pve-edk2-firmware:all 1.20181023-1 1.20190312-1 2019-05-08 02:10:00 upgrade qemu-server:amd64 5.0-47 5.0-50 2019-05-08 02:10:00 upgrade libpve-common-perl:all 5.0-47 5.0-51 2019-05-08 02:10:00 upgrade libpve-access-control:amd64 5.1-3 5.1-8 2019-05-08 02:10:00 upgrade libpve-http-server-perl:all 2.0-12 2.0-13 2019-05-08 02:10:00 upgrade libssh2-1:amd64 1.7.0-1 1.7.0-1+deb9u1 2019-05-08 02:10:00 upgrade linux-libc-dev:amd64 4.9.144-3.1 4.9.168-1 2019-05-08 02:10:08 upgrade pve-kernel-4.15:all 5.3-3 5.4-1 2019-05-08 02:10:08 upgrade postfix-sqlite:amd64 3.1.9-0+deb9u2 3.1.12-0+deb9u1 2019-05-08 02:10:08 upgrade postfix:amd64 3.1.9-0+deb9u2 3.1.12-0+deb9u1 2019-05-08 02:10:10 upgrade proxmox-widget-toolkit:all 1.0-23 1.0-26 2019-05-08 02:10:10 upgrade pve-container:all 2.0-35 2.0-37 2019-05-08 02:10:10 upgrade pve-docs:all 5.3-3 5.4-2 2019-05-08 02:10:11 upgrade pve-i18n:all 1.0-9 1.1-4 2019-05-08 02:10:11 upgrade pve-xtermjs:amd64 3.10.1-2 3.12.0-1 2019-05-08 02:10:11 upgrade pve-manager:amd64 5.3-11 5.4-5 2019-05-08 02:10:11 upgrade proxmox-ve:all 5.3-1 5.4-1 2019-05-08 02:10:11 upgrade pve-kernel-4.15.18-12-pve:amd64 4.15.18-35 4.15.18-36 2019-05-08 02:10:19 upgrade python-cryptography:amd64 1.7.1-3 1.7.1-3+deb9u1 2019-05-08 02:10:19 upgrade unzip:amd64 6.0-21 6.0-21+deb9u1 2019-05-08 02:10:19 upgrade ruby2.3-dev:amd64 2.3.3-1+deb9u4 2.3.3-1+deb9u6 2019-05-08 02:10:19 upgrade libruby2.3:amd64 2.3.3-1+deb9u4 2.3.3-1+deb9u6 2019-05-08 02:10:20 upgrade publicsuffix:all 20181003.1334-0+deb9u1 20190415.1030-0+deb9u1 2019-05-08 02:10:20 upgrade ruby2.3:amd64 2.3.3-1+deb9u4 2.3.3-1+deb9u6 -- Mark Schouten <m...@tuxis.nl> Tuxis, Ede, https://www.tuxis.nl T: +31 318 200208 _______________________________________________ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user _______________________________________________ pve-user mailing list pve-user@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
