Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes
> >>I agree. Also that low level C/corosync hacking is no real fun ... > >> > >>Another option is to use pve-daemon to apply the settings. > > I wonder if we could re-use the pve-firewall daemon, and transform it to > something global for all network things. > > for now network config, but they are also frr router, why not in the futur: > distributed dhcp server, nat, Yes, that is also reasonable. ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes
>>I agree. Also that low level C/corosync hacking is no real fun ... >> >>Another option is to use pve-daemon to apply the settings. I wonder if we could re-use the pve-firewall daemon, and transform it to something global for all network things. for now network config, but they are also frr router, why not in the futur: distributed dhcp server, nat, - Mail original - De: "dietmar" À: "Stoiko Ivanov" , "aderumier" Cc: "pve-devel" Envoyé: Jeudi 4 Avril 2019 13:57:27 Objet: Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes > On 04 April 2019 at 12:16 Stoiko Ivanov wrote: > > > On Thu, 4 Apr 2019 11:57:38 +0200 (CEST) > Alexandre DERUMIER wrote: > > > > But how does it work ? who is currently listening for changes in > > > pmxcfs ? (through inotify?) > > > > >>This is low-level C-code inside pmxcfs (corosync). Please not that > > >>INotify does not work at all on /etc/pve/ - instead, we use > > >>versions numbers to track changes (see /etc/pve/.version). > > ok great ! > > > > And how can we generate the config && reload network from here ? > > Can we call an external perl script ? > > > From a quick glance - we do this with `corosync-cfgtool -R` when the > corosync.conf changes > (https://git.proxmox.com/?p=pve-cluster.git;a=blob;f=data/src/dcdb.c;h=6585a3443f0a8239cf668bb10eee1482b0859149;hb=HEAD#l409) > > > However I would keep in mind that perl-scripts (especially if they pull > in many modules and dependencies) can take a rather long time (and many > disk-hits) when being loaded - not too sure, but could imagine that > pmxcfs blocks until the run has completed. I agree. Also that low level C/corosync hacking is no real fun ... Another option is to use pve-daemon to apply the settings. ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes
> On 04 April 2019 at 12:16 Stoiko Ivanov wrote: > > > On Thu, 4 Apr 2019 11:57:38 +0200 (CEST) > Alexandre DERUMIER wrote: > > > > But how does it work ? who is currently listening for changes in > > > pmxcfs ? (through inotify?) > > > > >>This is low-level C-code inside pmxcfs (corosync). Please not that > > >>INotify does not work at all on /etc/pve/ - instead, we use > > >>versions numbers to track changes (see /etc/pve/.version). > > ok great ! > > > > And how can we generate the config && reload network from here ? > > Can we call an external perl script ? > > > From a quick glance - we do this with `corosync-cfgtool -R` when the > corosync.conf changes > (https://git.proxmox.com/?p=pve-cluster.git;a=blob;f=data/src/dcdb.c;h=6585a3443f0a8239cf668bb10eee1482b0859149;hb=HEAD#l409) > > However I would keep in mind that perl-scripts (especially if they pull > in many modules and dependencies) can take a rather long time (and many > disk-hits) when being loaded - not too sure, but could imagine that > pmxcfs blocks until the run has completed. I agree. Also that low level C/corosync hacking is no real fun ... Another option is to use pve-daemon to apply the settings. ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes
Am Mittwoch, den 03.04.2019, 07:03 +0200 schrieb Dietmar Maurer: > > I think, something easy, is that we could have a copy of each > > /etc/network/interfaces of each node in > > /etc/pve/nodes//interfaces. > > (could be done we a change is done in gui local netowrk, or local > > network daemon copy it at regular interval in case of manual change > > for example). > > > > > > Like this, it's very easy, when a network change is one at > > datacenter level, we can directly test it on all network interfaces > > of all nodes ( /etc/pve/nodes/*/interfaces). (in the api endpoint), /etc/network/interfaces is only a small part of actual network configuration. > I is still unclear to me how you do those tests? AFAIK, ifreload does > not have a --dry-run option. Even when it has such option, it would > need access to the local node? (to see what interfaces exists, ...). > > So if you really need/want to test before apply, we could add and API > call for that: > > POST /api2/json/nodes//test_network_changes > > We can then add a TEST button to the GUI, or call those this test API > on all nodes before we apply changes. > > > and then write directly the conf. (no need vnet.new tmp file). > > I think network configuration is really complex, and we should avoid > to do anything automatically. > I would prefer and "APPLY" button, so that I have full control over > when network changes happen. > Maybe an extra "TEST" button would be also helpful. Probably helpful but as you said, network configuration can be really complex (good luck finding my tinc bridges in the interfaces file - let alone developing a test). Why not have a static (host specific) part that never ever gets touched by pve. Usually all thats needed to get the node into the cluster. Additional parts can be managed via the cluster - selectable on which nodes (including all) to apply. If you select/mark a node, try to apply, if that fails, fall back to your static basic configuration above and show errors so the admin can fix, ideally with the node still in the cluster. So you can test on single test-nodes if you need or apply on all if you're sure about your changes. We don't try to prevent shooting our feet at other places either (like firewalling). Best, Tom ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes
On Thu, 4 Apr 2019 11:57:38 +0200 (CEST) Alexandre DERUMIER wrote: > > But how does it work ? who is currently listening for changes in > > pmxcfs ? (through inotify?) > > >>This is low-level C-code inside pmxcfs (corosync). Please not that > >>INotify does not work at all on /etc/pve/ - instead, we use > >>versions numbers to track changes (see /etc/pve/.version). > ok great ! > > And how can we generate the config && reload network from here ? > Can we call an external perl script ? > From a quick glance - we do this with `corosync-cfgtool -R` when the corosync.conf changes (https://git.proxmox.com/?p=pve-cluster.git;a=blob;f=data/src/dcdb.c;h=6585a3443f0a8239cf668bb10eee1482b0859149;hb=HEAD#l409) However I would keep in mind that perl-scripts (especially if they pull in many modules and dependencies) can take a rather long time (and many disk-hits) when being loaded - not too sure, but could imagine that pmxcfs blocks until the run has completed. just something to keep an eye on while testing. > > - Mail original - > De: "dietmar" > À: "aderumier" > Cc: "pve-devel" , "Stoiko Ivanov" > Envoyé: Jeudi 4 Avril 2019 11:44:47 > Objet: Re: [pve-devel] rfc : pve-network : idea to generate and > reload config accross the nodes > > > >>So the idea is to detect network.cfg changes inside pmxcfs, and > > >>if we detect changes do a network reload. > > >> > > >>That way we can apply the config without an additional daemon - > > >>sounds good. > > > > Sound good. (so we can do changes in network.cfg.tmp, still have > > the test button(api call to each nodes), and apply change to > > network.cfg) > > > > > > But how does it work ? who is currently listening for changes in > > pmxcfs ? (through inotify?) > > This is low-level C-code inside pmxcfs (corosync). Please not that > INotify does not work at all on /etc/pve/ - instead, we use versions > numbers to track changes (see /etc/pve/.version). > > ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes
> But how does it work ? who is currently listening for changes in pmxcfs ? > (through inotify?) >>This is low-level C-code inside pmxcfs (corosync). Please not that INotify >>does not work at all >>on /etc/pve/ - instead, we use versions numbers to track changes (see >>/etc/pve/.version). ok great ! And how can we generate the config && reload network from here ? Can we call an external perl script ? - Mail original - De: "dietmar" À: "aderumier" Cc: "pve-devel" , "Stoiko Ivanov" Envoyé: Jeudi 4 Avril 2019 11:44:47 Objet: Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes > >>So the idea is to detect network.cfg changes inside pmxcfs, and if we > >>detect changes > >>do a network reload. > >> > >>That way we can apply the config without an additional daemon - sounds > >>good. > > Sound good. (so we can do changes in network.cfg.tmp, still have the test > button(api call to each nodes), and apply change to network.cfg) > > > But how does it work ? who is currently listening for changes in pmxcfs ? > (through inotify?) This is low-level C-code inside pmxcfs (corosync). Please not that INotify does not work at all on /etc/pve/ - instead, we use versions numbers to track changes (see /etc/pve/.version). ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes
> >>So the idea is to detect network.cfg changes inside pmxcfs, and if we > >>detect changes > >>do a network reload. > >> > >>That way we can apply the config without an additional daemon - sounds good. > > Sound good. (so we can do changes in network.cfg.tmp, still have the test > button(api call to each nodes), and apply change to network.cfg) > > > But how does it work ? who is currently listening for changes in pmxcfs ? > (through inotify?) This is low-level C-code inside pmxcfs (corosync). Please not that INotify does not work at all on /etc/pve/ - instead, we use versions numbers to track changes (see /etc/pve/.version). ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes
>>So the idea is to detect network.cfg changes inside pmxcfs, and if we detect >>changes >>do a network reload. >> >>That way we can apply the config without an additional daemon - sounds good. Sound good. (so we can do changes in network.cfg.tmp, still have the test button(api call to each nodes), and apply change to network.cfg) But how does it work ? who is currently listening for changes in pmxcfs ? (through inotify?) - Mail original - De: "dietmar" À: "pve-devel" , "Stoiko Ivanov" , "aderumier" Envoyé: Jeudi 4 Avril 2019 10:58:09 Objet: Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes > Two ideas that came up in my head (not sure if they are good or > sensibly implementable): > > * The networking config has the common property with the corosync > configuration (the chicken and egg problem - if it's wrong the > cluster cannot push the corrected config to a broken node) so why > don't we use the same/a similar mechanism for pushing out changes to > the live-config and getting changes from the live-config into the > pmxcfs (if we keep the live-data in pmxcfs we know when a write to it > happens and can copy it over to /etc/network/interfaces(.d) (and run > some ifquery and other tests) before)? Also this would save us from > having yet another daemon running in the background and consuming > resources. So the idea is to detect network.cfg changes inside pmxcfs, and if we detect changes do a network reload. That way we can apply the config without an additional daemon - sounds good. > * from a very quick run with ifquery - it has the ability to read an > parse the complete config (including 'source' statements) - so we > could use this to get support for '/etc/network/interfaces.d/*' > snippets to the API and GUI (IIRC there have been a few requests from > users for such a functionality) The idea is that we are only responsible for a single file, so that makes no sense to me. ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes
> Two ideas that came up in my head (not sure if they are good or > sensibly implementable): > > * The networking config has the common property with the corosync > configuration (the chicken and egg problem - if it's wrong the > cluster cannot push the corrected config to a broken node) so why > don't we use the same/a similar mechanism for pushing out changes to > the live-config and getting changes from the live-config into the > pmxcfs (if we keep the live-data in pmxcfs we know when a write to it > happens and can copy it over to /etc/network/interfaces(.d) (and run > some ifquery and other tests) before)? Also this would save us from > having yet another daemon running in the background and consuming > resources. So the idea is to detect network.cfg changes inside pmxcfs, and if we detect changes do a network reload. That way we can apply the config without an additional daemon - sounds good. > * from a very quick run with ifquery - it has the ability to read an > parse the complete config (including 'source' statements) - so we > could use this to get support for '/etc/network/interfaces.d/*' > snippets to the API and GUI (IIRC there have been a few requests from > users for such a functionality) The idea is that we are only responsible for a single file, so that makes no sense to me. ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes
> What do you mean by locally assigned ? manually with ip command ? > because it's be overwritten by network service restart/reload. (if the > interface is define in /etc/network/interfaces) >>So you are sure that ifupdown2 can verify a network config on a remote node? I don't think we can be sure at 100%. (For example, it don't see ovs). and I think that the dry-run feature is not 100% complete. (That's why in current ifupdown2 network reload api, I simply catch errors on reload for each interface, and revert only non-applied changes to /etc/network/interfaces.new). other command, ifquery -a, only display running configuration, but from the ifupdown2 cache. So manual setup with ip command are not seen. >>Thats sounds reasonable. >> >>Maybe we can use the broadcast_rrd() code to distribute vnet/transport status >>among cluster nodes? yes. like for storage. - Mail original - De: "dietmar" À: "aderumier" Cc: "pve-devel" Envoyé: Jeudi 4 Avril 2019 09:36:12 Objet: Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes > >>or locally assigned ip addresses or routes. A copy of > >>/etc/network/interfaces does not >>provide all necessary information. > > What do you mean by locally assigned ? manually with ip command ? > because it's be overwritten by network service restart/reload. (if the > interface is define in /etc/network/interfaces) So you are sure that ifupdown2 can verify a network config on a remote node? > > I was thinking about diplaying transport/vnets in main tree (like storage), > > and display error on it. > > > >>You mix different things here. The configuration is cluster wide (for all > >>nodes). Opposed to that, vnet status is per node. > >> > >>We have the same thing with storages, so maybe we can implement it the same > >>way? Simply display vnet status in the left tree (like we do for storage > >>status)? > >> > >> (maybe with compare the running local network config and what is defined > >> in /etc/pve/network/ ). (for presentation, maybe like a storage with > >> volumes, we could have a >>transport with vnets). Not sure it's a good > >> idea ? > >> > >>Sorry, I do not really understand that suggestion? > > I mean, it's indeed like for storage, but we could have a lots of vnets. (for > example, I'm using around 150 differents vlans in production) > As the vnets are alwary associated to 1 transport, my idea was to only > display transports in the main tree (on each node), and then when you click > an transport, you display in the > right panel, the list of vnets. (to compare with storage, a transport = > storage, and when you click on storage, you Thats sounds reasonable. Maybe we can use the broadcast_rrd() code to distribute vnet/transport status among cluster nodes? ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes
> >>or locally assigned ip addresses or routes. A copy of > >>/etc/network/interfaces does not >>provide all necessary information. > > What do you mean by locally assigned ? manually with ip command ? > because it's be overwritten by network service restart/reload. (if the > interface is define in /etc/network/interfaces) So you are sure that ifupdown2 can verify a network config on a remote node? > > I was thinking about diplaying transport/vnets in main tree (like storage), > > and display error on it. > > > >>You mix different things here. The configuration is cluster wide (for all > >>nodes). Opposed to that, vnet status is per node. > >> > >>We have the same thing with storages, so maybe we can implement it the same > >>way? Simply display vnet status in the left tree (like we do for storage > >>status)? > >> > >> (maybe with compare the running local network config and what is defined > >> in /etc/pve/network/ ). (for presentation, maybe like a storage with > >> volumes, we could have a >>transport with vnets). Not sure it's a good > >> idea ? > >> > >>Sorry, I do not really understand that suggestion? > > I mean, it's indeed like for storage, but we could have a lots of vnets. (for > example, I'm using around 150 differents vlans in production) > As the vnets are alwary associated to 1 transport, my idea was to only > display transports in the main tree (on each node), and then when you click > an transport, you display in the > right panel, the list of vnets. (to compare with storage, a transport = > storage, and when you click on storage, you Thats sounds reasonable. Maybe we can use the broadcast_rrd() code to distribute vnet/transport status among cluster nodes? ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes
> >>Even when it has such option, it would need access to the local node? (to > >>see what interfaces exists, ...). > Yes, that's why my last proposition what to have a of local copy > configuration to /etc/pve/. (to be able to test with only 1api call, without > calling each node) >>My point is that you need more info, for example existing interfaces >>(/proc/net/dev), Yes, indeed. >>or locally assigned ip addresses or routes. A copy of /etc/network/interfaces >>does not >>provide all necessary information. What do you mean by locally assigned ? manually with ip command ? because it's be overwritten by network service restart/reload. (if the interface is define in /etc/network/interfaces) > I was thinking about diplaying transport/vnets in main tree (like storage), > and display error on it. > >>You mix different things here. The configuration is cluster wide (for all >>nodes). Opposed to that, vnet status is per node. >> >>We have the same thing with storages, so maybe we can implement it the same >>way? Simply display vnet status in the left tree (like we do for storage >>status)? >> >> (maybe with compare the running local network config and what is defined in >> /etc/pve/network/ ). (for presentation, maybe like a storage with volumes, >> we could have a >>transport with vnets). Not sure it's a good idea ? >> >>Sorry, I do not really understand that suggestion? I mean, it's indeed like for storage, but we could have a lots of vnets. (for example, I'm using around 150 differents vlans in production) As the vnets are alwary associated to 1 transport, my idea was to only display transports in the main tree (on each node), and then when you click an transport, you display in the right panel, the list of vnets. (to compare with storage, a transport = storage, and when you click on storage, you display the volumes (or vnet in this case). In my cas, this is only to avoid to display 150 vnets * 20 nodes. Maybe another way, to no flood the tree, is to have a child "vnet" on each node, and then under this child, add all vnets (and keep the parent close by default) - Mail original - De: "dietmar" À: "aderumier" Cc: "pve-devel" Envoyé: Jeudi 4 Avril 2019 07:48:54 Objet: Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes > >>I is still unclear to me how you do those tests? AFAIK, ifreload does not > >>have a --dry-run option. > with ifupdown2, ifreload -a --no-act. > (+ tests with our currrent read_networt_interface code) Ok, thanks. (This flag is not documented in the manual page). > > >>Even when it has such option, it would need access to the local node? (to > >>see what interfaces exists, ...). > Yes, that's why my last proposition what to have a of local copy > configuration to /etc/pve/. (to be able to test with only 1api call, without > calling each node) My point is that you need more info, for example existing interfaces (/proc/net/dev), or locally assigned ip addresses or routes. A copy of /etc/network/interfaces does not provide all necessary information. > >>I am not sure it this is a real problem. I think the cluster wide vnet > >>config is the relevant config. If a node is unable to apply that config, > >>the node needs to get fixed. > >>I think of this like deploying a network configuration with ansible (or > >>other tools). > > Do you have an idea where to report a local error configuration ? Maybe an extra file inside /etc/pve/nodes// ... (not sure about that). > I was thinking about diplaying transport/vnets in main tree (like storage), > and display error on it. You mix different things here. The configuration is cluster wide (for all nodes). Opposed to that, vnet status is per node. We have the same thing with storages, so maybe we can implement it the same way? Simply display vnet status in the left tree (like we do for storage status)? > (maybe with compare the running local network config and what is defined in > /etc/pve/network/ ). (for presentation, maybe like a storage with volumes, we > could have a transport with vnets). Not sure it's a good idea ? Sorry, I do not really understand that suggestion? > >>So if you really need/want to test before apply, we could add and API call > >>for that: > >>POST /api2/json/nodes//test_network_changes > >>We can then add a TEST button to the GUI, or call those this test API on > >>all nodes before we apply changes. > > Ok, I was not sure about this way. So let's go for this. > > > > > >>I think network configuration is really complex, and we should avoid to do > >>anything automatically. > >>I would prefer and "APPLY" button, so that I have full control over when > >>network changes happen. > >>Maybe an extra "TEST" button would be also helpful. > > Ok, I'll look this way. OK ___ pve-devel mailing list pve-devel@pve.proxmox.com
Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes
> > >>I think of this like deploying a network configuration with ansible (or > > >>other tools). > > > > Do you have an idea where to report a local error configuration ? > > Maybe an extra file inside /etc/pve/nodes// ... > > (not sure about that). Please ignore that suggestion. I guess it is a bad idea to store that on the cluster file system, because it may important apply/test network settings while the cluster is offline. So maybe we simply store the last error somewhere on the local node. You need to use the API to get the status. ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes
> >>I is still unclear to me how you do those tests? AFAIK, ifreload does not > >>have a --dry-run option. > with ifupdown2, ifreload -a --no-act. > (+ tests with our currrent read_networt_interface code) Ok, thanks. (This flag is not documented in the manual page). > > >>Even when it has such option, it would need access to the local node? (to > >>see what interfaces exists, ...). > Yes, that's why my last proposition what to have a of local copy > configuration to /etc/pve/. (to be able to test with only 1api call, without > calling each node) My point is that you need more info, for example existing interfaces (/proc/net/dev), or locally assigned ip addresses or routes. A copy of /etc/network/interfaces does not provide all necessary information. > >>I am not sure it this is a real problem. I think the cluster wide vnet > >>config is the relevant config. If a node is unable to apply that config, > >>the node needs to get fixed. > >>I think of this like deploying a network configuration with ansible (or > >>other tools). > > Do you have an idea where to report a local error configuration ? Maybe an extra file inside /etc/pve/nodes// ... (not sure about that). > I was thinking about diplaying transport/vnets in main tree (like storage), > and display error on it. You mix different things here. The configuration is cluster wide (for all nodes). Opposed to that, vnet status is per node. We have the same thing with storages, so maybe we can implement it the same way? Simply display vnet status in the left tree (like we do for storage status)? > (maybe with compare the running local network config and what is defined in > /etc/pve/network/ ). (for presentation, maybe like a storage with volumes, we > could have a transport with vnets). Not sure it's a good idea ? Sorry, I do not really understand that suggestion? > >>So if you really need/want to test before apply, we could add and API call > >>for that: > >>POST /api2/json/nodes//test_network_changes > >>We can then add a TEST button to the GUI, or call those this test API on > >>all nodes before we apply changes. > > Ok, I was not sure about this way. So let's go for this. > > > > > >>I think network configuration is really complex, and we should avoid to do > >>anything automatically. > >>I would prefer and "APPLY" button, so that I have full control over when > >>network changes happen. > >>Maybe an extra "TEST" button would be also helpful. > > Ok, I'll look this way. OK ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes
Hi, Since I'm quite interested in networking I would like to support you with this topic (and learn new things). Am still reading up on the theory and our current code (will try to setup a test-environment soon in order to get some hands-on experience). Two ideas that came up in my head (not sure if they are good or sensibly implementable): * The networking config has the common property with the corosync configuration (the chicken and egg problem - if it's wrong the cluster cannot push the corrected config to a broken node) so why don't we use the same/a similar mechanism for pushing out changes to the live-config and getting changes from the live-config into the pmxcfs (if we keep the live-data in pmxcfs we know when a write to it happens and can copy it over to /etc/network/interfaces(.d) (and run some ifquery and other tests) before)? Also this would save us from having yet another daemon running in the background and consuming resources. * from a very quick run with ifquery - it has the ability to read an parse the complete config (including 'source' statements) - so we could use this to get support for '/etc/network/interfaces.d/*' snippets to the API and GUI (IIRC there have been a few requests from users for such a functionality) Does this make any sense? What do you think? Looking forward to this topic! stoiko On Tue, 2 Apr 2019 06:35:57 +0200 (CEST) Alexandre DERUMIER wrote: > Hi, > > I have rethinked about it, I have (again ;) a new idea for > implementation. > > The main problem is how to test a change at datacenter level, as we > need to test the local configuration of each node. > > and it's not currently in /etc/pve , but in /etc/network/interfaces > of each node. > > > I think, something easy, is that we could have a copy of > each /etc/network/interfaces of each node > in /etc/pve/nodes//interfaces. (could be done we a change > is done in gui local netowrk, or local network daemon copy it at > regular interval in case of manual change for example). > > > Like this, it's very easy, when a network change is one at datacenter > level, we can directly test it on all network interfaces of all nodes > ( /etc/pve/nodes/*/interfaces). (in the api endpoint), and then write > directly the conf. (no need vnet.new tmp file). > > Then the local daemon simply reload the network configuration. > > What do you think about this ? > > > - Mail original - > De: "aderumier" > À: "pve-devel" > Envoyé: Lundi 1 Avril 2019 15:18:51 > Objet: Re: [pve-devel] rfc : pve-network : idea to generate and > reload config accross the nodes > > as alternative, > we could simply > > manage multiple change in /etc/pve/network/vnet.cfg.new > > apply button -> replace /etc/pve/network/vnet.cfg > > The the local daemon, > do test (dry-run,) and report error in his status file. (and it's > displayed at network level in datacenter) if ok, > it's apply change, and report error in his status file. > if ok, update status to ok. > > > So, user can wait some seconds, and check the status of nodes at > datacenter level. > > Seem to be simplier. What do you think about this ? > > > > - Mail original - > De: "Alexandre Derumier" > À: "dietmar" > Cc: "pve-devel" > Envoyé: Lundi 1 Avril 2019 15:05:07 > Objet: Re: [pve-devel] rfc : pve-network : idea to generate and > reload config accross the nodes > > >>I don't really get why you want to do that? There are so many ways > >>to damage a network, and I doubt that we can reliable verify > >>that > > ifupdown2 have a dry-run too, it's working not too bad (but not 100% > complete) > > But I would avoid some basic mistakes, > like a vlan interface already defined and enslaved in another bridge > for example, or look to not enslave an interface with ipmanagement in > a bridge (try to not break cluster connectivity) > > > But I don't want to manage rollback across all nodes. > (config correctly applied on 1 node, another node fail, I don't want > to rollback the first node) It's more best effort, if 1 node have > failed, it's simply report the error in his status file. > > > > > > >>Also, what if some nodes are offline ... > We could make an exception, if a node is offline (down, network > daemon down,...), Then don't wait for validation, and apply config. > > Then the local deamon will try to apply config when node is up again. > In case of error, It'll report it through his status file. > > - Mail original - > De: "dietmar" > À: "Alexandre Derumier" , "pve-devel" > Envoyé: Lundi 1 Avril 2019 12:00:13 > Objet: Re: [pve-devel] rfc : pve-network : idea to generate and > reload config accross the nodes > > > maybe better: > > > > in gui, at network,datacenter level > > > > at each change, make a > > /etc/pve/networks/vnet.cfg. > > > > > > on local node, the daemon detect the new version,make verification, > > and update
Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes
>>I is still unclear to me how you do those tests? AFAIK, ifreload does not >>have a --dry-run option. with ifupdown2, ifreload -a --no-act. (+ tests with our currrent read_networt_interface code) >>Even when it has such option, it would need access to the local node? (to see >>what interfaces exists, ...). Yes, that's why my last proposition what to have a of local copy configuration to /etc/pve/. (to be able to test with only 1api call, without calling each node) >>I am not sure it this is a real problem. I think the cluster wide vnet config >>is the relevant config. If a node is unable to apply that config, the node >>needs to get fixed. >>I think of this like deploying a network configuration with ansible (or other >>tools). Do you have an idea where to report a local error configuration ? I was thinking about diplaying transport/vnets in main tree (like storage), and display error on it. (maybe with compare the running local network config and what is defined in /etc/pve/network/ ). (for presentation, maybe like a storage with volumes, we could have a transport with vnets). Not sure it's a good idea ? >>So if you really need/want to test before apply, we could add and API call >>for that: >>POST /api2/json/nodes//test_network_changes >>We can then add a TEST button to the GUI, or call those this test API on all >>nodes before we apply changes. Ok, I was not sure about this way. So let's go for this. >>I think network configuration is really complex, and we should avoid to do >>anything automatically. >>I would prefer and "APPLY" button, so that I have full control over when >>network changes happen. >>Maybe an extra "TEST" button would be also helpful. Ok, I'll look this way. - Mail original - De: "dietmar" À: "pve-devel" , "aderumier" Envoyé: Mercredi 3 Avril 2019 07:03:09 Objet: Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes > I have rethinked about it, I have (again ;) a new idea for implementation. > > The main problem is how to test a change at datacenter level, as we need to > test the local configuration of each node. > > and it's not currently in /etc/pve , but in /etc/network/interfaces of each > node. I am not sure it this is a real problem. I think the cluster wide vnet config is the relevant config. If a node is unable to apply that config, the node needs to get fixed. I think of this like deploying a network configuration with ansible (or other tools). > I think, something easy, is that we could have a copy of each > /etc/network/interfaces of each node in /etc/pve/nodes//interfaces. > (could be done we a change is done in gui local netowrk, or local network > daemon copy it at regular interval in case of manual change for example). > > > Like this, it's very easy, when a network change is one at datacenter level, > we can directly test it on all network interfaces of all nodes ( > /etc/pve/nodes/*/interfaces). (in the api endpoint), I is still unclear to me how you do those tests? AFAIK, ifreload does not have a --dry-run option. Even when it has such option, it would need access to the local node? (to see what interfaces exists, ...). So if you really need/want to test before apply, we could add and API call for that: POST /api2/json/nodes//test_network_changes We can then add a TEST button to the GUI, or call those this test API on all nodes before we apply changes. > and then write directly the conf. (no need vnet.new tmp file). >>I think network configuration is really complex, and we should avoid to do >>anything automatically. >>I would prefer and "APPLY" button, so that I have full control over when >>network changes happen. >>Maybe an extra "TEST" button would be also helpful. Ok. (Anyway ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes
> I have rethinked about it, I have (again ;) a new idea for implementation. > > The main problem is how to test a change at datacenter level, as we need to > test the local configuration of each node. > > and it's not currently in /etc/pve , but in /etc/network/interfaces of each > node. I am not sure it this is a real problem. I think the cluster wide vnet config is the relevant config. If a node is unable to apply that config, the node needs to get fixed. I think of this like deploying a network configuration with ansible (or other tools). > I think, something easy, is that we could have a copy of each > /etc/network/interfaces of each node in /etc/pve/nodes//interfaces. > (could be done we a change is done in gui local netowrk, or local network > daemon copy it at regular interval in case of manual change for example). > > > Like this, it's very easy, when a network change is one at datacenter level, > we can directly test it on all network interfaces of all nodes ( > /etc/pve/nodes/*/interfaces). (in the api endpoint), I is still unclear to me how you do those tests? AFAIK, ifreload does not have a --dry-run option. Even when it has such option, it would need access to the local node? (to see what interfaces exists, ...). So if you really need/want to test before apply, we could add and API call for that: POST /api2/json/nodes//test_network_changes We can then add a TEST button to the GUI, or call those this test API on all nodes before we apply changes. > and then write directly the conf. (no need vnet.new tmp file). I think network configuration is really complex, and we should avoid to do anything automatically. I would prefer and "APPLY" button, so that I have full control over when network changes happen. Maybe an extra "TEST" button would be also helpful. ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes
Hi, I have rethinked about it, I have (again ;) a new idea for implementation. The main problem is how to test a change at datacenter level, as we need to test the local configuration of each node. and it's not currently in /etc/pve , but in /etc/network/interfaces of each node. I think, something easy, is that we could have a copy of each /etc/network/interfaces of each node in /etc/pve/nodes//interfaces. (could be done we a change is done in gui local netowrk, or local network daemon copy it at regular interval in case of manual change for example). Like this, it's very easy, when a network change is one at datacenter level, we can directly test it on all network interfaces of all nodes ( /etc/pve/nodes/*/interfaces). (in the api endpoint), and then write directly the conf. (no need vnet.new tmp file). Then the local daemon simply reload the network configuration. What do you think about this ? - Mail original - De: "aderumier" À: "pve-devel" Envoyé: Lundi 1 Avril 2019 15:18:51 Objet: Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes as alternative, we could simply manage multiple change in /etc/pve/network/vnet.cfg.new apply button -> replace /etc/pve/network/vnet.cfg The the local daemon, do test (dry-run,) and report error in his status file. (and it's displayed at network level in datacenter) if ok, it's apply change, and report error in his status file. if ok, update status to ok. So, user can wait some seconds, and check the status of nodes at datacenter level. Seem to be simplier. What do you think about this ? - Mail original - De: "Alexandre Derumier" À: "dietmar" Cc: "pve-devel" Envoyé: Lundi 1 Avril 2019 15:05:07 Objet: Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes >>I don't really get why you want to do that? There are so many ways to damage >>a network, and >>I doubt that we can reliable verify that ifupdown2 have a dry-run too, it's working not too bad (but not 100% complete) But I would avoid some basic mistakes, like a vlan interface already defined and enslaved in another bridge for example, or look to not enslave an interface with ipmanagement in a bridge (try to not break cluster connectivity) But I don't want to manage rollback across all nodes. (config correctly applied on 1 node, another node fail, I don't want to rollback the first node) It's more best effort, if 1 node have failed, it's simply report the error in his status file. >>Also, what if some nodes are offline ... We could make an exception, if a node is offline (down, network daemon down,...), Then don't wait for validation, and apply config. Then the local deamon will try to apply config when node is up again. In case of error, It'll report it through his status file. - Mail original - De: "dietmar" À: "Alexandre Derumier" , "pve-devel" Envoyé: Lundi 1 Avril 2019 12:00:13 Objet: Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes > maybe better: > > in gui, at network,datacenter level > > at each change, make a > /etc/pve/networks/vnet.cfg. > > > on local node, the daemon detect the new version,make verification, > and update /etc/pve/nodes//.networkconfigstatus > > version: verify:ok I don't really get why you want to do that? There are so many ways to damage a network, and I doubt that we can reliable verify that Also, what if some nodes are offline ... ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes
as alternative, we could simply manage multiple change in /etc/pve/network/vnet.cfg.new apply button -> replace /etc/pve/network/vnet.cfg The the local daemon, do test (dry-run,) and report error in his status file. (and it's displayed at network level in datacenter) if ok, it's apply change, and report error in his status file. if ok, update status to ok. So, user can wait some seconds, and check the status of nodes at datacenter level. Seem to be simplier. What do you think about this ? - Mail original - De: "Alexandre Derumier" À: "dietmar" Cc: "pve-devel" Envoyé: Lundi 1 Avril 2019 15:05:07 Objet: Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes >>I don't really get why you want to do that? There are so many ways to damage >>a network, and >>I doubt that we can reliable verify that ifupdown2 have a dry-run too, it's working not too bad (but not 100% complete) But I would avoid some basic mistakes, like a vlan interface already defined and enslaved in another bridge for example, or look to not enslave an interface with ipmanagement in a bridge (try to not break cluster connectivity) But I don't want to manage rollback across all nodes. (config correctly applied on 1 node, another node fail, I don't want to rollback the first node) It's more best effort, if 1 node have failed, it's simply report the error in his status file. >>Also, what if some nodes are offline ... We could make an exception, if a node is offline (down, network daemon down,...), Then don't wait for validation, and apply config. Then the local deamon will try to apply config when node is up again. In case of error, It'll report it through his status file. - Mail original - De: "dietmar" À: "Alexandre Derumier" , "pve-devel" Envoyé: Lundi 1 Avril 2019 12:00:13 Objet: Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes > maybe better: > > in gui, at network,datacenter level > > at each change, make a > /etc/pve/networks/vnet.cfg. > > > on local node, the daemon detect the new version,make verification, > and update /etc/pve/nodes//.networkconfigstatus > > version: verify:ok I don't really get why you want to do that? There are so many ways to damage a network, and I doubt that we can reliable verify that Also, what if some nodes are offline ... ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes
>>I don't really get why you want to do that? There are so many ways to damage >>a network, and >>I doubt that we can reliable verify that ifupdown2 have a dry-run too, it's working not too bad (but not 100% complete) But I would avoid some basic mistakes, like a vlan interface already defined and enslaved in another bridge for example, or look to not enslave an interface with ipmanagement in a bridge (try to not break cluster connectivity) But I don't want to manage rollback across all nodes. (config correctly applied on 1 node, another node fail, I don't want to rollback the first node) It's more best effort, if 1 node have failed, it's simply report the error in his status file. >>Also, what if some nodes are offline ... We could make an exception, if a node is offline (down, network daemon down,...), Then don't wait for validation, and apply config. Then the local deamon will try to apply config when node is up again. In case of error, It'll report it through his status file. - Mail original - De: "dietmar" À: "Alexandre Derumier" , "pve-devel" Envoyé: Lundi 1 Avril 2019 12:00:13 Objet: Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes > maybe better: > > in gui, at network,datacenter level > > at each change, make a > /etc/pve/networks/vnet.cfg. > > > on local node, the daemon detect the new version,make verification, > and update /etc/pve/nodes//.networkconfigstatus > > version: verify:ok I don't really get why you want to do that? There are so many ways to damage a network, and I doubt that we can reliable verify that Also, what if some nodes are offline ... ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes
> maybe better: > > in gui, at network,datacenter level > > at each change, make a > /etc/pve/networks/vnet.cfg. > > > on local node, the daemon detect the new version,make verification, > and update /etc/pve/nodes//.networkconfigstatus > > version: verify:ok I don't really get why you want to do that? There are so many ways to damage a network, and I doubt that we can reliable verify that Also, what if some nodes are offline ... ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes
maybe better: in gui, at network,datacenter level at each change, make a /etc/pve/networks/vnet.cfg. on local node, the daemon detect the new version,make verification, and update /etc/pve/nodes//.networkconfigstatus version: verify:ok Then at datacenter level, the apply button is grey-out by default, if all local daemon verify version are ok (matching the vnet.cfg.), enable the apply button. Then on apply, mv /etc/pve/networks/vnet.cfg. /etc/pve/networks/vnet.cfg and local node apply with new config and reload network - Mail original - De: "Alexandre Derumier" À: "dietmar" Cc: "pve-devel" Envoyé: Lundi 1 Avril 2019 07:48:07 Objet: Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes >>Can't we simply add a manual "apply" button for now? Yes, I have thinked a little bit more about it. But I really would like to have some kind of verification before apply. >>Just by using backup config files: >> >>vnet.cfg.new >>vnet.cfg I'm not sure, as we could apply the change, the change is not apply on some nodes, then we do other changes. Maybe: 1) do a change in vnet.cfg 2) the local daemon, could compute the md5 of the vnet.cfg and test if it's ok and store it in /etc/pve/nodes//.networkconfigstatus. (version:hash status:validateok) 3) then we are able to apply configuration ... This will avoid the "test button", as the daemon is always doing it on config change in background. - Mail original - De: "dietmar" À: "aderumier" Cc: "pve-devel" Envoyé: Vendredi 29 Mars 2019 17:03:02 Objet: Re: rfc : pve-network : idea to generate and reload config accross the nodes > I think it's better than blindly try to reload networkconfig on each change, > as network is critical, > and sometimes admin need to change multiple parameters and apply it once. Can't we simply add a manual "apply" button for now? Just by using backup config files: vnet.cfg.new vnet.cfg ? ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes
>>Can't we simply add a manual "apply" button for now? Yes, I have thinked a little bit more about it. But I really would like to have some kind of verification before apply. >>Just by using backup config files: >> >>vnet.cfg.new >>vnet.cfg I'm not sure, as we could apply the change, the change is not apply on some nodes, then we do other changes. Maybe: 1) do a change in vnet.cfg 2) the local daemon, could compute the md5 of the vnet.cfg and test if it's ok and store it in /etc/pve/nodes//.networkconfigstatus. (version:hash status:validateok) 3) then we are able to apply configuration ... This will avoid the "test button", as the daemon is always doing it on config change in background. - Mail original - De: "dietmar" À: "aderumier" Cc: "pve-devel" Envoyé: Vendredi 29 Mars 2019 17:03:02 Objet: Re: rfc : pve-network : idea to generate and reload config accross the nodes > I think it's better than blindly try to reload networkconfig on each change, > as network is critical, > and sometimes admin need to change multiple parameters and apply it once. Can't we simply add a manual "apply" button for now? Just by using backup config files: vnet.cfg.new vnet.cfg ? ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes
> I think it's better than blindly try to reload networkconfig on each change, > as network is critical, > and sometimes admin need to change multiple parameters and apply it once. Can't we simply add a manual "apply" button for now? Just by using backup config files: vnet.cfg.new vnet.cfg ? ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
Re: [pve-devel] rfc : pve-network : idea to generate and reload config accross the nodes
I have always tried to avoid such things. They are clumsy and error prone > I have thinked about a way to generate config and reload it to differents > nodes > > > " > > make changes in /etc/pve/network/*.cfg > > at datacenter level, network panel , click button ->verify config, > this create a > /etc/pve/nodes//.networkconfigstatus for all nodes (with something like > verify:pending) > > each local node (daemon) will generate the configuration and validate if they > are no conflict/error > The ok/error with message will be stored in .networkconfigstatus (verify:ok) > > Reporting of status of all nodes is displayed at datacenter level. > > > If all nodes verification state are Ok, we can click a button "apply config", > > then local daemon apply configuration and reload. > status ok/error is reported in /etc/pve/nodes//.networkconfigstatus > > " > > I think it's better than blindly try to reload networkconfig on each change, > as network is critical, > and sometimes admin need to change multiple parameters and apply it once. > > > What do you think about this ? > ___ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
