Re: [ovs-discuss] [OVN] Aging mechanism for MAC_Binding table
On Wed, Aug 21, 2019 at 6:47 PM Ben Pfaff wrote: > > On Tue, Aug 20, 2019 at 06:11:12PM -0700, Han Zhou wrote: > > On Tue, Aug 20, 2019 at 4:57 PM Ben Pfaff wrote: > > > > > > Let me see if I'm following this correctly. This is what currently > > > happens: > > > > > > - HV1 needs a MAC address for an IP so it broadcasts an ARP request. > > > > > > - The port with the IP address, on HV2, causes the MAC_Binding to be > > > inserted. > > > > > > - Every ovn-controller inserts an OF flow for the binding. HV1 and > > > perhaps other ovn-controllers use this flow to populate the MAC > > > address for subsequent packets destined to the IP address in question. > > > > > > This proposal augments that with: > > > > > > - After a while, the binding goes idle and isn't used. The > > > ovn-controllers gradually notice this and delete their OF flows for > > > it. > > > > > > - HV3 eventually needs the binding again. It broadcasts an ARP request. > > > > > > - The port with the IP address causes the MAC_Binding to be inserted. > > > This might still be on HV2 if the port hasn't moved, or it might be on > > > HV4 if it has. > > > > > > Is that what you mean? It might work OK. > > > > > > Please do update the lifetime description in ovn-sb(5) under the > > > MAC_Binding table regardless of what you implement. > > > > > > Thanks, > > > > > > Ben. > > > > > > On Tue, Aug 20, 2019 at 09:03:57AM +0200, Daniel Alvarez Sanchez wrote: > > > > Hi folks, > > > > > > > > Reviving this thread as we're seeing this more and more problematic. > > > > Combining the ideas mentioned up thread, Dumitru, Numan, Lucas and I > > > > had some internal discussion where we came up with a possible approach > > > > and we'd love to get feedback from you: > > > > > > > > - Local ovn-controller will always insert an OF rule per MAC_Binding > > > > entry to match on src_ip + src_mac that will be sampled with a meter > > > > to ovn-controller. > > > > - When ovn-controller sees that one entry has not been hit "for a > > > > while", it'll delete the OpenFlow rule in table 65 that fills the > > > > eth.dst field with the MAC_Binding info. > > > > I assume the rules in table 65 can be "extended" for this purpose, instead > > of adding extra rules for this. > > > > > > - This will result in further ARP requests from the instance(s) that > > > > will refresh the MAC_Binding entries in the database. > > > > > > > > This could make troubleshooting a bit harder so at some point it'll be > > > > great to have a mechanism in OVS where we could disable a flow instead > > > > of deleting it. This way, one can tell that the flows in table 65 have > > > > been disabled due to the aging mechanism in the local node. > > > > Sorry that I didn't understand this. Why do you want the flow being > > disabled instead of deleted? I think if we want to avoid stale entries, we > > do want to delete them, so that the stale data doesn't occupy the space in > > flow table, neither in SB DB. It may be ok to add debug log for deleting a > > aged entry in ovn-controller, for trouble shooting purpose? > > > > > > > > > > Thoughts? Is there any performance consideration regarding the extra > > > > flows and meters? > > > > Are you proposing shared meters or one meter per mac-binding? If it is per > > mac-binding, I would be worried about the scalability considering that we > > may have >10k of mac-bindings. Or should I be worried? Maybe Justin and Ben > > can comment on the meter scalability. If it is a concern, I would suggest > > the feature be configurable (i.e. enable/disable), so that it can be > > enabled in environments where aging is required but number of mac-bindings > > are not very high. > > That *is* a lot of meters. I'd suggest at least testing to make sure > that it scales. I don't recall enough about the implementation to > guess. I guess sharing meters is an option if we're worried about scaling issues. We could implement something similar to what Han suggested for ARP packets [1]. That could potentially trigger more ARP requests from ovn-controller and more load on ovn-controller so we should probably allow the user to configure the way meters are created for these flows (e.g., meter per flow or meter bucket size). Thanks, Dumitru [1] https://mail.openvswitch.org/pipermail/ovs-dev/2019-August/361482.html > ___ > discuss mailing list > disc...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] [OVN] Aging mechanism for MAC_Binding table
On Tue, Aug 20, 2019 at 06:11:12PM -0700, Han Zhou wrote: > On Tue, Aug 20, 2019 at 4:57 PM Ben Pfaff wrote: > > > > Let me see if I'm following this correctly. This is what currently > > happens: > > > > - HV1 needs a MAC address for an IP so it broadcasts an ARP request. > > > > - The port with the IP address, on HV2, causes the MAC_Binding to be > > inserted. > > > > - Every ovn-controller inserts an OF flow for the binding. HV1 and > > perhaps other ovn-controllers use this flow to populate the MAC > > address for subsequent packets destined to the IP address in question. > > > > This proposal augments that with: > > > > - After a while, the binding goes idle and isn't used. The > > ovn-controllers gradually notice this and delete their OF flows for > > it. > > > > - HV3 eventually needs the binding again. It broadcasts an ARP request. > > > > - The port with the IP address causes the MAC_Binding to be inserted. > > This might still be on HV2 if the port hasn't moved, or it might be on > > HV4 if it has. > > > > Is that what you mean? It might work OK. > > > > Please do update the lifetime description in ovn-sb(5) under the > > MAC_Binding table regardless of what you implement. > > > > Thanks, > > > > Ben. > > > > On Tue, Aug 20, 2019 at 09:03:57AM +0200, Daniel Alvarez Sanchez wrote: > > > Hi folks, > > > > > > Reviving this thread as we're seeing this more and more problematic. > > > Combining the ideas mentioned up thread, Dumitru, Numan, Lucas and I > > > had some internal discussion where we came up with a possible approach > > > and we'd love to get feedback from you: > > > > > > - Local ovn-controller will always insert an OF rule per MAC_Binding > > > entry to match on src_ip + src_mac that will be sampled with a meter > > > to ovn-controller. > > > - When ovn-controller sees that one entry has not been hit "for a > > > while", it'll delete the OpenFlow rule in table 65 that fills the > > > eth.dst field with the MAC_Binding info. > > I assume the rules in table 65 can be "extended" for this purpose, instead > of adding extra rules for this. > > > > - This will result in further ARP requests from the instance(s) that > > > will refresh the MAC_Binding entries in the database. > > > > > > This could make troubleshooting a bit harder so at some point it'll be > > > great to have a mechanism in OVS where we could disable a flow instead > > > of deleting it. This way, one can tell that the flows in table 65 have > > > been disabled due to the aging mechanism in the local node. > > Sorry that I didn't understand this. Why do you want the flow being > disabled instead of deleted? I think if we want to avoid stale entries, we > do want to delete them, so that the stale data doesn't occupy the space in > flow table, neither in SB DB. It may be ok to add debug log for deleting a > aged entry in ovn-controller, for trouble shooting purpose? > > > > > > > Thoughts? Is there any performance consideration regarding the extra > > > flows and meters? > > Are you proposing shared meters or one meter per mac-binding? If it is per > mac-binding, I would be worried about the scalability considering that we > may have >10k of mac-bindings. Or should I be worried? Maybe Justin and Ben > can comment on the meter scalability. If it is a concern, I would suggest > the feature be configurable (i.e. enable/disable), so that it can be > enabled in environments where aging is required but number of mac-bindings > are not very high. That *is* a lot of meters. I'd suggest at least testing to make sure that it scales. I don't recall enough about the implementation to guess. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] [OVN] Aging mechanism for MAC_Binding table
On Wed, Aug 21, 2019 at 3:11 AM Han Zhou wrote: > > > > On Tue, Aug 20, 2019 at 4:57 PM Ben Pfaff wrote: > > > > Let me see if I'm following this correctly. This is what currently > > happens: > > > > - HV1 needs a MAC address for an IP so it broadcasts an ARP request. > > > > - The port with the IP address, on HV2, causes the MAC_Binding to be > > inserted. > > > > - Every ovn-controller inserts an OF flow for the binding. HV1 and > > perhaps other ovn-controllers use this flow to populate the MAC > > address for subsequent packets destined to the IP address in question. > > > > This proposal augments that with: > > > > - After a while, the binding goes idle and isn't used. The > > ovn-controllers gradually notice this and delete their OF flows for > > it. > > > > - HV3 eventually needs the binding again. It broadcasts an ARP request. > > > > - The port with the IP address causes the MAC_Binding to be inserted. > > This might still be on HV2 if the port hasn't moved, or it might be on > > HV4 if it has. > > > > Is that what you mean? It might work OK. Yes, that's it. At some point we can look into enhancing this using the SB DB and if all ovn-controllers decided to ignore a particular MAC_Binding entry, then we can remove it from the DB from ovn-northd (or some other mechanism). > > > > Please do update the lifetime description in ovn-sb(5) under the > > MAC_Binding table regardless of what you implement. > > > > Thanks, > > > > Ben. > > > > On Tue, Aug 20, 2019 at 09:03:57AM +0200, Daniel Alvarez Sanchez wrote: > > > Hi folks, > > > > > > Reviving this thread as we're seeing this more and more problematic. > > > Combining the ideas mentioned up thread, Dumitru, Numan, Lucas and I > > > had some internal discussion where we came up with a possible approach > > > and we'd love to get feedback from you: > > > > > > - Local ovn-controller will always insert an OF rule per MAC_Binding > > > entry to match on src_ip + src_mac that will be sampled with a meter > > > to ovn-controller. > > > - When ovn-controller sees that one entry has not been hit "for a > > > while", it'll delete the OpenFlow rule in table 65 that fills the > > > eth.dst field with the MAC_Binding info. > > I assume the rules in table 65 can be "extended" for this purpose, instead of > adding extra rules for this. > > > > - This will result in further ARP requests from the instance(s) that > > > will refresh the MAC_Binding entries in the database. > > > > > > This could make troubleshooting a bit harder so at some point it'll be > > > great to have a mechanism in OVS where we could disable a flow instead > > > of deleting it. This way, one can tell that the flows in table 65 have > > > been disabled due to the aging mechanism in the local node. > > Sorry that I didn't understand this. Why do you want the flow being disabled > instead of deleted? I think if we want to avoid stale entries, we do want to > delete them, so that the stale data doesn't occupy the space in flow table, > neither in SB DB. It may be ok to add debug log for deleting a aged entry in > ovn-controller, for trouble shooting purpose? We can use traces as well, yes :) > > > > > > > Thoughts? Is there any performance consideration regarding the extra > > > flows and meters? > > Are you proposing shared meters or one meter per mac-binding? If it is per > mac-binding, I would be worried about the scalability considering that we may > have >10k of mac-bindings. Or should I be worried? Maybe Justin and Ben can > comment on the meter scalability. If it is a concern, I would suggest the > feature be configurable (i.e. enable/disable), so that it can be enabled in > environments where aging is required but number of mac-bindings are not very > high. I was talking about one meter per mac-binding but I'll defer the answer to others, as I don't know much about meters. I'm not a big fan of configuration options but unless we have a clear view on this, it makes sense to me to have a knob for the 'aging'. > > > > > > > Thanks a lot! > > > Daniel > > > > > > > > > On Tue, Jul 9, 2019 at 7:19 AM Ben Pfaff wrote: > > > > > > > > On Mon, Jul 08, 2019 at 06:19:23PM -0700, Han Zhou wrote: > > > > > On Thu, Jun 27, 2019 at 6:44 AM Ben Pfaff wrote: > > > > > > > > > > > > On Tue, Jun 25, 2019 at 01:05:21PM +0200, Daniel Alvarez Sanchez > > > > > > wrote: > > > > > > > Lately we've been trying to solve certain issues related to stale > > > > > > > entries in the MAC_Binding table (e.g. [0]). On the other hand, > > > > > > > for > > > > > > > the OpenStack + Octavia (Load Balancing service) use case, we see > > > > > > > that > > > > > > > a reused VIP can be as well affected by stale entries in this > > > > > > > table > > > > > > > due to the fact that it's never bound to a VIF so ovn-controller > > > > > > > won't > > > > > > > claim it and send the GARPs to update the neighbors. > > > > > > > > > > > > > > I'm not sure if other scenarios may suffer
Re: [ovs-discuss] [OVN] Aging mechanism for MAC_Binding table
On Tue, Aug 20, 2019 at 4:57 PM Ben Pfaff wrote: > > Let me see if I'm following this correctly. This is what currently > happens: > > - HV1 needs a MAC address for an IP so it broadcasts an ARP request. > > - The port with the IP address, on HV2, causes the MAC_Binding to be > inserted. > > - Every ovn-controller inserts an OF flow for the binding. HV1 and > perhaps other ovn-controllers use this flow to populate the MAC > address for subsequent packets destined to the IP address in question. > > This proposal augments that with: > > - After a while, the binding goes idle and isn't used. The > ovn-controllers gradually notice this and delete their OF flows for > it. > > - HV3 eventually needs the binding again. It broadcasts an ARP request. > > - The port with the IP address causes the MAC_Binding to be inserted. > This might still be on HV2 if the port hasn't moved, or it might be on > HV4 if it has. > > Is that what you mean? It might work OK. > > Please do update the lifetime description in ovn-sb(5) under the > MAC_Binding table regardless of what you implement. > > Thanks, > > Ben. > > On Tue, Aug 20, 2019 at 09:03:57AM +0200, Daniel Alvarez Sanchez wrote: > > Hi folks, > > > > Reviving this thread as we're seeing this more and more problematic. > > Combining the ideas mentioned up thread, Dumitru, Numan, Lucas and I > > had some internal discussion where we came up with a possible approach > > and we'd love to get feedback from you: > > > > - Local ovn-controller will always insert an OF rule per MAC_Binding > > entry to match on src_ip + src_mac that will be sampled with a meter > > to ovn-controller. > > - When ovn-controller sees that one entry has not been hit "for a > > while", it'll delete the OpenFlow rule in table 65 that fills the > > eth.dst field with the MAC_Binding info. I assume the rules in table 65 can be "extended" for this purpose, instead of adding extra rules for this. > > - This will result in further ARP requests from the instance(s) that > > will refresh the MAC_Binding entries in the database. > > > > This could make troubleshooting a bit harder so at some point it'll be > > great to have a mechanism in OVS where we could disable a flow instead > > of deleting it. This way, one can tell that the flows in table 65 have > > been disabled due to the aging mechanism in the local node. Sorry that I didn't understand this. Why do you want the flow being disabled instead of deleted? I think if we want to avoid stale entries, we do want to delete them, so that the stale data doesn't occupy the space in flow table, neither in SB DB. It may be ok to add debug log for deleting a aged entry in ovn-controller, for trouble shooting purpose? > > > > Thoughts? Is there any performance consideration regarding the extra > > flows and meters? Are you proposing shared meters or one meter per mac-binding? If it is per mac-binding, I would be worried about the scalability considering that we may have >10k of mac-bindings. Or should I be worried? Maybe Justin and Ben can comment on the meter scalability. If it is a concern, I would suggest the feature be configurable (i.e. enable/disable), so that it can be enabled in environments where aging is required but number of mac-bindings are not very high. > > > > Thanks a lot! > > Daniel > > > > > > On Tue, Jul 9, 2019 at 7:19 AM Ben Pfaff wrote: > > > > > > On Mon, Jul 08, 2019 at 06:19:23PM -0700, Han Zhou wrote: > > > > On Thu, Jun 27, 2019 at 6:44 AM Ben Pfaff wrote: > > > > > > > > > > On Tue, Jun 25, 2019 at 01:05:21PM +0200, Daniel Alvarez Sanchez wrote: > > > > > > Lately we've been trying to solve certain issues related to stale > > > > > > entries in the MAC_Binding table (e.g. [0]). On the other hand, for > > > > > > the OpenStack + Octavia (Load Balancing service) use case, we see that > > > > > > a reused VIP can be as well affected by stale entries in this table > > > > > > due to the fact that it's never bound to a VIF so ovn-controller won't > > > > > > claim it and send the GARPs to update the neighbors. > > > > > > > > > > > > I'm not sure if other scenarios may suffer from this issue but seems > > > > > > reasonable to have an aging mechanism (as we discussed at some point > > > > > > in the past) that makes unused/old entries to expire. After talking to > > > > > > Numan on IRC, since a new pinctrl thread has been introduced recently > > > > > > [1], it'd be nice to implement this aging mechanism there. > > > > > > At the same time we'd be also reducing the amount of entries for long > > > > > > lived systems as it'd grow indefinitely. > > > > > > > > > > > > Any thoughts? > > > > > > > > > > > > Thanks! > > > > > > Daniel > > > > > > > > > > > > PS. With regards to the 'unused' vs 'old' entries I think it has to be > > > > > > 'old' rather than 'unused' as I don't see a way to reset the TTL of a > > > > > > MAC_Binding entry when we see packets coming. The implication is that > > > > > > we'll be seeing
Re: [ovs-discuss] [OVN] Aging mechanism for MAC_Binding table
Let me see if I'm following this correctly. This is what currently happens: - HV1 needs a MAC address for an IP so it broadcasts an ARP request. - The port with the IP address, on HV2, causes the MAC_Binding to be inserted. - Every ovn-controller inserts an OF flow for the binding. HV1 and perhaps other ovn-controllers use this flow to populate the MAC address for subsequent packets destined to the IP address in question. This proposal augments that with: - After a while, the binding goes idle and isn't used. The ovn-controllers gradually notice this and delete their OF flows for it. - HV3 eventually needs the binding again. It broadcasts an ARP request. - The port with the IP address causes the MAC_Binding to be inserted. This might still be on HV2 if the port hasn't moved, or it might be on HV4 if it has. Is that what you mean? It might work OK. Please do update the lifetime description in ovn-sb(5) under the MAC_Binding table regardless of what you implement. Thanks, Ben. On Tue, Aug 20, 2019 at 09:03:57AM +0200, Daniel Alvarez Sanchez wrote: > Hi folks, > > Reviving this thread as we're seeing this more and more problematic. > Combining the ideas mentioned up thread, Dumitru, Numan, Lucas and I > had some internal discussion where we came up with a possible approach > and we'd love to get feedback from you: > > - Local ovn-controller will always insert an OF rule per MAC_Binding > entry to match on src_ip + src_mac that will be sampled with a meter > to ovn-controller. > - When ovn-controller sees that one entry has not been hit "for a > while", it'll delete the OpenFlow rule in table 65 that fills the > eth.dst field with the MAC_Binding info. > - This will result in further ARP requests from the instance(s) that > will refresh the MAC_Binding entries in the database. > > This could make troubleshooting a bit harder so at some point it'll be > great to have a mechanism in OVS where we could disable a flow instead > of deleting it. This way, one can tell that the flows in table 65 have > been disabled due to the aging mechanism in the local node. > > Thoughts? Is there any performance consideration regarding the extra > flows and meters? > > Thanks a lot! > Daniel > > > On Tue, Jul 9, 2019 at 7:19 AM Ben Pfaff wrote: > > > > On Mon, Jul 08, 2019 at 06:19:23PM -0700, Han Zhou wrote: > > > On Thu, Jun 27, 2019 at 6:44 AM Ben Pfaff wrote: > > > > > > > > On Tue, Jun 25, 2019 at 01:05:21PM +0200, Daniel Alvarez Sanchez wrote: > > > > > Lately we've been trying to solve certain issues related to stale > > > > > entries in the MAC_Binding table (e.g. [0]). On the other hand, for > > > > > the OpenStack + Octavia (Load Balancing service) use case, we see that > > > > > a reused VIP can be as well affected by stale entries in this table > > > > > due to the fact that it's never bound to a VIF so ovn-controller won't > > > > > claim it and send the GARPs to update the neighbors. > > > > > > > > > > I'm not sure if other scenarios may suffer from this issue but seems > > > > > reasonable to have an aging mechanism (as we discussed at some point > > > > > in the past) that makes unused/old entries to expire. After talking to > > > > > Numan on IRC, since a new pinctrl thread has been introduced recently > > > > > [1], it'd be nice to implement this aging mechanism there. > > > > > At the same time we'd be also reducing the amount of entries for long > > > > > lived systems as it'd grow indefinitely. > > > > > > > > > > Any thoughts? > > > > > > > > > > Thanks! > > > > > Daniel > > > > > > > > > > PS. With regards to the 'unused' vs 'old' entries I think it has to be > > > > > 'old' rather than 'unused' as I don't see a way to reset the TTL of a > > > > > MAC_Binding entry when we see packets coming. The implication is that > > > > > we'll be seeing ARPs sent out more often when perhaps they're not > > > > > needed. This also leads to the discussion of making the cache timeout > > > > > configurable. > > > > > > > > I've always considered the MAC_Binding implementation incomplete because > > > > of this issue and others. ovn/TODO.rst says: > > > > > > > > * Dynamic IP to MAC binding enhancements. > > > > > > > > OVN has basic support for establishing IP to MAC bindings > > > dynamically, using > > > > ARP. > > > > > > > > * Ratelimiting. > > > > > > > > From casual observation, Linux appears to generate at most one > > > ARP per > > > > second per destination. > > > > > > > > This might be supported by adding a new OVN logical action for > > > > rate-limiting. > > > > > > > > * Tracking queries > > > > > > > > It's probably best to only record in the database responses to > > > queries > > > > actually issued by an L3 logical router, so somehow they have > > > > to > > > be > > > > tracked, probably by putting a tentative binding without a MAC > > > address > > > > into the
Re: [ovs-discuss] [OVN] Aging mechanism for MAC_Binding table
Hi folks, Reviving this thread as we're seeing this more and more problematic. Combining the ideas mentioned up thread, Dumitru, Numan, Lucas and I had some internal discussion where we came up with a possible approach and we'd love to get feedback from you: - Local ovn-controller will always insert an OF rule per MAC_Binding entry to match on src_ip + src_mac that will be sampled with a meter to ovn-controller. - When ovn-controller sees that one entry has not been hit "for a while", it'll delete the OpenFlow rule in table 65 that fills the eth.dst field with the MAC_Binding info. - This will result in further ARP requests from the instance(s) that will refresh the MAC_Binding entries in the database. This could make troubleshooting a bit harder so at some point it'll be great to have a mechanism in OVS where we could disable a flow instead of deleting it. This way, one can tell that the flows in table 65 have been disabled due to the aging mechanism in the local node. Thoughts? Is there any performance consideration regarding the extra flows and meters? Thanks a lot! Daniel On Tue, Jul 9, 2019 at 7:19 AM Ben Pfaff wrote: > > On Mon, Jul 08, 2019 at 06:19:23PM -0700, Han Zhou wrote: > > On Thu, Jun 27, 2019 at 6:44 AM Ben Pfaff wrote: > > > > > > On Tue, Jun 25, 2019 at 01:05:21PM +0200, Daniel Alvarez Sanchez wrote: > > > > Lately we've been trying to solve certain issues related to stale > > > > entries in the MAC_Binding table (e.g. [0]). On the other hand, for > > > > the OpenStack + Octavia (Load Balancing service) use case, we see that > > > > a reused VIP can be as well affected by stale entries in this table > > > > due to the fact that it's never bound to a VIF so ovn-controller won't > > > > claim it and send the GARPs to update the neighbors. > > > > > > > > I'm not sure if other scenarios may suffer from this issue but seems > > > > reasonable to have an aging mechanism (as we discussed at some point > > > > in the past) that makes unused/old entries to expire. After talking to > > > > Numan on IRC, since a new pinctrl thread has been introduced recently > > > > [1], it'd be nice to implement this aging mechanism there. > > > > At the same time we'd be also reducing the amount of entries for long > > > > lived systems as it'd grow indefinitely. > > > > > > > > Any thoughts? > > > > > > > > Thanks! > > > > Daniel > > > > > > > > PS. With regards to the 'unused' vs 'old' entries I think it has to be > > > > 'old' rather than 'unused' as I don't see a way to reset the TTL of a > > > > MAC_Binding entry when we see packets coming. The implication is that > > > > we'll be seeing ARPs sent out more often when perhaps they're not > > > > needed. This also leads to the discussion of making the cache timeout > > > > configurable. > > > > > > I've always considered the MAC_Binding implementation incomplete because > > > of this issue and others. ovn/TODO.rst says: > > > > > > * Dynamic IP to MAC binding enhancements. > > > > > > OVN has basic support for establishing IP to MAC bindings > > dynamically, using > > > ARP. > > > > > > * Ratelimiting. > > > > > > From casual observation, Linux appears to generate at most one > > ARP per > > > second per destination. > > > > > > This might be supported by adding a new OVN logical action for > > > rate-limiting. > > > > > > * Tracking queries > > > > > > It's probably best to only record in the database responses to > > queries > > > actually issued by an L3 logical router, so somehow they have to > > be > > > tracked, probably by putting a tentative binding without a MAC > > address > > > into the database. > > > > > > * Renewal and expiration. > > > > > > Something needs to make sure that bindings remain valid and > > expire those > > > that become stale. > > > > > > One way to do this might be to add some support for time to the > > database > > > server itself. > > > > > > * Table size limiting. > > > > > > The table of MAC bindings must not be allowed to grow > > unreasonably large. > > > > > > * MTU handling (fragmentation on output) > > > > > > So, what do we do about it? First, I think that adding support for time > > > to the database server is a terrible idea (even though I think I wrote > > > the above originally). Let's not do that. The following is some > > > "thinking out loud" on the subject. > > > > > > I think there's a challenge around which ovn-controller should take care > > > of a given MAC_Binding. We don't want every ovn-controller expiring > > > every binding. Ideally, we want exactly one ovn-controller expiring a > > > binding. One way would be to add an owner column (but it would be > > > better if we don't need it). > > > > > > If we want to keep track of "unused" bindings, I can imagine a > > > statistical mechanism to do that. Any user of a binding occasionally >
Re: [ovs-discuss] [OVN] Aging mechanism for MAC_Binding table
On Mon, Jul 08, 2019 at 06:19:23PM -0700, Han Zhou wrote: > On Thu, Jun 27, 2019 at 6:44 AM Ben Pfaff wrote: > > > > On Tue, Jun 25, 2019 at 01:05:21PM +0200, Daniel Alvarez Sanchez wrote: > > > Lately we've been trying to solve certain issues related to stale > > > entries in the MAC_Binding table (e.g. [0]). On the other hand, for > > > the OpenStack + Octavia (Load Balancing service) use case, we see that > > > a reused VIP can be as well affected by stale entries in this table > > > due to the fact that it's never bound to a VIF so ovn-controller won't > > > claim it and send the GARPs to update the neighbors. > > > > > > I'm not sure if other scenarios may suffer from this issue but seems > > > reasonable to have an aging mechanism (as we discussed at some point > > > in the past) that makes unused/old entries to expire. After talking to > > > Numan on IRC, since a new pinctrl thread has been introduced recently > > > [1], it'd be nice to implement this aging mechanism there. > > > At the same time we'd be also reducing the amount of entries for long > > > lived systems as it'd grow indefinitely. > > > > > > Any thoughts? > > > > > > Thanks! > > > Daniel > > > > > > PS. With regards to the 'unused' vs 'old' entries I think it has to be > > > 'old' rather than 'unused' as I don't see a way to reset the TTL of a > > > MAC_Binding entry when we see packets coming. The implication is that > > > we'll be seeing ARPs sent out more often when perhaps they're not > > > needed. This also leads to the discussion of making the cache timeout > > > configurable. > > > > I've always considered the MAC_Binding implementation incomplete because > > of this issue and others. ovn/TODO.rst says: > > > > * Dynamic IP to MAC binding enhancements. > > > > OVN has basic support for establishing IP to MAC bindings > dynamically, using > > ARP. > > > > * Ratelimiting. > > > > From casual observation, Linux appears to generate at most one > ARP per > > second per destination. > > > > This might be supported by adding a new OVN logical action for > > rate-limiting. > > > > * Tracking queries > > > > It's probably best to only record in the database responses to > queries > > actually issued by an L3 logical router, so somehow they have to > be > > tracked, probably by putting a tentative binding without a MAC > address > > into the database. > > > > * Renewal and expiration. > > > > Something needs to make sure that bindings remain valid and > expire those > > that become stale. > > > > One way to do this might be to add some support for time to the > database > > server itself. > > > > * Table size limiting. > > > > The table of MAC bindings must not be allowed to grow > unreasonably large. > > > > * MTU handling (fragmentation on output) > > > > So, what do we do about it? First, I think that adding support for time > > to the database server is a terrible idea (even though I think I wrote > > the above originally). Let's not do that. The following is some > > "thinking out loud" on the subject. > > > > I think there's a challenge around which ovn-controller should take care > > of a given MAC_Binding. We don't want every ovn-controller expiring > > every binding. Ideally, we want exactly one ovn-controller expiring a > > binding. One way would be to add an owner column (but it would be > > better if we don't need it). > > > > If we want to keep track of "unused" bindings, I can imagine a > > statistical mechanism to do that. Any user of a binding occasionally > > and probabilistically changes a serial number column that we'd introduce > > into the MAC_Binding table (this could be optimized to not bother if it > > has changed recently). The owner checks the serial number every so > > often and if it hasn't changed then it deletes the row. > > > > Thanks Ben for the advice. Since the user of a binding is simply a OpenFlow > rule matching, I guess we will need "controller" action to trigger the > serial number column update in ovn-controller, combined with a meter action > so that only small number of packets trigger the update. Is this what you > are suggesting? I had not thought that far ahead! That approach would work, although the trigger percentage would be difficult to figure out--it seems like really we'd want "every Nth second", not "every Nth packet". Another approach that might work would be for ovn-controller to notice the statistics on appropriate OpenFlow flows changing, or to use "learn" actions as a way to make a controller action trigger only every so often. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] [OVN] Aging mechanism for MAC_Binding table
On Thu, Jun 27, 2019 at 6:44 AM Ben Pfaff wrote: > > On Tue, Jun 25, 2019 at 01:05:21PM +0200, Daniel Alvarez Sanchez wrote: > > Lately we've been trying to solve certain issues related to stale > > entries in the MAC_Binding table (e.g. [0]). On the other hand, for > > the OpenStack + Octavia (Load Balancing service) use case, we see that > > a reused VIP can be as well affected by stale entries in this table > > due to the fact that it's never bound to a VIF so ovn-controller won't > > claim it and send the GARPs to update the neighbors. > > > > I'm not sure if other scenarios may suffer from this issue but seems > > reasonable to have an aging mechanism (as we discussed at some point > > in the past) that makes unused/old entries to expire. After talking to > > Numan on IRC, since a new pinctrl thread has been introduced recently > > [1], it'd be nice to implement this aging mechanism there. > > At the same time we'd be also reducing the amount of entries for long > > lived systems as it'd grow indefinitely. > > > > Any thoughts? > > > > Thanks! > > Daniel > > > > PS. With regards to the 'unused' vs 'old' entries I think it has to be > > 'old' rather than 'unused' as I don't see a way to reset the TTL of a > > MAC_Binding entry when we see packets coming. The implication is that > > we'll be seeing ARPs sent out more often when perhaps they're not > > needed. This also leads to the discussion of making the cache timeout > > configurable. > > I've always considered the MAC_Binding implementation incomplete because > of this issue and others. ovn/TODO.rst says: > > * Dynamic IP to MAC binding enhancements. > > OVN has basic support for establishing IP to MAC bindings dynamically, using > ARP. > > * Ratelimiting. > > From casual observation, Linux appears to generate at most one ARP per > second per destination. > > This might be supported by adding a new OVN logical action for > rate-limiting. > > * Tracking queries > > It's probably best to only record in the database responses to queries > actually issued by an L3 logical router, so somehow they have to be > tracked, probably by putting a tentative binding without a MAC address > into the database. > > * Renewal and expiration. > > Something needs to make sure that bindings remain valid and expire those > that become stale. > > One way to do this might be to add some support for time to the database > server itself. > > * Table size limiting. > > The table of MAC bindings must not be allowed to grow unreasonably large. > > * MTU handling (fragmentation on output) > > So, what do we do about it? First, I think that adding support for time > to the database server is a terrible idea (even though I think I wrote > the above originally). Let's not do that. The following is some > "thinking out loud" on the subject. > > I think there's a challenge around which ovn-controller should take care > of a given MAC_Binding. We don't want every ovn-controller expiring > every binding. Ideally, we want exactly one ovn-controller expiring a > binding. One way would be to add an owner column (but it would be > better if we don't need it). > > If we want to keep track of "unused" bindings, I can imagine a > statistical mechanism to do that. Any user of a binding occasionally > and probabilistically changes a serial number column that we'd introduce > into the MAC_Binding table (this could be optimized to not bother if it > has changed recently). The owner checks the serial number every so > often and if it hasn't changed then it deletes the row. > Thanks Ben for the advice. Since the user of a binding is simply a OpenFlow rule matching, I guess we will need "controller" action to trigger the serial number column update in ovn-controller, combined with a meter action so that only small number of packets trigger the update. Is this what you are suggesting? > The owner could also occasionally revalidate the binding. > > Any thoughts? > > Thanks, > > Ben. > ___ > discuss mailing list > disc...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] [OVN] Aging mechanism for MAC_Binding table
On Tue, Jun 25, 2019 at 01:05:21PM +0200, Daniel Alvarez Sanchez wrote: > Lately we've been trying to solve certain issues related to stale > entries in the MAC_Binding table (e.g. [0]). On the other hand, for > the OpenStack + Octavia (Load Balancing service) use case, we see that > a reused VIP can be as well affected by stale entries in this table > due to the fact that it's never bound to a VIF so ovn-controller won't > claim it and send the GARPs to update the neighbors. > > I'm not sure if other scenarios may suffer from this issue but seems > reasonable to have an aging mechanism (as we discussed at some point > in the past) that makes unused/old entries to expire. After talking to > Numan on IRC, since a new pinctrl thread has been introduced recently > [1], it'd be nice to implement this aging mechanism there. > At the same time we'd be also reducing the amount of entries for long > lived systems as it'd grow indefinitely. > > Any thoughts? > > Thanks! > Daniel > > PS. With regards to the 'unused' vs 'old' entries I think it has to be > 'old' rather than 'unused' as I don't see a way to reset the TTL of a > MAC_Binding entry when we see packets coming. The implication is that > we'll be seeing ARPs sent out more often when perhaps they're not > needed. This also leads to the discussion of making the cache timeout > configurable. I've always considered the MAC_Binding implementation incomplete because of this issue and others. ovn/TODO.rst says: * Dynamic IP to MAC binding enhancements. OVN has basic support for establishing IP to MAC bindings dynamically, using ARP. * Ratelimiting. From casual observation, Linux appears to generate at most one ARP per second per destination. This might be supported by adding a new OVN logical action for rate-limiting. * Tracking queries It's probably best to only record in the database responses to queries actually issued by an L3 logical router, so somehow they have to be tracked, probably by putting a tentative binding without a MAC address into the database. * Renewal and expiration. Something needs to make sure that bindings remain valid and expire those that become stale. One way to do this might be to add some support for time to the database server itself. * Table size limiting. The table of MAC bindings must not be allowed to grow unreasonably large. * MTU handling (fragmentation on output) So, what do we do about it? First, I think that adding support for time to the database server is a terrible idea (even though I think I wrote the above originally). Let's not do that. The following is some "thinking out loud" on the subject. I think there's a challenge around which ovn-controller should take care of a given MAC_Binding. We don't want every ovn-controller expiring every binding. Ideally, we want exactly one ovn-controller expiring a binding. One way would be to add an owner column (but it would be better if we don't need it). If we want to keep track of "unused" bindings, I can imagine a statistical mechanism to do that. Any user of a binding occasionally and probabilistically changes a serial number column that we'd introduce into the MAC_Binding table (this could be optimized to not bother if it has changed recently). The owner checks the serial number every so often and if it hasn't changed then it deletes the row. The owner could also occasionally revalidate the binding. Any thoughts? Thanks, Ben. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
[ovs-discuss] [OVN] Aging mechanism for MAC_Binding table
Hi folks, Lately we've been trying to solve certain issues related to stale entries in the MAC_Binding table (e.g. [0]). On the other hand, for the OpenStack + Octavia (Load Balancing service) use case, we see that a reused VIP can be as well affected by stale entries in this table due to the fact that it's never bound to a VIF so ovn-controller won't claim it and send the GARPs to update the neighbors. I'm not sure if other scenarios may suffer from this issue but seems reasonable to have an aging mechanism (as we discussed at some point in the past) that makes unused/old entries to expire. After talking to Numan on IRC, since a new pinctrl thread has been introduced recently [1], it'd be nice to implement this aging mechanism there. At the same time we'd be also reducing the amount of entries for long lived systems as it'd grow indefinitely. Any thoughts? Thanks! Daniel PS. With regards to the 'unused' vs 'old' entries I think it has to be 'old' rather than 'unused' as I don't see a way to reset the TTL of a MAC_Binding entry when we see packets coming. The implication is that we'll be seeing ARPs sent out more often when perhaps they're not needed. This also leads to the discussion of making the cache timeout configurable. [0] https://github.com/openvswitch/ovs/commit/81e928526b8a9393b90785fb0a9c82d79570ef84 [1] https://github.com/openvswitch/ovs/commit/3594ffab6b4b423aa635a313f6b304180d7dbaf7 ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
