Re: [openstack-dev] [neutron][stable] metadata agent performance
On Oct 22, 2014, at 12:53 AM, Jakub Libosvar wrote: > On 10/22/2014 02:26 AM, Maru Newby wrote: >> We merged caching support for the metadata agent in juno, and backported to >> icehouse. It was enabled by default in juno, but disabled by default in >> icehouse to satisfy the stable maint requirement of not changing functional >> behavior. >> >> While performance of the agent was improved with caching enabled, it >> regressed a reported 8x when caching was disabled [1]. This means that by >> default, the caching backport severely impacts icehouse Neutron's >> performance. >> >> So, what is the way forward? We definitely need to document the problem for >> both icehouse and juno. Is documentation enough? Or can we enable caching >> by default in icehouse? Or remove the backport entirely. >> >> There is also a proposal to replace the metadata agent’s use of the neutron >> client in favor of rpc [2]. There were comments on an old bug suggesting we >> didn’t want to do this [3], but assuming that we want this change in Kilo, >> is backporting even a possibility given that it implies a behavioral change >> to be useful? >> >> Thanks, >> >> >> Maru >> >> >> >> 1: https://bugs.launchpad.net/cloud-archive/+bug/1361357 >> 2: https://review.openstack.org/#/c/121782 >> 3: https://bugs.launchpad.net/neutron/+bug/1092043 >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > I thought the performance regression was caused by wrong keystone token > caching leading to authentication per neutron client instance. Fix was > backported to Icehouse [1]. > > Does it mean this patch hasn't solved the problem and regression is > somewhere else? As you say (and as per Ihar and Akihiro’s responses), the problem was fixed. I was confused by the bug that Oleg’s RPC proposal hard targeted as a related bug and thought the problem wasn’t yet resolved, but looking at it again it appears the bug is a Ubuntu distro issue. Accordingly, I’ve removed Neutron as a target for that bug. Apologies for the confusion. Maru > Kuba > > [1] https://review.openstack.org/#/c/120418/ > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][stable] metadata agent performance
My understanding is same as from Ihar, and we no longer have the degradation in the latest Icehouse update. There was a degradation in 2014.1.2 [2] but the fix was backported in 2014.1.3 [1]. We don't need to take care of backporting when considering metadata RPC patch. [1] https://review.openstack.org/#/c/120418/ [2] https://review.openstack.org/#/c/95491/ Thanks, Akihiro On Wed, Oct 22, 2014 at 7:24 PM, Ihar Hrachyshka wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA512 > > On 22/10/14 02:26, Maru Newby wrote: >> We merged caching support for the metadata agent in juno, and >> backported to icehouse. It was enabled by default in juno, but >> disabled by default in icehouse to satisfy the stable maint >> requirement of not changing functional behavior. >> >> While performance of the agent was improved with caching enabled, >> it regressed a reported 8x when caching was disabled [1]. This >> means that by default, the caching backport severely impacts >> icehouse Neutron's performance. > > If I correctly follow the degradation scenario, it's caused by > unneeded tokens requested from keystone each time a request hits > neutron metadata agent. This should be already fixed as [1] (included > in the latest 2014.1.3 release). > > [1]: https://review.openstack.org/#/c/118996/ > >> >> So, what is the way forward? We definitely need to document the >> problem for both icehouse and juno. Is documentation enough? Or >> can we enable caching by default in icehouse? Or remove the >> backport entirely. > > If I'm correct, the issue is already solved in the latest Icehouse > release, so there seems to be no need to document the regression for > 2014.1.2. But yeah, sure we could put it in its release notes just in > case. > >> >> There is also a proposal to replace the metadata agent’s use of the >> neutron client in favor of rpc [2]. There were comments on an old >> bug suggesting we didn’t want to do this [3], but assuming that we >> want this change in Kilo, is backporting even a possibility given >> that it implies a behavioral change to be useful? > > We probably wouldn't consider backporting it to stable branches > because it touches RPC API, and we usually avoid it there. Anyway, it > shouldn't be an issue at all (as per my comment above). > >> >> Thanks, >> >> >> Maru >> >> >> >> 1: https://bugs.launchpad.net/cloud-archive/+bug/1361357 2: >> https://review.openstack.org/#/c/121782 3: >> https://bugs.launchpad.net/neutron/+bug/1092043 >> > -BEGIN PGP SIGNATURE- > Version: GnuPG/MacGPG2 v2.0.22 (Darwin) > > iQEcBAEBCgAGBQJUR4XAAAoJEC5aWaUY1u57rYAH/jDqluduRLxwgHykP/NMIesj > 0MnesaiFwfeHdE5z3YEnteVkxEtkDRnmTRaau9TuOJpVrUfSIA7Lpa3S79Rv4cT5 > CC82FlU32fbOkCVFiXqgQvadNc3wrqHMag9FD6fpbg/MZlvV/VWHMl/z55rwhNh/ > yL+CzXd9uNgZy+ng0LI1EY+u9UcLtVwF8xhLhRIu5NMRim3HeRFlFUSN41ccemRJ > TdJUxMdtlYls/nCuIUk2QpSOZt1Hk2bysrBPh0etV501vsSHCq3cYZ3vjmt+jNX9 > thTKlsOaFpSLWnTn5+ERXk3y7pvJxo1AGOli3sLXIDYYNYPK4Y8PRYPLm43U45o= > =o7SU > -END PGP SIGNATURE- > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Akihiro Motoki ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][stable] metadata agent performance
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 22/10/14 02:26, Maru Newby wrote: > We merged caching support for the metadata agent in juno, and > backported to icehouse. It was enabled by default in juno, but > disabled by default in icehouse to satisfy the stable maint > requirement of not changing functional behavior. > > While performance of the agent was improved with caching enabled, > it regressed a reported 8x when caching was disabled [1]. This > means that by default, the caching backport severely impacts > icehouse Neutron's performance. If I correctly follow the degradation scenario, it's caused by unneeded tokens requested from keystone each time a request hits neutron metadata agent. This should be already fixed as [1] (included in the latest 2014.1.3 release). [1]: https://review.openstack.org/#/c/118996/ > > So, what is the way forward? We definitely need to document the > problem for both icehouse and juno. Is documentation enough? Or > can we enable caching by default in icehouse? Or remove the > backport entirely. If I'm correct, the issue is already solved in the latest Icehouse release, so there seems to be no need to document the regression for 2014.1.2. But yeah, sure we could put it in its release notes just in case. > > There is also a proposal to replace the metadata agent’s use of the > neutron client in favor of rpc [2]. There were comments on an old > bug suggesting we didn’t want to do this [3], but assuming that we > want this change in Kilo, is backporting even a possibility given > that it implies a behavioral change to be useful? We probably wouldn't consider backporting it to stable branches because it touches RPC API, and we usually avoid it there. Anyway, it shouldn't be an issue at all (as per my comment above). > > Thanks, > > > Maru > > > > 1: https://bugs.launchpad.net/cloud-archive/+bug/1361357 2: > https://review.openstack.org/#/c/121782 3: > https://bugs.launchpad.net/neutron/+bug/1092043 > -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) iQEcBAEBCgAGBQJUR4XAAAoJEC5aWaUY1u57rYAH/jDqluduRLxwgHykP/NMIesj 0MnesaiFwfeHdE5z3YEnteVkxEtkDRnmTRaau9TuOJpVrUfSIA7Lpa3S79Rv4cT5 CC82FlU32fbOkCVFiXqgQvadNc3wrqHMag9FD6fpbg/MZlvV/VWHMl/z55rwhNh/ yL+CzXd9uNgZy+ng0LI1EY+u9UcLtVwF8xhLhRIu5NMRim3HeRFlFUSN41ccemRJ TdJUxMdtlYls/nCuIUk2QpSOZt1Hk2bysrBPh0etV501vsSHCq3cYZ3vjmt+jNX9 thTKlsOaFpSLWnTn5+ERXk3y7pvJxo1AGOli3sLXIDYYNYPK4Y8PRYPLm43U45o= =o7SU -END PGP SIGNATURE- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][stable] metadata agent performance
On 10/22/2014 02:26 AM, Maru Newby wrote: > We merged caching support for the metadata agent in juno, and backported to > icehouse. It was enabled by default in juno, but disabled by default in > icehouse to satisfy the stable maint requirement of not changing functional > behavior. > > While performance of the agent was improved with caching enabled, it > regressed a reported 8x when caching was disabled [1]. This means that by > default, the caching backport severely impacts icehouse Neutron's performance. > > So, what is the way forward? We definitely need to document the problem for > both icehouse and juno. Is documentation enough? Or can we enable caching > by default in icehouse? Or remove the backport entirely. > > There is also a proposal to replace the metadata agent’s use of the neutron > client in favor of rpc [2]. There were comments on an old bug suggesting we > didn’t want to do this [3], but assuming that we want this change in Kilo, is > backporting even a possibility given that it implies a behavioral change to > be useful? > > Thanks, > > > Maru > > > > 1: https://bugs.launchpad.net/cloud-archive/+bug/1361357 > 2: https://review.openstack.org/#/c/121782 > 3: https://bugs.launchpad.net/neutron/+bug/1092043 > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > I thought the performance regression was caused by wrong keystone token caching leading to authentication per neutron client instance. Fix was backported to Icehouse [1]. Does it mean this patch hasn't solved the problem and regression is somewhere else? Kuba [1] https://review.openstack.org/#/c/120418/ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][stable] metadata agent performance
It sounds like the only reasonable option we are left with right now is to document. Even if we enabled/removed the backport, it would take time until users can get their hands on a new cut of the stable branch. We would need to be more diligent in the future and limit backports to just bug fixes to prevent situations like this from occurring, or maybe we need to have better testing...um...definitely the latter :) My 2c Armando On 22 October 2014 05:56, Maru Newby wrote: > We merged caching support for the metadata agent in juno, and backported > to icehouse. It was enabled by default in juno, but disabled by default in > icehouse to satisfy the stable maint requirement of not changing functional > behavior. > > While performance of the agent was improved with caching enabled, it > regressed a reported 8x when caching was disabled [1]. This means that by > default, the caching backport severely impacts icehouse Neutron's > performance. > > So, what is the way forward? We definitely need to document the problem > for both icehouse and juno. Is documentation enough? Or can we enable > caching by default in icehouse? Or remove the backport entirely. > > There is also a proposal to replace the metadata agent’s use of the > neutron client in favor of rpc [2]. There were comments on an old bug > suggesting we didn’t want to do this [3], but assuming that we want this > change in Kilo, is backporting even a possibility given that it implies a > behavioral change to be useful? > > Thanks, > > > Maru > > > > 1: https://bugs.launchpad.net/cloud-archive/+bug/1361357 > 2: https://review.openstack.org/#/c/121782 > 3: https://bugs.launchpad.net/neutron/+bug/1092043 > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [neutron][stable] metadata agent performance
We merged caching support for the metadata agent in juno, and backported to icehouse. It was enabled by default in juno, but disabled by default in icehouse to satisfy the stable maint requirement of not changing functional behavior. While performance of the agent was improved with caching enabled, it regressed a reported 8x when caching was disabled [1]. This means that by default, the caching backport severely impacts icehouse Neutron's performance. So, what is the way forward? We definitely need to document the problem for both icehouse and juno. Is documentation enough? Or can we enable caching by default in icehouse? Or remove the backport entirely. There is also a proposal to replace the metadata agent’s use of the neutron client in favor of rpc [2]. There were comments on an old bug suggesting we didn’t want to do this [3], but assuming that we want this change in Kilo, is backporting even a possibility given that it implies a behavioral change to be useful? Thanks, Maru 1: https://bugs.launchpad.net/cloud-archive/+bug/1361357 2: https://review.openstack.org/#/c/121782 3: https://bugs.launchpad.net/neutron/+bug/1092043 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
