Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-07-24 Thread Eric W. Biederman
Rafael David Tinoco writes: > Hello Eric, > > Coming back to this... > > On Jun 16, 2014, at 12:01 PM, Rafael Tinoco > wrote: > >> ... >> >> On Fri, Jun 13, 2014 at 9:02 PM, Eric W. Biederman >> wrote: >>> Rafael Tinoco writes: >>> Okay, Tests with the same script were

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-07-24 Thread Eric W. Biederman
Rafael David Tinoco rafael.tin...@canonical.com writes: Hello Eric, Coming back to this... On Jun 16, 2014, at 12:01 PM, Rafael Tinoco rafael.tin...@canonical.com wrote: ... On Fri, Jun 13, 2014 at 9:02 PM, Eric W. Biederman ebied...@xmission.com wrote: Rafael Tinoco

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-07-17 Thread Rafael David Tinoco
Hello Eric, Coming back to this... On Jun 16, 2014, at 12:01 PM, Rafael Tinoco wrote: > ... > > On Fri, Jun 13, 2014 at 9:02 PM, Eric W. Biederman > wrote: >> Rafael Tinoco writes: >> >>> Okay, >>> >>> Tests with the same script were done. >>> I'm comparing : master + patch vs 3.15.0-rc5

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-07-17 Thread Rafael David Tinoco
Hello Eric, Coming back to this... On Jun 16, 2014, at 12:01 PM, Rafael Tinoco rafael.tin...@canonical.com wrote: ... On Fri, Jun 13, 2014 at 9:02 PM, Eric W. Biederman ebied...@xmission.com wrote: Rafael Tinoco rafael.tin...@canonical.com writes: Okay, Tests with the same script

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-16 Thread Rafael Tinoco
... On Fri, Jun 13, 2014 at 9:02 PM, Eric W. Biederman wrote: > Rafael Tinoco writes: > >> Okay, >> >> Tests with the same script were done. >> I'm comparing : master + patch vs 3.15.0-rc5 (last sync'ed rcu commit) >> and 3.9 last bisect good. >> >> Same tests were made. I'm comparing the

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-16 Thread Rafael Tinoco
... On Fri, Jun 13, 2014 at 9:02 PM, Eric W. Biederman ebied...@xmission.com wrote: Rafael Tinoco rafael.tin...@canonical.com writes: Okay, Tests with the same script were done. I'm comparing : master + patch vs 3.15.0-rc5 (last sync'ed rcu commit) and 3.9 last bisect good. Same tests

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-13 Thread Eric W. Biederman
Rafael Tinoco writes: > Okay, > > Tests with the same script were done. > I'm comparing : master + patch vs 3.15.0-rc5 (last sync'ed rcu commit) > and 3.9 last bisect good. > > Same tests were made. I'm comparing the following versions: > > 1) master + suggested patch > 2) 3.15.0-rc5 (last rcu

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-13 Thread Rafael Tinoco
Okay, Tests with the same script were done. I'm comparing : master + patch vs 3.15.0-rc5 (last sync'ed rcu commit) and 3.9 last bisect good. Same tests were made. I'm comparing the following versions: 1) master + suggested patch 2) 3.15.0-rc5 (last rcu commit in my clone) 3) 3.9-rc2 (last

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-13 Thread Rafael Tinoco
Okay, Tests with the same script were done. I'm comparing : master + patch vs 3.15.0-rc5 (last sync'ed rcu commit) and 3.9 last bisect good. Same tests were made. I'm comparing the following versions: 1) master + suggested patch 2) 3.15.0-rc5 (last rcu commit in my clone) 3) 3.9-rc2 (last

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-13 Thread Eric W. Biederman
Rafael Tinoco rafael.tin...@canonical.com writes: Okay, Tests with the same script were done. I'm comparing : master + patch vs 3.15.0-rc5 (last sync'ed rcu commit) and 3.9 last bisect good. Same tests were made. I'm comparing the following versions: 1) master + suggested patch 2)

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread Rafael Tinoco
Ok, some misconfiguration here probably, never mind. I'll finish the tests tomorrow, compare with existent ones and let you know asap. Tks. On Wed, Jun 11, 2014 at 10:09 PM, Eric W. Biederman wrote: > Rafael Tinoco writes: > >> I'm getting a kernel panic with your patch: >> >> -- panic >> --

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread Eric W. Biederman
Rafael Tinoco writes: > I'm getting a kernel panic with your patch: > > -- panic > -- mount_block_root > -- mount_root > -- prepare_namespace > -- kernel_init_freeable > > It is giving me an unknown block device for the same config file i > used on other builds. Since my test is running on a kvm

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread Rafael Tinoco
I'm getting a kernel panic with your patch: -- panic -- mount_block_root -- mount_root -- prepare_namespace -- kernel_init_freeable It is giving me an unknown block device for the same config file i used on other builds. Since my test is running on a kvm guest under a ramdisk, i'm still checking

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread Eric W. Biederman
"Paul E. McKenney" writes: > On Wed, Jun 11, 2014 at 04:12:15PM -0700, Eric W. Biederman wrote: >> "Paul E. McKenney" writes: >> >> > On Wed, Jun 11, 2014 at 01:46:08PM -0700, Eric W. Biederman wrote: >> >> On the chance it is dropping the old nsproxy which calls syncrhonize_rcu >> >> in

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread Paul E. McKenney
On Wed, Jun 11, 2014 at 04:12:15PM -0700, Eric W. Biederman wrote: > "Paul E. McKenney" writes: > > > On Wed, Jun 11, 2014 at 01:46:08PM -0700, Eric W. Biederman wrote: > >> On the chance it is dropping the old nsproxy which calls syncrhonize_rcu > >> in switch_task_namespaces that is causing

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread Eric W. Biederman
"Paul E. McKenney" writes: > On Wed, Jun 11, 2014 at 01:46:08PM -0700, Eric W. Biederman wrote: >> On the chance it is dropping the old nsproxy which calls syncrhonize_rcu >> in switch_task_namespaces that is causing you problems I have attached >> a patch that changes from rcu_read_lock to

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread Paul E. McKenney
On Wed, Jun 11, 2014 at 01:46:08PM -0700, Eric W. Biederman wrote: > Dave Chiluk writes: > > > On 06/11/2014 11:18 AM, Paul E. McKenney wrote: > >> On Wed, Jun 11, 2014 at 10:46:00AM -0500, David Chiluk wrote: > >>> Now think about what happens when a gateway goes down, the namespaces > >>> need

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread Dave Chiluk
On 06/11/2014 03:46 PM, Eric W. Biederman wrote: > ip netns add also performs a bind mount so we get into all of the vfs > level locking as well. It's actually quite a bit worse than that as ip netns exec creates a new mount namespace as well. That being said, the vfs issues have been healthily

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread Rafael Tinoco
Eric, I'll test the patch with the same testcase and let you all know. Really appreciate everybody's efforts. On Wed, Jun 11, 2014 at 5:55 PM, Eric W. Biederman wrote: > "Paul E. McKenney" writes: > >> On Wed, Jun 11, 2014 at 01:27:07PM -0500, Dave Chiluk wrote: >>> On 06/11/2014 11:18 AM,

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread Eric W. Biederman
"Paul E. McKenney" writes: > On Wed, Jun 11, 2014 at 01:27:07PM -0500, Dave Chiluk wrote: >> On 06/11/2014 11:18 AM, Paul E. McKenney wrote: >> > On Wed, Jun 11, 2014 at 10:46:00AM -0500, David Chiluk wrote: >> >> Now think about what happens when a gateway goes down, the namespaces >> >> need

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread Eric W. Biederman
Dave Chiluk writes: > On 06/11/2014 11:18 AM, Paul E. McKenney wrote: >> On Wed, Jun 11, 2014 at 10:46:00AM -0500, David Chiluk wrote: >>> Now think about what happens when a gateway goes down, the namespaces >>> need to be migrated, or a new machine needs to be brought up to replace >>> it.

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread Paul E. McKenney
On Wed, Jun 11, 2014 at 01:27:07PM -0500, Dave Chiluk wrote: > On 06/11/2014 11:18 AM, Paul E. McKenney wrote: > > On Wed, Jun 11, 2014 at 10:46:00AM -0500, David Chiluk wrote: > >> Now think about what happens when a gateway goes down, the namespaces > >> need to be migrated, or a new machine

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread Dave Chiluk
On 06/11/2014 11:18 AM, Paul E. McKenney wrote: > On Wed, Jun 11, 2014 at 10:46:00AM -0500, David Chiluk wrote: >> Now think about what happens when a gateway goes down, the namespaces >> need to be migrated, or a new machine needs to be brought up to replace >> it. When we're talking about 3000

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread Paul E. McKenney
On Wed, Jun 11, 2014 at 10:46:00AM -0500, David Chiluk wrote: > On 06/11/2014 10:17 AM, Rafael Tinoco wrote: > > This script simulates a failure on a cloud infrastructure, for ex. As soon > > as > > one virtualization host fails all its network namespaces have to be migrated > > to other node.

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread David Chiluk
On 06/11/2014 10:17 AM, Rafael Tinoco wrote: > This script simulates a failure on a cloud infrastructure, for ex. As soon as > one virtualization host fails all its network namespaces have to be migrated > to other node. Creating thousands of netns in the shortest time possible > is the objective

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread Rafael Tinoco
> I am having a really hard time distinguishing the colors on both charts > (yeah, red-green colorblind, go figure). Any chance of brighter colors, > patterned lines, or (better yet) the data in tabular form (for example, > with the configuration choices as columns and the releases/commits > as

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread Paul E. McKenney
On Wed, Jun 11, 2014 at 02:52:09AM -0300, Rafael Tinoco wrote: > Paul E. McKenney, Eric Biederman, David Miller (and/or anyone else > interested): > > It was brought to my attention that netns creation/execution might > have suffered scalability/performance regression after v3.8. > > I would

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread Eric W. Biederman
Rafael Tinoco writes: > Paul E. McKenney, Eric Biederman, David Miller (and/or anyone else > interested): > > It was brought to my attention that netns creation/execution might > have suffered scalability/performance regression after v3.8. > > I would like you, or anyone interested, to review

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread Eric W. Biederman
Rafael Tinoco rafael.tin...@canonical.com writes: Paul E. McKenney, Eric Biederman, David Miller (and/or anyone else interested): It was brought to my attention that netns creation/execution might have suffered scalability/performance regression after v3.8. I would like you, or anyone

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread Paul E. McKenney
On Wed, Jun 11, 2014 at 02:52:09AM -0300, Rafael Tinoco wrote: Paul E. McKenney, Eric Biederman, David Miller (and/or anyone else interested): It was brought to my attention that netns creation/execution might have suffered scalability/performance regression after v3.8. I would like you,

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread Rafael Tinoco
I am having a really hard time distinguishing the colors on both charts (yeah, red-green colorblind, go figure). Any chance of brighter colors, patterned lines, or (better yet) the data in tabular form (for example, with the configuration choices as columns and the releases/commits as rows)?

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread David Chiluk
On 06/11/2014 10:17 AM, Rafael Tinoco wrote: This script simulates a failure on a cloud infrastructure, for ex. As soon as one virtualization host fails all its network namespaces have to be migrated to other node. Creating thousands of netns in the shortest time possible is the objective

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread Paul E. McKenney
On Wed, Jun 11, 2014 at 10:46:00AM -0500, David Chiluk wrote: On 06/11/2014 10:17 AM, Rafael Tinoco wrote: This script simulates a failure on a cloud infrastructure, for ex. As soon as one virtualization host fails all its network namespaces have to be migrated to other node. Creating

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread Dave Chiluk
On 06/11/2014 11:18 AM, Paul E. McKenney wrote: On Wed, Jun 11, 2014 at 10:46:00AM -0500, David Chiluk wrote: Now think about what happens when a gateway goes down, the namespaces need to be migrated, or a new machine needs to be brought up to replace it. When we're talking about 3000

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread Paul E. McKenney
On Wed, Jun 11, 2014 at 01:27:07PM -0500, Dave Chiluk wrote: On 06/11/2014 11:18 AM, Paul E. McKenney wrote: On Wed, Jun 11, 2014 at 10:46:00AM -0500, David Chiluk wrote: Now think about what happens when a gateway goes down, the namespaces need to be migrated, or a new machine needs to be

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread Eric W. Biederman
Dave Chiluk chi...@canonical.com writes: On 06/11/2014 11:18 AM, Paul E. McKenney wrote: On Wed, Jun 11, 2014 at 10:46:00AM -0500, David Chiluk wrote: Now think about what happens when a gateway goes down, the namespaces need to be migrated, or a new machine needs to be brought up to replace

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread Eric W. Biederman
Paul E. McKenney paul...@linux.vnet.ibm.com writes: On Wed, Jun 11, 2014 at 01:27:07PM -0500, Dave Chiluk wrote: On 06/11/2014 11:18 AM, Paul E. McKenney wrote: On Wed, Jun 11, 2014 at 10:46:00AM -0500, David Chiluk wrote: Now think about what happens when a gateway goes down, the

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread Rafael Tinoco
Eric, I'll test the patch with the same testcase and let you all know. Really appreciate everybody's efforts. On Wed, Jun 11, 2014 at 5:55 PM, Eric W. Biederman ebied...@xmission.com wrote: Paul E. McKenney paul...@linux.vnet.ibm.com writes: On Wed, Jun 11, 2014 at 01:27:07PM -0500, Dave

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread Dave Chiluk
On 06/11/2014 03:46 PM, Eric W. Biederman wrote: ip netns add also performs a bind mount so we get into all of the vfs level locking as well. It's actually quite a bit worse than that as ip netns exec creates a new mount namespace as well. That being said, the vfs issues have been healthily

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread Paul E. McKenney
On Wed, Jun 11, 2014 at 01:46:08PM -0700, Eric W. Biederman wrote: Dave Chiluk chi...@canonical.com writes: On 06/11/2014 11:18 AM, Paul E. McKenney wrote: On Wed, Jun 11, 2014 at 10:46:00AM -0500, David Chiluk wrote: Now think about what happens when a gateway goes down, the namespaces

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread Eric W. Biederman
Paul E. McKenney paul...@linux.vnet.ibm.com writes: On Wed, Jun 11, 2014 at 01:46:08PM -0700, Eric W. Biederman wrote: On the chance it is dropping the old nsproxy which calls syncrhonize_rcu in switch_task_namespaces that is causing you problems I have attached a patch that changes from

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread Paul E. McKenney
On Wed, Jun 11, 2014 at 04:12:15PM -0700, Eric W. Biederman wrote: Paul E. McKenney paul...@linux.vnet.ibm.com writes: On Wed, Jun 11, 2014 at 01:46:08PM -0700, Eric W. Biederman wrote: On the chance it is dropping the old nsproxy which calls syncrhonize_rcu in switch_task_namespaces that

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread Eric W. Biederman
Paul E. McKenney paul...@linux.vnet.ibm.com writes: On Wed, Jun 11, 2014 at 04:12:15PM -0700, Eric W. Biederman wrote: Paul E. McKenney paul...@linux.vnet.ibm.com writes: On Wed, Jun 11, 2014 at 01:46:08PM -0700, Eric W. Biederman wrote: On the chance it is dropping the old nsproxy which

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread Rafael Tinoco
I'm getting a kernel panic with your patch: -- panic -- mount_block_root -- mount_root -- prepare_namespace -- kernel_init_freeable It is giving me an unknown block device for the same config file i used on other builds. Since my test is running on a kvm guest under a ramdisk, i'm still checking

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread Eric W. Biederman
Rafael Tinoco rafael.tin...@canonical.com writes: I'm getting a kernel panic with your patch: -- panic -- mount_block_root -- mount_root -- prepare_namespace -- kernel_init_freeable It is giving me an unknown block device for the same config file i used on other builds. Since my test is

Re: Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-11 Thread Rafael Tinoco
Ok, some misconfiguration here probably, never mind. I'll finish the tests tomorrow, compare with existent ones and let you know asap. Tks. On Wed, Jun 11, 2014 at 10:09 PM, Eric W. Biederman ebied...@xmission.com wrote: Rafael Tinoco rafael.tin...@canonical.com writes: I'm getting a kernel

Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-10 Thread Rafael Tinoco
Paul E. McKenney, Eric Biederman, David Miller (and/or anyone else interested): It was brought to my attention that netns creation/execution might have suffered scalability/performance regression after v3.8. I would like you, or anyone interested, to review these charts/data and check if there

Possible netns creation and execution performance/scalability regression since v3.8 due to rcu callbacks being offloaded to multiple cpus

2014-06-10 Thread Rafael Tinoco
Paul E. McKenney, Eric Biederman, David Miller (and/or anyone else interested): It was brought to my attention that netns creation/execution might have suffered scalability/performance regression after v3.8. I would like you, or anyone interested, to review these charts/data and check if there