Re: [openstack-dev] [devstack] openstack client slowness / client-as-a-service

[Morgan Fainberg]

On Wed, Apr 20, 2016 at 10:28 AM, Dean Troyer  wrote:

> On Wed, Apr 20, 2016 at 9:43 AM, Doug Hellmann 
> wrote:
>
>> Cliff looks for commands on demand. If we modify it's command loader to
>> support some "built in" commands, and then implement the commands in OSC
>> that way, we can avoid scanning the real plugin system until we hit a
>> command that isn't built in.
>>
>
> Maybe down the road once this becomes a bigger percentage of the startup
> time, for now I do not (yet) believe the plugins are the problem others
> do.  See below...
>
>
>> The last time I proposed that, though, someone (Dean?) pointed out that
>> loading the plugins wasn't actually where OSC was spending its time. So,
>> maybe we should profile the thing before proposing code changes.
>>
>
> It has been a while since I looked into this in detail, we made a couple
> of changes then that helped, and since then the world kept moving and we're
> behind again. Right now OSC is a mess WRT keystoneclient/keystoneauth and
> their interactions with os-client-config.  We have lost sight of who is
> supposed to be doing what here.  I know for a fact that there is
> duplication in auth setup, we often make duplicate REST auth calls.
>
> OSC 2.4.0 should be released today/tomorrow, following which we begin
> merging the cleanup starting with the ksc/ksa bits.
>
> Here are a couple of things to consider for those who want to investigate
> further:
> * OSC does not load _any_ project client (modulo the ksc bit mentioned
> above) unless/until it is actually needed to make a REST call.
> * Timing on a help command includes a complete scan of all entry points to
> generate the list of commands
> * The --timing option lists all REST calls that properly go through our
> TimingSession object.  That should be all of them unless a library doesn't
> use the session it is given (the ones used by the commands in the OSC repo
> all do this correctly).
> * Interactive mode can be useful to get timing on just the setup/teardown
> process without actually running a command:
>
>   time openstack 
> So time for a couple of fun baselines, using the OSC SHA proposed for
> 2.4.0 (4639148b1d) on a Lenovo T420s with Ubuntu 14.0.4 against a DevStack
> on an Ubuntu 14.0.3 VM on a moderately-sized local NUC:
>
> * time openstack --timing* py2: 0m0.307s
>   * py3: 0m0.376s
>
> * time openstack --timing help
>   * py2: 0m1.939s
>   * py3: 0m1.803s
>
> * time openstack --timing catalog list
>   * py2: 0m0.675s - 0.360 REST = 0.315s
>   * py3: 0m0.704s - 0.312 REST = 0.392s
>
> * time openstack --timing flavor list
>   * py2: 0m0.772s - 0.447 REST = 0.325s
>   * py3: 0m2.563s - 2.146 REST = 0.417s
>
> * time openstack --timing image list
>   * py2: 0m0.860s - 0.517 REST = 0.343s
>   * py3: 0m0.952s - 0.539 REST = 0.423s
>
> Are there any conclusions to draw from this seat-of-the-pants look?
>
> * The differences between py2 and py3 are small, and not consistent.
> * The time for OSC to load and then exit immediately is within 0.1s of the
> time to execute a near-trivial command when the REST round trip times are
> removed.
> * Two auth calls are consistently being made, this is one of the things
> actively being cleaned up with the ksc/ksa transition bits.  The additional
> REST round trip in these tests is consistently between 0.14s and 0.2s, so
> that gain will come soon.
>
> I also have buried in my notes some out-of-date results of using
> boris-42's profimp that lead me to keystoneclient and the largest single
> static import being done and accounting for nearly half of the total static
> load time.  The transition to using ksa will help here, I do not have
> profimp numbers for that yet.
>
> dt
>
>
This was one of the driving forces behind the move to KSA in general,
eliminating and isolating the auth library to as small of a footprint as
possible.

There is still some loading overhead due to the auth plugin system (hard to
eliminate completely), but simple back-of-the-napkin numbers we did
initially showed KeystoneAuth way ahead in the game (and much easier to
improve upon since it is not tied to keystone CRUD interactions to get auth
performance improvements released)
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] openstack client slowness / client-as-a-service

[Dean Troyer]

On Wed, Apr 20, 2016 at 9:43 AM, Doug Hellmann 
wrote:

> Cliff looks for commands on demand. If we modify it's command loader to
> support some "built in" commands, and then implement the commands in OSC
> that way, we can avoid scanning the real plugin system until we hit a
> command that isn't built in.
>

Maybe down the road once this becomes a bigger percentage of the startup
time, for now I do not (yet) believe the plugins are the problem others
do.  See below...


> The last time I proposed that, though, someone (Dean?) pointed out that
> loading the plugins wasn't actually where OSC was spending its time. So,
> maybe we should profile the thing before proposing code changes.
>

It has been a while since I looked into this in detail, we made a couple of
changes then that helped, and since then the world kept moving and we're
behind again. Right now OSC is a mess WRT keystoneclient/keystoneauth and
their interactions with os-client-config.  We have lost sight of who is
supposed to be doing what here.  I know for a fact that there is
duplication in auth setup, we often make duplicate REST auth calls.

OSC 2.4.0 should be released today/tomorrow, following which we begin
merging the cleanup starting with the ksc/ksa bits.

Here are a couple of things to consider for those who want to investigate
further:
* OSC does not load _any_ project client (modulo the ksc bit mentioned
above) unless/until it is actually needed to make a REST call.
* Timing on a help command includes a complete scan of all entry points to
generate the list of commands
* The --timing option lists all REST calls that properly go through our
TimingSession object.  That should be all of them unless a library doesn't
use the session it is given (the ones used by the commands in the OSC repo
all do this correctly).
* Interactive mode can be useful to get timing on just the setup/teardown
process without actually running a command:

  time openstack __
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] openstack client slowness / client-as-a-service

[Doug Hellmann]

Excerpts from Steve Baker's message of 2016-04-20 16:38:25 +1200:
> On 20/04/16 06:17, Monty Taylor wrote:
> > On 04/19/2016 10:16 AM, Daniel P. Berrange wrote:
> >> On Tue, Apr 19, 2016 at 09:57:56AM -0500, Dean Troyer wrote:
> >>> On Tue, Apr 19, 2016 at 9:06 AM, Adam Young  wrote:
> >>>
>  I wonder how much of that is Token caching.  In a typical CLI use 
>  patter,
>  a new token is created each time a client is called, with no 
>  passing of a
>  token between services.  Using a session can greatly decrease the 
>  number of
>  round trips to Keystone.
> 
> >>>
> >>> Not as much as you think (or hope?).  Persistent token caching to 
> >>> disk will
> >>> help some, at other expenses though.  Using --timing on OSC will 
> >>> show how
> >>> much time the Identity auth round trip cost.
> >>>
> >>> I don't have current numbers, the last time I instrumented OSC there 
> >>> were
> >>> significant load times for some modules, so we went a good distance to
> >>> lazy-load as much as possible.
> >>>
> >>> What Dan sees WRT a persistent client process, though, is a 
> >>> combination of
> >>> those two things: saving the Python loading and the Keystone round 
> >>> trips.
> >>
> >> The 1.5sec overhead I eliminated doesn't actually have anything todo
> >> with network round trips at all. Even if you turn off all network
> >> services and just run 'openstack ' and let it fail due
> >> to inability to connect it'll still have that 1.5 sec overhead. It
> >> is all related to python runtime loading and work done during module
> >> importing.
> >>
> >> eg run 'unstack.sh' and then compare the main openstack client:
> >>
> >> $ time /usr/bin/openstack server list
> >> Discovering versions from the identity service failed when creating 
> >> the password plugin. Attempting to determine version from URL.
> >> Unable to establish connection to 
> >> http://192.168.122.156:5000/v2.0/tokens
> >>
> >> real0m1.555s
> >> user0m1.407s
> >> sys0m0.147s
> >>
> >> Against my client-as-a-service version:
> >>
> >> $ time $HOME/bin/openstack server list
> >> [Errno 111] Connection refused
> >>
> >> real0m0.045s
> >> user0m0.029s
> >> sys0m0.016s
> >>
> >>
> >> I'm sure there is scope for also optimizing network traffic / round
> >> trips, but I didn't investigate that at all.
> >>
> >>> I have (had!) a version of DevStack that put OSC into a subprocess and
> >>> called it via pipes to do essentially what Dan suggests.  It saves some
> >>> time, at the expense of complexity that may or may not be worth the 
> >>> effort.
> >>
> >> devstack doesn't actually really need any significant changes beyond
> >> making sure $PATH pointed to the replacement client programs and that
> >> the server was running - the latter could be automated as a launch on
> >> demand thing which would limit devstack changes.
> >>
> >> It actually doesn't technically need any devstack change - these
> >> replacement clients could simply be put in some 3rd party git repo
> >> and let developers who want the speed benefit simply put them in
> >> their $PATH before running devstack.
> >>
> >>> One thing missing is any sort of transactional control in the I/O 
> >>> with the
> >>> subprocess, ie, an EOT marker.  I planned to add a -0 option (think 
> >>> xargs)
> >>> to handle that but it's still down a few slots on my priority list.  
> >>> Error
> >>> handling is another problem, and at this point (for DevStack purposes
> >>> anyway) I stopped the investigation, concluding that reliability 
> >>> trumped a
> >>> few seconds saved here.
> >>
> >> For I/O I simply replaced stdout + stderr with a new StringIO handle to
> >> capture the data when running each command, and for error handling I
> >> ensured the exit status was fed back & likewise stderr printed.
> >>
> >> It is more than just a few seconds saved - almost 4 minutes, or
> >> nearly 20% of entire time to run stack.sh on my machine
> >>
> >>
> >>> Ultimately, this is one of the two giant nails in the coffin of 
> >>> continuing
> >>> to persue CLIs in Python.  The other is co-installability. (See that
> >>> current thread on the ML for pain points).  Both are easily solved with
> >>> native-code-generating languages.  Go and Rust are at the top of my
> >>> personal list here...
> >
> > Using entrypoints and plugins in python is slow, so loading them is 
> > slow, as is loading all of the dependent libraries. Those were choices 
> > made for good reason back in the day, but I'm not convinced either are 
> > great anymore.
> >
> > A pluginless CLI that simply used REST calls rather than the 
> > python-clientlibs should be able to launch in get to the business of 
> > doing work in 0.2 seconds - counting time to load and parse 
> > clouds.yaml. That time could be reduced - the time spent in occ 
> > parsing vendor json files is not strictly necessary and certainly 
> > could go faster. It's not as fast as 0.004 seconds, but with 

Re: [openstack-dev] [devstack] openstack client slowness / client-as-a-service

[Sean Dague]

On 04/19/2016 11:03 PM, Dean Troyer wrote:
> 
> 
> On Tue, Apr 19, 2016 at 8:17 PM, Adam Young  > wrote:
> 
> Maybe it is time to revamp Devstack.  Is there some way that,
> without a major rewrite, it could take better advantage of the CLI?
> Could we group commands, or migrate sections to python scripts that
> really all need to be done together?  For example, most of the early
> prep of the Keystone server moved to keystone-manage bootstrap.  Is
> there more bootstrap-type behavior we can and should consolidate?
> 
> 
> This is what I was talking about, trying to take advantage of the
> interactive mode that also reads from stdin to do a series of comamnds
> with a single load/auth cycle.  It lacks a LOT of things for a resilient
> use case such as DevStack (error abort or error ignore?, branching,
> everything a DSL would bring).
> 
> And if you'd like to replace stach.sh with stack.py, I'll not stop you,
> just don't call it DevStack.  Now you are building yet another
> deployment tool.  We've also been down that road before. It may well be
> time to retire DevStack, be sure to let us know when those willing to
> sponsor that work show up so they can attempt to learn from some of our
> mistakes and not repeat them the hard way.

I agree that the CLI being slow is an issue. It's an issue that hits all
the developers because it's adding 3 minutes to devstack runs.

We've stated that openstack client is our strategic interface to lead
with. We've also all experienced that it's so terribly slow for a CLI,
that it leaves a bad taste in our mouths.

While there are a lot of things that Devstack could do better
(especially when it comes to loading all keystone data (users / sc
entries), which is the majority of the time spend in osc), it does seem
to paper over a real issue that doesn't seem to be priority #1 for OSC
right now (or any of our CLIs).

So, could we get back to the root question.

What's actually taking the time? Can that be improved? All these
assumptions that openstacksdk or occ make things better make assumptions
they aren't loading as much python code or have dynamic entry points
that contribute to the slowness. There seems to be a lot of assumptions
here, and only Dan brought real data to the table.

So the real question is:

1) is anyone sufficiently motivated to do a data driven analysis (and
propose guidelines to addressing) why our python CLIs are slow?

Dan provided a starting point, but I've seen no one actually volunteer
to complete this work, or decide it's a project priority.

All the statements here of "use Lang Foo", "use Library X instead" are
pretty shoot from the hip with no actual data. Yes, there are
fundamental issues with python module loading.  These are problems that
can be solved if people are willing to do the hard work to profile and
limit the extensibility of the code.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] openstack client slowness / client-as-a-service

[Steve Baker]

On 20/04/16 06:17, Monty Taylor wrote:

On 04/19/2016 10:16 AM, Daniel P. Berrange wrote:

On Tue, Apr 19, 2016 at 09:57:56AM -0500, Dean Troyer wrote:

On Tue, Apr 19, 2016 at 9:06 AM, Adam Young  wrote:

I wonder how much of that is Token caching.  In a typical CLI use 
patter,
a new token is created each time a client is called, with no 
passing of a
token between services.  Using a session can greatly decrease the 
number of

round trips to Keystone.



Not as much as you think (or hope?).  Persistent token caching to 
disk will
help some, at other expenses though.  Using --timing on OSC will 
show how

much time the Identity auth round trip cost.

I don't have current numbers, the last time I instrumented OSC there 
were

significant load times for some modules, so we went a good distance to
lazy-load as much as possible.

What Dan sees WRT a persistent client process, though, is a 
combination of
those two things: saving the Python loading and the Keystone round 
trips.


The 1.5sec overhead I eliminated doesn't actually have anything todo
with network round trips at all. Even if you turn off all network
services and just run 'openstack ' and let it fail due
to inability to connect it'll still have that 1.5 sec overhead. It
is all related to python runtime loading and work done during module
importing.

eg run 'unstack.sh' and then compare the main openstack client:

$ time /usr/bin/openstack server list
Discovering versions from the identity service failed when creating 
the password plugin. Attempting to determine version from URL.
Unable to establish connection to 
http://192.168.122.156:5000/v2.0/tokens


real0m1.555s
user0m1.407s
sys0m0.147s

Against my client-as-a-service version:

$ time $HOME/bin/openstack server list
[Errno 111] Connection refused

real0m0.045s
user0m0.029s
sys0m0.016s


I'm sure there is scope for also optimizing network traffic / round
trips, but I didn't investigate that at all.


I have (had!) a version of DevStack that put OSC into a subprocess and
called it via pipes to do essentially what Dan suggests.  It saves some
time, at the expense of complexity that may or may not be worth the 
effort.


devstack doesn't actually really need any significant changes beyond
making sure $PATH pointed to the replacement client programs and that
the server was running - the latter could be automated as a launch on
demand thing which would limit devstack changes.

It actually doesn't technically need any devstack change - these
replacement clients could simply be put in some 3rd party git repo
and let developers who want the speed benefit simply put them in
their $PATH before running devstack.

One thing missing is any sort of transactional control in the I/O 
with the
subprocess, ie, an EOT marker.  I planned to add a -0 option (think 
xargs)
to handle that but it's still down a few slots on my priority list.  
Error

handling is another problem, and at this point (for DevStack purposes
anyway) I stopped the investigation, concluding that reliability 
trumped a

few seconds saved here.


For I/O I simply replaced stdout + stderr with a new StringIO handle to
capture the data when running each command, and for error handling I
ensured the exit status was fed back & likewise stderr printed.

It is more than just a few seconds saved - almost 4 minutes, or
nearly 20% of entire time to run stack.sh on my machine


Ultimately, this is one of the two giant nails in the coffin of 
continuing

to persue CLIs in Python.  The other is co-installability. (See that
current thread on the ML for pain points).  Both are easily solved with
native-code-generating languages.  Go and Rust are at the top of my
personal list here...


Using entrypoints and plugins in python is slow, so loading them is 
slow, as is loading all of the dependent libraries. Those were choices 
made for good reason back in the day, but I'm not convinced either are 
great anymore.


A pluginless CLI that simply used REST calls rather than the 
python-clientlibs should be able to launch in get to the business of 
doing work in 0.2 seconds - counting time to load and parse 
clouds.yaml. That time could be reduced - the time spent in occ 
parsing vendor json files is not strictly necessary and certainly 
could go faster. It's not as fast as 0.004 seconds, but with very 
little effort it's 6x faster.


Rather than ditching python for something like go, I'd rather put 
together a CLI with no plugins and that only depended on keystoneauth 
and os-client-config as libraries. No?




There is a middle ground between discovering plugins on every run and 
not allowing plugins at all. There should be significant performance 
gains by having a plugin registry which caches the result of discovery. 
The cost is the complexity of managing registry rebuilds when plugins 
change. This should be mitigated for most users by having registry 
rebuild hooks triggered by package/pip installs. The needs of 

Re: [openstack-dev] [devstack] openstack client slowness / client-as-a-service

[Adam Young]

On 04/19/2016 11:03 PM, Dean Troyer wrote:



On Tue, Apr 19, 2016 at 8:17 PM, Adam Young > wrote:


Maybe it is time to revamp Devstack.  Is there some way that,
without a major rewrite, it could take better advantage of the
CLI? Could we group commands, or migrate sections to python
scripts that really all need to be done together? For example,
most of the early prep of the Keystone server moved to
keystone-manage bootstrap.  Is there more bootstrap-type behavior
we can and should consolidate?


This is what I was talking about, trying to take advantage of the 
interactive mode that also reads from stdin to do a series of comamnds 
with a single load/auth cycle.  It lacks a LOT of things for a 
resilient use case such as DevStack (error abort or error ignore?, 
branching, everything a DSL would bring).

Right, so lets get those as feature requests into the Client, I think.

The same DSL could be used server side for batch commands?




And if you'd like to replace stach.sh with stack.py, I'll not stop 
you, just don't call it DevStack.  Now you are building yet another 
deployment tool.  We've also been down that road before. It may well 
be time to retire DevStack, be sure to let us know when those willing 
to sponsor that work show up so they can attempt to learn from some of 
our mistakes and not repeat them the hard way.
Nope.  Not gonna do it. I have no desire to do that.  Nope nope nopey 
nope nope.






dt

--

Dean Troyer
dtro...@gmail.com 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] openstack client slowness / client-as-a-service

[Dean Troyer]

On Tue, Apr 19, 2016 at 8:17 PM, Adam Young  wrote:

> Maybe it is time to revamp Devstack.  Is there some way that, without a
> major rewrite, it could take better advantage of the CLI? Could we group
> commands, or migrate sections to python scripts that really all need to be
> done together?  For example, most of the early prep of the Keystone server
> moved to keystone-manage bootstrap.  Is there more bootstrap-type behavior
> we can and should consolidate?


This is what I was talking about, trying to take advantage of the
interactive mode that also reads from stdin to do a series of comamnds with
a single load/auth cycle.  It lacks a LOT of things for a resilient use
case such as DevStack (error abort or error ignore?, branching, everything
a DSL would bring).

And if you'd like to replace stach.sh with stack.py, I'll not stop you,
just don't call it DevStack.  Now you are building yet another deployment
tool.  We've also been down that road before. It may well be time to retire
DevStack, be sure to let us know when those willing to sponsor that work
show up so they can attempt to learn from some of our mistakes and not
repeat them the hard way.

dt

-- 

Dean Troyer
dtro...@gmail.com
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] openstack client slowness / client-as-a-service

[Adam Young]

On 04/19/2016 07:24 PM, Jamie Lennox wrote:


Rather than ditching python for something like go, I'd rather put 
together a CLI with no plugins and that only depended on keystoneauth 
and os-client-config as libraries. No?
Let me add that if you are doing anything non trivial withe the CLI, you 
might want to think about just coding the whole thing in Python...which 
is, as you recall, we suggested Devstack go back about 5 years ago.  
There was a firm "stay in bash" push at the time, and so, yeah, we have 
a one-CLI call at a time install process.



Maybe it is time to revamp Devstack.  Is there some way that, without a 
major rewrite, it could take better advantage of the CLI? Could we group 
commands, or migrate sections to python scripts that really all need to 
be done together?  For example, most of the early prep of the Keystone 
server moved to keystone-manage bootstrap.  Is there more bootstrap-type 
behavior we can and should consolidate?


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] openstack client slowness / client-as-a-service

[Jamie Lennox]

On 20 April 2016 at 04:17, Monty Taylor  wrote:

> On 04/19/2016 10:16 AM, Daniel P. Berrange wrote:
>
>> On Tue, Apr 19, 2016 at 09:57:56AM -0500, Dean Troyer wrote:
>>
>>> On Tue, Apr 19, 2016 at 9:06 AM, Adam Young  wrote:
>>>
>>> I wonder how much of that is Token caching.  In a typical CLI use patter,
 a new token is created each time a client is called, with no passing of
 a
 token between services.  Using a session can greatly decrease the
 number of
 round trips to Keystone.


>>> Not as much as you think (or hope?).  Persistent token caching to disk
>>> will
>>> help some, at other expenses though.  Using --timing on OSC will show how
>>> much time the Identity auth round trip cost.
>>>
>>> I don't have current numbers, the last time I instrumented OSC there were
>>> significant load times for some modules, so we went a good distance to
>>> lazy-load as much as possible.
>>>
>>> What Dan sees WRT a persistent client process, though, is a combination
>>> of
>>> those two things: saving the Python loading and the Keystone round trips.
>>>
>>
>> The 1.5sec overhead I eliminated doesn't actually have anything todo
>> with network round trips at all. Even if you turn off all network
>> services and just run 'openstack ' and let it fail due
>> to inability to connect it'll still have that 1.5 sec overhead. It
>> is all related to python runtime loading and work done during module
>> importing.
>>
>> eg run 'unstack.sh' and then compare the main openstack client:
>>
>> $ time /usr/bin/openstack server list
>> Discovering versions from the identity service failed when creating the
>> password plugin. Attempting to determine version from URL.
>> Unable to establish connection to http://192.168.122.156:5000/v2.0/tokens
>>
>> real0m1.555s
>> user0m1.407s
>> sys 0m0.147s
>>
>> Against my client-as-a-service version:
>>
>> $ time $HOME/bin/openstack server list
>> [Errno 111] Connection refused
>>
>> real0m0.045s
>> user0m0.029s
>> sys 0m0.016s
>>
>>
>> I'm sure there is scope for also optimizing network traffic / round
>> trips, but I didn't investigate that at all.
>>
>> I have (had!) a version of DevStack that put OSC into a subprocess and
>>> called it via pipes to do essentially what Dan suggests.  It saves some
>>> time, at the expense of complexity that may or may not be worth the
>>> effort.
>>>
>>
>> devstack doesn't actually really need any significant changes beyond
>> making sure $PATH pointed to the replacement client programs and that
>> the server was running - the latter could be automated as a launch on
>> demand thing which would limit devstack changes.
>>
>> It actually doesn't technically need any devstack change - these
>> replacement clients could simply be put in some 3rd party git repo
>> and let developers who want the speed benefit simply put them in
>> their $PATH before running devstack.
>>
>> One thing missing is any sort of transactional control in the I/O with the
>>> subprocess, ie, an EOT marker.  I planned to add a -0 option (think
>>> xargs)
>>> to handle that but it's still down a few slots on my priority list.
>>> Error
>>> handling is another problem, and at this point (for DevStack purposes
>>> anyway) I stopped the investigation, concluding that reliability trumped
>>> a
>>> few seconds saved here.
>>>
>>
>> For I/O I simply replaced stdout + stderr with a new StringIO handle to
>> capture the data when running each command, and for error handling I
>> ensured the exit status was fed back & likewise stderr printed.
>>
>> It is more than just a few seconds saved - almost 4 minutes, or
>> nearly 20% of entire time to run stack.sh on my machine
>>
>>
>> Ultimately, this is one of the two giant nails in the coffin of continuing
>>> to persue CLIs in Python.  The other is co-installability. (See that
>>> current thread on the ML for pain points).  Both are easily solved with
>>> native-code-generating languages.  Go and Rust are at the top of my
>>> personal list here...
>>>
>>
> Using entrypoints and plugins in python is slow, so loading them is slow,
> as is loading all of the dependent libraries. Those were choices made for
> good reason back in the day, but I'm not convinced either are great anymore.
>
> A pluginless CLI that simply used REST calls rather than the
> python-clientlibs should be able to launch in get to the business of doing
> work in 0.2 seconds - counting time to load and parse clouds.yaml. That
> time could be reduced - the time spent in occ parsing vendor json files is
> not strictly necessary and certainly could go faster. It's not as fast as
> 0.004 seconds, but with very little effort it's 6x faster.
>
> Rather than ditching python for something like go, I'd rather put together
> a CLI with no plugins and that only depended on keystoneauth and
> os-client-config as libraries. No?
>
>
I can feel Dean banging his head from here :)

If you extend this because 

Re: [openstack-dev] [devstack] openstack client slowness / client-as-a-service

[John Griffith]

On Tue, Apr 19, 2016 at 12:17 PM, Monty Taylor  wrote:

> On 04/19/2016 10:16 AM, Daniel P. Berrange wrote:
>
>> On Tue, Apr 19, 2016 at 09:57:56AM -0500, Dean Troyer wrote:
>>
>>> On Tue, Apr 19, 2016 at 9:06 AM, Adam Young  wrote:
>>>
>>> I wonder how much of that is Token caching.  In a typical CLI use patter,
 a new token is created each time a client is called, with no passing of
 a
 token between services.  Using a session can greatly decrease the
 number of
 round trips to Keystone.


>>> Not as much as you think (or hope?).  Persistent token caching to disk
>>> will
>>> help some, at other expenses though.  Using --timing on OSC will show how
>>> much time the Identity auth round trip cost.
>>>
>>> I don't have current numbers, the last time I instrumented OSC there were
>>> significant load times for some modules, so we went a good distance to
>>> lazy-load as much as possible.
>>>
>>> What Dan sees WRT a persistent client process, though, is a combination
>>> of
>>> those two things: saving the Python loading and the Keystone round trips.
>>>
>>
>> The 1.5sec overhead I eliminated doesn't actually have anything todo
>> with network round trips at all. Even if you turn off all network
>> services and just run 'openstack ' and let it fail due
>> to inability to connect it'll still have that 1.5 sec overhead. It
>> is all related to python runtime loading and work done during module
>> importing.
>>
>> eg run 'unstack.sh' and then compare the main openstack client:
>>
>> $ time /usr/bin/openstack server list
>> Discovering versions from the identity service failed when creating the
>> password plugin. Attempting to determine version from URL.
>> Unable to establish connection to http://192.168.122.156:5000/v2.0/tokens
>>
>> real0m1.555s
>> user0m1.407s
>> sys 0m0.147s
>>
>> Against my client-as-a-service version:
>>
>> $ time $HOME/bin/openstack server list
>> [Errno 111] Connection refused
>>
>> real0m0.045s
>> user0m0.029s
>> sys 0m0.016s
>>
>>
>> I'm sure there is scope for also optimizing network traffic / round
>> trips, but I didn't investigate that at all.
>>
>> I have (had!) a version of DevStack that put OSC into a subprocess and
>>> called it via pipes to do essentially what Dan suggests.  It saves some
>>> time, at the expense of complexity that may or may not be worth the
>>> effort.
>>>
>>
>> devstack doesn't actually really need any significant changes beyond
>> making sure $PATH pointed to the replacement client programs and that
>> the server was running - the latter could be automated as a launch on
>> demand thing which would limit devstack changes.
>>
>> It actually doesn't technically need any devstack change - these
>> replacement clients could simply be put in some 3rd party git repo
>> and let developers who want the speed benefit simply put them in
>> their $PATH before running devstack.
>>
>> One thing missing is any sort of transactional control in the I/O with the
>>> subprocess, ie, an EOT marker.  I planned to add a -0 option (think
>>> xargs)
>>> to handle that but it's still down a few slots on my priority list.
>>> Error
>>> handling is another problem, and at this point (for DevStack purposes
>>> anyway) I stopped the investigation, concluding that reliability trumped
>>> a
>>> few seconds saved here.
>>>
>>
>> For I/O I simply replaced stdout + stderr with a new StringIO handle to
>> capture the data when running each command, and for error handling I
>> ensured the exit status was fed back & likewise stderr printed.
>>
>> It is more than just a few seconds saved - almost 4 minutes, or
>> nearly 20% of entire time to run stack.sh on my machine
>>
>>
>> Ultimately, this is one of the two giant nails in the coffin of continuing
>>> to persue CLIs in Python.  The other is co-installability. (See that
>>> current thread on the ML for pain points).  Both are easily solved with
>>> native-code-generating languages.  Go and Rust are at the top of my
>>> personal list here...
>>>
>>
> Using entrypoints and plugins in python is slow, so loading them is slow,
> as is loading all of the dependent libraries. Those were choices made for
> good reason back in the day, but I'm not convinced either are great anymore.
>
> A pluginless CLI that simply used REST calls rather than the
> python-clientlibs should be able to launch in get to the business of doing
> work in 0.2 seconds - counting time to load and parse clouds.yaml. That
> time could be reduced - the time spent in occ parsing vendor json files is
> not strictly necessary and certainly could go faster. It's not as fast as
> 0.004 seconds, but with very little effort it's 6x faster.
>
> Rather than ditching python for something like go, I'd rather put together
> a CLI with no plugins and that only depended on keystoneauth and
> os-client-config as libraries. No?

​
Yes, it would certainly seem more pragmatic than just dumping 

Re: [openstack-dev] [devstack] openstack client slowness / client-as-a-service

[Mooney, Sean K]

> -Original Message-
> From: Jay Pipes [mailto:jaypi...@gmail.com]
> Sent: Tuesday, April 19, 2016 8:10 PM
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] [devstack] openstack client slowness /
> client-as-a-service
> 
> On 04/19/2016 02:17 PM, Monty Taylor wrote:
> > Rather than ditching python for something like go, I'd rather put
> > together a CLI with no plugins and that only depended on keystoneauth
> > and os-client-config as libraries. No?
[Mooney, Sean K] that is similar to how shade works correct. There
Are no plugins all services are part of the core project. 

It might be a little out there but I was playing with http://nuitka.net/ at the 
weekend.
It's a python compiler that uses libpython to generate an AST that gets 
converted
To cpp and compile with the compiler of your choice to produce a binary.

I used clang and was able to compile the nova compute agent, the nova scheduler
and there oslo dependencies without any code modification. I haven't had a 
change
to use rally to see if it actually improved the performance but I was perfectly 
functional.

I was thinking of creating a tox env that would read the console_scripts entry 
in the setup.cfg
and generate a binary for each by compiling it with nuitka as a follow up but
If you want to still code in python but improve the speed then compiling it 
would
Also be worth considering before moving to something like go, especially since 
nuitka allows
You to mike compiled and uncompiled code.

> 
> Bingo.
> 
> -jay
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] openstack client slowness / client-as-a-service

[Jay Pipes]

On 04/19/2016 03:24 PM, Steve Martinelli wrote:

Or make use of openstacksdk instead of pulling in all the clients


This would be a good thing, yes.

-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] openstack client slowness / client-as-a-service

[Perry, Sean]

From: Fox, Kevin M [kevin@pnnl.gov]
Sent: Tuesday, April 19, 2016 12:59 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [devstack] openstack client slowness / 
client-as-a-service

What about a plugin cache like feature? if no cache, or cache older then 24 
hours, dump all discovered plugins into a python file that loads them more 
statically. if the cache is valid, just use it. and a hook in rpms/debs to 
remove the cache on plugin install/uninstall.

That is what I was thinking. But first we have to prove where the time is spent.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] openstack client slowness / client-as-a-service

[Fox, Kevin M]

What about a plugin cache like feature? if no cache, or cache older then 24 
hours, dump all discovered plugins into a python file that loads them more 
statically. if the cache is valid, just use it. and a hook in rpms/debs to 
remove the cache on plugin install/uninstall.

Thanks,
Kevin

From: Monty Taylor [mord...@inaugust.com]
Sent: Tuesday, April 19, 2016 11:17 AM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [devstack] openstack client slowness / 
client-as-a-service

On 04/19/2016 10:16 AM, Daniel P. Berrange wrote:
> On Tue, Apr 19, 2016 at 09:57:56AM -0500, Dean Troyer wrote:
>> On Tue, Apr 19, 2016 at 9:06 AM, Adam Young <ayo...@redhat.com> wrote:
>>
>>> I wonder how much of that is Token caching.  In a typical CLI use patter,
>>> a new token is created each time a client is called, with no passing of a
>>> token between services.  Using a session can greatly decrease the number of
>>> round trips to Keystone.
>>>
>>
>> Not as much as you think (or hope?).  Persistent token caching to disk will
>> help some, at other expenses though.  Using --timing on OSC will show how
>> much time the Identity auth round trip cost.
>>
>> I don't have current numbers, the last time I instrumented OSC there were
>> significant load times for some modules, so we went a good distance to
>> lazy-load as much as possible.
>>
>> What Dan sees WRT a persistent client process, though, is a combination of
>> those two things: saving the Python loading and the Keystone round trips.
>
> The 1.5sec overhead I eliminated doesn't actually have anything todo
> with network round trips at all. Even if you turn off all network
> services and just run 'openstack ' and let it fail due
> to inability to connect it'll still have that 1.5 sec overhead. It
> is all related to python runtime loading and work done during module
> importing.
>
> eg run 'unstack.sh' and then compare the main openstack client:
>
> $ time /usr/bin/openstack server list
> Discovering versions from the identity service failed when creating the 
> password plugin. Attempting to determine version from URL.
> Unable to establish connection to http://192.168.122.156:5000/v2.0/tokens
>
> real  0m1.555s
> user  0m1.407s
> sys   0m0.147s
>
> Against my client-as-a-service version:
>
> $ time $HOME/bin/openstack server list
> [Errno 111] Connection refused
>
> real  0m0.045s
> user  0m0.029s
> sys   0m0.016s
>
>
> I'm sure there is scope for also optimizing network traffic / round
> trips, but I didn't investigate that at all.
>
>> I have (had!) a version of DevStack that put OSC into a subprocess and
>> called it via pipes to do essentially what Dan suggests.  It saves some
>> time, at the expense of complexity that may or may not be worth the effort.
>
> devstack doesn't actually really need any significant changes beyond
> making sure $PATH pointed to the replacement client programs and that
> the server was running - the latter could be automated as a launch on
> demand thing which would limit devstack changes.
>
> It actually doesn't technically need any devstack change - these
> replacement clients could simply be put in some 3rd party git repo
> and let developers who want the speed benefit simply put them in
> their $PATH before running devstack.
>
>> One thing missing is any sort of transactional control in the I/O with the
>> subprocess, ie, an EOT marker.  I planned to add a -0 option (think xargs)
>> to handle that but it's still down a few slots on my priority list.  Error
>> handling is another problem, and at this point (for DevStack purposes
>> anyway) I stopped the investigation, concluding that reliability trumped a
>> few seconds saved here.
>
> For I/O I simply replaced stdout + stderr with a new StringIO handle to
> capture the data when running each command, and for error handling I
> ensured the exit status was fed back & likewise stderr printed.
>
> It is more than just a few seconds saved - almost 4 minutes, or
> nearly 20% of entire time to run stack.sh on my machine
>
>
>> Ultimately, this is one of the two giant nails in the coffin of continuing
>> to persue CLIs in Python.  The other is co-installability. (See that
>> current thread on the ML for pain points).  Both are easily solved with
>> native-code-generating languages.  Go and Rust are at the top of my
>> personal list here...

Using entrypoints and plugins in python is slow, so loading them is
slow, as is loading all of the dependent libraries. Those were choices
made for good reason back in the day, but I'm not convinced either are
great anym

Re: [openstack-dev] [devstack] openstack client slowness / client-as-a-service

[Steve Martinelli]

Or make use of openstacksdk instead of pulling in all the clients

stevemar



From:   Jay Pipes <jaypi...@gmail.com>
To: openstack-dev@lists.openstack.org
Date:   2016/04/19 03:11 PM
Subject:    Re: [openstack-dev] [devstack] openstack client slowness /
    client-as-a-service



On 04/19/2016 02:17 PM, Monty Taylor wrote:
> Rather than ditching python for something like go, I'd rather put
> together a CLI with no plugins and that only depended on keystoneauth
> and os-client-config as libraries. No?

Bingo.

-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] openstack client slowness / client-as-a-service

[Jay Pipes]

On 04/19/2016 02:17 PM, Monty Taylor wrote:

Rather than ditching python for something like go, I'd rather put
together a CLI with no plugins and that only depended on keystoneauth
and os-client-config as libraries. No?


Bingo.

-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] openstack client slowness / client-as-a-service

[Perry, Sean]

A pluginless CLI that simply used REST calls rather than the
python-clientlibs should be able to launch in get to the business of
doing work in 0.2 seconds - counting time to load and parse clouds.yaml.
That time could be reduced - the time spent in occ parsing vendor json
files is not strictly necessary and certainly could go faster. It's not
as fast as 0.004 seconds, but with very little effort it's 6x faster.

Rather than ditching python for something like go, I'd rather put
together a CLI with no plugins and that only depended on keystoneauth
and os-client-config as libraries. No?


Largely I agree. See the numbers I posted a few moments ago. Most of the 1 
second initialization is time spent loading plugins and the machinery of the 
CLI. After that we have issues with how the communication is handled.

If we can avoid re-implementing the wheel for each service client that would be 
nice.
An interesting experiment I think would be to have the CLI cache the entry 
points it finds so it only needs to loook for them once.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] openstack client slowness / client-as-a-service

[Perry, Sean]

From: Ian Cordasco [sigmaviru...@gmail.com]
Sent: Tuesday, April 19, 2016 11:11 AM
To: Perry, Sean; OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [devstack] openstack client slowness / 
client-as-a-service

-Original Message-
From: Perry, Sean <sean.pe...@hpe.com>
Reply: OpenStack Development Mailing List (not for usage questions) 
<openstack-dev@lists.openstack.org>
Date: April 19, 2016 at 12:41:02
To: OpenStack Development Mailing List (not for usage questions) 
<openstack-dev@lists.openstack.org>
Subject:  Re: [openstack-dev] [devstack] openstack client slowness /
client-as-a-service

> What Dan sees WRT a persistent client process, though, is a combination of 
> those two things:
> saving the Python loading and the Keystone round trips.
>
> If I replace the import of openstack.shell with a main that just returns 0 in 
> the OSC entry
> point and time it the result floats around 0.030s on my VMs. Python itself is 
> not the issue.
> The cost is in the imports.
>
> openstack --help >/dev/null takes over 4 seconds.
>
> Dean, have you tried replacing all of the stevedore/cliff magic with explicit 
> imports
> of the common OpenStack clients? I am curious if the issue is disk access, 
> Python's egg
> loading, etc.

Define "common". I don't think you'll franly find a common definition of that. 
Further, if you're using an even remotely recent version of pip (probably 1.4 
or newer) eggs are no longer a thing. If you're using vanilla setup.py to 
install everything, though, then yes maybe you're having problems related to 
that.

While it does not make (every|any)one happy, for benchmarking the list could be 
keystone nova, glance, cinder, and neutron. This is NOT about choosing projects 
but about setting up a reasonable baseline number.

By "egg" I mean all of the bits involved in Python loading our code. While 
zipped up eggs may not be involved there is the egg-info to parse for the entry 
points for instance.

A quick test this morning of replacing clientmanager and commandmanager with 
Mock and then having OSC exit before it invokes run() shows that loading the 
potential clients in client manager add half a second. So right now I have:

0.03s just to start the process
0.5s to start and exit
1.1s to start and exit if clientmanager rounds up potential client modules.

Then we start hitting token issue, chatting with services, etc. In my devstack 
'token issue' for admin takes 7.5 seconds of which 6.6s is network according to 
--timing evenly split between POST and GET.

I tried commenting out / mocking the other modules in openstackclient.shell to 
little effect.

This tells me that potentially we can save time

* optimizing the shell initialization
* optimizing the plugin loading / entry point handling
* the round tripping to the services

Whether our time is better spent optimizing / refactoring what we have or 
rewriting, introducing bugs, finding out the performance issues, plugin loading 
issues, etc. of a new language is good discussion. But just saying , yeah 
Python is slow is not selling it based on the numbers I am seeing.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] openstack client slowness / client-as-a-service

[Monty Taylor]

On 04/19/2016 10:16 AM, Daniel P. Berrange wrote:

On Tue, Apr 19, 2016 at 09:57:56AM -0500, Dean Troyer wrote:

On Tue, Apr 19, 2016 at 9:06 AM, Adam Young  wrote:


I wonder how much of that is Token caching.  In a typical CLI use patter,
a new token is created each time a client is called, with no passing of a
token between services.  Using a session can greatly decrease the number of
round trips to Keystone.



Not as much as you think (or hope?).  Persistent token caching to disk will
help some, at other expenses though.  Using --timing on OSC will show how
much time the Identity auth round trip cost.

I don't have current numbers, the last time I instrumented OSC there were
significant load times for some modules, so we went a good distance to
lazy-load as much as possible.

What Dan sees WRT a persistent client process, though, is a combination of
those two things: saving the Python loading and the Keystone round trips.


The 1.5sec overhead I eliminated doesn't actually have anything todo
with network round trips at all. Even if you turn off all network
services and just run 'openstack ' and let it fail due
to inability to connect it'll still have that 1.5 sec overhead. It
is all related to python runtime loading and work done during module
importing.

eg run 'unstack.sh' and then compare the main openstack client:

$ time /usr/bin/openstack server list
Discovering versions from the identity service failed when creating the 
password plugin. Attempting to determine version from URL.
Unable to establish connection to http://192.168.122.156:5000/v2.0/tokens

real0m1.555s
user0m1.407s
sys 0m0.147s

Against my client-as-a-service version:

$ time $HOME/bin/openstack server list
[Errno 111] Connection refused

real0m0.045s
user0m0.029s
sys 0m0.016s


I'm sure there is scope for also optimizing network traffic / round
trips, but I didn't investigate that at all.


I have (had!) a version of DevStack that put OSC into a subprocess and
called it via pipes to do essentially what Dan suggests.  It saves some
time, at the expense of complexity that may or may not be worth the effort.


devstack doesn't actually really need any significant changes beyond
making sure $PATH pointed to the replacement client programs and that
the server was running - the latter could be automated as a launch on
demand thing which would limit devstack changes.

It actually doesn't technically need any devstack change - these
replacement clients could simply be put in some 3rd party git repo
and let developers who want the speed benefit simply put them in
their $PATH before running devstack.


One thing missing is any sort of transactional control in the I/O with the
subprocess, ie, an EOT marker.  I planned to add a -0 option (think xargs)
to handle that but it's still down a few slots on my priority list.  Error
handling is another problem, and at this point (for DevStack purposes
anyway) I stopped the investigation, concluding that reliability trumped a
few seconds saved here.


For I/O I simply replaced stdout + stderr with a new StringIO handle to
capture the data when running each command, and for error handling I
ensured the exit status was fed back & likewise stderr printed.

It is more than just a few seconds saved - almost 4 minutes, or
nearly 20% of entire time to run stack.sh on my machine



Ultimately, this is one of the two giant nails in the coffin of continuing
to persue CLIs in Python.  The other is co-installability. (See that
current thread on the ML for pain points).  Both are easily solved with
native-code-generating languages.  Go and Rust are at the top of my
personal list here...


Using entrypoints and plugins in python is slow, so loading them is 
slow, as is loading all of the dependent libraries. Those were choices 
made for good reason back in the day, but I'm not convinced either are 
great anymore.


A pluginless CLI that simply used REST calls rather than the 
python-clientlibs should be able to launch in get to the business of 
doing work in 0.2 seconds - counting time to load and parse clouds.yaml. 
That time could be reduced - the time spent in occ parsing vendor json 
files is not strictly necessary and certainly could go faster. It's not 
as fast as 0.004 seconds, but with very little effort it's 6x faster.


Rather than ditching python for something like go, I'd rather put 
together a CLI with no plugins and that only depended on keystoneauth 
and os-client-config as libraries. No?


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] openstack client slowness / client-as-a-service

[Ian Cordasco]

 

-Original Message-
From: Perry, Sean <sean.pe...@hpe.com>
Reply: OpenStack Development Mailing List (not for usage questions) 
<openstack-dev@lists.openstack.org>
Date: April 19, 2016 at 12:41:02
To: OpenStack Development Mailing List (not for usage questions) 
<openstack-dev@lists.openstack.org>
Subject:  Re: [openstack-dev] [devstack] openstack client slowness /
client-as-a-service

> What Dan sees WRT a persistent client process, though, is a combination of 
> those two things:  
> saving the Python loading and the Keystone round trips.
>  
> If I replace the import of openstack.shell with a main that just returns 0 in 
> the OSC entry  
> point and time it the result floats around 0.030s on my VMs. Python itself is 
> not the issue.  
> The cost is in the imports.
>  
> openstack --help >/dev/null takes over 4 seconds.
>  
> Dean, have you tried replacing all of the stevedore/cliff magic with explicit 
> imports  
> of the common OpenStack clients? I am curious if the issue is disk access, 
> Python's egg  
> loading, etc.

Define "common". I don't think you'll franly find a common definition of that. 
Further, if you're using an even remotely recent version of pip (probably 1.4 
or newer) eggs are no longer a thing. If you're using vanilla setup.py to 
install everything, though, then yes maybe you're having problems related to 
that.

> Yeah Rust and friends are fun. But it presents a significant barrier to 
> entry. If nothing  
> else, Rust is not Enterprise ready. Go is not much better. We need to 
> remember that not 

Define "Enterprise ready". Considering the number and size of users/backers of 
each, I'm sure you have a different definition of Enterprise. If we're going to 
compare the two, Rust wins by simple virtue of actually having a packaging 
story.

> everyone will have their systems connected to the full Internet during system 
> install  
> time. Pypi mirrors with selected packages are possible today. Sure we could 
> implement  
> a cargo mirror too (or whatever Go calls theirs). But is that really better 
> than improving  
> the current situation?

Given the current status of CPython 2.7 (which I suspect most people are going 
to be using) what kind of speed improvements do you think you'll be able to add 
there? You could certainly work on fixing up Python 3.6 onwards (if it hasn't 
already been improved) but Enterprises aren't really looking to migrate to 
Python 3 anytime soon. If they were eager/willing to do that, the CPython 
developers wouldn't have extended the support lifetime of Python 2.7. You can 
try to optimize osc a whole lot, but as I understand it, a lot of effort has 
gone into that already.

I like OSC and it's goals and everything, but sometimes you have to wonder 
exactly how much more time and effort you are going to be willing to throw at a 
problem where the limiting factors are potentially out of scope for the project 
itself.

--  
Ian Cordasco


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] openstack client slowness / client-as-a-service

[Perry, Sean]

What Dan sees WRT a persistent client process, though, is a combination of 
those two things: saving the Python loading and the Keystone round trips.

If I replace the import of openstack.shell with a main that just returns 0 in 
the OSC entry point and time it the result floats around 0.030s on my VMs. 
Python itself is not the issue. The cost is in the imports.

openstack --help >/dev/null takes over 4 seconds.

Dean, have you tried replacing all of the stevedore/cliff magic with explicit 
imports of the common OpenStack clients? I am curious if the issue is disk 
access, Python's egg loading, etc.

Yeah Rust and friends are fun. But it presents a significant barrier to entry. 
If nothing else, Rust is not Enterprise ready. Go is not much better. We need 
to remember that not everyone will have their systems connected to the full 
Internet during system install time. Pypi mirrors with selected packages are 
possible today. Sure we could implement a cargo mirror too (or whatever Go 
calls theirs). But is that really better than improving the current situation?

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] openstack client slowness / client-as-a-service

[Perry, Sean]

From: Adam Young [ayo...@redhat.com]
Sent: Tuesday, April 19, 2016 7:06 AM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [devstack] openstack client slowness / 
client-as-a-service

On 04/18/2016 09:19 AM, Daniel P. Berrange wrote:

There have been threads in the past about the slowness of the "openstack"
client tool such as this one by Sean last year:

  http://lists.openstack.org/pipermail/openstack-dev/2015-April/061317.html

Sean mentioned a 1.5s fixed overhead on openstack client, and mentions it
is significantly slower than the equivalent nova command. In my testing
I don't see any real speed difference between openstack & nova client
programs, so maybe that differential has been addressed since Sean's
original thread, or maybe nova has got slower.

Overall though, I find it is way too sluggish considering it is running
on a local machine with 12 cpus and 30 GB of RAM.

I had a quick go at trying to profile the tools with cprofile and analyse
with KCacheGrind as per this blog:

  
https://julien.danjou.info/blog/2015/guide-to-python-profiling-cprofile-concrete-case-carbonara

And notice that in profiling 'nova help' for example, the big sink appears
to come from the 'pkg_resource' module and its use of pyparsing. I didn't
spend any real time to dig into this in detail, because it got me wondering
whether we can easily just avoid the big startup penalty by not having to
startup a new python interpretor for each command we run.

I traced devstack and saw it run 'openstack' and 'neutron' commands approx
140 times in my particular configuration. If each one of those has a 1.5s
overhead, we could potentially save 3 & 1/2 minutes off devstack execution
time.

So as a proof of concept I have created an 'openstack-server' command
which listens on a unix socket for requests and then invokes the
OpenStackShell.run / OpenStackComputeShell.main / NeutronShell.run
methods as appropriate.

I then replaced the 'openstack', 'nova' and 'neutron' commands with
versions that simply call to the 'openstack-server' service over the
UNIX socket. Since devstack will always recreate these commands in
/usr/bin, I simply put my replacements in $HOME/bin and then made
sure $HOME/bin was first in the $PATH

You might call this 'command line as a service' :-)

Anyhow, with my devstack setup a traditional install takes

  real  21m34.050s
  user  7m8.649s
  sys   1m57.865s

And when using openstack-server it only takes

  real  17m47.059s
  user  3m51.087s
  sys   1m42.428s

So that has cut 18% off the total running time for devstack, which
is quite considerable really.

I'm attaching the openstack-server & replacement openstack commands
so you can see what I did. You have to manually run the openstack-server
command ahead of time and it'll print out details of every command run
on stdout.

Anyway, I'm not personally planning to take this experiment any further.
I'll probably keep using this wrapper in my own local dev env since it
does cut down on devstack time significantly. This mail is just to see
if it'll stimulate any interesting discussion or motivate someone to
explore things further.

I wonder how much of that is Token caching.  In a typical CLI use patter, a new 
token is created each time a client is called, with no passing of a token 
between services.  Using a session can greatly decrease the number of round 
trips to Keystone.


In combination with a Session cache in ~/, maybe a modules found cache so 
stevedore et al do not need to hunt during initialization?

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] openstack client slowness / client-as-a-service

[Daniel P. Berrange]

On Tue, Apr 19, 2016 at 09:57:56AM -0500, Dean Troyer wrote:
> On Tue, Apr 19, 2016 at 9:06 AM, Adam Young  wrote:
> 
> > I wonder how much of that is Token caching.  In a typical CLI use patter,
> > a new token is created each time a client is called, with no passing of a
> > token between services.  Using a session can greatly decrease the number of
> > round trips to Keystone.
> >
> 
> Not as much as you think (or hope?).  Persistent token caching to disk will
> help some, at other expenses though.  Using --timing on OSC will show how
> much time the Identity auth round trip cost.
> 
> I don't have current numbers, the last time I instrumented OSC there were
> significant load times for some modules, so we went a good distance to
> lazy-load as much as possible.
> 
> What Dan sees WRT a persistent client process, though, is a combination of
> those two things: saving the Python loading and the Keystone round trips.

The 1.5sec overhead I eliminated doesn't actually have anything todo
with network round trips at all. Even if you turn off all network
services and just run 'openstack ' and let it fail due
to inability to connect it'll still have that 1.5 sec overhead. It
is all related to python runtime loading and work done during module
importing.

eg run 'unstack.sh' and then compare the main openstack client:

$ time /usr/bin/openstack server list
Discovering versions from the identity service failed when creating the 
password plugin. Attempting to determine version from URL.
Unable to establish connection to http://192.168.122.156:5000/v2.0/tokens

real0m1.555s
user0m1.407s
sys 0m0.147s

Against my client-as-a-service version:

$ time $HOME/bin/openstack server list
[Errno 111] Connection refused

real0m0.045s
user0m0.029s
sys 0m0.016s


I'm sure there is scope for also optimizing network traffic / round
trips, but I didn't investigate that at all.

> I have (had!) a version of DevStack that put OSC into a subprocess and
> called it via pipes to do essentially what Dan suggests.  It saves some
> time, at the expense of complexity that may or may not be worth the effort.

devstack doesn't actually really need any significant changes beyond
making sure $PATH pointed to the replacement client programs and that
the server was running - the latter could be automated as a launch on
demand thing which would limit devstack changes.

It actually doesn't technically need any devstack change - these
replacement clients could simply be put in some 3rd party git repo
and let developers who want the speed benefit simply put them in
their $PATH before running devstack.

> One thing missing is any sort of transactional control in the I/O with the
> subprocess, ie, an EOT marker.  I planned to add a -0 option (think xargs)
> to handle that but it's still down a few slots on my priority list.  Error
> handling is another problem, and at this point (for DevStack purposes
> anyway) I stopped the investigation, concluding that reliability trumped a
> few seconds saved here.

For I/O I simply replaced stdout + stderr with a new StringIO handle to
capture the data when running each command, and for error handling I
ensured the exit status was fed back & likewise stderr printed.

It is more than just a few seconds saved - almost 4 minutes, or
nearly 20% of entire time to run stack.sh on my machine


> Ultimately, this is one of the two giant nails in the coffin of continuing
> to persue CLIs in Python.  The other is co-installability. (See that
> current thread on the ML for pain points).  Both are easily solved with
> native-code-generating languages.  Go and Rust are at the top of my
> personal list here...

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] openstack client slowness / client-as-a-service

[Morgan Fainberg]

On Tue, Apr 19, 2016 at 7:57 AM, Dean Troyer  wrote:

> On Tue, Apr 19, 2016 at 9:06 AM, Adam Young  wrote:
>
>> I wonder how much of that is Token caching.  In a typical CLI use patter,
>> a new token is created each time a client is called, with no passing of a
>> token between services.  Using a session can greatly decrease the number of
>> round trips to Keystone.
>>
>
> Not as much as you think (or hope?).  Persistent token caching to disk
> will help some, at other expenses though.  Using --timing on OSC will show
> how much time the Identity auth round trip cost.
>
> I don't have current numbers, the last time I instrumented OSC there were
> significant load times for some modules, so we went a good distance to
> lazy-load as much as possible.
>
> What Dan sees WRT a persistent client process, though, is a combination of
> those two things: saving the Python loading and the Keystone round trips.
>
> I have (had!) a version of DevStack that put OSC into a subprocess and
> called it via pipes to do essentially what Dan suggests.  It saves some
> time, at the expense of complexity that may or may not be worth the effort.
>
> One thing missing is any sort of transactional control in the I/O with the
> subprocess, ie, an EOT marker.  I planned to add a -0 option (think xargs)
> to handle that but it's still down a few slots on my priority list.  Error
> handling is another problem, and at this point (for DevStack purposes
> anyway) I stopped the investigation, concluding that reliability trumped a
> few seconds saved here.
>
> Ultimately, this is one of the two giant nails in the coffin of continuing
> to persue CLIs in Python.  The other is co-installability. (See that
> current thread on the ML for pain points).  Both are easily solved with
> native-code-generating languages.  Go and Rust are at the top of my
> personal list here...
>
> dt
>

+1 for Rust (though I'd totally support Go as well) as personal choices.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] openstack client slowness / client-as-a-service

[Dean Troyer]

On Tue, Apr 19, 2016 at 9:06 AM, Adam Young  wrote:

> I wonder how much of that is Token caching.  In a typical CLI use patter,
> a new token is created each time a client is called, with no passing of a
> token between services.  Using a session can greatly decrease the number of
> round trips to Keystone.
>

Not as much as you think (or hope?).  Persistent token caching to disk will
help some, at other expenses though.  Using --timing on OSC will show how
much time the Identity auth round trip cost.

I don't have current numbers, the last time I instrumented OSC there were
significant load times for some modules, so we went a good distance to
lazy-load as much as possible.

What Dan sees WRT a persistent client process, though, is a combination of
those two things: saving the Python loading and the Keystone round trips.

I have (had!) a version of DevStack that put OSC into a subprocess and
called it via pipes to do essentially what Dan suggests.  It saves some
time, at the expense of complexity that may or may not be worth the effort.

One thing missing is any sort of transactional control in the I/O with the
subprocess, ie, an EOT marker.  I planned to add a -0 option (think xargs)
to handle that but it's still down a few slots on my priority list.  Error
handling is another problem, and at this point (for DevStack purposes
anyway) I stopped the investigation, concluding that reliability trumped a
few seconds saved here.

Ultimately, this is one of the two giant nails in the coffin of continuing
to persue CLIs in Python.  The other is co-installability. (See that
current thread on the ML for pain points).  Both are easily solved with
native-code-generating languages.  Go and Rust are at the top of my
personal list here...

dt
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack] openstack client slowness / client-as-a-service

[Adam Young]

On 04/18/2016 09:19 AM, Daniel P. Berrange wrote:

There have been threads in the past about the slowness of the "openstack"
client tool such as this one by Sean last year:

   http://lists.openstack.org/pipermail/openstack-dev/2015-April/061317.html

Sean mentioned a 1.5s fixed overhead on openstack client, and mentions it
is significantly slower than the equivalent nova command. In my testing
I don't see any real speed difference between openstack & nova client
programs, so maybe that differential has been addressed since Sean's
original thread, or maybe nova has got slower.

Overall though, I find it is way too sluggish considering it is running
on a local machine with 12 cpus and 30 GB of RAM.

I had a quick go at trying to profile the tools with cprofile and analyse
with KCacheGrind as per this blog:

   
https://julien.danjou.info/blog/2015/guide-to-python-profiling-cprofile-concrete-case-carbonara

And notice that in profiling 'nova help' for example, the big sink appears
to come from the 'pkg_resource' module and its use of pyparsing. I didn't
spend any real time to dig into this in detail, because it got me wondering
whether we can easily just avoid the big startup penalty by not having to
startup a new python interpretor for each command we run.

I traced devstack and saw it run 'openstack' and 'neutron' commands approx
140 times in my particular configuration. If each one of those has a 1.5s
overhead, we could potentially save 3 & 1/2 minutes off devstack execution
time.

So as a proof of concept I have created an 'openstack-server' command
which listens on a unix socket for requests and then invokes the
OpenStackShell.run / OpenStackComputeShell.main / NeutronShell.run
methods as appropriate.

I then replaced the 'openstack', 'nova' and 'neutron' commands with
versions that simply call to the 'openstack-server' service over the
UNIX socket. Since devstack will always recreate these commands in
/usr/bin, I simply put my replacements in $HOME/bin and then made
sure $HOME/bin was first in the $PATH

You might call this 'command line as a service' :-)

Anyhow, with my devstack setup a traditional install takes

   real 21m34.050s
   user 7m8.649s
   sys  1m57.865s

And when using openstack-server it only takes

   real 17m47.059s
   user 3m51.087s
   sys  1m42.428s

So that has cut 18% off the total running time for devstack, which
is quite considerable really.

I'm attaching the openstack-server & replacement openstack commands
so you can see what I did. You have to manually run the openstack-server
command ahead of time and it'll print out details of every command run
on stdout.

Anyway, I'm not personally planning to take this experiment any further.
I'll probably keep using this wrapper in my own local dev env since it
does cut down on devstack time significantly. This mail is just to see
if it'll stimulate any interesting discussion or motivate someone to
explore things further.


I wonder how much of that is Token caching.  In a typical CLI use 
patter, a new token is created each time a client is called, with no 
passing of a token between services.  Using a session can greatly 
decrease the number of round trips to Keystone.







Regards,
Daniel


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [devstack] openstack client slowness / client-as-a-service

[Daniel P. Berrange]

There have been threads in the past about the slowness of the "openstack"
client tool such as this one by Sean last year:

  http://lists.openstack.org/pipermail/openstack-dev/2015-April/061317.html

Sean mentioned a 1.5s fixed overhead on openstack client, and mentions it
is significantly slower than the equivalent nova command. In my testing
I don't see any real speed difference between openstack & nova client
programs, so maybe that differential has been addressed since Sean's
original thread, or maybe nova has got slower.

Overall though, I find it is way too sluggish considering it is running
on a local machine with 12 cpus and 30 GB of RAM.

I had a quick go at trying to profile the tools with cprofile and analyse
with KCacheGrind as per this blog:

  
https://julien.danjou.info/blog/2015/guide-to-python-profiling-cprofile-concrete-case-carbonara

And notice that in profiling 'nova help' for example, the big sink appears
to come from the 'pkg_resource' module and its use of pyparsing. I didn't
spend any real time to dig into this in detail, because it got me wondering
whether we can easily just avoid the big startup penalty by not having to
startup a new python interpretor for each command we run.

I traced devstack and saw it run 'openstack' and 'neutron' commands approx
140 times in my particular configuration. If each one of those has a 1.5s
overhead, we could potentially save 3 & 1/2 minutes off devstack execution
time.

So as a proof of concept I have created an 'openstack-server' command
which listens on a unix socket for requests and then invokes the
OpenStackShell.run / OpenStackComputeShell.main / NeutronShell.run
methods as appropriate.

I then replaced the 'openstack', 'nova' and 'neutron' commands with
versions that simply call to the 'openstack-server' service over the
UNIX socket. Since devstack will always recreate these commands in
/usr/bin, I simply put my replacements in $HOME/bin and then made
sure $HOME/bin was first in the $PATH

You might call this 'command line as a service' :-)

Anyhow, with my devstack setup a traditional install takes

  real  21m34.050s
  user  7m8.649s
  sys   1m57.865s

And when using openstack-server it only takes

  real  17m47.059s
  user  3m51.087s
  sys   1m42.428s

So that has cut 18% off the total running time for devstack, which
is quite considerable really.

I'm attaching the openstack-server & replacement openstack commands
so you can see what I did. You have to manually run the openstack-server
command ahead of time and it'll print out details of every command run
on stdout.

Anyway, I'm not personally planning to take this experiment any further.
I'll probably keep using this wrapper in my own local dev env since it
does cut down on devstack time significantly. This mail is just to see
if it'll stimulate any interesting discussion or motivate someone to
explore things further.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|
#!/usr/bin/python

import socket
import sys
import os
import os.path
import json

server_address = "/tmp/openstack.sock"

sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)

try:
sock.connect(server_address)
except socket.error, msg:
print >>sys.stderr, msg
sys.exit(1)


def send(sock, doc):
jdoc = json.dumps(doc)
sock.send('%d\n' % len(jdoc))
sock.sendall(jdoc)

def recv(sock):
length_str = ''

char = sock.recv(1)
if len(char) == 0:
print >>sys.stderr, "Unexpected end of file"
sys.exit(1)

while char != '\n':
length_str += char
char = sock.recv(1)
if len(char) == 0:
print >>sys.stderr, "Unexpected end of file"
sys.exit(1)

total = int(length_str)

# use a memoryview to receive the data chunk by chunk efficiently
jdoc = memoryview(bytearray(total))
next_offset = 0
while total - next_offset > 0:
recv_size = sock.recv_into(jdoc[next_offset:], total - next_offset)
next_offset += recv_size
try:
doc = json.loads(jdoc.tobytes())
except (TypeError, ValueError), e:
raise Exception('Data received was not in JSON format')
return doc

try:
env = {}
passenv = ["CINDER_VERSION",
   "OS_AUTH_URL",
   "OS_IDENTITY_API_VERSION",
   "OS_NO_CACHE",
   "OS_PASSWORD",
   "OS_PROJECT_NAME",
   "OS_REGION_NAME",
   "OS_TENANT_NAME",
   "OS_USERNAME",
   "OS_VOLUME_API_VERSION"]
for name in passenv:
if name in os.environ:
env[name] = os.environ[name]

cmd = {
"app": os.path.basename(sys.argv[0]),
"env": env,
"argv":