Re: [openstack-dev] Future-based api for openstack clients calls, that starts background tasks

2015-01-12 Thread Konstantin Danilov 
On Mon, Jan 12, 2015 at 3:33 PM, Angus Salkeld asalk...@mirantis.com
wrote:


 On Mon, Jan 12, 2015 at 10:17 PM, Konstantin Danilov 
 kdani...@mirantis.com wrote:

 Boris,

 Move from sync http to something like websocket requires a lot of work
 and not directly connected
 with API issue. When openstack api servers begin to support
 websockets - it would be easy to change implementation of monitoring
 thread without breaking compatibility.
 At the moment periodical pooling from additional thread looks reasonable
 for me
 and it creates same amount of http requests as all current
 implementation.

 BP is not about improving performance,
 but about providing convenient and common API to handle background tasks.

  So we won't need to retrieve 100500 times information about object.
 As I sad before - this API creates same amount of load as any code which
 we use to check background task currently.
 It even can decrease load due to requests aggregation in some cases (but
 there a points to discuss).

  As well this pattern doesn't look great.
  I would prefer to see something like:
  vm = novaclient.servers.create(, sync=True)

 This is completely different pattern. It is blocking call, which don't
 allows you to start two(or more) background tasks
 and from same thread and make some calculations while they running in
 background.


 Except if you use threads (eventlet or other) - I am still struggling to
 enjoy Futures/yield based flowcontrol, lost battle i guess:(.






 On Mon, Jan 12, 2015 at 1:42 PM, Boris Pavlovic bpavlo...@mirantis.com
 wrote:

 Konstantin,


 I believe it's better to work on server side, and use some modern
 approach like web sockets for async operations. So we won't need to
 retrieve 100500 times information about object. And then use this feature
 in clients.

  create_future = novaclient.servers.create_async()
 .
 vm = create_future.result()


 As well this pattern doesn't look great.

 I would prefer to see something like:

   vm = novaclient.servers.create(, sync=True)


 Best regards,
 Boris Pavlovic





 On Mon, Jan 12, 2015 at 2:30 PM, Konstantin Danilov 
 kdani...@mirantis.com wrote:

 Hi all.

 There a set of openstack api functions which starts background actions
 and return preliminary results - like 'novaclient.create'. Those
 functions
 requires periodically check results and handle timeouts/errors
 (and often cleanup + restart help to fix an error).

 Check/retry/cleanup code duplicated over a lot of core projects.
 As examples - heat, tempest, rally, etc and definitely in many
 third-party scripts.


 We have some very similar code at the moment, but we are keen to move away
 from it to
 something like making use of rpc .{start,end} notifications to reduce
 the load we put on keystone and
 friends.


This is nice approach for core projects, yet novaclient users are typically
can't use such approach.

But the nice things about futures is that we can have different engines
(websockets, sync http, rpc with callbacks, etc)
behind same API, and hide all implementation details behind it.  It's even
possible to use them in simultaneously -
different engine would used to handle different calls.





 I propose to provide common higth-level API for such functions, which
 uses
 'futures' (http://en.wikipedia.org/wiki/Futures_and_promises) as a way
 to
 present background task.

 Idea is to add to each background-task-starter function a complimentary
 call,
 that returns 'future' object. E.g.

 create_future = novaclient.servers.create_async()
 .
 vm = create_future.result()


 Is that going to return on any state change or do you pass in a list of
 acceptable states?


In general it should returns result if background task is completed
successfully or raise exception
if it fails. For server I currently use 'active' as success marker and
'error' or timeout for exception
( https://github.com/koder-ua/os_api/blob/master/os_api/nova.py#L74 ), and
hope that expected states
can be calculated from api call/parameters automatically, but not 100%
sure. So - yes, additional
parameter might be required.



 -Angus



 This allows to unify(and optimize) monitoring cycles, retries, etc.
 Please found complete BP at
 https://github.com/koder-ua/os_api/blob/master/README.md

 Thanks
 --
 Kostiantyn Danilov aka koder http://koder.ua
 Principal software engineer, Mirantis

 skype:koder.ua
 http://koder-ua.blogspot.com/
 http://mirantis.com


 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

Re: [openstack-dev] Future-based api for openstack clients calls, that starts background tasks

2015-01-12 Thread Angus Salkeld 
On Mon, Jan 12, 2015 at 10:17 PM, Konstantin Danilov kdani...@mirantis.com
wrote:

 Boris,

 Move from sync http to something like websocket requires a lot of work and
 not directly connected
 with API issue. When openstack api servers begin to support
 websockets - it would be easy to change implementation of monitoring
 thread without breaking compatibility.
 At the moment periodical pooling from additional thread looks reasonable
 for me
 and it creates same amount of http requests as all current implementation.

 BP is not about improving performance,
 but about providing convenient and common API to handle background tasks.

  So we won't need to retrieve 100500 times information about object.
 As I sad before - this API creates same amount of load as any code which
 we use to check background task currently.
 It even can decrease load due to requests aggregation in some cases (but
 there a points to discuss).

  As well this pattern doesn't look great.
  I would prefer to see something like:
  vm = novaclient.servers.create(, sync=True)

 This is completely different pattern. It is blocking call, which don't
 allows you to start two(or more) background tasks
 and from same thread and make some calculations while they running in
 background.


Except if you use threads (eventlet or other) - I am still struggling to
enjoy Futures/yield based flowcontrol, lost battle i guess:(.







 On Mon, Jan 12, 2015 at 1:42 PM, Boris Pavlovic bpavlo...@mirantis.com
 wrote:

 Konstantin,


 I believe it's better to work on server side, and use some modern
 approach like web sockets for async operations. So we won't need to
 retrieve 100500 times information about object. And then use this feature
 in clients.

  create_future = novaclient.servers.create_async()
 .
 vm = create_future.result()


 As well this pattern doesn't look great.

 I would prefer to see something like:

   vm = novaclient.servers.create(, sync=True)


 Best regards,
 Boris Pavlovic





 On Mon, Jan 12, 2015 at 2:30 PM, Konstantin Danilov 
 kdani...@mirantis.com wrote:

 Hi all.

 There a set of openstack api functions which starts background actions
 and return preliminary results - like 'novaclient.create'. Those
 functions
 requires periodically check results and handle timeouts/errors
 (and often cleanup + restart help to fix an error).

 Check/retry/cleanup code duplicated over a lot of core projects.
 As examples - heat, tempest, rally, etc and definitely in many
 third-party scripts.


We have some very similar code at the moment, but we are keen to move away
from it to
something like making use of rpc .{start,end} notifications to reduce the
load we put on keystone and
friends.



 I propose to provide common higth-level API for such functions, which
 uses
 'futures' (http://en.wikipedia.org/wiki/Futures_and_promises) as a way
 to
 present background task.

 Idea is to add to each background-task-starter function a complimentary
 call,
 that returns 'future' object. E.g.

 create_future = novaclient.servers.create_async()
 .
 vm = create_future.result()


Is that going to return on any state change or do you pass in a list of
acceptable states?

-Angus



 This allows to unify(and optimize) monitoring cycles, retries, etc.
 Please found complete BP at
 https://github.com/koder-ua/os_api/blob/master/README.md

 Thanks
 --
 Kostiantyn Danilov aka koder http://koder.ua
 Principal software engineer, Mirantis

 skype:koder.ua
 http://koder-ua.blogspot.com/
 http://mirantis.com


 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




 --
 Kostiantyn Danilov aka koder.ua
 Principal software engineer, Mirantis

 skype:koder.ua
 http://koder-ua.blogspot.com/
 http://mirantis.com

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Future-based api for openstack clients calls, that starts background tasks

2015-01-12 Thread Boris Pavlovic 
Konstantin,


I believe it's better to work on server side, and use some modern approach
like web sockets for async operations. So we won't need to retrieve 100500
times information about object. And then use this feature in clients.

 create_future = novaclient.servers.create_async()
 .
 vm = create_future.result()


As well this pattern doesn't look great.

I would prefer to see something like:

  vm = novaclient.servers.create(, sync=True)


Best regards,
Boris Pavlovic





On Mon, Jan 12, 2015 at 2:30 PM, Konstantin Danilov kdani...@mirantis.com
wrote:

 Hi all.

 There a set of openstack api functions which starts background actions
 and return preliminary results - like 'novaclient.create'. Those functions
 requires periodically check results and handle timeouts/errors
 (and often cleanup + restart help to fix an error).

 Check/retry/cleanup code duplicated over a lot of core projects.
 As examples - heat, tempest, rally, etc and definitely in many third-party
 scripts.

 I propose to provide common higth-level API for such functions, which uses
 'futures' (http://en.wikipedia.org/wiki/Futures_and_promises) as a way to
 present background task.

 Idea is to add to each background-task-starter function a complimentary
 call,
 that returns 'future' object. E.g.

 create_future = novaclient.servers.create_async()
 .
 vm = create_future.result()

 This allows to unify(and optimize) monitoring cycles, retries, etc.
 Please found complete BP at
 https://github.com/koder-ua/os_api/blob/master/README.md

 Thanks
 --
 Kostiantyn Danilov aka koder http://koder.ua
 Principal software engineer, Mirantis

 skype:koder.ua
 http://koder-ua.blogspot.com/
 http://mirantis.com

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Future-based api for openstack clients calls, that starts background tasks

2015-01-12 Thread Konstantin Danilov 
Hi all.

There a set of openstack api functions which starts background actions
and return preliminary results - like 'novaclient.create'. Those functions
requires periodically check results and handle timeouts/errors
(and often cleanup + restart help to fix an error).

Check/retry/cleanup code duplicated over a lot of core projects.
As examples - heat, tempest, rally, etc and definitely in many third-party
scripts.

I propose to provide common higth-level API for such functions, which uses
'futures' (http://en.wikipedia.org/wiki/Futures_and_promises) as a way to
present background task.

Idea is to add to each background-task-starter function a complimentary
call,
that returns 'future' object. E.g.

create_future = novaclient.servers.create_async()
.
vm = create_future.result()

This allows to unify(and optimize) monitoring cycles, retries, etc.
Please found complete BP at
https://github.com/koder-ua/os_api/blob/master/README.md

Thanks
-- 
Kostiantyn Danilov aka koder http://koder.ua
Principal software engineer, Mirantis

skype:koder.ua
http://koder-ua.blogspot.com/
http://mirantis.com
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Future-based api for openstack clients calls, that starts background tasks

2015-01-12 Thread Konstantin Danilov 
Boris,

Move from sync http to something like websocket requires a lot of work and
not directly connected
with API issue. When openstack api servers begin to support
websockets - it would be easy to change implementation of monitoring thread
without breaking compatibility.
At the moment periodical pooling from additional thread looks reasonable
for me
and it creates same amount of http requests as all current implementation.

BP is not about improving performance,
but about providing convenient and common API to handle background tasks.

 So we won't need to retrieve 100500 times information about object.
As I sad before - this API creates same amount of load as any code which we
use to check background task currently.
It even can decrease load due to requests aggregation in some cases (but
there a points to discuss).

 As well this pattern doesn't look great.
 I would prefer to see something like:
 vm = novaclient.servers.create(, sync=True)

This is completely different pattern. It is blocking call, which don't
allows you to start two(or more) background tasks
and from same thread and make some calculations while they running in
background.





On Mon, Jan 12, 2015 at 1:42 PM, Boris Pavlovic bpavlo...@mirantis.com
wrote:

 Konstantin,


 I believe it's better to work on server side, and use some modern approach
 like web sockets for async operations. So we won't need to retrieve 100500
 times information about object. And then use this feature in clients.

  create_future = novaclient.servers.create_async()
 .
 vm = create_future.result()


 As well this pattern doesn't look great.

 I would prefer to see something like:

   vm = novaclient.servers.create(, sync=True)


 Best regards,
 Boris Pavlovic





 On Mon, Jan 12, 2015 at 2:30 PM, Konstantin Danilov kdani...@mirantis.com
  wrote:

 Hi all.

 There a set of openstack api functions which starts background actions
 and return preliminary results - like 'novaclient.create'. Those functions
 requires periodically check results and handle timeouts/errors
 (and often cleanup + restart help to fix an error).

 Check/retry/cleanup code duplicated over a lot of core projects.
 As examples - heat, tempest, rally, etc and definitely in many
 third-party scripts.

 I propose to provide common higth-level API for such functions, which uses
 'futures' (http://en.wikipedia.org/wiki/Futures_and_promises) as a way to
 present background task.

 Idea is to add to each background-task-starter function a complimentary
 call,
 that returns 'future' object. E.g.

 create_future = novaclient.servers.create_async()
 .
 vm = create_future.result()

 This allows to unify(and optimize) monitoring cycles, retries, etc.
 Please found complete BP at
 https://github.com/koder-ua/os_api/blob/master/README.md

 Thanks
 --
 Kostiantyn Danilov aka koder http://koder.ua
 Principal software engineer, Mirantis

 skype:koder.ua
 http://koder-ua.blogspot.com/
 http://mirantis.com

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




-- 
Kostiantyn Danilov aka koder.ua
Principal software engineer, Mirantis

skype:koder.ua
http://koder-ua.blogspot.com/
http://mirantis.com
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev