Re: [openstack-dev] [horizon] REST and Django

2014-12-15 Thread Tihomir Trifonov
 for today, so can’t actually try that out
 at the moment.

 Thanks,
 Travis

 From: Thai Q Tran tqt...@us.ibm.commailto:tqt...@us.ibm.com
 Reply-To: OpenStack List openstack-dev@lists.openstack.orgmailto:
 openstack-dev@lists.openstack.org
 Date: Friday, December 12, 2014 at 11:05 AM
 To: OpenStack List openstack-dev@lists.openstack.orgmailto:
 openstack-dev@lists.openstack.org
 Subject: Re: [openstack-dev] [horizon] REST and Django


 In your previous example, you are posting to a certain URL (i.e.
 /keystone/{ver:=x.0}/{method:=update}).
 client: POST /keystone/{ver:=x.0}/{method:=update} = middleware: just
 forward to clients[ver].getattr(method)(**kwargs) = keystone: update

 Correct me if I'm wrong, but it looks like you have a unique URL for each
 /service/version/method.
 I fail to see how that is different than what we have today? Is there a
 view for each service? each version?

 Let's say for argument sake that you have a single view that takes care of
 all URL routing. All requests pass through this view and contain a JSON
 that contains instruction on which API to invoke and what parameters to
 pass.
 And lets also say that you wrote some code that uses reflection to map the
 JSON to an action. What you end up with is a client-centric application,
 where all of the logic resides client-side. If there are things we want to
 accomplish server-side, it will be extremely hard to pull off. Things like
 caching, websocket, aggregation, batch actions, translation, etc What
 you end up with is a client with no help from the server.

 Obviously the other extreme is what we have today, where we do everything
 server-side and only using client-side for binding events. I personally
 prefer a more balance approach where we can leverage both the server and
 client. There are things that client can do well, and there are things that
 server can do well. Going the RPC way restrict us to just client
 technologies and may hamper any additional future functionalities we want
 to bring server-side. In other words, using REST over RPC gives us the
 opportunity to use server-side technologies to help solve problems should
 the need for it arises.

 I would also argue that the REST approach is NOT what we have today. What
 we have today is a static webpage that is generated server-side, where API
 is hidden from the client. What we end up with using the REST approach is a
 dynamic webpage generated client-side, two very different things. We have
 essentially striped out the rendering logic from Django templating and
 replaced it with Angular.


 -Tihomir Trifonov t.trifo...@gmail.commailto:t.trifo...@gmail.com
 wrote: -
 To: OpenStack Development Mailing List (not for usage questions) 
 openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
 
 From: Tihomir Trifonov t.trifo...@gmail.commailto:t.trifo...@gmail.com
 Date: 12/12/2014 04:53AM
 Subject: Re: [openstack-dev] [horizon] REST and Django

 Here's an example: Admin user Joe has an Domain open and stares at it for
 15 minutes while he updates the description. Admin user Bob is asked to go
 ahead and enable it. He opens the record, edits it, and then saves it. Joe
 finished perfecting the description and saves it. Doing this action would
 mean that the Domain is enabled and the description gets updated. Last man
 in still wins if he updates the same fields, but if they update different
 fields then both of their changes will take affect without them stomping on
 each other. Whether that is good or bad may depend on the situation…


 That's a great example. I believe that all of the Openstack APIs support
 PATCH updates of arbitrary fields. This way - the frontend(AngularJS) can
 detect which fields are being modified, and to submit only these fields for
 update. If we however use a form with POST, although we will load the
 object before updating it, the middleware cannot find which fields are
 actually modified, and will update them all, which is more likely what PUT
 should do. Thus having full control in the frontend part, we can submit
 only changed fields. If however a service API doesn't support PATCH, it is
 actually a problem in the API and not in the client...



 The service API documentation almost always lags (although, helped by
 specs now) and the service team takes on the burden of exposing a
 programmatic way to access the API.  This is tested and easily consumable
 via the python clients, which removes some guesswork from using the service.

 True. But what if the service team modifies a method signature from let's
 say:

 def add_something(self, request,
 ​ field1, field2):


 to

 def add_something(self, request,
 ​ field1, field2, field3):


 and in the middleware we have the old signature:

 ​def add_something(self, request,
 ​ field1, field2):

 we still need to modify the middleware to add the new field. If however
 the middleware is transparent and just passes **kwargs, it will pass
 through whatever

Re: [openstack-dev] [horizon] REST and Django

2014-12-12 Thread Tihomir Trifonov
 need the lines:​
 
  project = api.keystone.tenant_get(request, id)
  kwargs = _tenant_kwargs_from_DATA(request.DATA, enabled=None)
 ​
 I agree that if you already have all the data it may be bad to have to do
 another call. I do think there is room for discussing the reasoning,
 though.
 As far as I can tell, they do this so that if you are updating an entity,
 you have to be very specific about the fields you are changing. I
 actually see this as potentially a protectionary measure against data
 loss and sometimes a very nice to have feature. It perhaps was intended
 to *help* guard against race conditions (no locking and no transactions
 with many users simultaneously accessing the data).
 
 Here's an example: Admin user Joe has a Domain open and stares at it for
 15 minutes while he updates just the description. Admin user Bob is asked
 to go ahead and enable it. He opens the record, edits it, and then saves
 it. Joe finished perfecting the description and saves it. They could in
 effect both edit the same domain independently. Last man in still wins if
 he updates the same fields, but if they update different fields then both
 of their changes will take affect without them stomping on each other. Or
 maybe it is intended to encourage client users to compare their current
 and previous to see if they should issue a warning if the data changed
 between getting and updating the data. Or maybe like you said, it is just
 overhead API calls.



 
 From: Tihomir Trifonov t.trifo...@gmail.commailto:t.trifo...@gmail.com
 
 Reply-To: OpenStack List
 openstack-dev@lists.openstack.orgmailto:
 openstack-...@lists.openstack.or
 g
 Date: Thursday, December 11, 2014 at 7:53 AM
 To: OpenStack List
 openstack-dev@lists.openstack.orgmailto:
 openstack-...@lists.openstack.or
 g
 Subject: Re: [openstack-dev] [horizon] REST and Django
 
 ​​
 Client just needs to know which URL to hit in order to invoke a certain
 API, and does not need to know the procedure name or parameters ordering.
 
 
 ​That's where the difference is. I think the client has to know the
 procedure name and parameters. Otherwise​ we have a translation factory
 pattern, that converts one naming convention to another. And you won't be
 able to call any service API if there is no code in the middleware to
 translate it to the service API procedure name and parameters. To avoid
 this - we can use a transparent proxy model - direct mapping of a client
 call to service API naming, which can be done if the client invokes the
 methods with the names in the service API, so that the middleware will
 just pass parameters, and will not translate. Instead of:
 
 
 updating user data:
 
 client: POST /user/=middleware: convert to
 /keystone/update/=   keystone: update
 
 we may use:
 
 client: POST /keystone/{ver:=x.0}/{method:=update}=
 middleware: just forward to clients[ver].getattr(method)(**kwargs) 
 =   keystone: update
 
 
 ​The idea here is that if we have keystone 4.0 client, ​we will have to
 just add it to the clients [] list and nothing more is required at the
 middleware level. Just create the frontend code to use the new Keystone
 4.0 methods. Otherwise we will have to add all new/different signatures
 of 4.0 against 2.0/3.0 in the middleware in order to use Keystone 4.0.
 
 There is also a great example of using a pluggable/new feature in
 Horizon. Do you remember the volume types support patch? The patch was
 pending in Gerrit for few months - first waiting the cinder support for
 volume types to go upstream, then waiting few more weeks for review. I am
 not sure, but as far as I remember, the Horizon patch even missed a
 release milestone and was introduced in the next release.
 
 If we have a transparent middleware - this will be no more an issue. As
 long as someone has written the frontend modules(which should be easy to
 add and customize), and they install the required version of the service
 API - they will not need updated Horizon to start using the feature.
 Maybe I am not the right person to give examples here, but how many of
 you had some kind of Horizon customization being locally merged/patched
 in your local distros/setups, until the patch is being pushed upstream?
 
 I will say it again. Nova, Keystone, Cinder, Glance etc. already have
 stable public APIs. Why do we want to add the translation middleware and
 to introduce another level of REST API? This layer will often hide new
 features, added to the service APIs and will delay their appearance in
 Horizon. That's simply not needed. I believe it is possible to just wrap
 the authentication in the middleware REST, but not to translate anything
 as RPC methods/parameters.
 
 
 ​And one more example:
 
 ​@rest_utils.ajax()
 def put(self, request, id):
 Update a single project.
 
 The POST data should be an application/json object containing the
 parameters to update: name (string),  description (string),
 domain_id (string) and enabled

Re: [openstack-dev] [horizon] REST and Django

2014-12-12 Thread Thai Q Tran
ts in the patch were very helpful for me to understand your
concerns about the ease of customizing without requiring upstream
changes. It also reminded me that Ive also previously questioned the
python middleman.

However, here are a couple of bullet points for Devils Advocate
consideration.


 * Will we take on auto-discovery of API extensions in two spots
(python for legacy and JS for new)?
 * The Horizon team will have to keep an even closer eye on every
single project and be ready to react if there are changes to the API that
break things. Right now in Glance, for example, they are working on some
fixes to the v2 API (soon to become v2.3) that will allow them to
deprecate v1 somewhat transparently to users of the client library.
 * The service API documentation almost always lags (although, helped
by specs now) and the service team takes on the burden of exposing a
programmatic way to access the API. This is tested and easily consumable
via the python clients, which removes some guesswork from using the
service.
 * This is going to be an incremental approach with legacy support
requirements anyway. So, incorporating python side changes wont just go
away.
 * Which approach would be better if we introduce a server side
caching mechanism or a new source of data such as elastic search to
improve performance? Would the client side code have to be changed
dramatically to take advantage of those improvements or could it be done
transparently on the server side if we own the exposed API?

Im not sure I fully understood your example about Cinder. Was it the
cinder client that held up delivery of horizon support, the cinder API or
both? If the API isnt in, then it would hold up delivery of the feature
in any case. There still would be timing pressures to react and build a
new view that supports it. For customization, with Richards approach new
views could be supported by just dropping in a new REST API decorated
module with the APIs you want, including direct pass through support if
desired to new APIs. Downstream customizations / Upstream changes to
views seem a bit like a bit of a related, but different issue to me as
long as their is an easy way to drop in new API support.

Finally, regarding the client making two calls to do an update:

Do we really need the lines:

 project = api.keystone.tenant_get(request, id)
 kwargs = _tenant_kwargs_from_DATA(request.DATA, enabled=None)

I agree that if you already have all the data it may be bad to have to do
another call. I do think there is room for discussing the reasoning,
though.
As far as I can tell, they do this so that if you are updating an entity,
you have to be very specific about the fields you are changing. I
actually see this as potentially a protectionary measure against data
loss and sometimes a very nice to have feature. It perhaps was intended
to *help* guard against race conditions (no locking and no transactions
with many users simultaneously accessing the data).

Here's an example: Admin user Joe has a Domain open and stares at it for
15 minutes while he updates just the description. Admin user Bob is asked
to go ahead and enable it. He opens the record, edits it, and then saves
it. Joe finished perfecting the description and saves it. They could in
effect both edit the same domain independently. Last man in still wins if
he updates the same fields, but if they update different fields then both
of their changes will take affect without them stomping on each other. Or
maybe it is intended to encourage client users to compare their current
and previous to see if they should issue a warning if the data changed
between getting and updating the data. Or maybe like you said, it is just
overhead API calls.
From: Tihomir Trifonov t.trifo...@gmail.commailto:t.trifo...@gmail.com
Reply-To: OpenStack List
openstack-dev@lists.openstack.orgmailto:openstack-...@lists.openstack.or
g
Date: Thursday, December 11, 2014 at 7:53 AM
To: OpenStack List
openstack-dev@lists.openstack.orgmailto:openstack-...@lists.openstack.or
g
Subject: Re: [openstack-dev] [horizon] REST and Django


Client just needs to know which URL to hit in order to invoke a certain
API, and does not need to know the procedure name or parameters ordering.


That's where the difference is. I think the client has to know the
procedure name and parameters. Otherwise we have a translation factory
pattern, that converts one naming convention to another. And you won't be
able to call any service API if there is no code in the middleware to
translate it to the service API procedure name and parameters. To avoid
this - we can use a transparent proxy model - direct mapping of a client
call to service API naming, which can be done if the client invokes the
methods with the names in the service API, so that the middleware will
just pass parameters, and will not translate. Instead of:


updating user data:

  client: POST /user/  =  middleware: convert to
/keystone/update/  = keystone: update

we may use:

  client: P

Re: [openstack-dev] [horizon] REST and Django

2014-12-12 Thread Tripp, Travis S
Tihomir,

Today I added one glance call based on Richard’s decorator pattern[1] and 
started to play with incorporating some of your ideas. Please note, I only had 
limited time today.  That is passing the kwargs through to the glance client. 
This was an interesting first choice, because it immediately highlighted a 
concrete example of the horizon glance wrapper post-processing still being 
useful (rather than be a direct pass-through with no wrapper). See below. If 
you have some some concrete code examples of your ideas it would be helpful.

[1] 
https://review.openstack.org/#/c/141273/2/openstack_dashboard/api/rest/glance.py

With the patch, basically, you can call the following and all of the GET 
parameters get passed directly through to the horizon glance client and you get 
results back as expected.

http://localhost:8002/api/glance/images/?sort_dir=descsort_key=created_atpaginate=Truemarker=bb2cfb1c-2234-4f54-aec5-b4916fe2d747

If you pass in an incorrect sort_key, the glance client returns the following 
error message which propagates back to the REST caller as an error with the 
message:

sort_key must be one of the following: name, status, container_format, 
disk_format, size, id, created_at, updated_at.

This is done by passing **request.GET.dict() through.

Please note, that if you try this (with POSTMAN, for example), you need to set 
the header of X-Requested-With = XMLHttpRequest

So, what issues did it immediately call out with directly invoking the client?

The python-glanceclient internally handles pagination by returning a generator. 
 Each iteration on the generator will handle making a request for the next page 
of data. If you were to just do something like return list(image_generator) to 
serialize it back out to the caller, it would actually end up making a call 
back to the server X times to fetch all pages before serializing back (thereby 
not really paginating). The horizon glance client wrapper today handles this by 
using islice intelligently along with honoring the API_RESULT_LIMIT setting in 
Horizon. So, this gives a direct example of where the wrapper does something 
that a direct passthrough to the client would not allow.

That said, I can see a few ways that we could use the same REST decorator code 
and provide direct access to the API.  We’d simply provide a class where the 
url_regex maps to the desired path and gives direct passthrough. Maybe that 
kind of passthrough could always be provided for ease of customization / 
extensibility and additional methods with wrappers provided when necessary.  I 
need to leave for today, so can’t actually try that out at the moment.

Thanks,
Travis

From: Thai Q Tran tqt...@us.ibm.commailto:tqt...@us.ibm.com
Reply-To: OpenStack List 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Friday, December 12, 2014 at 11:05 AM
To: OpenStack List 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [horizon] REST and Django


In your previous example, you are posting to a certain URL (i.e. 
/keystone/{ver:=x.0}/{method:=update}).
client: POST /keystone/{ver:=x.0}/{method:=update} = middleware: just 
forward to clients[ver].getattr(method)(**kwargs) = keystone: update

Correct me if I'm wrong, but it looks like you have a unique URL for each 
/service/version/method.
I fail to see how that is different than what we have today? Is there a view 
for each service? each version?

Let's say for argument sake that you have a single view that takes care of all 
URL routing. All requests pass through this view and contain a JSON that 
contains instruction on which API to invoke and what parameters to pass.
And lets also say that you wrote some code that uses reflection to map the JSON 
to an action. What you end up with is a client-centric application, where all 
of the logic resides client-side. If there are things we want to accomplish 
server-side, it will be extremely hard to pull off. Things like caching, 
websocket, aggregation, batch actions, translation, etc What you end up 
with is a client with no help from the server.

Obviously the other extreme is what we have today, where we do everything 
server-side and only using client-side for binding events. I personally prefer 
a more balance approach where we can leverage both the server and client. There 
are things that client can do well, and there are things that server can do 
well. Going the RPC way restrict us to just client technologies and may hamper 
any additional future functionalities we want to bring server-side. In other 
words, using REST over RPC gives us the opportunity to use server-side 
technologies to help solve problems should the need for it arises.

I would also argue that the REST approach is NOT what we have today. What we 
have today is a static webpage that is generated server-side, where API is 
hidden from the client. What we end up with using the REST approach is a 
dynamic

Re: [openstack-dev] [horizon] REST and Django

2014-12-11 Thread Tihomir Trifonov
 deletions.

 -Tihomir Trifonov t.trifo...@gmail.com wrote: -
 To: OpenStack Development Mailing List (not for usage questions) 
 openstack-dev@lists.openstack.org
 From: Tihomir Trifonov t.trifo...@gmail.com
 Date: 12/10/2014 03:04AM

 Subject: Re: [openstack-dev] [horizon] REST and Django

 Richard, thanks for the reply,


 I agree that the given example is not a real REST. But we already have the
 REST API - that's Keystone, Nova, Cinder, Glance, Neutron etc, APIs. So
 what we plan to do here? To add a new REST layer to communicate with other
 REST API? Do we really need Frontend-REST-REST architecture ? My opinion is
 that we don't need another REST layer as we currently are trying to go away
 from the Django layer, which is the same - another processing layer.
 Although we call it REST proxy or whatever - it doesn't need to be a real
 REST, but just an aggregation proxy that combines and forwards some
 requests with adding minimal processing overhead. What makes sense for me
 is to keep the authentication in this layer as it is now - push a cookie to
 the frontend, but the REST layer will extract the auth tokens from the
 session storage and prepare the auth context for the REST API request to OS
 services. This way we will not expose the tokens to the JS frontend, and
 will have strict control over the authentication. The frontend will just
 send data requests, they will be wrapped with auth context and forwarded.

 Regarding the existing issues with versions in the API - for me the
 existing approach is wrong. All these fixes were made as workarounds. What
 should have been done is to create abstractions for each version and to use
 a separate class for each version. This was partially done for the
 keystoneclient in api/keystone.py, but not for the forms/views, where we
 still have if-else for versions. What I suggest here is to have
 different(concrete) views/forms for each version, and to use them according
 the context. If the Keystone backend is v2.0 - then in the Frontend use
 keystone2() object, otherwise use keystone3() object. This of course needs
 some more coding, but is much cleaner in terms of customization and
 testing. For me the current hacks with 'if keystone.version == 3.0' are
 wrong at many levels. And this can be solved now. *The problem till now
 was that we had one frontend that had to be backed by different versions of
 backend components*. *Now we can have different frontends that map to
 specific backend*. That's how I understand the power of Angular with it's
 views and directives. That's where I see the real benefit of using
 full-featured frontend. Also imagine how easy will be then to deprecate a
 component version, compared to what we need to do now for the same.

 Otherwise we just rewrite the current Django middleware with another
 DjangoRest middleware and don't change anything, we don't fix the problems.
 We just move them to another place.

 I still think that in Paris we talked about a new generation of the
 Dashboard, a different approach on building the frontend for OpenStack.
 What I've heard there from users/operators of Horizon was that it was
 extremely hard to add customizations and new features to the Dashboard, as
 all these needed to go through upstream changes and to wait until next
 release cycle to get them. Do we still want to address these concerns and
 how? Please, correct me if I got things wrong.


 On Wed, Dec 10, 2014 at 11:56 AM, Richard Jones r1chardj0...@gmail.com
 wrote:

 Sorry I didn't respond to this earlier today, I had intended to.

 What you're describing isn't REST, and the principles of REST are what
 have been guiding the design of the new API so far. I see a lot of value in
 using REST approaches, mostly around clarity of the interface.

 While the idea of a very thin proxy seemed like a great idea at one
 point, my conversations at the summit convinced me that there was value in
 both using the client interfaces present in the openstack_dashboard/api
 code base (since they abstract away many issues in the apis including
 across versions) and also value in us being able to clean up (for example,
 using project_id rather than project in the user API we've already
 implemented) and extend those interfaces (to allow batched operations).

 We want to be careful about what we expose in Horizon to the JS clients
 through this API. That necessitates some amount of code in Horizon. About
 half of the current API for keysone represents that control (the other half
 is docstrings :)


  Richard


 On Tue Dec 09 2014 at 9:37:47 PM Tihomir Trifonov t.trifo...@gmail.com
 wrote:

 Sorry for the late reply, just few thoughts on the matter.

 IMO the REST middleware should be as thin as possible. And I mean thin
 in terms of processing - it should not do pre/post processing of the
 requests, but just unpack/pack. So here is an example:

 instead of making AJAX calls that contain instructions:

 ​​
 POST --json --data {action: delete, data

Re: [openstack-dev] [horizon] REST and Django

2014-12-11 Thread Tripp, Travis S
Tihomir,

Your comments in the patch were the actually the clearest to me about ease of 
customizing without requiring upstream changes and really made me think more 
about your points.

Here are a couple of bullet points for consideration.


  *   Will we take on auto-discovery of API extensions in two spots (python for 
legacy and JS for new)?
  *   As teams move towards deprecating / modifying APIs who will be 
responsible for ensuring the JS libraries stay up to date and keep tabs on 
every single project?  Right now in Glance, for example, they are working on 
some fixes to v2 API (soon to become v2.3) that will allow them to deprecate v1 
so that Nova can migrate from v1.  Part of this includes making simultaneous 
improvements to the client library so that the switch can happen more 
transparently to client users. This testing and maintenance of the service team 
already takes on.
  *   The service API documentation almost always lags (helped by specs now) 
and the service team takes on the burden of exposing a programmatic way to 
access which is tested and easily consumable via the python clients which 
removes some guesswork from using the service.
  *   This is going to be an incremental approach with legacy support 
requirements anyway, I think.  So, incorporating python side changes won’t just 
go away.
  *   A tangent that needs to be considered IMO since I’m working on some 
elastic search things right now. Which of these would be better if we introduce 
a server side caching mechanism or a new source of data such as elastic search 
to improve performance?
 *   Would the client just be able to handle changing whether or not it 
used cache with a header and in either case the server side appropriately uses 
the cache? (e.g. Cache-Control: no-cache)

I’m not sure I fully understood your example about Cinder.  Was it the 
cinderclient that held up delivery of that horizon support or there cinder API 
or both?  If the API isn’t in, then it would hold up delivery of the feature in 
any case. If it is just about delivering new functionality, all that would be 
required in Richard’s approach is to drop in a new file of decorated classes / 
functions from his utility with the API’s you want? None of the API calls have 
anything to do with how your view actually replaces the upstream view.  These 
are all just about accessing the data.

Finally, I mentioned the following in the patch related to your example below 
about the client making two calls to do an update, but wanted to mention here 
to see if it is an approach that was purposeful (I don’t know the history):

​Do we really need the lines:​

 project = api.keystone.tenant_get(request, id)
 kwargs = _tenant_kwargs_from_DATA(request.DATA, enabled=None)
​
I agree that if you already have all the data it is really bad to have to do 
another call. I do think there is room for discussing the reasoning, though.
As far as I can tell, they do this so that if you are updating an entity, you 
have to be very specific about the fields you are changing. I actually see this 
as potentially a protectionary measure against data loss and a sometimes very 
nice to have feature. It perhaps was intended to *help* guard against race 
conditions *sometimes*.

Here's an example: Admin user Joe has an Domain open and stares at it for 15 
minutes while he updates the description. Admin user Bob is asked to go ahead 
and enable it. He opens the record, edits it, and then saves it. Joe finished 
perfecting the description and saves it. Doing this action would mean that the 
Domain is enabled and the description gets updated. Last man in still wins if 
he updates the same fields, but if they update different fields then both of 
their changes will take affect without them stomping on each other. Whether 
that is good or bad may depend on the situation…

From: Tihomir Trifonov t.trifo...@gmail.commailto:t.trifo...@gmail.com
Reply-To: OpenStack List 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Thursday, December 11, 2014 at 7:53 AM
To: OpenStack List 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [horizon] REST and Django

​​
Client just needs to know which URL to hit in order to invoke a certain API, 
and does not need to know the procedure name or parameters ordering.


​That's where the difference is. I think the client has to know the procedure 
name and parameters. Otherwise​ we have a translation factory pattern, that 
converts one naming convention to another. And you won't be able to call any 
service API if there is no code in the middleware to translate it to the 
service API procedure name and parameters. To avoid this - we can use a 
transparent proxy model - direct mapping of a client call to service API 
naming, which can be done if the client invokes the methods with the names in 
the service API, so that the middleware will just pass parameters

Re: [openstack-dev] [horizon] REST and Django

2014-12-11 Thread Richard Jones
On Fri Dec 12 2014 at 1:06:08 PM Tripp, Travis S travis.tr...@hp.com
wrote:

 ​Do we really need the lines:​

  project = api.keystone.tenant_get(request, id)
  kwargs = _tenant_kwargs_from_DATA(request.DATA, enabled=None)
 ​
 I agree that if you already have all the data it is really bad to have to
 do another call. I do think there is room for discussing the reasoning,
 though.
 As far as I can tell, they do this so that if you are updating an entity,
 you have to be very specific about the fields you are changing. I actually
 see this as potentially a protectionary measure against data loss and a
 sometimes very nice to have feature. It perhaps was intended to *help*
 guard against race conditions *sometimes*.


Yep, it looks like I broke this API by implementing it the way I did, and
I'll alter the API so that you pass both the current object (according to
the client) and the parameters to alter.

Thanks everyone for the great reviewing!


 Richard
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [horizon] REST and Django

2014-12-10 Thread Richard Jones
 in the above cases the client is still going to be hanging,
 waiting for those server-backend calls, with no feedback until it's all
 done. I would hope that the client-server call overhead is minimal, but I
 guess that's probably wishful thinking when in the land of random Internet
 users hitting some provider's Horizon :)

  So yeah, having mulled it over myself I agree that it's useful to have
 batch operations implemented in the POST handler, the most common operation
 being DELETE.

  Maybe one day we could transition to a batch call with user feedback
 using a websocket connection.


   Richard

  Richard Jones ---11/27/2014 05:38:53 PM---On Fri Nov 28 2014 at
 5:58:00 AM Tripp, Travis S travis.tr...@hp.com wrote:

 From: Richard Jones r1chardj0...@gmail.com
 To: Tripp, Travis S travis.tr...@hp.com, OpenStack List 
 openstack-dev@lists.openstack.org
 Date: 11/27/2014 05:38 PM
 Subject: Re: [openstack-dev] [horizon] REST and Django

 --




 On Fri Nov 28 2014 at 5:58:00 AM Tripp, Travis S *travis.tr...@hp.com*
 travis.tr...@hp.com wrote:

Hi Richard,

You are right, we should put this out on the main ML, so copying
thread out to there.  ML: FYI that this started after some impromptu IRC
discussions about a specific patch led into an impromptu google hangout
discussion with all the people on the thread below.


 Thanks Travis!



As I mentioned in the review[1], Thai and I were mainly discussing
the possible performance implications of network hops from client to
horizon server and whether or not any aggregation should occur server 
 side.
  In other words, some views  require several APIs to be queried before 
 any
data can displayed and it would eliminate some extra network requests 
 from
client to server if some of the data was first collected on the server 
 side
across service APIs.  For example, the launch instance wizard will need 
 to
collect data from quite a few APIs before even the first step is 
 displayed
(I’ve listed those out in the blueprint [2]).

The flip side to that (as you also pointed out) is that if we keep
the API’s fine grained then the wizard will be able to optimize in one
place the calls for data as it is needed. For example, the first step may
only need half of the API calls. It also could lead to perceived
performance increases just due to the wizard making a call for different
data independently and displaying it as soon as it can.


 Indeed, looking at the current launch wizard code it seems like you
 wouldn't need to load all that data for the wizard to be displayed, since
 only some subset of it would be necessary to display any given panel of the
 wizard.



I tend to lean towards your POV and starting with discrete API calls
and letting the client optimize calls.  If there are performance problems
or other reasons then doing data aggregation on the server side could be
considered at that point.


 I'm glad to hear it. I'm a fan of optimising when necessary, and not
 beforehand :)



Of course if anybody is able to do some performance testing between
the two approaches then that could affect the direction taken.


 I would certainly like to see us take some measurements when performance
 issues pop up. Optimising without solid metrics is bad idea :)


 Richard



[1]

 *https://review.openstack.org/#/c/136676/8/openstack_dashboard/api/rest/urls.py*

 https://review.openstack.org/#/c/136676/8/openstack_dashboard/api/rest/urls.py
[2]
*https://blueprints.launchpad.net/horizon/+spec/launch-instance-redesign*
https://blueprints.launchpad.net/horizon/+spec/launch-instance-redesign

-Travis

*From: *Richard Jones *r1chardj0...@gmail.com*
r1chardj0...@gmail.com
 * Date: *Wednesday, November 26, 2014 at 11:55 PM
 * To: *Travis Tripp *travis.tr...@hp.com* travis.tr...@hp.com, Thai
Q Tran/Silicon Valley/IBM *tqt...@us.ibm.com* tqt...@us.ibm.com,
David Lyle *dkly...@gmail.com* dkly...@gmail.com, Maxime Vidori 
*maxime.vid...@enovance.com* maxime.vid...@enovance.com,
Wroblewski, Szymon *szymon.wroblew...@intel.com*
szymon.wroblew...@intel.com, Wood, Matthew David (HP Cloud -
Horizon) *matt.w...@hp.com* matt.w...@hp.com, Chen, Shaoquan 
*sean.ch...@hp.com* sean.ch...@hp.com, Farina, Matt (HP Cloud) 
*matthew.far...@hp.com* matthew.far...@hp.com, Cindy Lu/Silicon
Valley/IBM *c...@us.ibm.com* c...@us.ibm.com, Justin
Pomeroy/Rochester/IBM *jpom...@us.ibm.com* jpom...@us.ibm.com,
Neill Cox *neill@ingenious.com.au* neill@ingenious.com.au
 * Subject: *Re: REST and Django

I'm not sure whether this is the appropriate place to discuss this,
or whether I should be posting to the list under [Horizon] but I think we
need to have a clear idea of what goes in the REST API and what goes in 
 the
client (angular) code.

In my mind, the thinner the REST API

Re: [openstack-dev] [horizon] REST and Django

2014-12-10 Thread Tihomir Trifonov
:
 ​
 {name: item
 ​2
 }
 ​,
 {​
 
 ​
 action
 ​ : delete​
 ,
 ​
 payload:
 ​
 {name: item
 ​3
 }
 ​ ] ​
 ​
 ​
 }


 ​The idea is that the middleware should not know the actual data. It
 should ideally just unpack the data:

 ​​responses = []


 for cmd in
 ​ ​
 ​
 ​
 request.POST['batch']:​


 ​
 ​​responses
 ​.append(​
 ​
 getattr(controller, cmd['action']
 ​)(**
 cmd['​payload']
 ​)​)


 ​return responses​



 ​and the frontend(JS) will just send batches of simple commands, and will
 receive a list of responses for each command in the batch. The error
 handling will be done in the frontend​(JS) as well.

 ​

 For the more complex example of 'put()' where we have dependent objects:

 project = api.keystone.tenant_get(request, id)
 kwargs = self._tenant_kwargs_from_DATA(request.DATA, enabled=None)
 api.keystone.tenant_update(request, project, **kwargs)



 In practice the project data should be already present in the
 frontend(assuming that we already loaded it to render the project
 form/view), so

 ​
 ​
 POST --json --data {
 ​batch
 :
 ​[
 {​
 
 ​
 action
 ​ : tenant_update​
 ,
 ​payload: ​
 {project: js_project_object.id, name: some name, prop1: some
 prop, prop2: other prop, etc.}
 ​
 ​ ] ​
 ​
 ​
 }​

 So in general we don't need to recreate the full state on each REST call,
 if we make the Frontent full-featured application. This way - the frontend
 will construct the object, will hold the cached value, and will just send
 the needed requests as single ones or in batches, will receive the response
 from the API backend, and will render the results. The whole processing
 logic will be held in the Frontend(JS), while the middleware will just act
 as proxy(un/packer). This way we will maintain just the logic in the
 frontend, and will not need to duplicate some logic in the middleware.




 On Tue, Dec 2, 2014 at 4:45 PM, Adam Young ayo...@redhat.com wrote:

  On 12/02/2014 12:39 AM, Richard Jones wrote:

 On Mon Dec 01 2014 at 4:18:42 PM Thai Q Tran tqt...@us.ibm.com wrote:

  I agree that keeping the API layer thin would be ideal. I should add
 that having discrete API calls would allow dynamic population of table.
 However, I will make a case where it *might* be necessary to add
 additional APIs. Consider that you want to delete 3 items in a given table.

 If you do this on the client side, you would need to perform: n * (1
 API request + 1 AJAX request)
 If you have some logic on the server side that batch delete actions: n
 * (1 API request) + 1 AJAX request

 Consider the following:
 n = 1, client = 2 trips, server = 2 trips
 n = 3, client = 6 trips, server = 4 trips
 n = 10, client = 20 trips, server = 11 trips
 n = 100, client = 200 trips, server 101 trips

 As you can see, this does not scale very well something to
 consider...

  This is not something Horizon can fix.  Horizon can make matters
 worse, but cannot make things better.

 If you want to delete 3 users,   Horizon still needs to make 3 distinct
 calls to Keystone.

 To fix this, we need either batch calls or a standard way to do
 multiples of the same operation.

 The unified API effort it the right place to drive this.







  Yep, though in the above cases the client is still going to be
 hanging, waiting for those server-backend calls, with no feedback until
 it's all done. I would hope that the client-server call overhead is
 minimal, but I guess that's probably wishful thinking when in the land of
 random Internet users hitting some provider's Horizon :)

  So yeah, having mulled it over myself I agree that it's useful to have
 batch operations implemented in the POST handler, the most common operation
 being DELETE.

  Maybe one day we could transition to a batch call with user feedback
 using a websocket connection.


   Richard

  Richard Jones ---11/27/2014 05:38:53 PM---On Fri Nov 28 2014 at
 5:58:00 AM Tripp, Travis S travis.tr...@hp.com wrote:

 From: Richard Jones r1chardj0...@gmail.com
 To: Tripp, Travis S travis.tr...@hp.com, OpenStack List 
 openstack-dev@lists.openstack.org
 Date: 11/27/2014 05:38 PM
 Subject: Re: [openstack-dev] [horizon] REST and Django

 --




 On Fri Nov 28 2014 at 5:58:00 AM Tripp, Travis S *travis.tr...@hp.com*
 travis.tr...@hp.com wrote:

Hi Richard,

You are right, we should put this out on the main ML, so copying
thread out to there.  ML: FYI that this started after some impromptu IRC
discussions about a specific patch led into an impromptu google hangout
discussion with all the people on the thread below.


 Thanks Travis!



As I mentioned in the review[1], Thai and I were mainly discussing
the possible performance implications of network hops from client to
horizon server and whether or not any aggregation should occur server 
 side.
  In other words, some views  require several APIs to be queried before 
 any
data can displayed and it would eliminate some extra network requests 
 from
client to server if some

Re: [openstack-dev] [horizon] REST and Django

2014-12-10 Thread Thai Q Tran
 
Yep, though in the above cases the client is still going to
  be hanging, waiting for those server-backend calls, with no
  feedback until it's all done. I would hope that the
  client-server call overhead is minimal, but I guess that's
  probably wishful thinking when in the land of random Internet
  users hitting some provider's Horizon :)


So yeah, having mulled it over myself I agree that it's
  useful to have batch operations implemented in the POST
  handler, the most common operation being DELETE.


Maybe one day we could transition to a batch call with user
  feedback using a websocket connection.
  
  




  Richard
  
  

  
Richard Jones ---11/27/2014 05:38:53
    PM---On Fri Nov 28 2014 at 5:58:00 AM Tripp, Travis S
    travis.tr...@hp.com
wrote:
  
  From: Richard Jones r1chardj0...@gmail.com
  To: "Tripp, Travis S" travis.tr...@hp.com,
OpenStack List openstack-dev@lists.openstack.org
  Date: 11/27/2014 05:38 PM
  Subject:
  Re:
[openstack-dev] [horizon] REST and Django


  

  
  
  On Fri Nov 28 2014 at 5:58:00
AM Tripp, Travis S travis.tr...@hp.com wrote:


  Hi Richard,
  
  You are right, we should put
this out on the main ML, so copying thread out to
there. ML: FYI that this started after some impromptu
IRC discussions about a specific patch led into an
impromptu google hangout discussion with all the people
on the thread below.


Thanks Travis!



  As I mentioned in the
review[1], Thai and I were mainly discussing the
possible performance implications of network hops from
client to horizon server and whether or not any
aggregation should occur server side.  In other words,
some views require several APIs to be queried before
any data can displayed and it would eliminate some extra
network requests from client to server if some of the
data was first collected on the server side across
service APIs. For example, the launch instance wizard
will need to collect data from quite a few APIs before
even the first step is displayed (Ive listed those out
in the blueprint [2]).
  
  The flip side to that (as
you also pointed out) is that if we keep the APIs fine
grained then the wizard will be able to optimize in one
place the calls for data as it is needed. For example,
the first step may only need half of the API calls. It
also could lead to perceived performance increases just
due to the wizard making a call for different data
independently and displaying it as soon as it can.


Indeed, looking at the current
  launch wizard code it seems like you wouldn't need to load
  all that data for the wizard to be displayed, since only
  some subset of it would be necessary to display any given
  panel of the wizard. 



  I tend to lean towards your
POV and starting with discrete API calls and letting the
client optimize calls. If there are performance
problems or other reasons then doing data aggregation on
the server side could be considered at that point.


I'm glad to hear it. I'm a fan
  of optimising when necessary, and not beforehand :)



  Of course if anybody is able
to do some performance testing between the two
approaches then that could affect the direction taken.


I would certainly like to see us
  take some measurements when performance issues pop up.
  Optimising without solid metrics is bad idea :)


  Richard

  
  

  
  [1]https://review.openstack.org/#/c/136676/8/openstack_dashboard/api/rest/urls.py
  [2]https://blueprints.launchpad.net/horizon/+spec/launch-instance-redesign
  
  -Travis
   

Re: [openstack-dev] [horizon] REST and Django

2014-12-09 Thread Tihomir Trifonov
Sorry for the late reply, just few thoughts on the matter.

IMO the REST middleware should be as thin as possible. And I mean thin in
terms of processing - it should not do pre/post processing of the requests,
but just unpack/pack. So here is an example:

instead of making AJAX calls that contain instructions:

​​
 POST --json --data {action: delete, data: [ {name:
 item1}, {name: item2}, {name: item3 ]}


I think a better approach is just to pack/unpack batch commands, and leave
execution to the frontend/backend and not middleware:


​​
 POST --json --data {
 ​batch
 :
 ​[
 {​
 
 ​
 action
 ​ : delete​
 ,
 ​payload: ​
 {name: item1}
 ​,
 {​
 
 ​
 action
 ​ : delete​
 ,
 ​
 payload:
 ​
 {name: item
 ​2
 }
 ​,
 {​
 
 ​
 action
 ​ : delete​
 ,
 ​
 payload:
 ​
 {name: item
 ​3
 }
 ​ ] ​
 ​
 ​
 }


​The idea is that the middleware should not know the actual data. It should
ideally just unpack the data:

​​responses = []


for cmd in
 ​ ​
 ​
 ​
 request.POST['batch']:​


 ​
 ​​responses
 ​.append(​
 ​
 getattr(controller, cmd['action']
 ​)(**
 cmd['​payload']
 ​)​)


 ​return responses​



​and the frontend(JS) will just send batches of simple commands, and will
receive a list of responses for each command in the batch. The error
handling will be done in the frontend​(JS) as well.

​

For the more complex example of 'put()' where we have dependent objects:

project = api.keystone.tenant_get(request, id)
 kwargs = self._tenant_kwargs_from_DATA(request.DATA, enabled=None)
 api.keystone.tenant_update(request, project, **kwargs)



In practice the project data should be already present in the
frontend(assuming that we already loaded it to render the project
form/view), so

​
​
POST --json --data {
​batch
:
​[
{​

​
action
​ : tenant_update​
,
​payload: ​
{project: js_project_object.id, name: some name, prop1: some
prop, prop2: other prop, etc.}
​
​ ] ​
​
​
}​

So in general we don't need to recreate the full state on each REST call,
if we make the Frontent full-featured application. This way - the frontend
will construct the object, will hold the cached value, and will just send
the needed requests as single ones or in batches, will receive the response
from the API backend, and will render the results. The whole processing
logic will be held in the Frontend(JS), while the middleware will just act
as proxy(un/packer). This way we will maintain just the logic in the
frontend, and will not need to duplicate some logic in the middleware.




On Tue, Dec 2, 2014 at 4:45 PM, Adam Young ayo...@redhat.com wrote:

  On 12/02/2014 12:39 AM, Richard Jones wrote:

 On Mon Dec 01 2014 at 4:18:42 PM Thai Q Tran tqt...@us.ibm.com wrote:

  I agree that keeping the API layer thin would be ideal. I should add
 that having discrete API calls would allow dynamic population of table.
 However, I will make a case where it *might* be necessary to add
 additional APIs. Consider that you want to delete 3 items in a given table.

 If you do this on the client side, you would need to perform: n * (1 API
 request + 1 AJAX request)
 If you have some logic on the server side that batch delete actions: n *
 (1 API request) + 1 AJAX request

 Consider the following:
 n = 1, client = 2 trips, server = 2 trips
 n = 3, client = 6 trips, server = 4 trips
 n = 10, client = 20 trips, server = 11 trips
 n = 100, client = 200 trips, server 101 trips

 As you can see, this does not scale very well something to consider...

  This is not something Horizon can fix.  Horizon can make matters worse,
 but cannot make things better.

 If you want to delete 3 users,   Horizon still needs to make 3 distinct
 calls to Keystone.

 To fix this, we need either batch calls or a standard way to do multiples
 of the same operation.

 The unified API effort it the right place to drive this.







  Yep, though in the above cases the client is still going to be hanging,
 waiting for those server-backend calls, with no feedback until it's all
 done. I would hope that the client-server call overhead is minimal, but I
 guess that's probably wishful thinking when in the land of random Internet
 users hitting some provider's Horizon :)

  So yeah, having mulled it over myself I agree that it's useful to have
 batch operations implemented in the POST handler, the most common operation
 being DELETE.

  Maybe one day we could transition to a batch call with user feedback
 using a websocket connection.


   Richard

  [image: Inactive hide details for Richard Jones ---11/27/2014 05:38:53
 PM---On Fri Nov 28 2014 at 5:58:00 AM Tripp, Travis S travis.tr]Richard
 Jones ---11/27/2014 05:38:53 PM---On Fri Nov 28 2014 at 5:58:00 AM Tripp,
 Travis S travis.tr...@hp.com wrote:

 From: Richard Jones r1chardj0...@gmail.com
 To: Tripp, Travis S travis.tr...@hp.com, OpenStack List 
 openstack-dev@lists.openstack.org
 Date: 11/27/2014 05:38 PM
 Subject: Re: [openstack-dev] [horizon] REST and Django
  --




 On Fri Nov 28 2014 at 5:58:00 AM Tripp

Re: [openstack-dev] [horizon] REST and Django

2014-12-02 Thread Adam Young

On 12/02/2014 12:39 AM, Richard Jones wrote:
On Mon Dec 01 2014 at 4:18:42 PM Thai Q Tran tqt...@us.ibm.com 
mailto:tqt...@us.ibm.com wrote:


I agree that keeping the API layer thin would be ideal. I should
add that having discrete API calls would allow dynamic population
of table. However, I will make a case where it */might/* be
necessary to add additional APIs. Consider that you want to delete
3 items in a given table.

If you do this on the client side, you would need to perform: n *
(1 API request + 1 AJAX request)
If you have some logic on the server side that batch delete
actions: n * (1 API request) + 1 AJAX request

Consider the following:
n = 1, client = 2 trips, server = 2 trips
n = 3, client = 6 trips, server = 4 trips
n = 10, client = 20 trips, server = 11 trips
n = 100, client = 200 trips, server 101 trips

As you can see, this does not scale very well something to
consider...

This is not something Horizon can fix.  Horizon can make matters worse, 
but cannot make things better.


If you want to delete 3 users,   Horizon still needs to make 3 distinct 
calls to Keystone.


To fix this, we need either batch calls or a standard way to do 
multiples of the same operation.


The unified API effort it the right place to drive this.






Yep, though in the above cases the client is still going to be 
hanging, waiting for those server-backend calls, with no feedback 
until it's all done. I would hope that the client-server call overhead 
is minimal, but I guess that's probably wishful thinking when in the 
land of random Internet users hitting some provider's Horizon :)


So yeah, having mulled it over myself I agree that it's useful to have 
batch operations implemented in the POST handler, the most common 
operation being DELETE.


Maybe one day we could transition to a batch call with user feedback 
using a websocket connection.



 Richard

Inactive hide details for Richard Jones ---11/27/2014 05:38:53
PM---On Fri Nov 28 2014 at 5:58:00 AM Tripp, Travis S
travis.trRichard Jones ---11/27/2014 05:38:53 PM---On Fri Nov 28
2014 at 5:58:00 AM Tripp, Travis S travis.tr...@hp.com
mailto:travis.tr...@hp.com wrote:

From: Richard Jones r1chardj0...@gmail.com
mailto:r1chardj0...@gmail.com
To: Tripp, Travis S travis.tr...@hp.com
mailto:travis.tr...@hp.com, OpenStack List
openstack-dev@lists.openstack.org
mailto:openstack-dev@lists.openstack.org
Date: 11/27/2014 05:38 PM
Subject: Re: [openstack-dev] [horizon] REST and Django






On Fri Nov 28 2014 at 5:58:00 AM Tripp, Travis S
_travis.tripp@hp.com_ mailto:travis.tr...@hp.com wrote:

Hi Richard,

You are right, we should put this out on the main ML, so
copying thread out to there.  ML: FYI that this started after
some impromptu IRC discussions about a specific patch led into
an impromptu google hangout discussion with all the people on
the thread below. 



Thanks Travis!

As I mentioned in the review[1], Thai and I were mainly
discussing the possible performance implications of network
hops from client to horizon server and whether or not any
aggregation should occur server side.   In other words, some
views  require several APIs to be queried before any data can
displayed and it would eliminate some extra network requests
from client to server if some of the data was first collected
on the server side across service APIs.  For example, the
launch instance wizard will need to collect data from quite a
few APIs before even the first step is displayed (I’ve listed
those out in the blueprint [2]).

The flip side to that (as you also pointed out) is that if we
keep the API’s fine grained then the wizard will be able to
optimize in one place the calls for data as it is needed. For
example, the first step may only need half of the API calls.
It also could lead to perceived performance increases just due
to the wizard making a call for different data independently
and displaying it as soon as it can. 



Indeed, looking at the current launch wizard code it seems like
you wouldn't need to load all that data for the wizard to be
displayed, since only some subset of it would be necessary to
display any given panel of the wizard.

I tend to lean towards your POV and starting with discrete API
calls and letting the client optimize calls.  If there are
performance problems or other reasons then doing data
aggregation on the server side could be considered at that point. 



I'm glad to hear it. I'm a fan of optimising when necessary, and
not beforehand :)

Of course if anybody is able to do some

Re: [openstack-dev] [horizon] REST and Django

2014-12-01 Thread Richard Jones
On Mon Dec 01 2014 at 4:18:42 PM Thai Q Tran tqt...@us.ibm.com wrote:

 I agree that keeping the API layer thin would be ideal. I should add that
 having discrete API calls would allow dynamic population of table. However,
 I will make a case where it *might* be necessary to add additional APIs.
 Consider that you want to delete 3 items in a given table.

 If you do this on the client side, you would need to perform: n * (1 API
 request + 1 AJAX request)
 If you have some logic on the server side that batch delete actions: n *
 (1 API request) + 1 AJAX request

 Consider the following:
 n = 1, client = 2 trips, server = 2 trips
 n = 3, client = 6 trips, server = 4 trips
 n = 10, client = 20 trips, server = 11 trips
 n = 100, client = 200 trips, server 101 trips

 As you can see, this does not scale very well something to consider...

Yep, though in the above cases the client is still going to be hanging,
waiting for those server-backend calls, with no feedback until it's all
done. I would hope that the client-server call overhead is minimal, but I
guess that's probably wishful thinking when in the land of random Internet
users hitting some provider's Horizon :)

So yeah, having mulled it over myself I agree that it's useful to have
batch operations implemented in the POST handler, the most common operation
being DELETE.

Maybe one day we could transition to a batch call with user feedback using
a websocket connection.


 Richard

 [image: Inactive hide details for Richard Jones ---11/27/2014 05:38:53
 PM---On Fri Nov 28 2014 at 5:58:00 AM Tripp, Travis S travis.tr]Richard
 Jones ---11/27/2014 05:38:53 PM---On Fri Nov 28 2014 at 5:58:00 AM Tripp,
 Travis S travis.tr...@hp.com wrote:

 From: Richard Jones r1chardj0...@gmail.com
 To: Tripp, Travis S travis.tr...@hp.com, OpenStack List 
 openstack-dev@lists.openstack.org
 Date: 11/27/2014 05:38 PM
 Subject: Re: [openstack-dev] [horizon] REST and Django
 --




 On Fri Nov 28 2014 at 5:58:00 AM Tripp, Travis S *travis.tr...@hp.com*
 travis.tr...@hp.com wrote:

Hi Richard,

You are right, we should put this out on the main ML, so copying
thread out to there.  ML: FYI that this started after some impromptu IRC
discussions about a specific patch led into an impromptu google hangout
discussion with all the people on the thread below.


 Thanks Travis!



As I mentioned in the review[1], Thai and I were mainly discussing the
possible performance implications of network hops from client to horizon
server and whether or not any aggregation should occur server side.   In
other words, some views  require several APIs to be queried before any data
can displayed and it would eliminate some extra network requests from
client to server if some of the data was first collected on the server side
across service APIs.  For example, the launch instance wizard will need to
collect data from quite a few APIs before even the first step is displayed
(I’ve listed those out in the blueprint [2]).

The flip side to that (as you also pointed out) is that if we keep the
API’s fine grained then the wizard will be able to optimize in one place
the calls for data as it is needed. For example, the first step may only
need half of the API calls. It also could lead to perceived performance
increases just due to the wizard making a call for different data
independently and displaying it as soon as it can.


 Indeed, looking at the current launch wizard code it seems like you
 wouldn't need to load all that data for the wizard to be displayed, since
 only some subset of it would be necessary to display any given panel of the
 wizard.



I tend to lean towards your POV and starting with discrete API calls
and letting the client optimize calls.  If there are performance problems
or other reasons then doing data aggregation on the server side could be
considered at that point.


 I'm glad to hear it. I'm a fan of optimising when necessary, and not
 beforehand :)



Of course if anybody is able to do some performance testing between
the two approaches then that could affect the direction taken.


 I would certainly like to see us take some measurements when performance
 issues pop up. Optimising without solid metrics is bad idea :)


 Richard




[1]

 *https://review.openstack.org/#/c/136676/8/openstack_dashboard/api/rest/urls.py*

 https://review.openstack.org/#/c/136676/8/openstack_dashboard/api/rest/urls.py
[2]
*https://blueprints.launchpad.net/horizon/+spec/launch-instance-redesign*
https://blueprints.launchpad.net/horizon/+spec/launch-instance-redesign

-Travis

*From: *Richard Jones *r1chardj0...@gmail.com*
r1chardj0...@gmail.com
 * Date: *Wednesday, November 26, 2014 at 11:55 PM
 * To: *Travis Tripp *travis.tr...@hp.com* travis.tr...@hp.com, Thai Q
Tran/Silicon Valley/IBM *tqt...@us.ibm.com* tqt...@us.ibm.com

Re: [openstack-dev] [horizon] REST and Django

2014-11-30 Thread Thai Q Tran
I agree that keeping the API layer thin would be ideal. I should add that
having discrete API calls would allow dynamic population of table. However,
I will make a case where it might be necessary to add additional APIs.
Consider that you want to delete 3 items in a given table.

If you do this on the client side, you would need to perform: n * (1 API
request + 1 AJAX request)
If you have some logic on the server side that batch delete actions: n * (1
API request) + 1 AJAX request

Consider the following:
n = 1, client = 2 trips, server = 2 trips
n = 3, client = 6 trips, server = 4 trips
n = 10, client = 20 trips, server = 11 trips
n = 100, client = 200 trips, server 101 trips

As you can see, this does not scale very well something to consider...




From:   Richard Jones r1chardj0...@gmail.com
To: Tripp, Travis S travis.tr...@hp.com, OpenStack List
openstack-dev@lists.openstack.org
Date:   11/27/2014 05:38 PM
Subject:Re: [openstack-dev] [horizon] REST and Django



On Fri Nov 28 2014 at 5:58:00 AM Tripp, Travis S travis.tr...@hp.com
wrote:
  Hi Richard,

  You are right, we should put this out on the main ML, so copying thread
  out to there.  ML: FYI that this started after some impromptu IRC
  discussions about a specific patch led into an impromptu google hangout
  discussion with all the people on the thread below.

Thanks Travis!


  As I mentioned in the review[1], Thai and I were mainly discussing the
  possible performance implications of network hops from client to horizon
  server and whether or not any aggregation should occur server side.   In
  other words, some views  require several APIs to be queried before any
  data can displayed and it would eliminate some extra network requests
  from client to server if some of the data was first collected on the
  server side across service APIs.  For example, the launch instance wizard
  will need to collect data from quite a few APIs before even the first
  step is displayed (I’ve listed those out in the blueprint [2]).

  The flip side to that (as you also pointed out) is that if we keep the
  API’s fine grained then the wizard will be able to optimize in one place
  the calls for data as it is needed. For example, the first step may only
  need half of the API calls. It also could lead to perceived performance
  increases just due to the wizard making a call for different data
  independently and displaying it as soon as it can.

Indeed, looking at the current launch wizard code it seems like you
wouldn't need to load all that data for the wizard to be displayed, since
only some subset of it would be necessary to display any given panel of the
wizard.


  I tend to lean towards your POV and starting with discrete API calls and
  letting the client optimize calls.  If there are performance problems or
  other reasons then doing data aggregation on the server side could be
  considered at that point.

I'm glad to hear it. I'm a fan of optimising when necessary, and not
beforehand :)


  Of course if anybody is able to do some performance testing between the
  two approaches then that could affect the direction taken.

I would certainly like to see us take some measurements when performance
issues pop up. Optimising without solid metrics is bad idea :)


    Richard


  [1]
  https://review.openstack.org/#/c/136676/8/openstack_dashboard/api/rest/urls.py
  [2]
  https://blueprints.launchpad.net/horizon/+spec/launch-instance-redesign

  -Travis

  From: Richard Jones r1chardj0...@gmail.com
  Date: Wednesday, November 26, 2014 at 11:55 PM
  To: Travis Tripp travis.tr...@hp.com, Thai Q Tran/Silicon Valley/IBM 
  tqt...@us.ibm.com, David Lyle dkly...@gmail.com, Maxime Vidori 
  maxime.vid...@enovance.com, Wroblewski, Szymon 
  szymon.wroblew...@intel.com, Wood, Matthew David (HP Cloud - Horizon)
  matt.w...@hp.com, Chen, Shaoquan sean.ch...@hp.com, Farina, Matt
  (HP Cloud) matthew.far...@hp.com, Cindy Lu/Silicon Valley/IBM 
  c...@us.ibm.com, Justin Pomeroy/Rochester/IBM jpom...@us.ibm.com, Neill
  Cox neill@ingenious.com.au
  Subject: Re: REST and Django

  I'm not sure whether this is the appropriate place to discuss this, or
  whether I should be posting to the list under [Horizon] but I think we
  need to have a clear idea of what goes in the REST API and what goes in
  the client (angular) code.

  In my mind, the thinner the REST API the better. Indeed if we can get
  away with proxying requests through without touching any *client code,
  that would be great.

  Coding additional logic into the REST API means that a developer would
  need to look in two places, instead of one, to determine what was
  happening for a particular call. If we keep it thin then the API
  presented to the client developer is very, very similar to the API
  presented by the services. Minimum surprise.

  Your thoughts?


       Richard


  On Wed Nov 26 2014 at 2:40:52 PM Richard Jones r1chardj0...@gmail.com
  wrote:
   Thanks

Re: [openstack-dev] [horizon] REST and Django

2014-11-27 Thread Richard Jones
On Fri Nov 28 2014 at 5:58:00 AM Tripp, Travis S travis.tr...@hp.com
wrote:

  Hi Richard,

  You are right, we should put this out on the main ML, so copying thread
 out to there.  ML: FYI that this started after some impromptu IRC
 discussions about a specific patch led into an impromptu google hangout
 discussion with all the people on the thread below.


Thanks Travis!



 As I mentioned in the review[1], Thai and I were mainly discussing the
 possible performance implications of network hops from client to horizon
 server and whether or not any aggregation should occur server side.   In
 other words, some views  require several APIs to be queried before any data
 can displayed and it would eliminate some extra network requests from
 client to server if some of the data was first collected on the server side
 across service APIs.  For example, the launch instance wizard will need to
 collect data from quite a few APIs before even the first step is displayed
 (I’ve listed those out in the blueprint [2]).

  The flip side to that (as you also pointed out) is that if we keep the
 API’s fine grained then the wizard will be able to optimize in one place
 the calls for data as it is needed. For example, the first step may only
 need half of the API calls. It also could lead to perceived performance
 increases just due to the wizard making a call for different data
 independently and displaying it as soon as it can.


Indeed, looking at the current launch wizard code it seems like you
wouldn't need to load all that data for the wizard to be displayed, since
only some subset of it would be necessary to display any given panel of the
wizard.



 I tend to lean towards your POV and starting with discrete API calls and
 letting the client optimize calls.  If there are performance problems or
 other reasons then doing data aggregation on the server side could be
 considered at that point.


I'm glad to hear it. I'm a fan of optimising when necessary, and not
beforehand :)



 Of course if anybody is able to do some performance testing between the
 two approaches then that could affect the direction taken.


I would certainly like to see us take some measurements when performance
issues pop up. Optimising without solid metrics is bad idea :)


Richard



  [1]
 https://review.openstack.org/#/c/136676/8/openstack_dashboard/api/rest/urls.py
 [2]
 https://blueprints.launchpad.net/horizon/+spec/launch-instance-redesign

  -Travis

   From: Richard Jones r1chardj0...@gmail.com
 Date: Wednesday, November 26, 2014 at 11:55 PM
 To: Travis Tripp travis.tr...@hp.com, Thai Q Tran/Silicon Valley/IBM 
 tqt...@us.ibm.com, David Lyle dkly...@gmail.com, Maxime Vidori 
 maxime.vid...@enovance.com, Wroblewski, Szymon 
 szymon.wroblew...@intel.com, Wood, Matthew David (HP Cloud - Horizon) 
 matt.w...@hp.com, Chen, Shaoquan sean.ch...@hp.com, Farina, Matt
 (HP Cloud) matthew.far...@hp.com, Cindy Lu/Silicon Valley/IBM 
 c...@us.ibm.com, Justin Pomeroy/Rochester/IBM jpom...@us.ibm.com, Neill
 Cox neill@ingenious.com.au
 Subject: Re: REST and Django

  I'm not sure whether this is the appropriate place to discuss this, or
 whether I should be posting to the list under [Horizon] but I think we need
 to have a clear idea of what goes in the REST API and what goes in the
 client (angular) code.

  In my mind, the thinner the REST API the better. Indeed if we can get
 away with proxying requests through without touching any *client code, that
 would be great.

  Coding additional logic into the REST API means that a developer would
 need to look in two places, instead of one, to determine what was happening
 for a particular call. If we keep it thin then the API presented to the
 client developer is very, very similar to the API presented by the
 services. Minimum surprise.

  Your thoughts?


   Richard


 On Wed Nov 26 2014 at 2:40:52 PM Richard Jones r1chardj0...@gmail.com
 wrote:

 Thanks for the great summary, Travis.

  I've completed the work I pledged this morning, so now the REST API
 change set has:

  - no rest framework dependency
 - AJAX scaffolding in openstack_dashboard.api.rest.utils
 - code in openstack_dashboard/api/rest/
 - renamed the API from identity to keystone to be consistent
 - added a sample of testing, mostly for my own sanity to check things
 were working

  https://review.openstack.org/#/c/136676


Richard

 On Wed Nov 26 2014 at 12:18:25 PM Tripp, Travis S travis.tr...@hp.com
 wrote:

  Hello all,

  Great discussion on the REST urls today! I think that we are on track
 to come to a common REST API usage pattern.  To provide quick summary:

  We all agreed that going to a straight REST pattern rather than
 through tables was a good idea. We discussed using direct get / post in
 Django views like what Max originally used[1][2] and Thai also started[3]
 with the identity table rework or to go with djangorestframework [5] like
 what Richard was prototyping with[4].

  The main things we would