On Fri, Mar 24, 2017 at 3:58 PM, Martin Sivak <[email protected]> wrote:
> > I feel like every REST API I've ever worked with has had the aggregation > + > > projection problem. It's like we're trying to use REST as a replacement > for > > SQL -- but the logic that executes the "SQL" lives in a browser now, and > it > > used to live on a server close to the DB. And REST isn't expressive for > > selecting data like SQL is. > > The current industry solution I know about is called API gateway.. > most of the big players have internal API with lots of low level stuff > and then couple of external API gateways tailored to what the client > needs. > > http://microservices.io/patterns/apigateway.html (check the backend > for frontend section) > > This trend is also visible when you think about services that offer > API gateway management and billing like > https://aws.amazon.com/api-gateway/ or our very own > https://www.3scale.net/ right, but the api gateway solves 2 problems: 1: if you have a microservice architecture it is hard for frontend to talk to 20 different moving services. So the gateway hides this complexity behind it. This is not the problem we have. 2: you can have more api gateways (e.g. more apis) tailored for every frontend. I don't think we need this - the current API serves us pretty well in every FE Im involved in. The only thing which I miss is the data aggregation. So I would go back to the original topic of this thread - do some small change which has a chance to be merged to the project and helps us where it hurts. > > > Martin > > On Fri, Mar 24, 2017 at 3:47 PM, Greg Sheremeta <[email protected]> > wrote: > > I feel like every REST API I've ever worked with has had the aggregation > + > > projection problem. It's like we're trying to use REST as a replacement > for > > SQL -- but the logic that executes the "SQL" lives in a browser now, and > it > > used to live on a server close to the DB. And REST isn't expressive for > > selecting data like SQL is. > > > > There must be some industry solution to this "I want to do SQL over REST" > > problem. > > > > On Fri, Mar 24, 2017 at 5:54 AM, Martin Sivak <[email protected]> wrote: > >> > >> > for quite some time I have been more or less involved in development > of > >> > various UIs for oVirt based entirely on the oVirt's REST API ranging > >> > from > >> > the quite mature moVirt [1] through some cockpit extensions to a young > >> > and > >> > experimental user portal replacement [2]. > >> > >> oVirt optimizer has the same issue.. > >> > >> > 2: add some tiny service which would just accept a list of queries, > >> > execute > >> > them locally (but using real HTTP requests) and return in one bulk. A > >> > naive > >> > implementation just to give a sense of what I mean of this would be a > >> > shell > >> > script getting list of strings like > >> > "https://localhost/ovirt-engine/api/vms/123/sessions"; iterate over > them > >> > and > >> > do a curl request for each, mangle the results into one string and > >> > return > >> > (credits for this idea to msivak). Easy to implement, possibility to > add > >> > also projections later to save some bandwidth. But the API would > anyway > >> > be > >> > hammered by bunch of queries, only the network roundtrip would be > saved. > >> > >> The biggest cost for (especially mobile) clients is the cost of > >> establishing new SSL connection. SSL is also pretty expensive on the > >> server side. > >> > >> So running the aggregation service on the ovirt-engine machine (behind > >> Apache) means the client will do a single SSL request with list of N > >> urls and the local "reverse-proxy" will perform single authentication > >> and N plain HTTP requests (or even better - AJP). It won't remove any > >> time from the actual command run time, but it will reduce protocol > >> overhead. > >> > >> I think this is the simplest first step that requires almost no change > >> to existing infrastructure. > >> > >> -- > >> Martin Sivak > >> SLA / oVirt > >> > >> On Fri, Mar 24, 2017 at 10:20 AM, Tomas Jelinek <[email protected]> > >> wrote: > >> > Hi All, > >> > > >> > for quite some time I have been more or less involved in development > of > >> > various UIs for oVirt based entirely on the oVirt's REST API ranging > >> > from > >> > the quite mature moVirt [1] through some cockpit extensions to a young > >> > and > >> > experimental user portal replacement [2]. > >> > > >> > One issue we hit over and over again is the missing data aggregation. > In > >> > the > >> > 3.x era we used to use in moVirt the detail=something > >> > api to get the disks and nics of the VM, something like: > >> > > >> > GET /ovirt-engine/api/vms > >> > Accept: application/json; detail=disks > >> > > >> > This allowed us to store this data in local database leading to great > >> > user > >> > experience. Since this feature has been removed in 4.x API [3] > >> > we needed to retire to a different solution. When the VM detail is > >> > selected > >> > by the user, start loading the disks and nics and hope the user > >> > will not be fast enough to see the delay. The UX is slightly worse bug > >> > kinda > >> > acceptable. > >> > > >> > We hit this issue harder in the new user portal [2], because we > already > >> > have > >> > the VM cached and show the whole VM in one screen. So, if you pick it, > >> > you > >> > will get it's details immediately. > >> > But, since you don't have all the details, we need to do an additional > >> > call > >> > (two actually) to load this data and they start to appear later. > >> > So, something which would be very fast and smooth starts to feel > >> > sluggish. > >> > > >> > Recently, we hit this issue again which forced us to sacrifice the UX > >> > even > >> > more - it is the "console in use" feature of user portal. > >> > The use case is this: > >> > - if the console is already taken by some user, there are > complications > >> > if > >> > other current user tryes to take it as well (will avoid details about > >> > settings and permissins involved, but long story short, the user will > >> > probably not be allowed to connect to it. The "probably" is the key > here > >> > since we can not do any intelligent decision in advance, we can only > >> > warn > >> > the user that the console is taken). > >> > - in the current GWT user portal, if the VM's console is taken, it is > >> > shown > >> > on the VM's "box" that "console is taken". This was a highly requested > >> > feature > >> > - to get this information using the current REST API, we need to go to > >> > the > >> > /vms/<vmid>/sessions subcollection. To get this for all VMs, it would > be > >> > doing N queries per poll which we can not afford > >> > - so the current PR [4] will probably end up to only check it on the > >> > attempt > >> > to connect to the console warning the user. Maybe it will be also > shown > >> > in > >> > Vm details. But the UX in case the user will look for a VM which has > >> > free > >> > console will suffer significantly (e.g. try one by one until some > opens > >> > or > >> > look at details one by one to see if the warning appears (with a > delay)) > >> > > >> > I understand that embedding the details of the VM to the response > comes > >> > with > >> > a cost, namely: > >> > - performance hit > >> > - complexity of the API code > >> > - the "cleanness" of REST suffers > >> > > >> > But I think we should seriously consider to provide some option to > data > >> > aggregation. > >> > > >> > I know this has been discussed many times with no result, but I think > it > >> > is > >> > time to bring this topic up again. I'll try to summarize the (failed) > >> > attempts tried so far: > >> > - the detail=<something> parameter with ad-hoc embedding of data. This > >> > has > >> > been there and removed in 4.0 [3] > >> > - the DoctorREST project - e.g. a proxy above the current api. The > idea > >> > was > >> > to create a service which will be independent of the engine itself, > will > >> > locally poll the engine's REST, store all data in local (mongo)DB and > >> > provide a rich api with aggregations and projections and push > >> > notifications. > >> > This polling of everything to get the data to DoctorREST proved to be > >> > pretty > >> > costy, so also a more invasive approach of pushing data from engine to > >> > doctor has been discused [5]. None of this two approaches have been > >> > accepted > >> > (too complicated, too invasive). > >> > - writing some custom ad-hoc servlet serving only a purpose of one > >> > frontend > >> > - this is actually there for the dashboard, but it is not a generic > >> > solution > >> > for the other frontends and we really should not develop custom "APIs" > >> > for > >> > every frontend > >> > - there were some other proposals discussed (some 3th party solutions > >> > etc) > >> > but I think none of them made it even to a PoC > >> > > >> > So, now I would try again and try small to get at least some benefit. > I > >> > see > >> > 2 paths we could try: > >> > 1: embed something which burns us immediatly, e.g. the /sessions into > >> > VMs. I > >> > really liked the ;detail=sessions approach, could we move it back? > >> > 2: add some tiny service which would just accept a list of queries, > >> > execute > >> > them locally (but using real HTTP requests) and return in one bulk. A > >> > naive > >> > implementation just to give a sense of what I mean of this would be a > >> > shell > >> > script getting list of strings like > >> > "https://localhost/ovirt-engine/api/vms/123/sessions"; iterate over > them > >> > and > >> > do a curl request for each, mangle the results into one string and > >> > return > >> > (credits for this idea to msivak). Easy to implement, possibility to > add > >> > also projections later to save some bandwidth. But the API would > anyway > >> > be > >> > hammered by bunch of queries, only the network roundtrip would be > saved. > >> > 3: any other simple approaches? > >> > > >> > I honestly prefer the first approach. It is not beautiful, it is not > >> > REST-ful, but it is easy to implement, very pragmatic and useful. > >> > What do you think? > >> > > >> > Thank you and sorry for the long mail :) > >> > Tomas > >> > > >> > [1]: https://github.com/oVirt/moVirt > >> > [2]: https://github.com/oVirt/ovirt-web-ui > >> > [3]: https://gerrit.ovirt.org/#/c/61260 > >> > [4]: https://github.com/oVirt/ovirt-web-ui/pull/106/ > >> > [5]: https://gerrit.ovirt.org/#/c/45233/ > >> > > >> > > >> > _______________________________________________ > >> > Devel mailing list > >> > [email protected] > >> > http://lists.ovirt.org/mailman/listinfo/devel > >> _______________________________________________ > >> Devel mailing list > >> [email protected] > >> http://lists.ovirt.org/mailman/listinfo/devel > > > > > > > > > > -- > > Greg Sheremeta, MBA > > Red Hat, Inc. > > Sr. Software Engineer > > [email protected] >
_______________________________________________ Devel mailing list [email protected] http://lists.ovirt.org/mailman/listinfo/devel
