Re: [openstack-dev] [Ceilometer] Complex query BP implementation
On 12/26/2013 04:24 AM, Julien Danjou wrote: On Tue, Dec 24 2013, Jay Pipes wrote: I think you may have switched the above? Do you mean POST for sanity (short URLs) and GET for compatibility/standards? I don't think so, but maybe my sentence wasn't clear, sorry. I meant that in theory, the right verb to use would be GET as you pointed out. However, since using GET with a body is not a common practice, also supporting the POST verb as a replacement is a good idea for compatibility. Ah, OK, got it. Can you elaborate why this is not something Ceilometer would be interested in supporting? Would you prefer this kind of thing live in some other component? I don't think there's any gain in storing data that are tight to a consumer application. Storing and retrieving this kind of data, and then executing this as a higher cost than just executing a query. Storing this kind of data to be executed later looks to me like we would start building some sort of light PaaS platform. I really don't think that's needed at this stage. K, understood. If we stick with just GET, then I could implement the saved query using bit.ly ;) -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ceilometer] Complex query BP implementation
On Tue, Dec 24 2013, Jay Pipes wrote: > I think you may have switched the above? Do you mean POST for sanity (short > URLs) and GET for compatibility/standards? I don't think so, but maybe my sentence wasn't clear, sorry. I meant that in theory, the right verb to use would be GET as you pointed out. However, since using GET with a body is not a common practice, also supporting the POST verb as a replacement is a good idea for compatibility. > Can you elaborate why this is not something Ceilometer would be interested > in supporting? Would you prefer this kind of thing live in some other > component? I don't think there's any gain in storing data that are tight to a consumer application. Storing and retrieving this kind of data, and then executing this as a higher cost than just executing a query. Storing this kind of data to be executed later looks to me like we would start building some sort of light PaaS platform. I really don't think that's needed at this stage. -- Julien Danjou // Free Software hacker / independent consultant // http://julien.danjou.info signature.asc Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ceilometer] Complex query BP implementation
Hi, Thank you all for the comments, see mines inline. Best Regards and Merry Christmas, Ildiko -Original Message- From: Jay Pipes [mailto:jaypi...@gmail.com] Sent: Tuesday, December 24, 2013 2:14 PM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Ceilometer] Complex query BP implementation On 12/24/2013 04:23 AM, Julien Danjou wrote: > On Tue, Dec 24 2013, Jay Pipes wrote: > >> My main objection to the proposed solution is that it violates the >> principle in all of the OpenStack REST APIs that a POST request *creates* a >> resource. >> In the proposed API, you use: >> >> POST /query/$resource >> >> to actually retrieve records of type $resource. In all the other >> OpenStack REST APIs, the above request would create a $resource >> subresource of a "query" resource. And, to be honest, people expect >> HTTP REST APIs to use the GET HTTP method for querying, not POST. >> It's an anti-pattern to use POST in this way. > > I thought the same, but it seems that actually using GET with a body > isn't terribly standard, though not strictly disallowed AFAIU. So I > think supporting both, GET for sanity and POST for compatibility, > would make sense. I think you may have switched the above? Do you mean POST for sanity (short URLs) and GET for compatibility/standards? Using GET with a request body is really not a common practice. ildikov: There is a risk with using GET request with a body that some routers or proxies will drop the body as it is so unexpected. It is more common to use POST for rich queries, than GET with body. ildikov: Currently nothing blocks us to use both. As I mentioned in my previous mail, the actual simple query feature and our solution are independent from each other, so using the new implementation will not affect the currently existing one. >> Now, that said... I think that the advanced query interface you >> propose does indeed have real-world, demanded use cases. Right now, >> you are 100% correct that the existing GET request filters are >> simplistic and don't support either aggregation or advanced union or >> intersection queries. >> >> I would definitely be supportive of using POST to store "saved >> queries" (as you mention in your wiki page). However, the main query >> interface should remain the GET HTTP method, including for ad-hoc advanced >> querying. > > Storing queries doesn't sound like something we would want to do. Can you elaborate why this is not something Ceilometer would be interested in supporting? Would you prefer this kind of thing live in some other component? ildikov: I think stored query support can be a user value in the future. But first we need to see the behavior of the databases under heavy load and then we can plan with features like this. But correct me, if I'm wrong here. Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ceilometer] Complex query BP implementation
Hi Jay, Thank you for the comments, see my replies inline, I used my IRC nick. Best Regards, Ildiko -Original Message- From: Jay Pipes [mailto:jaypi...@gmail.com] Sent: Tuesday, December 24, 2013 3:45 AM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Ceilometer] Complex query BP implementation On 12/16/2013 03:54 PM, Ildikó Váncsa wrote: > Hi guys, > > The first working version of the Complex filter expressions in API > queries blueprint [1] was pushed for review[2]. > > We implemented a new query REST resource in order to provide rich > query functionality for samples, alarms and alarm history. The future > plans (in separated blueprints) with this new functionality is > extending it to support Statistics and stored queries. The new feature > is documented on Launchpad wiki[3], with an example for how to use the new > query on the API. > > What is your opinion about this solution? > > I would appreciate some review comments and/or feedback on the > implementation. :) Hi Ildiko, thanks for your proposed API for complex querying in Ceilometer. Unfortunately, I'm not a fan of the approach taken, but I do see some definite need/use cases here. My main objection to the proposed solution is that it violates the principle in all of the OpenStack REST APIs that a POST request *creates* a resource. In the proposed API, you use: POST /query/$resource to actually retrieve records of type $resource. In all the other OpenStack REST APIs, the above request would create a $resource subresource of a "query" resource. And, to be honest, people expect HTTP REST APIs to use the GET HTTP method for querying, not POST. It's an anti-pattern to use POST in this way. ildikov: It is not an uncommon solution, even in case of REST APIs, to use POST for rich queries in this way, as we proposed it. It is not a forbidden approach in any standards. We did not want to violate any rules of OpenStack with this solution, but we did not find any document, which would contain a definite rule against our implementation. Now, that said... I think that the advanced query interface you propose does indeed have real-world, demanded use cases. Right now, you are 100% correct that the existing GET request filters are simplistic and don't support either aggregation or advanced union or intersection queries. I would definitely be supportive of using POST to store "saved queries" (as you mention in your wiki page). However, the main query interface should remain the GET HTTP method, including for ad-hoc advanced querying. ildikov: The current grammar, which is supported by the query part of the GET HTTP request, is not easily extendable. We would like to create a solution, which can be further extended, if our grammar is not able to fulfill all the requirements against it. Our principle was also to create a module, which supports the query as a feature and also easy to maintain, extend and reuse. We ended up in a JSON representation, which can be processed by built-in functions, we do not need to write and maintain a new code base, just in order to process the parts of a query expression. We cannot send JSON expressions in the URL of GET. It is very uncommon to use the body of a GET request, however it is also not forbidden by any standards. Probably proxies and some routers would drop the body of the GET request as it is so unexpected, so we did not have other options, but to use POST. ildikov: It is a future blueprint, to have stored query support in Ceilometer, currently it is out of scope. So, what I would like to see is essentially a removal of the "query" resource, and instead tack on your advanced Ceilometer domain-specific language to the supported GET query arguments. This would have two advantages: 1) You will not need to re-implement the orderby and limit expressions. Virtually all other OpenStack APIs (including Ceilometer) use the "limit" and "sort_by" query parameters already, so those should be used as-is. ildikov: If I know right, the orderby parameter is currently not provided by the query in the URL of GET. There is a default sort with descending order based on timestamps, but it is not a feature, only a default behavior. Our solution provides the possibility to define the fields, by which the user would like to sort the results and it is also the user's choice, if the ordering is ascending or descending for each specified orderby criteria. This implementation has user value, see the discussion on the mailing list: http://www.gossamer-threads.com/lists/openstack/dev/33926?page=last ildikov: In case of limit, you are right, but we are talking about approximately six or ten lines of code for the whole new implementation, probably this amount can be acceptable as it is only a value check and one more parameter in the quer
Re: [openstack-dev] [Ceilometer] Complex query BP implementation
On 12/24/2013 04:23 AM, Julien Danjou wrote: On Tue, Dec 24 2013, Jay Pipes wrote: My main objection to the proposed solution is that it violates the principle in all of the OpenStack REST APIs that a POST request *creates* a resource. In the proposed API, you use: POST /query/$resource to actually retrieve records of type $resource. In all the other OpenStack REST APIs, the above request would create a $resource subresource of a "query" resource. And, to be honest, people expect HTTP REST APIs to use the GET HTTP method for querying, not POST. It's an anti-pattern to use POST in this way. I thought the same, but it seems that actually using GET with a body isn't terribly standard, though not strictly disallowed AFAIU. So I think supporting both, GET for sanity and POST for compatibility, would make sense. I think you may have switched the above? Do you mean POST for sanity (short URLs) and GET for compatibility/standards? Using GET with a request body is really not a common practice. Now, that said... I think that the advanced query interface you propose does indeed have real-world, demanded use cases. Right now, you are 100% correct that the existing GET request filters are simplistic and don't support either aggregation or advanced union or intersection queries. I would definitely be supportive of using POST to store "saved queries" (as you mention in your wiki page). However, the main query interface should remain the GET HTTP method, including for ad-hoc advanced querying. Storing queries doesn't sound like something we would want to do. Can you elaborate why this is not something Ceilometer would be interested in supporting? Would you prefer this kind of thing live in some other component? Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ceilometer] Complex query BP implementation
On Tue, Dec 24 2013, Jay Pipes wrote: > My main objection to the proposed solution is that it violates the principle > in all of the OpenStack REST APIs that a POST request *creates* a resource. > In the proposed API, you use: > > POST /query/$resource > > to actually retrieve records of type $resource. In all the other OpenStack > REST APIs, the above request would create a $resource subresource of a > "query" resource. And, to be honest, people expect HTTP REST APIs to use the > GET HTTP method for querying, not POST. It's an anti-pattern to use POST in > this way. I thought the same, but it seems that actually using GET with a body isn't terribly standard, though not strictly disallowed AFAIU. So I think supporting both, GET for sanity and POST for compatibility, would make sense. > Now, that said... I think that the advanced query interface you propose does > indeed have real-world, demanded use cases. Right now, you are 100% correct > that the existing GET request filters are simplistic and don't support > either aggregation or advanced union or intersection queries. > > I would definitely be supportive of using POST to store "saved queries" (as > you mention in your wiki page). However, the main query interface should > remain the GET HTTP method, including for ad-hoc advanced querying. Storing queries doesn't sound like something we would want to do. -- Julien Danjou ;; Free Software hacker ; independent consultant ;; http://julien.danjou.info signature.asc Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Ceilometer] Complex query BP implementation
On 12/16/2013 03:54 PM, Ildikó Váncsa wrote: Hi guys, The first working version of the Complex filter expressions in API queries blueprint [1] was pushed for review[2]. We implemented a new query REST resource in order to provide rich query functionality for samples, alarms and alarm history. The future plans (in separated blueprints) with this new functionality is extending it to support Statistics and stored queries. The new feature is documented on Launchpad wiki[3], with an example for how to use the new query on the API. What is your opinion about this solution? I would appreciate some review comments and/or feedback on the implementation. :) Hi Ildiko, thanks for your proposed API for complex querying in Ceilometer. Unfortunately, I'm not a fan of the approach taken, but I do see some definite need/use cases here. My main objection to the proposed solution is that it violates the principle in all of the OpenStack REST APIs that a POST request *creates* a resource. In the proposed API, you use: POST /query/$resource to actually retrieve records of type $resource. In all the other OpenStack REST APIs, the above request would create a $resource subresource of a "query" resource. And, to be honest, people expect HTTP REST APIs to use the GET HTTP method for querying, not POST. It's an anti-pattern to use POST in this way. Now, that said... I think that the advanced query interface you propose does indeed have real-world, demanded use cases. Right now, you are 100% correct that the existing GET request filters are simplistic and don't support either aggregation or advanced union or intersection queries. I would definitely be supportive of using POST to store "saved queries" (as you mention in your wiki page). However, the main query interface should remain the GET HTTP method, including for ad-hoc advanced querying. So, what I would like to see is essentially a removal of the "query" resource, and instead tack on your advanced Ceilometer domain-specific language to the supported GET query arguments. This would have two advantages: 1) You will not need to re-implement the orderby and limit expressions. Virtually all other OpenStack APIs (including Ceilometer) use the "limit" and "sort_by" query parameters already, so those should be used as-is. 2) Users will already be familiar with the standard GET /samples, GET /alarms, etc query interface, and all they would need to learn is how to encode the advanced query parameters properly. No need to learn/implement new resource endpoints. You used this English-language example of an advanced query: "Check for cpu_util samples reported between 18:00-18:15 or between 18:30 - 18:45 where the utilization is between 23 and 26 percent." and had the following POST request JSON-ified body: POST /query/meters ["and", ["and", ["and", ["=", "counter_name", "cpu_util"], [">", "counter_volume", 0.23]], ["and", ["=", "counter_name", "cpu_util"], ["<", "counter_volume", 0.26]]], ["or", ["and", [">", "timestamp", "2013-12-01T18:00:00"], ["<", "timestamp", "2013-12-01T18:15:00"]], ["and", [">", "timestamp", "2013-12-01T18:30:00"], ["<", "timestamp", "2013-12-01T18:45:00" (note that the above doesn't actually correspond to the English-language query... you would not want the third-level "and"s and you would want >= and <= for the temporal BETWEEN clause...) The equivalent GET request might be encoded like so: GET /meters?expr=(expr1%20or%20expr2)%20and%20expr3&expr1=(timestamp%3E%3D2013-12-01T18%3A00%3A00%20and%20timestamp%3C%3D2013-12-01T18%3A15%3A00)&expr2=(timestamp%3E%3D2013-12-01T18%3A30%3A00%20and%20timestamp%3C%3D2013-12-01T18%3A45%3A00)&expr3=(counter_name%3D%27cpu_util%27%20and%20(counter_volume%20%3E%200.23%20and%20counter_volume%20%3C%200.26)) Which is just the following, with the values url-encoded: expr = (expr1 or expr2) and expr3 expr1 = (timestamp>=2013-12-01T18:00:00 and timestamp<=2013-12-01T18:15:00 ) expr2 = (timestamp>=2013-12-01T18:30:00 and timestamp<=2013-12-01T18:45:00) expr3 = (counter_name='cpu_util' and (counter_volume>0.23 and counter_volume<0.26)) I know the expression might not look as nice as POST /query/meters, but it is in-line with Internet custom. I would definitely support the use of your POST with JSON-encoded query DSL for stored views, though. For example, to save a stored query for the above report: POST /reports ["and", ["and", ["=", "counter_name", "cpu_util"], ["between", "counter_volume", 0.23, 0.26], ["or", ["and", [">", "timestamp", "2013-12-01T18:00:00"], ["<", "timestamp", "2013-12-01T18:15:00"]], ["and", [">", "timestamp", "2013-12-01T18:30:00"], ["<", "timestamp", "2013-12-01T18:45:00" And then you could issue the same query using the GET of the returned report UUID or report name... GET /reports/$REPORT
[openstack-dev] [Ceilometer] Complex query BP implementation
Hi guys, The first working version of the Complex filter expressions in API queries blueprint [1] was pushed for review[2]. We implemented a new query REST resource in order to provide rich query functionality for samples, alarms and alarm history. The future plans (in separated blueprints) with this new functionality is extending it to support Statistics and stored queries. The new feature is documented on Launchpad wiki[3], with an example for how to use the new query on the API. What is your opinion about this solution? I would appreciate some review comments and/or feedback on the implementation. :) [1] https://blueprints.launchpad.net/ceilometer/+spec/complex-filter-expressions-in-api-queries [2] https://review.openstack.org/#/q/status:open+project:openstack/ceilometer+branch:master+topic:bp/complex-filter-expressions-in-api-queries,n,z [3] https://wiki.openstack.org/wiki/Ceilometer/ComplexFilterExpressionsInAPIQueries Thanks and Best Regards, Ildiko ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
