Re: [openstack-dev] Search Project - summit follow up
Dmitri, If you intend to make this middleware generic and reusable between different OpenStack services, your best shot, to my understanding, will be to propose a new library in oslo-incubator. -- Best regards, Oleg Gelbukh On Fri, Nov 22, 2013 at 4:05 AM, Dmitri Zimin(e) | StackStorm d...@stackstorm.com wrote: On Wed, Nov 20, 2013 at 2:11 PM, Dolph Mathews dolph.math...@gmail.comwrote: On Wed, Nov 20, 2013 at 1:06 PM, Dmitri Zimin(e) | StackStorm d...@stackstorm.com wrote: Thanks Terry for highlighting this: Yes, tenant isolation is the must. It's not reflected in the prototype - it queries Solr directly; but the proper implementation will go through the query API service, where ACL will be applied. UX folks are welcome to comment on expected queries. I think the key benefit of cross-resource index over querying DBs is that it saves the clients from implementing complex queries case by case, leaving flexibility to the user. I question the need for this service, as this service **should** very much be dependent on the clients for this functionality. Expecting to query backends directly must be a misunderstanding somewhere... Start with a specification for filtering across all services and advocate for it on both existing and new APIs. Dolph, thanks for the suggestion: we will begin drafting the API on the wiki. Just to be clear: this is not filtering. Existing filtering APIs are [getting] sufficient. This is a full text search, which doesn't exist yet. SWIFT is now considering Search API, ideologically similar, but limited to Object Storage metadata [1]. Search middleware can make it generic and extensible. And yes, middleware may be a better term, as this is not a service like nova or cinder, but a layer on top. Do we need to clarify where search middleware shall live? Or do we question wether there is the need for search functionality? What else shall we do to make the discussion forward? [1] http://lists.openstack.org/pipermail/openstack-dev/2013-November/019014.html -- Dmitri. On Wed, Nov 20, 2013 at 2:27 AM, Thierry Carrez thie...@openstack.orgwrote: Dmitri Zimin(e) | StackStorm wrote: Hi Stackers, The project Search is a service providing fast full-text search for resources across OpenStack services. [...] At first glance this looks slightly scary from a security / tenant isolation perspective. Most search results would be extremely user-specific (and leaking data from one user to another would be catastrophic), so the benefits of indexing (vs. querying DB) would be very limited ? -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- -Dolph ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Search Project - summit follow up
On Wednesday, November 20, 2013 3:12 PM Dolph Mathews dolph.math...@gmail.com wrote: On Wed, Nov 20, 2013 at 1:06 PM, Dmitri Zimin(e) | StackStorm d...@stackstorm.com wrote: Thanks Terry for highlighting this: Yes, tenant isolation is the must. It's not reflected in the prototype - it queries Solr directly; but the proper implementation will go through the query API service, where ACL will be applied. UX folks are welcome to comment on expected queries. I think the key benefit of cross-resource index over querying DBs is that it saves the clients from implementing complex queries case by case, leaving flexibility to the user. I question the need for this service, as this service **should** very much be dependent on the clients for this functionality. Expecting to query backends directly must be a misunderstanding somewhere... Start with a specification for filtering across all services and advocate for it on both existing and new APIs. First, I am all in favor of extensive and common filtering across services in OpenStack. Any improvements there would be extremely useful. The benefit of a specific search service is an exhaustive list of all resources in the stack where any field in the data is queryable. So names, ids, IPs, etc. So as an admin knowing one piece of data, I can get results across services that may match. It returns data I may have known about, but also data I may have overlooked. When purging a project, I can find all resources tied to that project. Same for a user, or IP. It's the difference between needing to know where to look for every piece of data vs knowing a piece of data and being able to do something useful with it. The performance and usability difference is huge. -- Dmitri. -- -Dolph As mentioned before, this has to be guarded by role based access controls, which is not a trivial addition, knowing what piece of data is available to whom. And maintaining up-to-date data is also a lingering question. -David ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Search Project - summit follow up
On Wed, Nov 20, 2013 at 2:11 PM, Dolph Mathews dolph.math...@gmail.comwrote: On Wed, Nov 20, 2013 at 1:06 PM, Dmitri Zimin(e) | StackStorm d...@stackstorm.com wrote: Thanks Terry for highlighting this: Yes, tenant isolation is the must. It's not reflected in the prototype - it queries Solr directly; but the proper implementation will go through the query API service, where ACL will be applied. UX folks are welcome to comment on expected queries. I think the key benefit of cross-resource index over querying DBs is that it saves the clients from implementing complex queries case by case, leaving flexibility to the user. I question the need for this service, as this service **should** very much be dependent on the clients for this functionality. Expecting to query backends directly must be a misunderstanding somewhere... Start with a specification for filtering across all services and advocate for it on both existing and new APIs. Dolph, thanks for the suggestion: we will begin drafting the API on the wiki. Just to be clear: this is not filtering. Existing filtering APIs are [getting] sufficient. This is a full text search, which doesn't exist yet. SWIFT is now considering Search API, ideologically similar, but limited to Object Storage metadata [1]. Search middleware can make it generic and extensible. And yes, middleware may be a better term, as this is not a service like nova or cinder, but a layer on top. Do we need to clarify where search middleware shall live? Or do we question wether there is the need for search functionality? What else shall we do to make the discussion forward? [1] http://lists.openstack.org/pipermail/openstack-dev/2013-November/019014.html -- Dmitri. On Wed, Nov 20, 2013 at 2:27 AM, Thierry Carrez thie...@openstack.orgwrote: Dmitri Zimin(e) | StackStorm wrote: Hi Stackers, The project Search is a service providing fast full-text search for resources across OpenStack services. [...] At first glance this looks slightly scary from a security / tenant isolation perspective. Most search results would be extremely user-specific (and leaking data from one user to another would be catastrophic), so the benefits of indexing (vs. querying DB) would be very limited ? -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- -Dolph ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Search Project - summit follow up
Dmitri Zimin(e) | StackStorm wrote: Hi Stackers, The project Search is a service providing fast full-text search for resources across OpenStack services. [...] At first glance this looks slightly scary from a security / tenant isolation perspective. Most search results would be extremely user-specific (and leaking data from one user to another would be catastrophic), so the benefits of indexing (vs. querying DB) would be very limited ? -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Search Project - summit follow up
On Wed, Nov 20, 2013 at 1:06 PM, Dmitri Zimin(e) | StackStorm d...@stackstorm.com wrote: Thanks Terry for highlighting this: Yes, tenant isolation is the must. It's not reflected in the prototype - it queries Solr directly; but the proper implementation will go through the query API service, where ACL will be applied. UX folks are welcome to comment on expected queries. I think the key benefit of cross-resource index over querying DBs is that it saves the clients from implementing complex queries case by case, leaving flexibility to the user. I question the need for this service, as this service **should** very much be dependent on the clients for this functionality. Expecting to query backends directly must be a misunderstanding somewhere... Start with a specification for filtering across all services and advocate for it on both existing and new APIs. -- Dmitri. On Wed, Nov 20, 2013 at 2:27 AM, Thierry Carrez thie...@openstack.orgwrote: Dmitri Zimin(e) | StackStorm wrote: Hi Stackers, The project Search is a service providing fast full-text search for resources across OpenStack services. [...] At first glance this looks slightly scary from a security / tenant isolation perspective. Most search results would be extremely user-specific (and leaking data from one user to another would be catastrophic), so the benefits of indexing (vs. querying DB) would be very limited ? -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- -Dolph ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] Search Project - summit follow up
Hey OpenStackers, I hope you all enjoyed Hong Kong Summit and for those who couldn't attend, I hope we will see each other next time! For all of you, I'd like to follow up on conversation from Summit about Search project [0], which Dmitri introduced and I believe it has great potential for enhancing OpenStack's user experience, especially in Horizon, but not just in there. I'd like to thank to Dmitri and all folks who were working on this project for sharing and their will for contributing. It would be great, if we can follow up on this topic, start discussion around the project here on mailing list (since not everybody could attend the summit) and gather some feedback. There are lot of questions like where to accommodate the code, what are next steps, etc. I am cc'ing Dmitri, so he can bring more light into this discussion and I hope we can move forward. Thanks -- Jarda [0] https://wiki.openstack.org/wiki/SearchProject ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Search Project - summit follow up
Thanks Jarda for the intro; Hi Stackers, The project Search is a service providing fast full-text search for resources across OpenStack services. The idea was introduced and discussed at the Summit. We confirmed the need for Search service: 1) from core Horizon team and UX 2) from integrators building custom consoles 3) some suggested CLI use-cases. We established it’s likely be a new service. I spoke with Julien @ Ceilometer about leveraging some overlapping functionality and he suggested an integration point to address it. Now let’s discuss it with the broader community and have you guys guide it through the proper next steps. DETAILS: * Wiki - basic project description, and 2 min video showing Search in action. https://wiki.openstack.org/wiki/SearchProject * Etherpad - captures Summit discussions: https://etherpad.openstack.org/p/search * Github - the rough prototype: * Search backend: https://github.com/StackStorm/search * Horizon plugin: https://github.com/StackStorm/search-horizon QUESTIONS: 1) Is it relevant to OpenStack vision and directions? -- yes, based on Summit discussions 2) What is a good home for it? A a new standalone program or a part of existing program? -- So far: Standalone program (input from David Lyle, Matthias Runge, Doug Helmann) 3) Who else wants to be involved? Pigs? Chickens? 4) What do we do next? -- Create an Launchpad? -- Form the team? -- Give it a name? -- Begin flashing out the design? Thanks to David Lyle, Mattihas Runge, Nicolas Simonds, Julien Danjou, Noy Itzikowitz, Ryan Lane and Doug Helmann for your help and active input. Cheers, Dmitri. On Tue, Nov 19, 2013 at 7:56 AM, Jaromir Coufal jcou...@redhat.com wrote: Hey OpenStackers, I hope you all enjoyed Hong Kong Summit and for those who couldn't attend, I hope we will see each other next time! For all of you, I'd like to follow up on conversation from Summit about Search project [0], which Dmitri introduced and I believe it has great potential for enhancing OpenStack's user experience, especially in Horizon, but not just in there. I'd like to thank to Dmitri and all folks who were working on this project for sharing and their will for contributing. It would be great, if we can follow up on this topic, start discussion around the project here on mailing list (since not everybody could attend the summit) and gather some feedback. There are lot of questions like where to accommodate the code, what are next steps, etc. I am cc'ing Dmitri, so he can bring more light into this discussion and I hope we can move forward. Thanks -- Jarda [0] https://wiki.openstack.org/wiki/SearchProject ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
