(Posting to the mailing list rather than writing a spec or making code because I think it is important to get some input and feedback before going off on something wild. Below I'm talking about speculative plans and seeking feedback, not reporting decisions about the future. Some of this discussion is intentionally naive about how things are because that's not really relevant, what's relevant is how things should be or could be.
tl;dr: I want to make the configuration of the pollsters more explicit and not conflate and overlap the entry_points.txt and pipeline.yaml in confusing and inefficient ways. * entry_points.txt should define what measurements are possible, not what measurements are loaded * something new should define what measurements are loaded and polled (and their intervals) (sources in pipeline.yaml speak) * pipeline.yaml should define transformations and publishers Would people like something like this?) The longer version: Several of the outcomes of the Liberty Design Summit were related to making changes to the agents which gather or hear measurements and events. Some of these changes have pending specs: * Ceilometer Collection Agents Split https://review.openstack.org/#/c/186964/ Splitting the collection agents into their own repo to allow use and evolution separate from the rest of Ceilometer. * Adding Meta-Data Caching Spec https://review.openstack.org/#/c/185084/ Adding metadata caching to the compute agent so the Nova-API is less assaulted than it currently is. * Declarative notification handling https://review.openstack.org/#/c/178399/ Be able to hear and transform a notification to an event without having to write code. Reviewing these and other specs and doing some review of the code points out that we have an opportunity to make some architectural and user interface improvements (while still maintain existing functionality). For example: The current ceilometer polling agent has an interesting start up process: 1 It determines which namespaces it is operating in ('compute', 'central', 'ipmi'). 2 Using entry_points defined in setup.cfg it initializes all the polling extensions and all the discovery extensions (independent of sources defined in pipeline.yaml) 3 Every source in pipeline.yaml is given a list of pollsters that match the meters defined by the source, creating long running tasks to do the polling. 4 Each task does resource discovery and partitioning coordination. 5 measurements/samples are gathered and are transformed and published according the sink rules in pipeline.yaml A couple things about this seem less than ideal: * 2 means we load redundant stuff unless we edit entry_points.txt. We do not want to encourage this sort of behavior. entry_points is not configuration[1]. We should configure elsewhere to declare "I care about things X (including the option of "all things")" and then load the tools to do so, on demand. * Two things are happening in the same context in step 5 and that seems quite limiting with regard to opportunities for effective maintenance and optimizing. My intuition (which often needs to sanity checked, thus my posting here) tells me there are some things we could change: * Separate polling and publishing/transforming into separate workers/processes. * Extract the definition of sources to be polled from pipeline.yaml to its own file and use that to be the authority of which extensions are loaded for polling and discovery. What do people think? [1] This is really the core of my concern and the main part I want to see change. -- Chris Dent tw:@anticdent freenode:cdent https://tank.peermore.com/tanks/cdent _______________________________________________ OpenStack-operators mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
