Thanks, this should be pretty easy to test out and benchmark. I'm classifying this as P2 so we can get it some attention earlier.
On Sat, Jul 19, 2014 at 12:59 PM, Josh Drake <[email protected]> wrote: > Ergh, meant to submit this to ansible-devel, sorry about that. Anyway, I > definitely agree that some potential backends (eg: SQL database) might not > be suited for the type of workload here, but that redis should perform well > in any conceivable use-case. I've submitted a pull request #8203 ( > https://github.com/ansible/ansible/pull/8203) for continuing the > discussion. > > > On Friday, July 18, 2014 2:53:40 PM UTC-5, Michael DeHaan wrote: > >> Let me reverse my earlier logic here - with something like Redis, this is >> probably 100% fine. >> >> We need to put this in queue, so please send us a PR. >> >> We can test it out to see how it does, databases may be hard, but Redis >> is not. >> >> Let me know and we can put this through it's paces. >> >> I think with the initial there may be cache invalidation logic that needs >> overhaul, so we'll have to just be really careful about it. >> >> >> >> On Thu, Jul 17, 2014 at 9:49 PM, Michael DeHaan <[email protected]> >> wrote: >> >>> To elaborate: the vars_plugin theoretically could be returned via a >>> specialized inventory plugin. It's more efficient to do so for small >>> numbers of hosts. But there are dangers - lots of accesses, if not lazily >>> done, could bog down the system immensely, and make it intractable for >>> large numbers of hosts. >>> >>> I'm worried about that. >>> >>> If we have a playbook of 10,000 systems, and we have 50 tasks in that >>> playbook, and -f 200, how does Redis hold up, etc. >>> >>> Anyway, more of a topic for ansible-devel really. >>> >>> >>> >>> >>> >>> >>> >>> On Thu, Jul 17, 2014 at 9:31 PM, Michael DeHaan <[email protected]> >>> wrote: >>> >>>> While I appreciate the interest, fact caching will need to have very >>>> rigid design requirements so we are unlikely to take a pull request on it >>>> at this time. >>>> >>>> Ultimately I see this happening as a combination of a callback plugin >>>> to intercept facts, and a vars plugin to provide them. >>>> >>>> And it will need to be optimized for database usage. >>>> >>>> >>>> >>>> >>>> >>>> On Thu, Jul 17, 2014 at 9:26 PM, Josh Drake <[email protected]> wrote: >>>> >>>>> Greetings, >>>>> >>>>> I know it has been attempted before and is still slated for the >>>>> future, but I recently needed fact caching in my personal use of Ansible. >>>>> I >>>>> leveraged the work that was already done to fix the bugs that were present >>>>> and complete a handful of working caching backends: redis, memcached, and >>>>> a >>>>> simple file backend. I have been using them in my environments for a >>>>> couple >>>>> of weeks now (mostly redis, but testing the others as well), and haven't >>>>> had any issues. I am still extremely new to Ansible, and basically only >>>>> have enough knowledge of the internals to implement the aforementioned >>>>> functionality. That said, I figured I'd re-open discussion on this topic >>>>> here before submitting a pull request. I've included a link here and below >>>>> to a feature branch diff >>>>> <https://github.com/joshdrake/ansible/compare/feature/fact_caching?expand=1&w=1> >>>>> against the devel branch for review. Things of note: >>>>> >>>>> >>>>> 1. Only SETUP_CACHE is leveraging caching backends. VARS_CACHE is >>>>> untouched as I'm not quite sure I understand the use-case behind >>>>> caching >>>>> play variables between playbook executions. >>>>> 2. Caching backends have a base class they should extend to ensure >>>>> the API is properly implemented. All the heavy lifting is done by each >>>>> caching backend. >>>>> 3. Given the existing usage of SETUP_CACHE (eg: dictionary based >>>>> access), caching backends must be able to return the keys that are >>>>> being >>>>> held in cache. There are various ways of doing that can be seen in the >>>>> diff. Redis is perhaps the most interesting and optimal since it allows >>>>> usage of sorted sets. >>>>> 4. All unit tests pass and the sample playbooks noted as issues in >>>>> the previous threads are not present. I haven't had time recently to >>>>> do so, >>>>> but I'll work on running the integration tests as well. >>>>> >>>>> Hopefully I'm not encroaching on any plans of major refactoring for >>>>> fact caching since I know it's been in the pipeline for awhile. I don't >>>>> have any strong opinions on the matter, but I figured that I would make >>>>> what I've done available in the event it might be useful. >>>>> >>>>> Diff for Fact Caching Feature Branch: https://github.com/ >>>>> joshdrake/ansible/compare/feature/fact_caching?expand=1&w=1 >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Ansible Project" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> >>>>> To view this discussion on the web visit https://groups.google.com/d/ >>>>> msgid/ansible-project/0ad9bef2-a918-45d5-9bcb- >>>>> a0bbb83a3a7e%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/ansible-project/0ad9bef2-a918-45d5-9bcb-a0bbb83a3a7e%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> >>> >> -- > You received this message because you are subscribed to the Google Groups > "Ansible Project" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/ansible-project/84e96d2a-1c91-4bd2-a746-bb033cda9cec%40googlegroups.com > <https://groups.google.com/d/msgid/ansible-project/84e96d2a-1c91-4bd2-a746-bb033cda9cec%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Ansible Project" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CA%2BnsWgzjOC7UGH39U7RmcL%3D51PRdvP7Q-LS0e7tO3JhMOKyDJw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
