Are all of the files you're referencing vault-encrypted, or does the slowness come from the initialization of the VaultLib? It might be better to cache that object based on the encryption method and/or hashed password rather than the unencrypted contents of the YAML files themselves.
This is definitely something better discussed on ansible-devel, so copying that list on this. On Thu, Jun 5, 2014 at 7:06 AM, <[email protected]> wrote: > Hello, > > We recently started using a vault and discovered that using it slowed > down ansible start up significantly. A bit of digging made it obvious why: > perhaps because of how we were using it, the vault opened and decrypted > once for every one of the ~110 YAML files ansible was reading through on > launch, adding about 0.5s each time it did so for a total start up time of > > 1 minute. As a kind of band aid/sledgehammer solution, we modified > ansible utils slightly to cache parsed YAML data and return a deepcopy of > the cached data every time ansible wanted to access the same file again > (see code) below, which reduced the start up time to under 10 seconds and > seems to work for us. Just thought I'd share in case someone else has run > into the same problem. > > Regards > J Kling > > --- > > # > lib/ansible/utils/__init__.py,55 > YAML_CACHE = {} > > # lib/ansible/utils/__init__.py,542 > def parse_yaml_from_file(path, vault_password=None): > ''' convert a yaml file to a data structure ''' > > global YAML_CACHE > data = None > show_content = True > > if path in YAML_CACHE: > return copy.deepcopy(YAML_CACHE[path]) > > try: > data = open(path).read() > except IOError: > raise errors.AnsibleError("file could not read: %s" % path) > > vault = VaultLib(password=vault_password) > if vault.is_encrypted(data): > data = vault.decrypt(data) > show_content = False > > try: > > YAML_CACHE[path] = parse_yaml(data, path_hint=path) > return copy.deepcopy(YAML_CACHE[path]) > > except yaml.YAMLError, exc: > process_yaml_error(exc, data, path, show_content) > > -- > You received this message because you are subscribed to the Google Groups > "Ansible Project" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/ansible-project/9ca89236-ecfe-4930-baa0-f397a12af488%40googlegroups.com > <https://groups.google.com/d/msgid/ansible-project/9ca89236-ecfe-4930-baa0-f397a12af488%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Ansible Project" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CAMFyvFipKq-2twgAY%2BYrke_pCjHWSun9zqWEprOPhzdNRGf0ZQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
