There's been an idea floating in my mind for quite a while now about using a job queue for compiling puppet catalogs. I just mentioned the idea on IRC and a few people really liked the idea, so I thought I'd bring it up here and get other thoughts on it.
The idea is that instead of puppet masters compiling catalogs on demand, they would operate out of a job queue: - When a node's cert is signed, the compilation queue gets a job for that node. - When a compile job finishes, that node gets added to the queue at the very end. This results in the puppet master constantly compiling catalogs. - When a catalog changes, the puppet master notifies the node that it needs to fetch the updated catalog and do a puppet run. - When an exported resource changes, any nodes which collect that resource should get their compilation jobs moved to the front of the queue. (This should be optional, as the behavior might not be desired) - Nodes would still run puppet out of a cron job, but they would use the cached catalog, not request one from the master. - If any of the facts used in the catalog change, the node would notify the master and the compilation job would move to the front of the queue. In the world of cloud computing, this becomes extremely beneficial, though it still is advantageous for traditional datacenter environments: - Puppet runs are computationally expensive for the client. With this design, the client could have its cron job set to a very infrequent value (once per hour or such). This way small instances without much available resources wont waste them on puppet runs that don't change anything. - The masters could be very beefy instances with a lot of CPU. By constantly generating catalogs, you ensure that the system's resources aren't sitting idle and being wasted. - By using a queuing mechanism, you can determine how loaded the masters are. If the oldest job in the queue is X minutes old, you can spin up another master. - With the current method of generating catalogs on demand, if a lot of nodes request a catalog at the same time, it can cause compilations to go very slow. If they go too slow, the client will get a timeout. The client will then go to the master again to request another catalog when the first generation completed fine, it just took a while. With queuing, the master can compile exactly X amount of catalogs at the same time. It can even be configured to only start a new compilation job if the system load is less than X. - Since puppet has to serve up both files and catalogs, if all the available processes are used compiling catalogs, requests for files end up hanging. By moving catalog compilations to a background worker, file requests will be faster. (you can implement a workaround for this: I have 2 puppet master pools, catalog requests get routed to one pool, everything else to the other pool, but this isn't a simple or standard design) Now most of this could be done on top of puppet. You could create a system which reads from a queue, performs a `puppet master compile`, notifies the client if it changes, etc. But there are a few sticky points which it wouldn't be able to do (just features, none would prevent the system from working): - There is no way of determining which facts were used in a catalog compilation. A client could issue a catalog compilation request when any fact changes, but facts like `uptime_seconds` always change, so it would always result in a catalog compilation. The proper way to handle this would be to monitor when a fact variable is read. Then add a list of "facts used" and their value to the resulting catalog. Then the puppet agent can use that to see if the facts used have changed. This may not be a significant issue though. If the `puppet agent` cron job is set to something very infrequent, it won't be requesting catalog compilations very often. - The catalog doesn't indicate whether a resource was collected from another node. So when an exported resource changes, you wouldn't be able to find nodes which collect those exported resources and recompile those catalogs. Now this isn't a big deal since the current method of using cron jobs can't even do this. The catalog does indicate that a resource is exported, so you could look for nodes with resources of the same type & title as the exported resource. But it's possible for a resource to have the same type & title of an exported resource as long as the exported resource isn't collected during the same catalog compilation, so this method might end up recompiling catalogs which don't use the exported resource. Puppet could probably be monkey patched to address these and add the data to the catalog. The script system could then extract the info, while the puppet agent would just ignore the data. Thoughts? I might be up for coding this, but it's would be low on the priority list and I wouldn't get to it for a long time. -Patrick -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/a40c4cdb-766d-4cb6-957e-31594aed1371%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
