You are right that in general, this is an anti-pattern, but the data we will be sending is write only logging data. It's essentially a log sink, but for arbitrary data (blobs, files, name it you want). In clustered setup, you simply keep the logs on nodes along with other logging data. On the other hand, we aim at using syslog for everything in the future, this could be additional thing to keep in mind.
We also discussed storing this in database schema in a formal way (new table rendered_template with references to host/template_kind) but arguments against are this is already quite complex part of our schema, keeping history could easily grow the database and performance could be problem for Remote Execution when performing large-scale script runs. We ended up discussing actually removing audit trail data from database, but this turned to be problem as we need to keep reference to taxonomy, users and records. This brings me to alternative proposal: I plan to work on full syslog support for Rails and Smart Proxy in January next year. The goal is to achieve structured logging using some standard format (CEE/CEF). We would send those rendered templates to Rails log (syslog) and there could be an optional syslog module to extract those log entries and put them into similar directory structure. This could be the default setup for small deployments, but for large-scale we will recommend to send logs over network to central logging point anyway, so the module could be installed there or depending on setup this could be loaded into ELK stack. (*) In short, let's send the rendered templates to syslog, it deals with multiple lines just fine, we can simply filter these from production.log and put them into a separate file or even directory structure. (*) http://www.rsyslog.com/writing-rsyslog-plugins-in-2-minutes/ On Tue, Dec 5, 2017 at 6:34 PM, Timo Goebel <[email protected]> wrote: > I don't think, this is a good idea. Storing files on the local filesystem is > generally consideres an anti-pattern imho. Both for web applications and > expecially in a container world. > > In a clustered setup, you'd have to attach some kind of shared-storage to > the foreman hosts to make this work. Possibly via NFS or something. This is > meant to cause trouble. I generally advise not to have any state on an > application server. > > I think, this can and should be done with a database. I agree, that the > audit model might not be the best place for this. But a database definitely > is. > > Just my 2 cents. > > - Timo > > Am 05.12.17 um 18:01 schrieb Lukas Zapletal: > >> Hello, >> >> Foreman does have a nice audit trail for many operations, what's >> missing is ability to find how a template (e.g. kickstart) was >> rendered. Storing whole template text in audit table is probably not >> the best thing to do, production.log is also not a good fit, so here >> is a proposal. >> >> I want to create a small API called File audit (*) with ability to >> store arbitrary files into >> /var/log/foreman/file_audit/id/text-timestamp where "id" is record id >> (or multiple ids concatenated with dots), "text" is arbitrary alphanum >> text and "timestamp" is unix epoch number. Example: >> >> /var/log/foreman/file_audit/1.33.7/my.host.lan-pxelinux-1512492603 >> >> That API will be used to store contents of all templates rendered, so >> users can easily go "back in time" and see how templates are being >> rendered. The directory would be root-only reading and files will be >> created with restricted permissions (foreman user rw only). On system >> with SELinux, security would be more tightened allowing writing only >> to foreman_t domain and reading to nobody. For the example above, this >> would mean: >> >> 1.33.7/my.host.lan-pxelinux-1512492603 >> >> 1 - file audit type (static list, 1 stands for "template audit entry") >> 33 - host id >> 7 - template id >> my.host.lan-pxelinux - extra data so users can work and search from >> command line >> 1512492603 - UNIX epoch timestamp >> >> Everytime new record is added, a log entry is created into >> production.log containing file path. By default, there will be a cron >> job deleting all files older than one month. In documentation, we will >> ask users to rsync the directory to different location if long-term >> archival is needed. >> >> This API could be used for other audit logging, for example when user >> uploads a manifest ZIP file in katello or new version of RPM/Puppet >> file. This will be the first step to improve audit around templates, >> later on we can create a plugin showing the data in audit/host pages >> if needed. But in the first phase, administrators could easily >> search/grep/diff those files when necessary. >> >> (*) if you have a better name, please do propose >> > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. -- Later, Lukas @lzap Zapletal -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
