Hi David, thanks for the feedback, comments inline below
[template generators...] > > this sounds very promising. I question if you really would use the same > format for writing to a local file as you do when forwarding, the local > file normall doesn't log the severity info (or at least not in the same > format) > > I believe you already have differing templates for standard vs > forwarding > in many cases. > > rather than trying to do it in the config, is there a way to let the C > module say "have that module do it's think, then I'll tweak the result" > so > that the code doesn't need to get duplicated between modules for the > most > common cases? Actually, this complicates things AND adds some overhead again. On the other hand, I have now checked, code duplication is very small and really a one-time job. So I think I will accept that duplication for the time being. In any case, generators could load common code via the usual module loader (in just the same way as e.g. imtcp loads the relevant network stream driver). [...] > > Also, it would be useful to have the ability to persist already- > generated > > properties with the message while it is continued to be processed in > the rule > > engine. So far, we do not have this ability, and the reason is > processing > > time (plus, as usual, implementation effort): for that, we would need > to > > maintain a list (or hash, ...) of name/value pairs, store them to > disk for > > disk queues and shuffle them through the rule engine as processing is > carried > > out. As I said, quite doable, but another big addition. > > I expect that this isn't worthwhile for a couple of reasons. > > 1. with something like this you need to worry about multi-thread > protection and locks, which will kill your performance Not really, only if the locks need actually to be applied. Note that we already have some locking inside message processing (e.g. when a date string is generated), but in general a real lock is never taken, so this is not causing problems. Even then, we could improve the situation by spinlocks or other very light mechanisms. > > 2. with modern CPUs you really want to only work in the cache if you > can. > Any access to additional memory will stall the CPU long enough that you > could have done a significant amount of processing instead. It's to the > point where the kernel developers have measured and said that if the > CPU > needs to copy the contents of a TCP packet, they can do the checksum > calculation at the same time and the overhead of the memory I/O will > make > the time taken for the calculateion to be a net zero additional time > (the > CPU internally processes things in parallel) yes, and this was actually one of the performance improvements done to the output part. But it really depends on the question if memory is reused or not. Think about writing the same message to more than one file (a common scenario). Without the ability to store anything inside the message object, we actually need to generate the string twice. That means we need to write it twice to memory, and this means that the cache cannot be re-used. If we now have a capability to store the generated string together with the message, we could re-use it on the second invokation. Most probably, it is still available inside the cache. So we do not only not need to generate it, we can also access it at highest speed directly from the CPU cache. That should be a big saving. But it comes at some expense, and that is maintaining the property inside the message, which involves at least some writes (and maybe rebalancing operations, based on data structure used). So it would probably best to make persisting the string optional so that it will only be done when there is value in it. [...] > well, rather than creating an entire interface, what about creating a > patch to hard-code TraditionalFormat and TraditionalForwardFormat (or > pick > a couple) and we can benchmark the system with the hard-coded C formats > vs > the current process. good idea -- and as you have seen, already done :) Rainer _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
