For almost all of the parameters to the module, they _must_ vary by message. The only exceptions are things like TLS settings, or the remote host endpoint. Everything else is structured data about an event that happened elsewhere. Most fields can be omitted if there's no parameter set - it's unusual that we set a description on a metric for example. Really we only require host/metric/service - I think we should error if you try to send an event that doesn't contain these three fields at least.
I'm absolutely happy with a json blob for setting custom fields; you're right to question their flexibility - they're just string key/value pairs appended to the end of the protobuf message, so a json blob is perfect. Thanks for the second opinion. I prefer the structured approach anyway. On Fri, 2 Dec 2016 at 07:50 David Lang <da...@lang.hm> wrote: > On Fri, 2 Dec 2016, Bob Gregory wrote: > > > Evening all, > > > > I've mostly finished my last personal project, so my thoughts are turning > > to omriemann. > > > > I'm trying to work out how we might configure the module. Riemann > requires > > that we send a protobuf encoded message containing a few pre-set fields, > > plus whatever additional fields we feel like forwarding. > > > > host: localhost > > service: cpu-load-average/1m > > state: ok > > time: 1480661786 > > description: "everything is perfectly fine" > > tags: ["laptop", "personal"] > > metric: 0.58 > > ttl: 120 > > my-custom-field: 27 > > > > This makes it unusual for an rsyslog module: usually rsyslog is happy to > > ship arbitrary strings to a destination and only cares about the > _framing_ > > of your data: omelasticsearch, ommysql, omkafka, omrelp etc. all accept > > some number of static parameters, plus a free-form template for the > actual > > message. > > > > Omriemann, in order to be useful, will need to impose some structure on > the > > message itself. > > > > How do people think we should configure the module so that people have > > flexibility over the host, metric value, metric name, and tags on a > > per-message basis? > > use a parameter to pass the variable name to use for the field, and have a > default if they aren't set. > > Also, think hard about the need to set them on a per-message basis. > > > I guess the simplest thing that could possibly work is defining a simple > > message format, eg. `host=foo; metric_f=0.6; > > service=rsyslog.impstats/utime; timestamp=1480661786` that messages need > to > > conform to. We can then parse out the key/value pairs in the module and > > encode them to protobuf. > > no, that way lies madness (I did something very similar in the first > iteration > of omudpspoof, but in my defense that was before we had the action() cal) > > > Alternatively, we could set up the structure of the message in the config > > itself, like this: > > > > action( > > type="omriemann" > > host="$hostname" > > metric="$!metric.value" > > service="$!metric.name") > > > > That seems more user-friendly, but rules out using custom fields. I guess > > I'd have to create a new template per-field during module begin. > > this is the right approach for the fixed fields. For defining custom > fields, can > you accept a JSON structure and do the right thing? > > Given that the protobuf needs to be pre-defined and exist on both sides, > how > much flexibility do you really have? > > > On a related note, I think I remember seeing some discussion of > conversion > > functions recently. Some of the fields need to valid integers, floats, > unix > > timestamps etc. What's the best way of parsing those out? > > you will be passed strings [1] and need to validate them (and figure out > what to > do if you are passed garbage) > > [1] timestamps are a possible exception to this. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
