A syslog processor would be useful for log aggregation. I'm pretty sure that log4j, etc. have native syslog appenders.
-Joey On Sat, Apr 25, 2015 at 12:13 Bryan Bende <bbe...@gmail.com> wrote: > Joe, > > Thanks for the background on ListenUDP. > > The use case I was thinking of was log aggregation... most logging > frameworks like logback, log4j, etc., have a UDP appender, and they also > generally have a json format/layout that conforms with the "logstash" > format. I was thinking it would be cool to be able to use NiFi as an > alternative to logstash, flume, and whatever other technologies are being > used to get logs into a central location. There are obviously other options > besides udp, but it seemed easy and well supported. > > Maybe a property on the processor could control whether or not it buffered > datagrams vs producing a new FlowFile for each datagram? > > -Bryan > > > > On Fri, Apr 24, 2015 at 8:45 PM, Joe Witt <joe.w...@gmail.com> wrote: > > > Mike Moser: Great thinking! > > > > Bryan > > > > Taken from listen udp docs: "This processor listens for Datagram > > Packets on a given port and concatenates the contents of those packets > > together generating flow files roughly as often as the internal buffer > > fills up or until no more data is currently available." > > > > Quite honestly when this processor was originally built NiFi didn't > > have the ability to do the sort of fancy 'slab allocation' mechanism > > it supports today when generating a stream of flow files. So we could > > probably pretty easily reimplement this to behave more like you were > > thinking it should. But it is probably worth a bit of > > discussion/exploration to see what makes sense. The case we built it > > for was data arriving in UDP packets and it was structured in such a > > way that simple binary concatenation was sufficient because the data > > was inherently demarcatable/stream processing friendly. We could, > > however, implement it now such that each UDP datagram becomes a flow > > file. But not sure that makes sense either. This is sort of the > > inherent challenge of providing a raw socket listener. If the 'thing' > > being exchanged is not clear then we're not sure what the boundary of > > a given flow file should be. > > > > I'll stop rambling: Please if you would describe the use case a bit > > more we can think about whether providing a mode of 'datagram = > > flowfile' makes sense. > > > > Thanks! > > Joe > > > > On Fri, Apr 24, 2015 at 7:44 PM, Bryan Bende <bbe...@gmail.com> wrote: > > > Thanks for the suggestions... looks like it is in fact coming out of > > > ListenUDP like that. I'll try to figure out if this is expected > behavior, > > > or possibly something with how the messages are being sent. > > > > > > Sorry for the false alarm about MergeContent. > > > > > > On Fri, Apr 24, 2015 at 9:48 AM, Michael Moser <moser...@gmail.com> > > wrote: > > > > > >> At first glance, I would suspect ListenUDP is placing more than one > UDP > > >> datagram into one flowfile. It might be worth spending some time > > checking > > >> if that can happen. > > >> > > >> -- Mike > > >> > > >> > > >> On Thu, Apr 23, 2015 at 9:35 PM, Joe Witt <joe.w...@gmail.com> wrote: > > >> > > >> > Are you sure you're not sending the [ , ] over UDP as well ;-) > > >> > > > >> > Can you create a template of your flow and send it over? Perhaps > just > > >> > attach to a JIRA for this. MergeContent is a powerful and useful > > >> > thing so if you're seeing funky behavior we want to sort it out > > >> > quickly. > > >> > > > >> > On Thu, Apr 23, 2015 at 8:47 PM, Bryan Bende <bbe...@gmail.com> > > wrote: > > >> > > I'm trying to use MergeContent to merge json documents. I have the > > >> > Header. > > >> > > Demarcator, and Footer properties pointing to files with [ , ] > > >> > > respectively. I left all other properties the same, and set Max > > Entries > > >> > to > > >> > > 5 and Max Bin Age to 10 seconds. > > >> > > > > >> > > I have a simple flow with ListenUDP -> MergeContent -> > > >> > PutSolrContentStream > > >> > > (from the pull request). If I send a bunch of json documents over > > UDP, > > >> > most > > >> > > of them will merge correctly, but I'll see a couple where the > > >> demarcator > > >> > > didn't get inserted between two json documents. > > >> > > > > >> > > Any thoughts as to why this would happen? > > >> > > > > >> > > I added a significant amount of logging to the > > >> getDescriptorFileContent() > > >> > > method in MergeContent to see if there was a reason why it would > > return > > >> > > null for the demarcator, but nothing obvious is really jumping out > > at > > >> me. > > >> > > > >> > > >
