Hello Andre, Wanted to follow up with you and see how things are developing here and offer any assistance.
Thanks Joe On Mon, Nov 30, 2015 at 11:09 AM, Andre <[email protected]> wrote: > Hi Joe, > > Damn! I should have suspected it... :-( > > Anyhow, the code is starting to take shape here : > > https://github.com/trixpan/nifi-lumberjack-bundle > (I will send a PR when I get closer to complete) > > I believe I've cleaned most of the netty dependencies, except for the > issues I faced trying to get TLS to work (netty is so much easier for > this sort of things! ), so meanwhile I am simply using SOCAT to play > the TLS termination point until I can get my head around. > > The code heavily follows the ListenSyslog pattern (don't pay attention > to the mess, I will clean it later) but still doesn't inject > flowfiles. So far I have been able to receive the initial window size > packets, however, no traffic is processed yet (I suspect it is a > problem with the framedecoder switch statement but will have to > confirm tomorrow morning. > > Cheers > > > On Mon, Nov 30, 2015 at 1:45 PM, Joe Witt <[email protected]> wrote: >> Hello >> >> I think it is fair to say that building listeners, where NiFi is >> acting as a recipient of data being pushed to it, is among the harder >> extension points to build. I also think that the outline of the >> general pattern is fair at a high level but that a good API to make >> that more repeatable generically is pretty hard for the reasons Tony >> mentions. >> >> That said there are things we can do and should do to just keep making >> such things a little bit easier over time. We're happy to help you >> walk through NIFI-856 and if we find things to chip away at >> generically then let's identify them and work them off. >> >> Thanks >> Joe >> >> On Sun, Nov 29, 2015 at 9:37 PM, Tony Kurc <[email protected]> wrote: >>> Many of these use blockingqueues to pass between the "listener" and the >>> "workers". I do think some library support for this would be a good idea, >>> to help provide thread safe publication and make exception handling a >>> little easier. I think providing any of the networking support may be >>> tricky because of the diversity of client apis and libraries. But a nice >>> interprocessor pub/sub api seems like a good move. I noticed Joe Witt >>> mentioned adding a little delay on poll in his listensyslog, I mentioned >>> how coverity and findbugs complain about ignoring the return code of >>> offer... Can you put a ticket in for this? I wouldn't mind working it. >>> On Nov 29, 2015 9:01 PM, "Andre" <[email protected]> wrote: >>> >>>> Hi, >>>> >>>> I am trying to give it a go on NIFI-856 (I'm not a coder either but >>>> decided to take the challenge). >>>> >>>> As I try to get my head around it I've noticed that coding "Listeners" >>>> for NiFi is sort of a nightmare. (please take no offense, it is just a >>>> sincere unskilled code statement). >>>> >>>> I've looked at ListenHTTP, ListenUDP and ListenSyslog for inspiration, >>>> I got the impression they all seem to have a need to tackle the >>>> transmission of messages between the Listener "daemon" and the >>>> onTrigger method. (e.g. syslogEvents within ListenSyslog). >>>> >>>> Is this understanding correct? >>>> >>>> If so, Shouldn't the framework contain something that makes it easier >>>> to write this sort of processor instead of having to recreate them? >>>> >>>> I say that because nearly every Listen processors will: >>>> >>>> 1. Bind to a socket; >>>> 2. Receive messages; >>>> 3. Read messages; >>>> 4. Add them flowfile; >>>> 5. Acknowledge receipt; >>>> >>>> Where 4 and 5 are usually applicable to some protocols (like Logstash, >>>> RELP, RLTP, etc). >>>> >>>> Keen to hear your thoughts >>>> >>>> Andre >>>>
