Dan I do recommend going the custom processor route. You could initially do that using InvokeScriptedProcessor to figure out any intracacies then turn it into a full blown proc later.
Thanks Joe On Sep 4, 2016 2:51 PM, "dgm" <[email protected]> wrote: > Hi Andre, > > > > Thanks for the reply. Agree with you. I’m not implementing a memory > resident AV approach…but rather an ingress “kill chain” as we ingest > data. We are looking at metascan, but have had some mixed results with it > in a couple of other systems it was used in. But for the time being, I > have other tools, e.g. clam, mcafee, Symantec, etc that I can use through > their API’s or command line tools. Because as you said, the goal is to get > a pass/fail assertion from the tool before allowing the content to be > ingested. > > > > So with that in mind, and as I’m trying to learn more about Nifi… I’m > wondering if existing processors would be good for this use case, or if I > need to pursue building a custom processor. > > > > I haven’t tried the stream command processor yet, but in reading about it, > seems like this would require me to use the command line on-demand scanning > of these tools. And that I may need to pursue writing a custom processor > to interface with external tools like clam, mcafee, metascan in order to > get low-latency, higher throughput with these tools. > > > > > > Thanks, > > Dan M > > > > *From: *Andre <[email protected]> > *Reply-To: *<[email protected]> > *Date: *Sunday, September 4, 2016 at 12:56 AM > *To: *<[email protected]> > *Subject: *Re: AV Scanning a Flowfile > > > > Dan, > > > > Using a memory resident AV would not provide you with certainty of the > result as the file would just fail to write or fail to read, you want > something that is telling you, this is bad|good. > > > > There used to be many AVs with Java APIs but nowadays they tend to be > reasonably rarer... Some people as you noted use a daemon based approach > like: > > > > http://nsinfra.blogspot.com.au/2011/06/java-api-to-detect- > virus-using-clamav.html > > > > Having said that, have you considered using an RESTful AV API such as > Metascan? It is a paid product but would fit your need. > > > > You can also use a combination of Hash processor and multi AV lookup > providers like Virustotal (RESTful) and Cymru (QueryDNS) but this would not > detect automatically generated malware variants. This is a technique widely > used to escape hashing, where the hostile party changes a single bit of the > malicious payload via padding or other approach, causing the resulting > md5/sha hash to be completely different from the hash of the same malware > delivered to another party. > > > > Worst case comes, you could use PutEmail followed by ListenSMTP but this > is nuclear waste level of dirty... :-) > > > > Cheers > > > > > > On Sun, Sep 4, 2016 at 10:41 AM, dgm <[email protected]> wrote: > > Hello – > > > > Has anyone had any experience integrating an AV Scanner with NiFi, such as > clamav or mcafee? Both support running as service listening to a port as > well command line options… > > > > I’m currently thinking of trying to use the ExecuteStreamCommand > processor, however, if using command line tools, the external app will have > to initialize for each invocation. This is where I think being able to use > an external service call/api might be significantly faster. > > > > Thanks, > > Dan M > > > > >
