We should have an ASF location. I share Uwe's view that Wiki is not the ideal place. But it is our current place.
We'll need someone to push that along. On Thu, Mar 24, 2016 at 12:51 PM, Simon Ball <[email protected]> wrote: > There is a collection of templates for common patterns at > https://github.com/hortonworks-gallery/nifi-templates > > Feel free to submit any interesting use cases as PRs to that GitHub. > > Simon > > > On 24 Mar 2016, at 17:50, Uwe Geercken <[email protected]> wrote: > > Joe > > yes I know this one. but with obviously so much interest in nifi there must > be lots of good ideas around. > > can we not collect templates on github somewhere? > > rgds > > uwe > -- > Diese Nachricht wurde von meinem Android Mobiltelefon mit WEB.DE Mail > gesendet. > Am 24.03.2016, 18:03, Joe Witt <[email protected]> schrieb: >> >> Totally agree with you. >> >> How about his for now >> >> https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates >> >> And we can come up with something better (like the registry) in the >> future. >> >> Thanks >> Joe >> >> On Thu, Mar 24, 2016 at 11:00 AM, Uwe Geercken <[email protected]> >> wrote: >> > I believe it would be cool to have a central place where we can store >> > and >> > download templates. today so many people do cool stuff but its all >> > decentralized. >> > >> > maybe it is just a matter of deciding where >> > >> > uwe >> > -- >> > Diese Nachricht wurde von meinem Android Mobiltelefon mit WEB.DE Mail >> > gesendet. >> > Am 23.03.2016, 23:57, Russell Bateman >> > <[email protected]> schrieb: >> >> >> >> Hope not to make copy/paste mistakes. I can share the code snippets, >> >> but >> >> unfortunately not a trace. I didn't keep one and my efforts to >> >> restore >> >> the broken code only leave me frustrated. (I've obliterated my NAR >> >> several times, but the debugger tells me that the lines don't >> >> match up >> >> when I step through, can't set breakpoints, etc. Sorry.) >> >> >> >> >> How I was doing it: >> >> >> >> public void onTrigger( final ProcessSession session, final >> >> DataExtractor filter, final String regexPattern ) >> >> throws ProcessException >> >> { >> >> final FlowFile flowfile = session.get(); >> >> * final AtomicReference< FlowFile > flowFileHolder = new >> >> AtomicReference<>();** >> >> ** flowFileHolder.set( flowfile );* >> >> >> >> session.read( flowfile, new InputStreamCallback() >> >> { >> >> @Override >> >> public void process( InputStream in ) throws IOException >> >> { >> >> RegularExpressionMatch match; >> >> >> >> try >> >> { >> >> // slurp the in-coming stream to match into... >> >> Matcher matcher = Pattern.compile( regexPattern ).matcher( >> >> StreamUtilities.slurp( in, -1 ) ); >> >> if( !matcher.find() ) >> >> throw new IOException( "Failed to match regular expression >> >> pattern in the document" ); >> >> match = new RegularExpressionMatch( matcher ); >> >> } >> >> catch( Exception e ) >> >> { >> >> throw new IOException( "Failed to create regular expression >> >> matcher with passed properties" ); >> >> } >> >> >> >> Tag tag = filter.buildTag( match ); >> >> String xml; >> >> xml = ( Utilities.isEmpty( tag ) ) >> >> ? "(No tag built for this document.)" >> >> : new TagUtilities( tag, true /*properties.debug*/ >> >> ).getXmlFromTag( 0 ); >> >> * try** >> >> ** {** >> >> ** FlowFile flowFileWithAttributes = flowFileHolder.get();** >> >> ** flowFileWithAttributes = session.putAttribute( >> >> flowFileWithAttributes, "concept", xml );** >> >> ** flowFileHolder.set( flowFileWithAttributes );** >> >> } >> >> catch( Throwable e ) >> >> { >> >> -----> e.printStackTrace(); // IllegalStateException >> >> } >> >> * } >> >> } ); >> >> >> >> // the flowfile now has attributes we put on it... >> >> session.transfer( flowFileHolder.get(), new >> >> ProcessorRelationships().getSuccess() ); >> >> } >> >> >> >> How I'm doing it now: >> >> >> >> >> public void onTrigger( final ProcessSession session, final >> >> DataExtractor filter, final String regexPattern ) >> >> throws ProcessException >> >> { >> >> FlowFile flowfile = session.get(); >> >> * final String attribute;* >> >> * final AtomicReference< String > attributeHolder = new >> >> AtomicReference<>();* >> >> >> >> session.read( flowfile, new InputStreamCallback() >> >> { >> >> @Override >> >> public void process( InputStream in ) throws IOException >> >> { >> >> RegularExpressionMatch match; >> >> >> >> try >> >> { >> >> // slurp the in-coming stream to match into... >> >> String incoming = StreamUtilities.slurp( in, -1 ); >> >> Matcher matcher = Pattern.compile( regexPattern ).matcher( >> >> incoming ); >> >> if( !matcher.find() ) >> >> throw new IOException( "Failed to match regular >> >> expression pattern in the document" ); >> >> match = new RegularExpressionMatch( matcher ); >> >> } >> >> catch( Exception e ) >> >> { >> >> throw new IOException( "Failed to create regular expression >> >> matcher with passed properties" ); >> >> } >> >> >> >> Tag tag = filter.buildTag( match ); >> >> String xml; >> >> xml = ( Utilities.isEmpty( tag ) ) >> >> ? "(No tag built for this document.)" >> >> : new TagUtilities( tag, true /*properties.debug*/ >> >> ).getXmlFromTag( 0 ); >> >> * attributeHolder.set( xml );* >> >> } >> >> } ); >> >> >> >> /* Can't change anything about the session right inside here: >> >> don't change flowfiles >> >> * in the callback! Set the attribute here instead of inside the >> >> read! >> >> */ >> >> * attribute = attributeHolder.get();* >> >> * flowfile = session.putAttribute( flowfile, "concept", attribute );* >> >> >> >> // the flowfile now has attributes we put on it... >> >> session.transfer( flowfile, new >> >> ProcessorRelationships().getSuccess() ); >> >> } >> >> >> >> On 03/23/2016 03:42 PM, Oleg Zhurakousky wrote: >> >> > Russell >> >> > >> >> > This doesn’t sound right. Would you care to share a code snippet on >> >> > how >> >> > you set the attribute as well as stack trace? >> >> > >> >> > Cheers >> >> > Oleg >> >> > >> >> >> On Mar 23, 2016, at 5:33 PM, Russell Bateman >> >> >> <[email protected]> wrote: >> >> >> >> >> >> I stumbled upon something peculiar in NiFi. I had been attaching an >> >> >> attribute to a flowfile in the session.read() call-back. The NiFi >> >> >> unit >> >> >> testing framework tolerated this, but when I finally ran my >> >> >> processor for >> >> >> real, it blew chunks (IllegalStateException). I solved the problem >> >> >> by saving >> >> >> the new attribute and attaching it instead outside the call-back >> >> >> just before >> >> >> calling session.transfer(). >> >> >> >> >> >> The only reason I'm pointing this out is because I somewhat >> >> >> rely on >> >> >> unit testing and hope to catch this level of error/gotcha earlier >> >> >> than >> >> >> button-pushing testing >> >> >> >> >> >> --in case anyone cares. >> >> >> >> >> >> Open to comments, cat-calls, etc. >> >> >> >> >> >> ;-) >> >> >> >> >> >> Best regards, >> >> >> >> >> >> Russ
