Re: End of stream?

Sounds very promising, thank you!! I'll share what I find out :) Are there other group-related use cases? Maybe some non-incremental statistical measures? Regards, Matt Sent from my iPhone > On Nov 6, 2015, at 9:48 PM, Michael Moser wrote: > > Matt, > > There is the

Re: WebAppContext Failed startup of context

? Sent from my iPhone > On Dec 6, 2015, at 5:49 PM, Tony Kurc <trk...@gmail.com> wrote: > > @Matt, I'm compiling now too > >> On Sun, Dec 6, 2015 at 5:46 PM, Matt Burgess <mattyb...@gmail.com> wrote: >> >> Yes sir. I think they changed the way constructo

Re: WebAppContext Failed startup of context

n: "2.6.32-573.8.1.el6.x86_64", arch: "amd64", >>>> family: "unix" >>>> >>>> >>>> $ ps aux | grep mvn >>>> /etc/alternatives/java_sdk_1.8.0//bin/java -classpath >>>> /usr/local/apache-maven-3.3.

Re: WebAppContext Failed startup of context

Sun, Dec 6, 2015 at 5:31 PM, Matt Burgess <mattyb...@gmail.com> wrote: >> >> I think I've seen that with Java 7 compiled code (not compiled with 8 >> using 7 target) loaded on a Java 8 VM. >> >> Sent from my iPhone >> >>> On Dec 6, 2015, at 4:58

Re: WebAppContext Failed startup of context

I think I've seen that with Java 7 compiled code (not compiled with 8 using 7 target) loaded on a Java 8 VM. Sent from my iPhone > On Dec 6, 2015, at 4:58 PM, Andre wrote: > > Matt, > > $ mvn -version > Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; >

Re: MergeContent processor - set demarcator as new line

I'm thinking a PutElasticsearch processor to do bulk uploads of 1+ flow files (the batch size configurable), and another Get/Search Elasticsearch to use the search capability. Do you picture others, and/or more advanced options like specifying the index and creating it if it does not exist?

Re: MergeContent processor - set demarcator as new line

Speaking of Elasticsearch, are you using InvokeHttp for the bulk insert, or something else? There is a Jira case (NIFI-1275) to add Elasticsearch processor(s) to NiFi, we welcome all thoughts and comments there on how to make things easier. Regards, Matt Sent from my iPhone > On Dec 13,

Re: Groovy unit tests

+1, Groovy is a great way to rapidly generate tests, remove boilerplate, and enable powerful test frameworks like Spock. Sent from my iPhone > On Jan 4, 2016, at 7:29 PM, Andy LoPresto wrote: > > I am considering writing unit tests in for new development/regression

Re: Syslog Classes

Hadoop uses this approach to mark API methods as Stable/Unstable/etc. I think the rule of thumb is that Major.Minor.Patch versions correspond to Breaking.Feature.Fix Sent from my iPhone > On Jan 5, 2016, at 7:01 PM, Tony Kurc wrote: > > I'm trying to read code on my phone,

Re: Syslog Classes

Roger that. This is what Hadoop does, for an API method (class, etc.) in Java it is annotated as @Stable or @Unstable. I was just referring to the semantics of when you might expect an @Unstable method to change, for example. Or am I still misunderstanding what you mean? Regards, Matt Sent

Re: Testing handling of static class methods

You could move the one static call into an instance method of PutJMS, and use Mockito.spy() to get a partial mock of the processor, then use when() to override the instance method in the test. Not sure if that's how it's done in other places but it's worked for me in the past. Regards, Matt

Re: Limiting a queue

Shaine, I was about to start work on processor(s) almost exactly like the ones you are describing. Is this something you'd be interested in collaborating on? I've worked on the SQL generation piece before (though not for NiFi). Regards, Matt On Wed, Jun 8, 2016 at 5:29 PM, Shaine Berube

Re: Limiting a queue

any database being > in source and any database being in target. > > As far as collaboration, what files should I send and where? > > On Thu, Jun 9, 2016 at 7:58 AM, Matt Burgess <mattyb...@gmail.com> wrote: > >> Shaine, >> >> I was about to start

Re: Consuming web services through NiFi

saikrishnat, There are multiple processors that can get data from public web services, such as InvokeHttp [1]. There are sample templates at [2], including "Working_With_CSV.xml" which consumes a RESTful web service at http://randomuser.me. If you are looking to discover/invoke WSDL/SOAP

Re: Apache NiFi 0.7.0 Release date ?

en around for a while and just never knew to keep these things > in mind. Now, if these are things I did ... well then ... please > don't send those examples :-) > > Thanks > Joe > > On Tue, Jun 7, 2016 at 9:50 AM, Matt Burgess <mattyb...@gmail.com> wrote: >> +1 for k

Re: Apache NiFi 0.7.0 Release date ?

I merged an HL7 PR last night, I will take a look. The tests passed for me but maybe I screwed something up with the patch application. On Tue, Jun 7, 2016 at 10:00 AM, Andre wrote: > + 1 on the build errors on HL7. > > Seems to break things in here as well. > > Cheers > >

Re: Apache NiFi 0.7.0 Release date ?

+1 for keeping JIRA accurate, I'll take a look too. On Tue, Jun 7, 2016 at 9:48 AM, Michael Moser wrote: > All, > > I'm noticing a few JIRA tickets in the Resolved state, with commits against > them, but the Fix Version is none. I believe JIRA drives the release > notes, so

Re: Apache NiFi 0.7.0 Release date ?

There was a whitespace issue in the test HL7 file. Joey moved the content into the test class itself and removed the file. I reviewed and merged to 0.x and master, so things should be back to normal. Sorry for the hiccup. Regards, Matt On Tue, Jun 7, 2016 at 10:03 AM, Matt Burgess <mat

Re: Consuming web services through NiFi

> URL..but not thru NiFi.. > > do i have to change any settings.?? > > Regards,Sai > > On Thu, Jun 9, 2016 at 3:02 PM, Matt Burgess [via Apache NiFi Developer > List] <ml-node+s39713n11300...@n7.nabble.com> wrote: > >> Which other sites did you try? I've notic

Re: Consuming web services through NiFi

tting same error.. > > any idea.?? > > On Tue, Jun 7, 2016 at 4:55 PM, Matt Burgess [via Apache NiFi Developer > List] <ml-node+s39713n11201...@n7.nabble.com> wrote: > >> saikrishnat, >> >> There are multiple processors that can get data from p

NiFi Script Tester v1.1.1 available

All, Just wanted to let you know that my NiFi Script Tester has been updated, turned out the original version didn't work so well :-P I've fixed the bugs I found and added some helpful things (such as Apache Commons IO to the fat JAR, for IOUtils.toString()) and the "-module" option to add

Re: [DISCUSS] Closing in on the Apache NiFi 0.7.0 Release

ee addressing Mark Payne's comments. > Needs another round of review > > - "Allow user to specify file filter regex when unpacking zip/tar archives" > NIFI-1586[3] [status] Ricky Saltzer created a PR that addressed a couple > comments and is rebased (request by Matt Burge

Re: [DISCUSS] - Markdown option for documentation artifacts

+1 with template On Wed, Jun 8, 2016 at 10:39 AM, dan bress wrote: > +1 > > On Wed, Jun 8, 2016 at 7:05 AM Andre wrote: > >> +1 on this + a template that matches existing additional.html >> On 8 Jun 2016 04:28, "Bryan Rosander"

Re: Use VFS with (Put|List|.*)HDFS

As a former Pentaho employee, I can add some details around this: - Pentaho does/did have a fork of Apache VFS. Mostly it was the application of bugs in the 2.x line against the 1.x codebase. Since then they have contributed fixes from their branch back to the 2.x fork, and when I left they were

Re: [DISCUSS] Closing in on the Apache NiFi 0.7.0 Release

ache.org > alopresto.apa...@gmail.com > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > >> On Jun 13, 2016, at 3:26 PM, Matt Burgess <mattyb...@gmail.com> wrote: >> >> I'll take the PutSlack PR and Ricky's unpack PR, and if there's a

Re: NIFI-2026: Opinion needed about adding non-Apache repos

Your hadoop.version could also be set by this profile, right? And conceivably hive.version and hbase.version (et al) if all has been set up correctly (and we can change if not, as we should be setting all dependency versions from properties). Do you need the java.arg.15 for this? I think it's

Re: convert syslog into .json format?

PutElasticsearch currently requires you have the id for the document in a flowfile attribute. It does not yet support auto-increment, but you can emulate that with an UpdateAttribute processor before PutElasticsearch, set something like "es.docid" to ${nextInt()}, then in PutElasticsearch the

Re: java.lang.UnsatisfiedLinkError in PutHDFS with snappy compression.

To add to Jeremy's last point, even after the library is present, the files must be greater than the HDFS block size (default is 64 MB I think?) or Hadoop-snappy will also not compress them. Sent from my iPhone > On Feb 6, 2016, at 5:41 PM, Jeremy Dyer wrote: > > Shweta, >

Re: Are there Travis CI interface issues?

GitHub was down at that time, probably prevented the Travis webhook from updating the status. Sent from my iPhone > On Jan 28, 2016, at 7:44 AM, Joe Skora wrote: > > *Is there a problem with GitHub server pulling status from the Travis CI > build server?* > > I entered pull

Re: Auto-Organize Layout

+1 for "snap to grid" feature Sent from my iPhone > On Jan 19, 2016, at 4:20 PM, dan bress wrote: > > Maybe not exactly "auto-layout" but I would back a notion of having the > components snap to a coarser grain grid than what we currently have. > Sometimes I care a lot

Re: [VOTE] Release Apache NiFi 0.5.0 (RC3)

Ran the helper, verified the artifacts look good, ran with some scripting processors. +1 (non-binding) On Fri, Feb 12, 2016 at 6:27 PM, Tony Kurc wrote: > Hello > I am pleased to be calling this vote for the source release of Apache NiFi > nifi-0.5.0. > > The source zip,

Re: Accessing the name of a Processor programatically

I think Aldrin's points about being able to use properties for the UI are pretty cool, but I also think there's room for the inverse (which is what I think the OP meant). The name of the processor is tied to its instance (though not guaranteed unique between instances, and definitely not unique

Re: Scripted controllers

Sumo et al, I thought about scripted Controller Services when I added NIFI-1458 for scripting reporting tasks. The thing I ran into was that Processors and Reporting Tasks are technically "consumers", at least in the sense that no other NiFi "object" knows about their interfaces, innards, etc.

Re: Process Creation

Josh, Happy to help, could you provide some more details around the use case? For example the JSON content (scrubbed if need be of course) and a stack trace or other logging/bulletin text about the error? Regarding PutFile to view output, an alternative is to use a LogAttribute processor with

Re: [ANNOUNCE] New Apache NiFi Committer Oleg Zhurakousky

Congratulations! Well deserved. > On Mar 15, 2016, at 8:17 PM, Tony Kurc wrote: > > On behalf of the Apache NiFI PMC, I am very pleased to announce that Oleg > Zhurakousky has accepted the PMC's invitation to become a committer on the > Apache NiFi project. We greatly

Re: Cross NAR Controller Services

Devin, DBCPConnectionPool is an implementation of the DBCPService interface. The DBCPService interface is in the nifi-dbcp-service-api and is the way you'd interact with the instance(s), which currently happen to be DBCPConnectionPool objects. If you implement your own class that implements

Re: [DISCUSS] - Class loader isolation per instance of the component

Why not make UGI/Kerb stuff a Controller Service? Each CS instance can have class loader isolation and can be shared among processor instances that want to share UGI/Kerb/service config? Sent from my iPhone > On Mar 9, 2016, at 8:50 AM, Oleg Zhurakousky > wrote:

Re: Establishment of MiNiFi repo and supporting tools

I'm a +1 as well. > On Mar 11, 2016, at 8:26 AM, Tony Kurc wrote: > > Should this be marked as a [VOTE]? In any case, I'm +1 > >> On Fri, Mar 11, 2016 at 1:11 AM, Aldrin Piri wrote: >> >> All, >> >> Didn't explicitly mention this, but was aiming for a

Re: [DISCUSS] - Class loader isolation per instance of the component

need a mechanism to provide CL isolation per instance > when it’s due. > > Oleg > > > On Mar 9, 2016, at 8:54 AM, Matt Burgess <mattyb...@gmail.com> wrote: > > > > Why not make UGI/Kerb stuff a Controller Service? Each CS instance can > have class loader isolatio

Re: [DRAFT][REPORT] Apache NiFi - April 2016

y strong. >> >> ## PMC changes: >> >> - Currently 13 PMC members. >> - No new PMC members in the last three months. >> - Last PMC addition was Sean Busbey on Wed Nov 25 2015 >> >> ## Committer base changes: >> >> - Currently 21 committers.

Re: [VOTE] Establish Apache MiNiFi, a subproject of Apache NIFi

+1 > On Apr 9, 2016, at 1:13 AM, Joe Witt wrote: > > Team, > > Following a solid discussion [1] in early January regarding > establishment of MiNiFi as a subproject of NiFi I'd like to call a > formal vote to record this important community decision and > established

Re: Re: Re: Re: Re: Re: Processor additional documentation

; > artifact org.apache.nifi:nifi-nar-bundles:pom:0.6.0-SNAPSHOT and > > 'parent.relativePath' points at wrong local POM @ line 19, column 13 -> > > [Help 2] > > > > > > Any quick advice before I start diggin in? > > > > Greetings Uwe > > > &

Re: [VOTE] Release Apache NiFi 0.6.0 (RC1)

e: > Thanks to Aldrin and Matt Burgess, we were able to push a new signature > for each artifact to the repository and verify it. Please resume release > verification. > > hw12203:/Users/alopresto/Workspace/scratch/release_verification/0.6.0 > alopresto >  0s @ 20:01:08 $ rmf nifi-0.

Re: [VOTE] Release Apache NiFi 0.6.0 (RC1)

I'm getting an error on Aldrin's signature during gpg --verify: gpg: BAD signature from "Aldrin Piri (Code Signing Key) " [unknown] Is anyone else seeing this? Thanks, Matt On Mon, Mar 21, 2016 at 4:37 PM, Aldrin Piri wrote: > Hello, > > I am pleased

Re: Closing in on the Apache NiFi 0.6.0 release

All, We found an issue with the PutElasticsearch processor as well, it doesn't evaluate EL expressions using the flow file attributes for the Index and Document Type properties. This is a simple fix, I've written it up as NIFI-1666 [1] and will have a PR very shortly. I'd like to get this into

Re: NiFi-1660

Chris, Great to hear! For the following steps, I'm assuming it will be a one-argument function called "jsonPath" that takes a String containing a JSON Path expression to be evaluated on the "subject" of the function. So I am picturing it used like: ${

Re: [VOTE] Release Apache NiFi 0.6.0 (RC2)

+1 (non-binding) Ran through all build verification steps successfully, tested locally with various templates, processors, and controller services. On Thu, Mar 24, 2016 at 12:46 AM, Joe Witt wrote: > +1 (binding) > > Ran through all build verification steps and all checks

Re: Re: Re: Re: Processor additional documentation

Uwe, The additional details piece appears to be a result of your ".nar" file actually being more like a ".jar", rather than a bundle that includes a JAR which in turn includes your source code and docs. Since you did all the hard work with creating some useful processors, I took the liberty of

Re: Aw: Re: Re: Re: Re: Processor additional documentation

nything I have to specifically do with maven? And is there documentation > available? > > greetings and thanks for help. > > Uwe > >> Gesendet: Montag, 21. März 2016 um 14:04 Uhr >> Von: "Matt Burgess" <mattyb...@gmail.com> >> An: dev@nifi

Re: Error at startup

I can't see them, perhaps the attachment is being removed. Can you paste the text from the logs into the email? Thanks, Matt On Tue, Mar 22, 2016 at 10:10 AM, Pierre Villard < pierre.villard...@gmail.com> wrote: > Erf that's strange, I do see logs from my side. > Is it better? > > 2016-03-22

Re: ExecuteSQL and NiFi 0.5.1 - Error org.apache.avro.SchemaParseException: Empty name

That's on me, that commit went into 0.5.0 and looks like a negative logic error. I thought I had unit tested it but I guess not :( Sent from my iPhone > On Mar 5, 2016, at 6:57 PM, Bryan Bende wrote: > > I think this a legitimate bug that was introduced in 0.5.0. > > I

Re: ExecuteSQL and NiFi 0.5.1 - Error org.apache.avro.SchemaParseException: Empty name

Actually on second thought it's not negative logic, it should be checking against tableNameFromMeta. Sent from my iPhone > On Mar 5, 2016, at 6:57 PM, Bryan Bende wrote: > > I think this a legitimate bug that was introduced in 0.5.0. > > I created this ticket:

Re: Variable FlowFile Attributes Defined by FlowFile Content

This is an interesting idea, can you elaborate on what such a file would look like and how it would be used? Would it contain values to be used as attributes in ExtractText as well as the content from which to extract the values for these attributes? In general, I don't believe a property name

Re: Import Kafka messages into Titan

I'll let someone else have a go at question 2 :) If you're using ExecuteScript with Groovy, you don't need EvaluateJsonPath, Groovy has a JSONSlurper that works nicely (examples on my blog). To put directly into Titan you don't need to convert the format, instead you'll want Gremlin (part of

Re: [DISCUSS] create apache nifi 0.6.1

Sounds good to me. > On Mar 31, 2016, at 10:29 AM, Joe Witt wrote: > > Team, > > I propose that we do an Apache NiFi 0.6.1 release. There are a few > important findings we've made and put JIRAs/PRs in for over the past > couple of days. The most concerning is that I

Re: [ANNOUNCE] New Apache NiFi Committer Andy LoPresto

Congratulations Andy!! Well-deserved, looking forward to continued great work! > On Apr 2, 2016, at 11:39 AM, Tony Kurc wrote: > > On behalf of the Apache NiFI PMC, I am very pleased to announce that Andy > LoPresto has accepted the PMC's invitation to become a committer on

Re: [DISCUSS] git branching model

I like option 1 as well. In the case where a fix is to be put into both branches, will the developer be responsible for issuing 2 PRs / patches, one against each branch? This would help in the case that the PR/patch against 0.x won't merge cleanly into master; however the reviewer(s) would need

Re: Import Kafka messages into Titan

Idioma, There is not yet a JSON-to-JSON translator, although there is a Jira case to add it (https://issues.apache.org/jira/browse/NIFI-361) However you have a handful of options here: 1) If you don't have a specific output format in mind, use Simon's approach to generate a JSON file with the

Re: catch commit error in OnTrigger to diversify session behaviour

Upon a TitanException in your load() method, you could wrap it in an IOException and re-throw, and rather than print the stack trace, you can catch the IOException in onTrigger() and send to failure. You likely won't want to rollback the session if you're transferring to failure; the user can

Re: Slack for 'apachenifi'?

, Joe Witt <joe.w...@gmail.com> wrote: > Perfect. Thanks Matt. Will sync off list and transfer to PMC. > > Joe > On Apr 19, 2016 7:26 AM, "Matt Burgess" <mattyb...@gmail.com> wrote: > >> I made it (so we would have that team name when we wanted to get it &

Re: NiFi Rest API Start

It's part of the "Update Processor" API, check out this thread from the NiFi dev list archive: http://apache-nifi-developer-list.39713.n7.nabble.com/Reg-starting-and-Stopping-processor-tp7930p7949.html Regards, Matt On Mon, Apr 25, 2016 at 12:17 PM, dale.chang13

Re: Identifier Attribute for PutElasticSearch

Identifier Attribute is a little confusing and different, I struggled to decide whether to have an "Identifier" property or an "Identifier Attribute" property, and chose the latter because if you had an Identifier property and didn't use expression language, then the processor would immediately

Re: Nifi to Titan... How?

Pat, No worries, this discussion is relevant to the devs group :) I appreciate and share your interest in getting connected data into graph databases in order to get more insight out of the data. It may be possible to put the data directly into the backing store (Cassandra, e.g.) via NiFi, but

Re: [DISCUSS] nifi meetup/hackathon ground rules?

One thing that could be done to enable demos while still having PRs/patches go through the Apache process is to have the Organizer create a hackathon branch off their fork, and merge in any patches/PRs that are demo-able, then show the hackathon goodness at the end. Then the regular process

Re: [ANNOUNCE] New Apache NiFi Committer James Wing

Congrats and welcome James! On Wed, May 11, 2016 at 4:10 PM, Joe Witt wrote: > On behalf of the Apache NiFi PMC, I am pleased to announce that James > Wing has accepted the PMC's inviation to become a committer on the > Apache NiFi project. James has been an excellent

Re: [discuss] PropertyDescriptor name and displayName attributes

Same here. Internationalization is often implemented as properties files/resources, where you possibly load in a file based on the system setting for Locale (like processor_names_en_US.properties). If we were to do internationalization this way (i.e. a non-code based solution, which is more

Re: Seek nifi help

Welcome to NiFi, we are glad you are finding it useful. The ExecuteScript processor works with Jython (a Python interpreter for the Java Virtual Machine), not the native Python language. That means that native (C/C++) modules for Python will not be available, but modules written in pure Python

Re: [DISCUSS] Extraction of Extension API

+1 > On May 6, 2016, at 3:30 PM, Aldrin Piri wrote: > > All, > > I would like to propose a refactoring of the nifi-api for our master/1.0 > branch. In summary, a lightweight and concise view of this module allows > for reduced footprint of the NIFI API for components and

Re: Slack for 'apachenifi'?

I made it (so we would have that team name when we wanted to get it going). Not sure how to transfer it but I sent you an invite a while back. I can send invites to PMC members then hand over control or whatever needs to be done. > On Apr 18, 2016, at 11:39 PM, Joe Witt

Re: [VOTE] Incorporate SHA256 part of release process

+1 > On Apr 13, 2016, at 6:13 PM, Aldrin Piri wrote: > > This was mentioned in the vote thread for the RC2 release and wanted to > separate it out to keep the release messaging streamlined. As mentioned by > Andy, the MD5 and SHA1 are subject to collisions. From another

Re: [ANNOUNCE] New Apache NiFi Committer Pierre Villard

Congratulations Pierre, well deserved! Looking forward to more great contributions like you've already had :) Regards, Matt > On May 25, 2016, at 11:50 AM, Joe Witt wrote: > > On behalf of the Apache NiFi PMC, I am pleased to announce that Pierre > Villard has accepted the

Re: Apache Yetus annotations : clarifying public vs private api

agreed, big +1 here, will make things much more clear re: stability and extensibility. On Mon, Jul 25, 2016 at 7:43 PM, Tony Kurc wrote: > awesome! > > On Mon, Jul 25, 2016 at 7:32 PM, Joe Witt wrote: > >> Team >> >> For those interested I am going to take

Re: DistributedMapCacheClient Groovy example

okup = context.controllerServiceLookup >> def cacheServerName = distributedMapCacheServerName.value >> >> def cacheServerId = >> lookup.getControllerServiceIdentifiers(ControllerService).find { >> cs -> lookup.getControllerServiceName(cs) == cacheServerName >&g

Re: [ANNOUNCE] New Apache NiFi Committer Yolanda Davis

Congratulations and welcome!! > On Jul 30, 2016, at 8:16 AM, Tony Kurc wrote: > > On behalf of the Apache NiFI PMC, I am very pleased to announce that > Yolanda Davis has accepted the PMC's invitation to become a committer on > the Apache NiFi project. We greatly appreciate

Re: [DISCUSS] 1.0 Beta Release

+1 > On Aug 1, 2016, at 10:14 PM, Joe Percivall > wrote: > > Hey Team, > > The big ticket elements of 1.0 look to be in a pretty good state. Lots of > things left to tighten up, primarily docs and bug fixes, but, it seems wise > for a major release like this

Re: DB2 to MS SQL data transfer

Sumo, One-time migrations are possible with NiFi (although probably not a common use case). Are the target tables created already? Some DBs (like MySQL) support queries that return the Create Table statement, if you get those you can use PutSQL to execute them on your target system. If the

Re: DistributedMapCacheClient Groovy example

Sumo, I have some example code at http://funnifi.blogspot.com/2016/04/inspecting-your-nifi.html. Although it's not expressly for ExecuteScript it should be pretty usable as-is. Also you might be able to use the technique outlined in my other post:

Re: DistributedMapCacheClient Groovy example

ributed.cache.client.Serializer, >> org.apache.nifi.distributed.cache.client.Deserializer), wait(), any(), >> getAt(java.lang.String), every(): >> org.apache.nifi.processor.exception.ProcessException: >> javax.script.ScriptException: javax.script.ScriptException: >&g

Re: ExecuteScript and flowfiles

Dave, Both options are possible, though note the scripting processors use Jython not Python, so hopefully the scripts would work but might need a little refactor. The first option is probably best solved with the ExecuteScript processor, the second is probably best solved with

Re: Apache NiFi - Supported file formats/types

Gitendra, NiFi is technically a format-agnostic system, meaning it can support any type of content. Having said that, there are processors to make transformation/manipulation easier for various formats such as Avro, JSON, XML, CSV, SQL, and for others there are processors like ReplaceText and

Re: QueryDatabaseTable

That sounds like a bug, mind filing a Jira case? Thanks, Matt P.S. Not sure the where filter will work even after the fix but I will take a look. > On Jul 7, 2016, at 3:20 PM, Nabegh wrote: > > I'm trying to load a table from Netezza database using QueryDatabaseTable >

Re: [ANNOUNCE] New Apache NiFi PMC Member - Andy LoPresto

Congratulations Andy! Well deserved! > On Jul 7, 2016, at 12:44 PM, Joe Witt wrote: > > On behalf of the Apache NiFI PMC, I am very pleased to announce that > Andy LoPresto has accepted the PMC's invitation to join the Apache > NiFi PMC. We greatly appreciate all of Andy's

Re: [DISCUSS] MiNiFi C++ Initial Release - cpp-0.0.1

+1, also we need documentation on the S2S protocol so we can further extend the reach (Go, Python, Erlang/Elixir, etc) > On Aug 4, 2016, at 11:42 PM, Aldrin Piri wrote: > > Hey folks, > > There's been great efforts to get an initial working implementation of > MiNiFi in

Re: [ANNOUNCE] New Apache NiFi Committer Andre Fucs de Miranda

Congratulations and welcome aboard Andre! > On Aug 6, 2016, at 10:40 PM, Tony Kurc wrote: > > On behalf of the Apache NiFI PMC, I am very pleased to announce that Andre > Fucs de Miranda has accepted the PMC's invitation to become a committer on > the Apache NiFi project. We

Re: Externally update processor State

Peter/Joe, Updating an "Initial Max Value" property could set the state when the processor is started, so you wouldn't need to clear state unless your Initial Max Value is less than what is in state currently. Regards, Matt On Mon, Aug 15, 2016 at 4:16 PM, Peter Wicks (pwicks)

Re: IllegalStateException: FlowFile[1,71039521870109.mockFlowFile,0B] has already been transferred

Alessio, Without seeing your modifications, I am guessing that session.transfer() is being called multiple times for the same flow file. Can you share your onTrigger() code? If not, take a look to see if there is a loop or other path that could cause session.transfer() to be called on the same

Re: Thoughts on change to PutCassandraQL (nifi-cassandra-processors)

Michael, Thank you for your contribution! I took the liberty of writing up NIFI-3425 [1], and adding your patch file. I haven't had a chance to take a look yet but will review soon. Regards, Matt [1] https://issues.apache.org/jira/browse/NIFI-3425 On Wed, Feb 1, 2017 at 7:58 AM, Michael

Re: what to do

Bernie, Aldrin replied to your initial email yesterday (see below), did you subscribe to the dev list (by sending an email to dev-subscr...@nifi.apache.org)? If not then you may not receive responses (I've CC'd your email here this time). Regards, Matt Hi Bernie, Sorry you have had some

Re: Custom Processor

I answered on the users list but will copy here in case there are folks on dev but not users: Are your values Strings or numbers? Meaning does the JSON look like: { "a": "2.1234567891E10" } or { "a" : 2.1234567891E10 } If the latter, would the output field ("a" or "new_a" or whatever) have to

Re: [ANNOUNCE] New Apache NiFi Committer Jeff Storck

Congrats Jeff, well deserved. Welcome aboard! On Tue, Feb 21, 2017 at 3:05 PM, Rob Moran wrote: > Thanks and congrats, Jeff! > > Rob > > On Tue, Feb 21, 2017 at 3:01 PM, Pierre Villard > wrote: > >> Congrats ! >> >> 2017-02-21 20:58 GMT+01:00 Joe

Re: NiFi PutElasticsearch Processor

Shanka, The Fetch/PutElasticsearch processors are built to be part of the ES cluster, and IIRC Elasticsearch says that this should be compatible against dot releases for a particular major/minor version, so I think ours are built against 2.1.x. These might work with ES 2.2.0 but they do not

Re: Creating a custom DatabaseAdapter

Stanislav, This is not your fault, it is a current limitation due to the organization of the NiFi submodules with respect to the location of DatabaseAdapter. In general, when a NiFi interface is to be extensible, there is a submodule for the API JAR to be available to other NARs/submodules/etc.

Re: [DISCUSS] Scale-out/Object Storage - taming the diversity of processors

I agree with Andrew in the operations sense, and would like to add that the user experience around dynamic properties (and even "conditional" properties that are not dynamic but can be exposed when other properties are "Applied") can be less-than-ideal and IMHO should be used sparingly. Full

Re: NiFi PutElasticsearch Processor

I agree with Joe, it sounds like the flow file content is the plain-text extract of the PDF, and the ES processors are expecting JSON documents. You can use ReplaceText to wrap your content in a JSON object, with something like: { "content" : $1} (I didn't try that but the idea is just to put

Re: [VOTE] Establish Registry, a sub-project of Apache NiFi

+1 binding On Fri, Feb 10, 2017 at 11:40 AM, Bryan Bende wrote: > All, > > Following a solid discussion for the past few days [1] regarding the > establishment of Registry as a sub-project of Apache NiFi, I'd like to > call a formal vote to record this important community

Re: NIFI-329 - IRC Processors

I had looked around a while ago for IRC clients (was thinking of contributing IRC processors as my first contribution to NiFi, but got sidetracked), I agree that Kitteh seems like the best choice (others like Pircbotx, JerkLib, and Martyr don't have friendly licenses). They all seem to have one

Re: Regular Expression, or Similar Processor

to Matt's list: > > https://blogs.apache.org/nifi/entry/storing_syslog_events_in_hbase > > Thanks, > > Bryan > > On Tue, Jan 17, 2017 at 6:35 PM, Matt Burgess <mattyb...@apache.org> wrote: > >> Oziel, >> >> The ListenSyslog processor [1] will ap

Re: Regular Expression, or Similar Processor

Oziel, The ListenSyslog processor [1] will apply the regular expression and extract each of the fields into flow file attributes. From there you could use AttributesToJSON [2] with "Include Core Attributes" set to false, that should give you fields named "syslog.hostname" for example. You could

Re: User-supplied attributes for processors

Kirk, The processors have to explicitly know about dynamic properties (and their intent) in order to use them appropriately. For a processor like ListSFTP, it could be beneficial to have custom attributes (as parameters to the SFTP session perhaps?) but the domain knowledge on how they'd be used

Re: How can we validate a data flow.?

I've used a handful of techniques/scripts to get the data into an analysis tool: 1) NiFi REST API to get the Provenance data 2) Groovy script to transform into Apache Tinkerpop 3.x format (can provide if desired) 3) Gremlin script to get the Tinkerpop file into Neo4J (can't find it but probably a

  1   2   3   4   5   6   >