Re: PutSolrContentStream Doc and DocID confusion

Hello, Just to clarify, do you have LogAttribute with the "Log Payload" property to true? If the actual output of LogAttribute is /[*docid*="95663ced-6a3b-4356-877a-7c5707c046e7_779"]/ then that is not a valid JSON document. You would need the content of the flow file to be something like the

Re: [DISCUSS] From Contributor to Committer

So are we saying as a community that a contributor has to first become a committer, and then only after continued consistent engagement could then be considered for PMC? I don't have any issue with that approach, although it is not exactly what I thought when we first created the two tiers. On

Re: Compression of Data in HDFS

Hi James, It looks like there may be a typo in what I wrote... I had $[path} but it should be ${path} Sorry about that, can you let us know if that worked. Thanks, Bryan On Thu, Apr 7, 2016 at 3:05 AM, jamesgreen wrote: > Hi Bryan > > I tried what you

Re: [VOTE] Release Apache NiFi 0.6.1 (RC1)

+1 (binding) Ran through release helper and everything checked out. Ran example flows and functioned as expected. On Wed, Apr 6, 2016 at 11:16 PM, Aldrin Piri wrote: > +1, binding > > Build and tests worked on OS X and Windows 7. Branch looked good on our > Travis build

Re: Compression of Data in HDFS

Ok one more question... On GetHDFS are you setting the Directory to "\landing\databasename\prodeiw_arc\" and then setting Recurse Sub-Directories to true to have it go into each table's directory? The reason I ask is because the FlowFiles coming out of GetHDFS have an attribute on them called

Re: Compression of Data in HDFS

Hello, Can you describe your flow a bit more? Are you using ListHDFS + FetchHDFS to retrieve the data from HDFS? What value do you have for the Directory property in PutHDFS? Thanks, Bryan On Wed, Apr 6, 2016 at 7:12 AM, jamesgreen wrote: > I am trying to

Re: How to pass variables to nifi Processor

Hello, For properties that support expression language, they can reference properties passed in through the bootstrap.conf file. So you could define a property in bootstrap.conf like -Dfoo=abc and in a processor property reference that as $[foo}. There are plans for this be improved in the

Re: Error setting up environment for Custom Processor

Hello, I think there are two options... You could try going through your Pom files and anywhere you see 0.1.0-incubating change it to the latest NiFi release of 0.6.0. A second option is to not rely on the nifi-nar-bundles parent, the steps to remove that are described here:

Re: Can't connect to Secure HBase cluster

>From doing some Googling it seems like the problem is similar to the one described here where Hive could no longer talk to HBase after installing Phoenix: https://community.hortonworks.com/questions/1652/how-can-i-query-hbase-from-hive.html The solution in that scenario was to add a phoenix jar

Re: Import Kafka messages into Titan

For #2, you can use templates to move the flow (or parts of it) to another instance. A possible approach is to organize the flow into process groups and create a template per process group, making it potentially easier to update parts of the flow independently. This project might be helpful to

Re: Can't connect to Secure HBase cluster

wrote: > Hi, > > > > Yes, I can connect using that user. > > > > Had to test it on HBase master as HBase not installed on NiFi server. > > > > Regards > > Guillaume > > > > Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=5509

Re: Splitting Incoming FlowFile, Output Multiple FlowFiles

Hello, SplitText and SplitContent should be producing individual FlowFiles. Are you seeing something different? For SplitText you would set "Line Split Count" to 1 in order to get a FlowFile for each line of the incoming CSV. If you are doing extremely large files, it is generally recommended

Re: [VOTE] Release Apache NiFi 0.6.0 (RC2)

+1 (binding) I did run into the problem previously mentioned with TestListFile. For me this test never passes on OSX, but worked fine when building on a Linux VM. I tested ListFile from the running application and it appears to work as expected, so I don't see this as a reason to vote against the

Re: Custom Graphs/Metrics reporting

Hi Michael, Currently the graphing functionality is not extensible, as far as I know. One option that might be helpful is a ReportingTask. ReportingTasks are components that get access to all of the metrics and provenance events and can push them out to an external system. Currently there are

Re: Closing in on the Apache NiFi 0.6.0 release

All, While testing the original RC I came across a couple of issues with the new GetSplunk processor. The first issue relates to being able to specify the timezone through a processor property, so that the timezone used for searching will match with splunk's timezone. The second is is that

Processor additional documentation

gt;>>>>> Dan, > > > > >>>>>> > > > > >>>>>> but maybe I have a wrong understanding: do I have to create an > > > > >>> index.html > > > > >>>>>> file? Currently I have only created an additionalDet

Re: quick reminder on new nars

n-archetypes/nifi-processor-bundle-archetype/src/main/resources/archetype-resources/nifi-__artifactBaseName__-nar/pom.xml > > Thanks > Joe > > On Wed, Mar 16, 2016 at 9:19 AM, Bryan Bende <bbe...@gmail.com> wrote: > > Also sounds like we need to update the archetype ba

Re: Cross NAR Controller Services

Devin, This WIki page shows how to create the appropriate dependencies between your NAR and the ControllerService: https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-LinkingProcessorsandControllerServices I also created an example project

Re: Unable to connect Nifi to SQL Server

Hello, If you are specifying the URI for the driver jar, then you would generally put it outside of NiFi lib directory, somewhere else. The URI also needs the file:/// prefix, so for windows I think it would be: file:///D:/some/path/to/driver/sqljdbc4.jar Let us know if that works. -Bryan On

Re: ExecuteSQL and NiFi 0.5.1 - Error org.apache.avro.SchemaParseException: Empty name

I think this a legitimate bug that was introduced in 0.5.0. I created this ticket: https://issues.apache.org/jira/browse/NIFI-1596 For those interested, I think the line of code causing the problem is this:

Re: GetMongo GC Overflow

Hello, I'm not that familiar with MongoDB, but from looking at the existing GetMongo processor, it seems to create a FlowFile per Document and only calls session.commit() once at the very end, which could possibly be a problem when producing a very significant amount of flow files. When you

Re: Installing NIFI through ambari.

It may have something to do with the APP_ID in the metrics being 'nificluster'. It needs to line up with the APP_ID in the service. I would recommend leaving it as the default value and see if that works. On Friday, February 26, 2016, davi wrote: > I'm have similar

Re: [VOTE] Release Apache NiFi 0.5.1 (RC2)

+1 Release this package as nifi-0.5.1 (binding) Ran through release helper and everything checked out. Tested flows against HDFS with and without Kerberos enabled. On Wed, Feb 24, 2016 at 11:22 PM, Joe Witt wrote: > +1 Release this package as nifi-0.5.1 (binding) > > Have

Re: Issue connecting to Oracle-HPSM using NIFI

Naveen, Can you check nifi_home/logs/nifi-app.log to see if there is a full stack trace beyond the message that says "unsupported feature"? Would be curious to see what the full stack trace looks like. -Bryan On Mon, Feb 22, 2016 at 2:26 PM, Kilaru, Naveen Kumar <

Re: Set default yield duration

Adam, I think the idea was for the user to always control these values through the user interface, and therefore there isn't a way that I know of for a specific processor to control the default values for yield duration and run schedule. I have always wanted to be able to do this though,

Re: Nifi - Adding attributes to flow file results in FlowFileHandlingException when flow file is transferred

Hello, The error message is indicating that you are trying to transfer an unknown FlowFile because it is transferring a reference to the original FlowFile before you updated the attributes. You would need to assign the result of putAllAttributes (or putAttribute) and then transfer that: flowFile

Re: Using NiFI with a Secured HBASE setup

Laxman, The current HBase integration does not support Kerberized HBase installs at the moment. I created a JIRA to track this: https://issues.apache.org/jira/browse/NIFI-1488 -Bryan On Mon, Feb 8, 2016 at 10:36 AM, wrote: > Hi, > > I have configured a Hbase

Re: Using NiFI with a Secured HBASE setup

, Bryan Bende <bbe...@gmail.com> wrote: > Laxman, > > The current HBase integration does not support Kerberized HBase installs > at the moment. > > I created a JIRA to track this: > https://issues.apache.org/jira/browse/NIFI-1488 > > -Bryan > > On Mo

Re: Installing NIFI through ambari.

Hello, The Apache NiFi community only provides an example service for viewing metrics in Ambari [1]. This example does not actually install and manage NiFi through Ambari, it only acts as placeholder to view metrics. -Bryan [1] https://cwiki.apache.org/confluence/display/NIFI/Ambari+Metrics

Re: Does Nifi-ambari reporting supports Ambari version 2.1.1

Joe/Shweta, I view the Ambari stuff as two different pieces... the AmbariReportingTask in NiFi which was developed against Ambari 2.1, but I believe should work against any 2.X version because the Ambari Metrics Service was introduced in 2.0.0. The reporting task can send over metrics independent

Re: How to capture more than 40 groups in Extract Text

Hi Shweta, You may want to consider a custom processor at this point. The csv-to-json example works ok for smaller csv files, but admittedly is not a great solution when there are a lot of columns. There has been interest from the community in the past on having a ConvertCsvToJson processor, but

Re: Are we thinking about Penalization all wrong?

I really like the idea of being able to have different penalty durations for each connection, and I think this would make things a bit clearer then relying on the developer to determine when to penalize things. On Thu, Jan 28, 2016 at 11:34 AM, Mark Payne wrote: > All, > >

Re: ListenLumberjack processor is working

Andre, Very cool that you have made progress here. Being able to integrate with logstash will be very useful. I think the refactoring I'm doing for the RELP stuff should help reduce the amount of code that had to be carried over from ListenSyslog. I'm happy to help you update your code once my

Re: Direction for Integration Tests

I also like Mans categories, as I know I have personally written some unit tests that don't require any external dependencies, but might have veered closer to an integration test than a true unit test. Some examples I am thinking of are the tests for Put/GetSolr which use and EmbeddedSolrServer,

Syslog Classes

All, I'm working on NIFI-1273 to add support for the RELP protocol (Reliable Event Logging Protocol) to the syslog processors. In order to do this I'll likely have to add at least one more channel reader implementation to the inner classes that already exist in ListenSyslog. I'm starting to think

Re: Syslog Classes

ith a registry > model then we will revisit these and many others anyway. > > Thanks > Joe > > On Tue, Jan 5, 2016 at 10:22 AM, Bryan Bende <bbe...@gmail.com> wrote: > > All, > > > > I'm working on NIFI-1273 to add support for the RELP protocol (Reliable > &

Re: about the debug of nifi

Hello, The nifi-ide-integration project will not have any src, it just generates the project files for Eclipse or IntelliJ (in your case the .iml, .ipr, and .iws files). You will need to have the nifi source code somewhere else, and the remainder of the instructions here [1] explain how to link

Re: Syslog Classes

/main/java/org/apache/nifi/processors/standard/syslog On Tue, Jan 5, 2016 at 6:14 PM, Tony Kurc <trk...@gmail.com> wrote: > Excellent questions, Bryan. I'll give it some thought tonight. > > On Tue, Jan 5, 2016 at 11:22 AM, Bryan Bende <bbe...@gmail.com> wrote: > > > Al

Re: Standing a REST interface on GetMongo processor

Hello, You are correct that GetMongo is currently a source processor that does not accept input. I don't know of any existing plans for it to support incoming FlowFIles, but it sounds like a useful feature and would be a good request to make in JIRA [1]. -Bryan [1]

Re: [VOTE] Release Apache NiFi 0.4.1 (rc1)

+1 (binding) Release this package as Apache NiFi 0.4.1 On Sat, Dec 19, 2015 at 4:55 PM, Tony Kurc wrote: > I've verified the signature, hashes, built on Ubuntu 14.04 (x86_64) with > Oracle java 7. Built binaries worked as expected. > > Licence and notice look good > > +1 > +1

Re: Cluster Setup

Hello, You can run the NCM on the same machine as a node, but they still have to be two separate processes. You would create two copies of the directory where you extracted nifi and set one to be the manager and one to be the node, you'll also have to give them different nifi.web.http.port

Re: Testing handling of static class methods

If you get it into a protected instance method, you can also make an inner class in your test, something like TestablePutJMS extends PutJMS, and overrides that method to return a mock or whatever you want. That is a common pattern in a lot of the processor tests. On Fri, Dec 18, 2015 at 3:44 PM,

Re: Facing Issue while connecting with HDFS

Hello, By default NiFi will interact with HDFS as the user that started the NiFi process. If you can only access HDFS as the superuser then you will probably need to start NiFi as that user. There is a "run.as" property in conf/bootstrap.conf where you can specify a username to run NiFi as, but

Re: How to iterate through complex JSON objects.

As an alternative approach, could you use SplitJSON first to split on the items array? You would get a FlowFile for each item, then when you use EvaluateJSONPath you would be dealing with only a single FlowFile so you could extract the id and title and use ReplaceText like you were already doing.

Re: Moving BinFiles

Joe, Thanks for bringing this up. I remember running into this issue a long time ago when trying to extend BinFiles from a processor in another NAR. I'm wondering if there could be another artifact in nifi-standard-bundle like nifi-standard-processor-utils where stuff like this could go? There

Re: Facing Issue while connecting with HDFS

Digvijay, Are you talking about Kerberos authentication to the HDFS cluster? If so, in nifi.properties you specify your krb5.conf file: nifi.kerberos.krb5.file=/etc/krb5.conf (or wherever your conf file is) This is the file that would have your realms defined. Then on the HDFS processors

Re: Facing Issue while connecting with HDFS

Site-to-Site is a direct connection between NiFi instances/clusters over a socket, so TCP based. There will always have to be at least one local machine involved. When NiFi pulls/receives data from somewhere, it takes that data under control and stores it in the NiFi content repository on disk

Re: Facing Issue while connecting with HDFS

It doesn't necessarily have to be on the same machine, but the machine NiFi is on would have to be able to communicate with the name-node and data-nodes in order to push/pull data to/from HDFS. In your example this would mean your local machine would need to be able to access the name-node and

Re: Facing Issue while connecting with HDFS

Hello, The directory property should only need the path in HDFS. The other stuff like the file system, host, and port would be determined from the provided configuration files. Do you receive any different response if you set the directory to just "/root" (or some other path)? If you still

Re: Need access to edit Apache NiFi confluence pages.

David, I just granted you access, let us know if it doesn't work. Thanks, Bryan On Tue, Dec 1, 2015 at 2:30 PM, David Wynne wrote: > Please grant me access to the Confluence Apache-NiFi wiki page. >

Re: Coding a Processor that writes to multiple output flowfiles at once

Hi Salvatore, Have you looked at the append() method on ProcessSession which lets you append to the content of a FlowFile? You should be able to create several new FlowFiles, and then while reading lines from the incoming FlowFile, append the appropriate parts to each of the new FlowFIles. An

Re: Nifi startup error after I install my first custom processor

to add some logging. > > Mark > > On Thu, Nov 19, 2015 at 8:23 PM, Bryan Bende <bbe...@gmail.com> wrote: > > > Hi Mark, > > > > Glad to hear you were able to get started building processors, and glad > > that blog post helped! > > > > I

Re: nifi fails build on CentOS 7

George, Thanks for passing on this info about the syslog tests, those are new for this current release so it is good to know what people are running into. The ListenSyslog processor has a property that sets the SO_RCVBUF to allow increasing the OS socket receive buffer, it defaults to 1MB and it

Re: JSON / Avro issues

Jeff, Are you using the 0.3.0 release? I think this is the issue you ran into which is resolved for the next release: https://issues.apache.org/jira/browse/NIFI-944 With regards to ConvertJSONtoAvro, I believe it one json document per line with a new line at the end of each line (your second

Re: Gitting the hub right

Joe, One way to avoid the merge commits is to use rebase. I believe we have it outlined here: https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide#ContributorGuide-Keepingyourfeaturebranchcurrent In short, you basically... - checkout your master - fetch upstream to get the latest

Re: Next release?

Also, to answer Ricky's question about how to merge in the pull request once there is consensus... There are multiple ways to do it, but I believe what a lot of PMC members do is the following: - Get a patch of the pull request by appending .patch to the end of the url - git am --signoff <

Pull Request Comments and JIRA

Does anyone know why in-line comments on pull-requests don't post back to the JIRA? Comments on the overall pull request do post back. I feel like this might have been something that worked during incubation and no longer works, but could be wrong.

Re: Recommendation on getting started as a contibutor

Mark, I don't think there is a strong preference for patches vs. pull requests. A lot of contributors use pull requests, and I personally find it easier to review pull requests because you can give feedback in-line on the code. The more important thing is that whatever is being submitted should

Re: NiFi 0.3 : Query regarding HDFS processor

Hello, There is a property on PutHDFS where you can specify the Hadoop configuration files which tell the processor about your HDFS installation: Hadoop Configuration Resources - A file or comma separated list of files which contains the Hadoop file system configuration. Without this, Hadoop

Re: Ingest Original data from External system by data's dependent condition

FYI, in case someone wants to work it, the ticket for extracting from Avro is: https://issues.apache.org/jira/browse/NIFI-962 On Tue, Oct 13, 2015 at 9:29 AM, Andrew Grande wrote: > A typical pattern/workaround for this situation was to copy e.g. the json > _in full_

Re: Ingest Original data from External system by data's dependent condition

We do have an idea that we called HoldFile that hasn't been fully implemented yet, but has come up a few times: https://issues.apache.org/jira/browse/NIFI-190 The idea was basically for a processor to "hold" a FlowFile until it was signaled by another processor to release it. Seems like this is

Re: A flow to PutSQL the lines of a CSV file?

Russell, How big are these CSVs in terms of rows and columns? If they aren't too big, another option could be to use SplitText + ReplaceText to split the csv into a FlowFile per line, and then convert each line into SQL in ReplaceText. The downside is that this would create a lot of FlowFiles

Re: PutHDFS Configuration Issue

Glad that first issue was resolved! I am by no means a kerberos expert, but having set this up once before, the setup should be something like the following: nifi.kerberos.krb5.file=/etc/krb5.conf (or wherever your conf file is) This is the file that would have your realms defined. Then on

Re: PutHDFS Configuration Issue

Hi Domenic, It sounds like you are on the right path... just to confirm, did you restart NiFi after setting nifi.kerberos.krb5.file in nifi.properties? It will only pick up changes to nifi.properties on a restart. Also, On Wed, Sep 30, 2015 at 12:38 PM, DomenicPuzio

Re: Syslog processors?

Corey, No one was worked on this as far as I know, but there is definitely interest. There is a ticket that I believe you created :) [1] You can however receive syslog messages over UDP using the ListenUDP processor, and setting syslog to forward messages to the given port. -Bryan [1]

Re: NIFI DBCP connection pool not working for hive

Hello, Can you provide the error message you received? Is it about finding/loading the driver? or about actually creating the connection? Thanks, Bryan On Fri, Sep 18, 2015 at 2:11 PM, vthiru2006 wrote: > Hi, > > I'm trying to make a hive connection pool via controller

Re: Ways to migrate to 0.3.0

Rick, You should be able to move the conf/flow.xml.gz from your snapshot install to your new 0.3.0 conf directory. -Bryan On Tuesday, September 15, 2015, Rick Braddy wrote: > How can I migrate prior flow graph from 0.3.0-SNAPSHOT to the new 0.3.0 > tree? > > Rick > --

Re: [VOTE] Release Apache NiFi 0.3.0

Signature and checksums look good Build passes with contrib-check README/LICENSE/NOTICE all look good Test dataflows perform as expexted +1 (binding) - Release this package as nifi-0.3.0 On Tue, Sep 15, 2015 at 9:50 AM, Mark Payne wrote: > Downloaded source, verified

Re: Session Commits

Rick, I believe it depends on how you implement your processor, meaning what class you extend from. If you extend AbstractProcessor, then you can see that it creates a session and calls commit for you. If you extend AbstractSessionFactoryProcessor, or implemented the Processor interface, then you

Re: POM dependency failures

Rick, Can you check in your local Maven repository to see if the 0.3.0-SNAPSHOTs are in there? It is typically in HOME/.m2 As one example ~/.m2/repository/org/apache/nifi/nifi-api/ should have a sub-directory for 0.3.0-SNAPSHOT. -Bryan On Fri, Sep 4, 2015 at 2:50 PM, Rick Braddy

Re: POM dependency failures

ck/0.3.0-SNAPSHOT > ./nifi/0.3.0-SNAPSHOT > ./nifi-external/0.3.0-SNAPSHOT > ./nifi-processor-utils/0.3.0-SNAPSHOT > ./nifi-nar-bundles/0.3.0-SNAPSHOT > ./nifi-commons/0.3.0-SNAPSHOT > ./nifi-api/0.3.0-SNAPSHOT > ./nifi-framework-bundle/0.3.0-SNAPSHOT > > Rick > >

Re: Adding new processor to standard bundle, not showing in UI

Rick, Everything you described sounds like the correct approach. One thing to try, in the directory where you have nifi installed, there should be a work directory which has a nar sub-directory... you could try stopping nifi, deleting that nar directory, and starting again. That directory

Re: GetSolr NoClassDefFoundError

Srikanth/Joe, The NoClassDefFoundError was a side effect that resulted when we bumped several dependencies to newer versions in the 0.2.x release, specifically the httpclient. We have the fix ready to go for 0.3.0, the Jira is - https://issues.apache.org/jira/browse/NIFI-780 Thanks, Bryan On

Re: 0.3.0 In-depth Testing

All, I was testing a scenario this morning using the Flume processors to leverage Flume's Syslog sources, and ran into an issue that I created a ticket for [1]. Assuming the patch is reviewed and agreed upon, I would suggest this could be included in 0.3.0 as it only affects a single line of

Re: help with adding a process

Pat, If you are developing your bundle with in the nifi source code, you also need to update the nifi-assembly/pom.xml to include your new nar in the final package that gets built. You can check if it is included after doing a full build, and checking the lib directory (located wherever you are

Re: [VOTE] Release Apache NiFi nifi-nar-maven-plugin 1.1.0

+1 Release this package as nifi-nar-maven-plugin-1.1.0 Verified all steps in Matt's helper email, functions as expected. On Wed, Aug 19, 2015 at 11:25 PM, Matt Gilman matt.c.gil...@gmail.com wrote: +1 (binding) Release this package as nifi-nar-maven-plugin-1.1.0 On Wed, Aug 19, 2015 at

Re: Process to create multiple files

Pat, You don't need to extend anything special to produce multiple files. You can create as many FlowFiles as you want and transfer them to a relationship, such as... FlowFile childFlowFile = session.create(original); // create more ListFlowFile flowFiles = new ArrayListFlowFile();

Re: [DISCUSS] Feature proposal: First-class Avro Support

Ryan, Thanks for the feedback and suggestions! We will definitely factor all of this into the design, and when I get a chance I will update the Wiki page accordingly. Thanks, Bryan On Sat, Aug 15, 2015 at 5:45 PM, Ryan Blue b...@cloudera.com wrote: On 08/12/2015 06:09 PM, Bryan Bende wrote

Re: [DISCUSS] Removal of the 'master' vs 'develop' distinction

If we worked on master and had a prod branch that was the last release, then we have the same thing we do now, just with different names. This would be GitLab Flow as Brandon pointed out. That being said, I don't have experience with the release process, and maybe the prod branch does not provide

Re: Trouble sending files

Hi Patrick. There are generally two approaches to sending data between nifi instances... - The sending instance brings data to a remote process group which is pointing at the receiving nifi, the receiving nifi has an input port to receive the data - The sending instance brings data to an output

Re: [VOTE] Release Apache NiFi 0.2.1

+1 (binding) - Signature and hashes look good - LICENSE, NOTICE, README look good in source and assembly - Source builds with contrib-check - Started resulting binary with https and client auth, successfully prompted to request an account - Basic flow works On Thu, Jul 23, 2015 at 6:49 PM, Joe

<    3   4   5   6   7   8