Re: Common data exchange formats and tabular data

All, Benjamin has already done a lot of good work and it would very helpful we can agree how to move on. https://issues.apache.org/jira/browse/NIFI-901 My first post was naive, there are much more things to consider. It is probably impossible to select only one “correct data exchange format”

Re: Common data exchange formats and tabular data

Matt, Good overview. >Avro is similar, but the schema must always be provided with the data. In >the case of NiFi DataFlows, it's likely more efficient to send the schema >once as an initialization packet (I can't remember the real term in NiFi), >then the rows can be streamed individually,

Re: Avro tests are broken on Windows?

Same here, win7 --- T E S T S --- Running org.apache.nifi.processors.avro.TestConvertAvroToJSON Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.421 sec <<< FAILURE! - in

Re: [DISCUSS] Feature proposal: First-class Avro Support

Hello Avro Content Viewer would be great. Does anyone has time to implement this? Thanks toivo -- View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/DISCUSS-Feature-proposal-First-class-Avro-Support-tp2437p2676.html Sent from the Apache NiFi

Re: Streaming large result set - ExecuteSQL

Hi, ExecuteSql supports streaming and full result set in not loaded into memory. Data is read continuously from ResultSet and immediately written to FlowFile. But as Mark said result is one big FlowFile, so you can’t start reading FlowFile by next processor before it’s completed by ExecuteSql.

Long running processors, stopping and timeouts.

Hi, Sometimes long running processors cannot be easily stopped. See nifi-users, Mon, 23 Nov, ExecuteProcess processor cannot be stopped/started And often it is desirable to be able set running timeout before processor start. https://issues.apache.org/jira/browse/NIFI-528 Add support to specify

Data Manipulation Language for Avro, JSON, CSV, etc

Hi from user mailing list: nifi processor to parse+update the current json on the fly NIFI-361 Sometimes I miss DML in Java code. I mean something like SQL Maybe we can use DML in NiFi? Some universal DML for any data? Be it Avro, JSON, CSV, etc. DML should handle any tabular like data. And not

Re: Failed to execute goal org.apache.rat:apache-rat-plugin:0.11:check (default) on project nifi-mock: Too many files with unapproved license: 6 See RAT report in: /home/toivo/git/nifi/nifi-mock/targe

Hi Of course, I forgot to add to each new file /* * Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with . . . Silly me. Its OK now. thanks Toivo -- View this message in context:

Re: [ANNOUNCE] New Apache NiFi Committer Andy LoPresto

Congratulations! -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/ANNOUNCE-New-Apache-NiFi-Committer-Andy-LoPresto-tp8725p8729.html Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Failed to execute goal org.apache.rat:apache-rat-plugin:0.11:check (default) on project nifi-mock: Too many files with unapproved license: 6 See RAT report in: /home/toivo/git/nifi/nifi-mock/target/ra

Hi I am missing something obvious, but lately my clean install -Pcontrib-check builds failing: Failed to execute goal org.apache.rat:apache-rat-plugin:0.11:check (default) on project nifi-mock: Too many files with unapproved license: 6 See RAT report in:

Re: [GitHub] nifi pull request: Nifi 1214

To move on and clean a mess I created new pull request. https://github.com/apache/nifi/pull/388 nifi-1214b Mock Framework should allow order-independent assumptions on FlowFiles. This replaces previous nifi-1214 thanks toivo -- View this message in context:

Re: [DISCUSS] Using the new 'help wanted' tool from comdev

Helpwanted is currently fairly empty. This is our chance. :) Let’s add some 5-10 issues from Jira. At least this raise NiFi awareness. Thanks Toivo -- View this message in context:

Re: Data Manipulation Language for Avro, JSON, CSV, etc

Hi Working CSV processor is now done. SQL select statement can be used to manipulate CSV data. Input, CSV file: ID:int,AMOUNT1:float,AMOUNT2:float,AMOUNT3:float 008, 10.05, 15.45, 89.99 100, 20.25, 25.25, 45.25 200, 34.05, 25.05, 75.05 select ID, AMOUNT1+AMOUNT2+AMOUNT3 as TOTAL from CSV.A where

Re: BUILD FAILURE: java.lang.NoClassDefFoundError: org/apache/nifi/authorization/Authority

Matt, thanks toivo -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/BUILD-FAILURE-java-lang-NoClassDefFoundError-org-apache-nifi-authorization-Authority-tp9321p9353.html Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: ListenSMTP processor

I support Oleg opinion. Do one thing and do it well. Thanks Toivo -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/ListenSMTP-processor-tp10510p12891.html Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: In ListS3 processor, where does Nifi persists the state of objects?

Hi, As far I know ListS3 use NiFi built in StateManager which in turn use StateProvider's. NiFi may have different StateProvider implementations. Currently NiFi have 2 providers, ZooKeeper based and write-ahead log file based. ZooKeeper is used when NiFi cluster is configured and other is used

Bulletin and related FlowFile

I am creating custom Reporting Task. Reporting Task will read Bulletins. But I am missing something obvious, bulletin does not have reference to FlowFile? I mean Bulletin was generated during processing some FlowFile and I want to find FlowFile attributes. Is this possible? Thanks Toivo --

Re: Spring JdbcTemplate, SimpleJdbcCall and DataSource

Hi Koji, Thank you for reviewing. Great ideas. I created new PR. https://github.com/apache/nifi/pull/1450 Spring JDBCTemplate usage example is in DBCPServiceTest.java testGetDataSource() and createInsertSelectDrop(DataSource dataSource) Thanks Toivo -- View this message in context:

Careful, don't change processor package name.

I refactored code and split several processors to different packages. I did not change processors names. After restarting NiFi flow was otherwise as it was before but all processors lost configuration. All properties values was “Sensitive value set”, be it normal non-sensitive or controller

Re: Spring JdbcTemplate, SimpleJdbcCall and DataSource

Please, can anyone review PR? https://github.com/apache/nifi/pull/1417 Thanks Toivo -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Spring-JdbcTemplate-SimpleJdbcCall-and-DataSource-tp14405p14482.html Sent from the Apache NiFi Developer List mailing list

Re: ExecuteSql output formats.

Hi Andrew, My initial idea was something like. import java.io.OutputStream; import java.sql.ResultSet; public interface ResultSetConverter { public long convertToAvroStream(final ResultSet rs, final OutputStream outStream); } and implementation for example public class

Re: ExecuteSql output formats.

Hi Matt, I did not notice Peter work, thanks for pointing it out. You are right, controller service approach helps to solve dependency problems. I read “Looking for feedback on my WIP Design” and code and this looks much bigger than my simple solution. I have to think about it. Thanks Toivo

ExecuteSql output formats.

Hi, Currently Avro is supported, but often other formats are needed. Workaround is to let ExecuteSql create FlowFile in Avro and immediately after that (using another processor) convert Avro to some other format. This is a waste of time and resources. Maybe it's wise to add pluggable output

Conceptual flow view.

Hi, NiFi has one view for the designing flow and runtime. I wonder is it possible to have separate design time view? Which show simplified conceptual view. Conceptual view is without runtime metric (In, Read/Write bytes, etc). Such view has better readability and can be used to represent ideas to

Re: Conceptual flow view.

Hi Joe, I used WebCrawler.xml template as example https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates First as it is And conceptual view.

Re: Temporary FlowFile storage.

Hi Mark, Great, this should be sufficient. Thanks, Toivo -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Temporary-FlowFile-storage-tp14210p14215.html Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Spring JdbcTemplate, SimpleJdbcCall and DataSource

Hi Matt, Right, getConnection() was sufficient, and still is in the most of the cases. DataSource is needed rarely and is exception. https://issues.apache.org/jira/browse/NIFI-3339 I added getDataSource() and updated code in numerous places where API is used. Thanks Toivo -- View this

Spring JdbcTemplate, SimpleJdbcCall and DataSource

Spring JdbcTemplate, SimpleJdbcCall and DataSource Does anybody use Spring JdbcTemplate, SimpleJdbcCall? Both take DataSource constructor parameeter and not Connection. And later when SimpleJdbcCall (or JdbcTemplate) need Connection, it will ask it from DataSource and release afterwards.

Wormhole Connections

Hi, https://cwiki.apache.org/confluence/display/NIFI/Wormhole+Connections Does anyone working on this? I am interested to have Wormhole connections to improve flow readability. Any ideas how to move on? Thanks Toivo -- View this message in context:

Adding custom logging Appender to NiFi

Hi, I have custom logging appenders (extends ch.qos.logback.core.AppenderBase) I put custom appenders jar file to /lib directory. I added appenders to /conf/ logback.xml and this worked fine using NiFi 1.1.1 Few weeks ago I started to upgrade to NiFi 1.3.0 and along the process appenders are

Re: Adding custom logging Appender to NiFi

Solved. You need to put your custom jar files also to /lib/bootstrap directrory. Regards, Toivo -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Adding-custom-logging-Appender-to-NiFi-tp16408p16424.html Sent from the Apache NiFi Developer List mailing

Fail-fast

We have normal processing groups and special error handling group. If during any step something will go wrong, processing of all items by this id will be aborted. Error handling group will be invoked which will do some cleanup. Failure relationships are used to route to error handling group. But

Removing unneeded nar files from NiFi standard distribution.

Hi, In lib directory are a lot of nar files which we don’t use. I assume unneeded nar files can be deleted? Any negative consequences? I assume start time may improve and maybe RAM consumption? Or is it not worth of trouble? Thanks Toivo -- View this message in context:

Re: Adding Failure handling to ListFile

Content would be empty. One or more attributes contain error message, maybe error code, etc. -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: Importing templates which have reference to same controller service.

1.3.0 -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: Adding Failure handling to ListFile

We are run separate processes (implemented as Process Group) which have start and end. Same process should same unique id. We want to monitor process from start to end. Summary include only process name, when started, when ended, unique id and whatever it was successful or not. In case of failure

Adding Failure handling to ListFile

Hi ListFile has only SUCCESS relationship. It would useful to add also FAILURE. For example when input directory does not exist or can’t be read. What do you think? Thanks Toivo -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Importing templates which have reference to same controller service.

Hi Today we had annoying problem after importing template. Template has reference to same controller service (with same name) which already existed. As result controller service list shows 2 services with same name and one was disabled. One processors from new flow complained what controller

DBCPConnectionPool encrypted password

Hi, I need to encrypt DBCPConnectionPool password. I have working decryption code. Problem is how to make decrypted password available to DBCPConnectionPool. I thought to use expression language and JVM System Property’s. DBCPConnectionPool is capable to read System property value. But how to

Re: DBCPConnectionPool encrypted password

Hi Bryan, >> Are you saying you are trying to externalize the value outside the >> w and keep it encrypted somewhere else? Yes, exactly. We have different passwords on different environments (dev, test, production). After development flow (using template currently) will be deployed to test

Re: DBCPConnectionPool encrypted password

Hi Bryan, We start using Registry soon anyway, so this is useful info. But it would be event better when we don’t need to enter passwords manually each time we deploy template. Any ideas how to do this? Thank you Toivo -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/