Re: [ANNOUNCE] New Apache NiFi PMC Member - Oleg Zhurakousky

[Oleg Zhurakousky]

Thank you all!

 *Joe Percivall*
> e: joeperciv...@gmail.com

Re: [ANNOUNCE] New Apache NiFi Committer Bin Qiu

[Oleg Zhurakousky]

And of course I did it in the wrong thread ;)
Congrats Bin!!!

> On Apr 5, 2017, at 9:48 AM, Oleg Zhurakousky <o...@suitcase.io> wrote:
> 
> Thank you all!
> 
> > *Joe Percivall*
>> e: joeperciv...@gmail.com
> 

Re: [ANNOUNCE] New Apache NiFi Committer Bin Qiu

[Oleg Zhurakousky]

Thank you all!

 *Joe Percivall*
> e: joeperciv...@gmail.com

Re: When should MergeContent stop and proceed to next processor?

[Oleg Zhurakousky]

Ok, can you please set the “Correlation Identifier” to “fragment.identifier”?
That is what I was trying to explain in the previous email.

Cheers
Oleg

> On Mar 16, 2017, at 11:06 AM, srini  wrote:
> 
> Hi Oleg,
> 
> Here is MergetContent screenshot. My flowfiles don't give any clue about
> what record it belongs to. I have an attribute called recordId which
> distinguishes each record. But I shouldn't add recordId in the flowfiles to
> be merged.
> 
>  
> 
> thanks
> Srini
> 
> 
> 
> --
> View this message in context: 
> http://apache-nifi-developer-list.39713.n7.nabble.com/When-should-MergeContent-stop-and-proceed-to-next-processor-tp15148p15164.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
> 

Re: When should MergeContent stop and proceed to next processor?

[Oleg Zhurakousky]

Hi

Is there any chance you can share your processor’s configuration? I am curious 
as to what are you using as “Correlation Attribute Name” in the MergeContent 
processor.
Basically this attribute allows to distinguish groups of flow files so, since 
you have SplitJson as an upstream processor feeding MergeContent you can use 
“fragment.identifier” as correlation attribute.
Anyway, please share what you can.

Cheers
Oleg

> On Mar 15, 2017, at 5:55 PM, srini  wrote:
> 
> Hi,
> I have a subflow like this. From SplitJson to MergeContent it is in loop. I
> expect it loops based on the number of splits of that record. How it know
> that splits for that record is over, and it needs to be proceed to next
> processor that is ExtractText? 
> 
> I have 3 records. In my case it is merging 25 (8 + 5 + 12 = 25). It is
> merging all records data into one record. It shouldn't merge all, instead
> after each record it should proceed to next processor.
> 1st record: Merge 8 items
> 2nd record:  Merge 5 items.
> 3rd record: Merge 12 items.
> 
> 
>  
> 
> What changes do you recommend to my flow?
> Here is my MergeContent screenshot.
> 
>  
> 
> 
> 
> 
> --
> View this message in context: 
> http://apache-nifi-developer-list.39713.n7.nabble.com/When-should-MergeContent-stop-and-proceed-to-next-processor-tp15148.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
> 

Re: Connection Issue

[Oleg Zhurakousky]

Sorry, I do see the “unwrap” call now, but what I can’t correlate is your code 
to the stack trace. Actually, it appears from your stack trace that in your 
GESBOProcessor on line 183 you are attempting to cast the 
org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper to 
OracleConnection and the unwrap call is supposed to be called on such 
connection. Can you clarify?

Cheers
Oleg

> On Mar 15, 2017, at 2:29 PM, Anil Rai <anilrain...@gmail.com> wrote:
> 
> Hi Oleg, if you look at the java code and logs i sent in the previous
> email, it does have the code to unwrap. But it is not allowing to cast that
> connection object back to OracleConnection. It fails.
> 
> On Wed, Mar 15, 2017 at 2:14 PM, Oleg Zhurakousky <
> ozhurakou...@hortonworks.com> wrote:
> 
>> Ok, so it appears that DBCP does not like you to get access to the
>> underlying connection without additional configuration, but Java does since
>> java.sql.Connection extends java.sql.Wrapper from which you can do
>> something like this:
>> 
>> Connection dbConnection= d.getConnection();
>> OracleConnection orcConnection = null;
>> if (dbConnection.isWrapperFor(OracleConnection.class)) {
>>orcConnection = dbConnection.unwrap(OracleConnection.class);
>> }
>> . . .
>> 
>> Let me know
>> 
>> Cheers
>> Oleg
>> 
>> On Mar 15, 2017, at 1:33 PM, Anil Rai <anilrain...@gmail.com> anilrain...@gmail.com>> wrote:
>> 
>> Hi Oleg, we tried and no luck. This is another example where we are seeing
>> similar issue. Attached txt has the java code as well as the log.
>> 
>> Thanks in advance
>> Anil
>> 
>> 
>> On Wed, Mar 15, 2017 at 10:02 AM, Anil Rai <anilrain...@gmail.com> anilrain...@gmail.com>> wrote:
>> Thanks Oleg. Makes sense. Will try and keep you posted.
>> 
>> Regards
>> Anil
>> 
>> 
>> On Wed, Mar 15, 2017 at 9:56 AM, Oleg Zhurakousky <
>> ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote:
>> Anil
>> 
>> Thank you for details. That does help a lot.
>> 
>> First, I want to make sure that it is clear that this is not a NiFi issue,
>> since the problem is specific to the combination of DBCP and Oracle and the
>> expectations between the two.
>> 
>> Seems like Oracle JDBC connection is wrapped in an implementation specific
>> class (DBCP in this case I assume).
>> It is my believe that you need to obtain reference to native Oracle
>> connection to avoid "JMS-112: Connection is invalid".
>> So, I think you need to try to cast your Connection object to DBCPs
>> DelegatingConnection and then do something like this:
>> 
>> DelegatingConnection wrappedConn = (DelegatingConnection)con;
>> OracleConnection ocon =  null ;
>> if (wrappedConn != null)
>> ocon = (OracleConnection) wrappedConn.getDelegate();
>> . . .
>> 
>> Let me know how it goes
>> 
>> Cheers
>> Oleg
>> 
>>> On Mar 15, 2017, at 9:20 AM, Anil Rai <anilrain...@gmail.com> anilrain...@gmail.com>> wrote:
>>> 
>>> 2017-03-14 16:50:42,312 ERROR [NiFi logging handler]
>> org.apache.nifi.StdErr
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> 2017-03-14 16:50:42,313 ERROR [NiFi logging handler]
>> org.apache.nifi.StdErr
>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>>> 2017-03-14 16:50:42,313 ERROR [NiFi logging handler]
>> org.apache.nifi.StdErr
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$
>> ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>>> 2017-03-14 16:50:42,313 ERROR [NiFi logging handler]
>> org.apache.nifi.StdErr
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$
>> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>>> 2017-03-14 16:50:42,313 ERROR [NiFi logging handler]
>> org.apache.nifi.StdErr
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1142)
>>> 2017-03-14 16:50:42,313 ERROR [NiFi logging handler]
>> org.apache.nifi.StdErr
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:617)
>>> 2017-03-14 16:50:42,313 ERROR [NiFi logging handler]
>> org.apache.nifi.StdErr
>>> at java.lang.Thread.run(Thread.java:745)
>>> 2017-03-14 16:50:43,567 INFO [NiFi logging handler]
>> org.apache.nifi.StdOut
>>&

Re: Connection Issue

[Oleg Zhurakousky]

Ok, so it appears that DBCP does not like you to get access to the underlying 
connection without additional configuration, but Java does since 
java.sql.Connection extends java.sql.Wrapper from which you can do something 
like this:

Connection dbConnection= d.getConnection();
OracleConnection orcConnection = null;
if (dbConnection.isWrapperFor(OracleConnection.class)) {
orcConnection = dbConnection.unwrap(OracleConnection.class);
}
. . .

Let me know

Cheers
Oleg

On Mar 15, 2017, at 1:33 PM, Anil Rai 
<anilrain...@gmail.com<mailto:anilrain...@gmail.com>> wrote:

Hi Oleg, we tried and no luck. This is another example where we are seeing 
similar issue. Attached txt has the java code as well as the log.

Thanks in advance
Anil


On Wed, Mar 15, 2017 at 10:02 AM, Anil Rai 
<anilrain...@gmail.com<mailto:anilrain...@gmail.com>> wrote:
Thanks Oleg. Makes sense. Will try and keep you posted.

Regards
Anil


On Wed, Mar 15, 2017 at 9:56 AM, Oleg Zhurakousky 
<ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote:
Anil

Thank you for details. That does help a lot.

First, I want to make sure that it is clear that this is not a NiFi issue, 
since the problem is specific to the combination of DBCP and Oracle and the 
expectations between the two.

Seems like Oracle JDBC connection is wrapped in an implementation specific 
class (DBCP in this case I assume).
It is my believe that you need to obtain reference to native Oracle connection 
to avoid "JMS-112: Connection is invalid".
So, I think you need to try to cast your Connection object to DBCPs 
DelegatingConnection and then do something like this:

DelegatingConnection wrappedConn = (DelegatingConnection)con;
OracleConnection ocon =  null ;
if (wrappedConn != null)
 ocon = (OracleConnection) wrappedConn.getDelegate();
 . . .

Let me know how it goes

Cheers
Oleg

> On Mar 15, 2017, at 9:20 AM, Anil Rai 
> <anilrain...@gmail.com<mailto:anilrain...@gmail.com>> wrote:
>
> 2017-03-14 16:50:42,312 ERROR [NiFi logging handler] org.apache.nifi.StdErr
>  at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 2017-03-14 16:50:42,313 ERROR [NiFi logging handler] org.apache.nifi.StdErr
>  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> 2017-03-14 16:50:42,313 ERROR [NiFi logging handler] org.apache.nifi.StdErr
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> 2017-03-14 16:50:42,313 ERROR [NiFi logging handler] org.apache.nifi.StdErr
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> 2017-03-14 16:50:42,313 ERROR [NiFi logging handler] org.apache.nifi.StdErr
>  at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 2017-03-14 16:50:42,313 ERROR [NiFi logging handler] org.apache.nifi.StdErr
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 2017-03-14 16:50:42,313 ERROR [NiFi logging handler] org.apache.nifi.StdErr
>  at java.lang.Thread.run(Thread.java:745)
> 2017-03-14 16:50:43,567 INFO [NiFi logging handler] org.apache.nifi.StdOut
> Databse Connection :- 
> jdbc:oracle:thin:@xxxog.yy.com:1521/DB1<http://jdbc:oracle:thin:@xxxog.yy.com:1521/DB1>,
> UserName=user, Oracle JDBC driver
> 2017-03-14 16:50:43,567 ERROR [NiFi logging handler] org.apache.nifi.StdErr
> oracle.AQ.AQException: JMS-112: Connection is invalid
> 2017-03-14 16:50:43,567 INFO [NiFi logging handler] org.apache.nifi.StdOut
> AQ Driver Class ---> oracle.AQ.AQOracleDriver
> 2017-03-14 16:50:43,567 ERROR [NiFi logging handler] org.apache.nifi.StdErr
> 2017-03-14 16:50:43,568 INFO [NiFi logging handler] org.apache.nifi.StdOut
> Aq Sesssion ---> null
> 2017-03-14 16:50:43,568 ERROR [NiFi logging handler] org.apache.nifi.StdErr
> 2017-03-14 16:50:43,568 INFO [NiFi logging handler] org.apache.nifi.StdOut
> Queue Owner ---> APPS
> 2017-03-14 16:50:43,568 ERROR [NiFi logging handler] org.apache.nifi.StdErr
>  at oracle.AQ.AQDriverManager.createAQSession(AQDriverManager.java:193)
> 2017-03-14 16:50:43,569 INFO [NiFi logging handler] org.apache.nifi.StdOut
> QueueName ---> WF_BPEL_Q
> 2017-03-14 16:50:43,569 ERROR [NiFi logging handler] org.apache.nifi.StdErr
>  at
> com.oracle.xx.connector.processors.xx_SCon_ConsumeAQ.xx_Scon_ConsumeAQ.createSession(xx_Scon_ConsumeAQ.java:183)
> 2017-03-14 16:50:43,569 INFO [NiFi logging handler] org.apache.nifi.StdOut
> EventName ---> oracle.apps.ar.hz.CustAcctSite.update
> 2017-03-14 16:50:43,569 ERROR [NiFi logging handler] org.apache.nifi.StdErr
>  at
> com.oracle.xx.connector.processors.XX_SCon_ConsumeAQ.XX_Scon_Consume

Re: [VOTE] Release Apache NiFi nifi-nar-maven-plugin-1.2.0

[Oleg Zhurakousky]

Build successful, built sample NAR all is good
+1

> On Mar 15, 2017, at 10:25 AM, Matt Burgess  wrote:
> 
> +1 Release this package as nifi-nar-maven-plugin-1.2.0
> 
> Verified checksums, verified and built from commit, built a NAR with
> the updated plugin.
> 
> On Tue, Mar 14, 2017 at 12:21 PM, Bryan Bende  wrote:
>> Hello,
>> 
>> I am pleased to be calling this vote for the source release of Apache
>> NiFi nifi-nar-maven-plugin-1.2.0.
>> 
>> The source zip, including signatures, digests, etc. can be found at:
>> https://repository.apache.org/content/repositories/orgapachenifi-1101
>> 
>> The Git tag is nifi-nar-maven-plugin-1.2.0-RC1
>> The Git commit ID is d0c9d46d25a3eb8d3dbeb2783477b1a7c5b2f345
>> https://git-wip-us.apache.org/repos/asf?p=nifi-maven.git;a=commit;h=d0c9d46d25a3eb8d3dbeb2783477b1a7c5b2f345
>> 
>> Checksums of nifi-nar-maven-plugin-1.2.0-source-release.zip:
>> MD5: a20b62075f79bb890c270445097dc337
>> SHA1: 68e4739c9a4c4b2c69ff4adab8e1fdb0e7840923
>> SHA256: f5d4acbaa38460bcf19e9b33f385aa643798788026875bd034ee837e5d9d45a8
>> 
>> Release artifacts are signed with the following key:
>> https://people.apache.org/keys/committer/bbende.asc
>> 
>> KEYS file available here:
>> https://dist.apache.org/repos/dist/release/nifi/KEYS
>> 
>> 3 issues were closed/resolved for this release:
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12339193
>> 
>> Release note highlights can be found here:
>> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-NiFiNARMavenPluginVersion1.2.0
>> 
>> The vote will be open for 72 hours.
>> Please download the release candidate and evaluate the necessary items
>> including checking hashes, signatures, build from source, and test.
>> 
>> Then please vote:
>> 
>> [ ] +1 Release this package as nifi-nar-maven-plugin-1.2.0
>> [ ] +0 no opinion
>> [ ] -1 Do not release this package because because...
> 

Re: Connection Issue

[Oleg Zhurakousky]

16:50:43,570 ERROR [NiFi logging handler] org.apache.nifi.StdErr
>  at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 2017-03-14 16:50:43,570 ERROR [NiFi logging handler] org.apache.nifi.StdErr
>  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> 2017-03-14 16:50:43,570 ERROR [NiFi logging handler] org.apache.nifi.StdErr
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> 2017-03-14 16:50:43,570 ERROR [NiFi logging handler] org.apache.nifi.StdErr
>  at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> 2017-03-14 16:50:43,570 ERROR [NiFi logging handler] org.apache.nifi.StdErr
>  at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 2017-03-14 16:50:43,570 ERROR [NiFi logging handler] org.apache.nifi.StdErr
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 2017-03-14 16:50:43,570 ERROR [NiFi logging handler] org.apache.nifi.StdErr
>  at java.lang.Thread.run(Thread.java:745)
> 2017-03-14 16:50:43,570 ERROR [NiFi logging handler] org.apache.nifi.StdErr
> java.lang.NullPointerException
> 
> 
> On Wed, Mar 15, 2017 at 9:09 AM, Oleg Zhurakousky <
> ozhurakou...@hortonworks.com> wrote:
> 
>> Anil
>> 
>> Unfortunately the attachment didn’t come thru. Perhaps you can just paste
>> the relevant part of the exception.
>> 
>> Cheers
>> Oleg
>> 
>> On Mar 15, 2017, at 8:58 AM, Anil Rai <anilrain...@gmail.com> anilrain...@gmail.com>> wrote:
>> 
>> Hi Oleg, Thanks. attached is the log. Let me know if you want us to change
>> the log levels and re-run and send you additional logs?
>> 
>> 
>> On Tue, Mar 14, 2017 at 5:12 PM, Oleg Zhurakousky <
>> ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote:
>> Anil
>> 
>> I understand that you are having an issue and we are here to help, but we
>> can only do this if you help us just a little more, so it would be very
>> helpful if you provided a stack trace (I understand if you have to mask
>> sensitive information).
>> The “. . .fails saying cannot create AQSession. . .” could be due to
>> various reasons and until we see the stack trace everything here would be
>> speculation.
>> I hope you understand
>> 
>> Cheers
>> Oleg
>> 
>>> On Mar 14, 2017, at 4:59 PM, Anil Rai <anilrain...@gmail.com> anilrain...@gmail.com>> wrote:
>>> 
>>> Here is the behaviour that we have seen so forhope this helps
>>> 
>>>  1. When we run the java code in eclipse, it works and this is the
>>>  connection object that is printed ->
>>>  oracle.jdbc.driver.T4CConnection@6f75e721
>>>  2. When we hard code all the values as mentioned in my first email in a
>>>  custom processor, deploy that. It works as well. The above connection
>>>  object gets printed.
>>>  3. When we change the code in the custom processor to use the DBCP
>>>  connection service, deploy that. The connection object that gets
>> printed is
>>>  jdbc:oracle:thin:@oged-scan.og.ge.com:1521/ORPOGPB1<http:/
>> /jdbc:oracle:thin:@oged-scan.og.ge.com:1521/ORPOGPB1> and this does not
>>>  work. aqSession = AQDriverManager.createAQSession(connection) fails
>>>  saying cannot create AQSession.
>>> 
>>> Thanks
>>> Anil
>>> 
>>> 
>>> On Tue, Mar 14, 2017 at 4:13 PM, Oleg Zhurakousky <
>>> ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>>
>> wrote:
>>> 
>>>> Anil
>>>> 
>>>> When you say "it does not like the connection object. . .” what do you
>>>> mean by that?
>>>> Can you please provide stack trace or some other details?
>>>> 
>>>> Cheers
>>>> Oleg
>>>> 
>>>>> On Mar 14, 2017, at 4:06 PM, Anil Rai <anilrain...@gmail.com> anilrain...@gmail.com>> wrote:
>>>>> 
>>>>> Thanks Russ. Yes, we are doing exactly the same thing.
>>>>> 
>>>>>  driverClass = context.getProperty(DRIVER_CLASS).getValue();
>>>>>  queueName = context.getProperty(QUEUE_NAME).getValue();
>>>>>  databaseSchema = context.getProperty(DATABASE_
>> SCHEMA).getValue();
>>>>>  consumerName = context.getProperty(CONSUMER_NAME).getValue();
>>>>>  event

Re: Connection Issue

[Oleg Zhurakousky]

Anil

Unfortunately the attachment didn’t come thru. Perhaps you can just paste the 
relevant part of the exception.

Cheers
Oleg

On Mar 15, 2017, at 8:58 AM, Anil Rai 
<anilrain...@gmail.com<mailto:anilrain...@gmail.com>> wrote:

Hi Oleg, Thanks. attached is the log. Let me know if you want us to change the 
log levels and re-run and send you additional logs?


On Tue, Mar 14, 2017 at 5:12 PM, Oleg Zhurakousky 
<ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote:
Anil

I understand that you are having an issue and we are here to help, but we can 
only do this if you help us just a little more, so it would be very helpful if 
you provided a stack trace (I understand if you have to mask sensitive 
information).
The “. . .fails saying cannot create AQSession. . .” could be due to various 
reasons and until we see the stack trace everything here would be speculation.
I hope you understand

Cheers
Oleg

> On Mar 14, 2017, at 4:59 PM, Anil Rai 
> <anilrain...@gmail.com<mailto:anilrain...@gmail.com>> wrote:
>
> Here is the behaviour that we have seen so forhope this helps
>
>   1. When we run the java code in eclipse, it works and this is the
>   connection object that is printed ->
>   oracle.jdbc.driver.T4CConnection@6f75e721
>   2. When we hard code all the values as mentioned in my first email in a
>   custom processor, deploy that. It works as well. The above connection
>   object gets printed.
>   3. When we change the code in the custom processor to use the DBCP
>   connection service, deploy that. The connection object that gets printed is
>   
> jdbc:oracle:thin:@oged-scan.og.ge.com:1521/ORPOGPB1<http://jdbc:oracle:thin:@oged-scan.og.ge.com:1521/ORPOGPB1>
>  and this does not
>   work. aqSession = AQDriverManager.createAQSession(connection) fails
>   saying cannot create AQSession.
>
> Thanks
> Anil
>
>
> On Tue, Mar 14, 2017 at 4:13 PM, Oleg Zhurakousky <
> ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote:
>
>> Anil
>>
>> When you say "it does not like the connection object. . .” what do you
>> mean by that?
>> Can you please provide stack trace or some other details?
>>
>> Cheers
>> Oleg
>>
>>> On Mar 14, 2017, at 4:06 PM, Anil Rai 
>>> <anilrain...@gmail.com<mailto:anilrain...@gmail.com>> wrote:
>>>
>>> Thanks Russ. Yes, we are doing exactly the same thing.
>>>
>>>   driverClass = context.getProperty(DRIVER_CLASS).getValue();
>>>   queueName = context.getProperty(QUEUE_NAME).getValue();
>>>   databaseSchema = context.getProperty(DATABASE_SCHEMA).getValue();
>>>   consumerName = context.getProperty(CONSUMER_NAME).getValue();
>>>   eventName = context.getProperty(EVENT_NAME).getValue();
>>>   DBCPService connection =
>>> context.getProperty(JDBC_CONNECTION_POOL).asControllerService(
>> DBCPService.class);
>>>   Connection conn = connection.getConnection();
>>> *aqSession = AQDriverManager.createAQSession(connection);*
>>>
>>> The underlined code above fails as it does not like the connection object
>>> that is returned by the DBCPService.
>>>
>>> Regards
>>> Anil
>>>
>>>
>>> On Tue, Mar 14, 2017 at 2:43 PM, Russell Bateman 
>>> <r...@windofkeltia.com<mailto:r...@windofkeltia.com>>
>>> wrote:
>>>
>>>> Anil,
>>>>
>>>> Typically, your custom processor should have a property, something like
>>>>
>>>>   public static final PropertyDescriptor DBCP_SERVICE = new
>>>> PropertyDescriptor.Builder()
>>>>   .name("Database Connection Pooling Service")
>>>>   .description("The Controller Service that is used to obtain
>>>> connection to database")
>>>>   .required(true)
>>>>   .identifiesControllerService(DBCPService.class)
>>>>   .build();
>>>>
>>>> When your NiFi user sets up the flow, he or she puts a reference to
>> NiFi's
>>>> DBCPConnectionPool in it. In configuring that (a ControllerService, you
>>>> tell it that it's Oracle, location, etc.)
>>>>
>>>> Then, your onTrigger() code would do something like this:
>>>>
>>>>   final DBCPService dbcpService = context.getProperty(DBCP_SERVI
>>>> CE).asControllerService(DBCPService.class);
>>>>
>>>>
>>>> and later...
>>>>
>>>>
>

Re: Connection Issue

[Oleg Zhurakousky]

Anil

I understand that you are having an issue and we are here to help, but we can 
only do this if you help us just a little more, so it would be very helpful if 
you provided a stack trace (I understand if you have to mask sensitive 
information). 
The “. . .fails saying cannot create AQSession. . .” could be due to various 
reasons and until we see the stack trace everything here would be speculation. 
I hope you understand

Cheers
Oleg

> On Mar 14, 2017, at 4:59 PM, Anil Rai <anilrain...@gmail.com> wrote:
> 
> Here is the behaviour that we have seen so forhope this helps
> 
>   1. When we run the java code in eclipse, it works and this is the
>   connection object that is printed ->
>   oracle.jdbc.driver.T4CConnection@6f75e721
>   2. When we hard code all the values as mentioned in my first email in a
>   custom processor, deploy that. It works as well. The above connection
>   object gets printed.
>   3. When we change the code in the custom processor to use the DBCP
>   connection service, deploy that. The connection object that gets printed is
>   jdbc:oracle:thin:@oged-scan.og.ge.com:1521/ORPOGPB1 and this does not
>   work. aqSession = AQDriverManager.createAQSession(connection) fails
>   saying cannot create AQSession.
> 
> Thanks
> Anil
> 
> 
> On Tue, Mar 14, 2017 at 4:13 PM, Oleg Zhurakousky <
> ozhurakou...@hortonworks.com> wrote:
> 
>> Anil
>> 
>> When you say "it does not like the connection object. . .” what do you
>> mean by that?
>> Can you please provide stack trace or some other details?
>> 
>> Cheers
>> Oleg
>> 
>>> On Mar 14, 2017, at 4:06 PM, Anil Rai <anilrain...@gmail.com> wrote:
>>> 
>>> Thanks Russ. Yes, we are doing exactly the same thing.
>>> 
>>>   driverClass = context.getProperty(DRIVER_CLASS).getValue();
>>>   queueName = context.getProperty(QUEUE_NAME).getValue();
>>>   databaseSchema = context.getProperty(DATABASE_SCHEMA).getValue();
>>>   consumerName = context.getProperty(CONSUMER_NAME).getValue();
>>>   eventName = context.getProperty(EVENT_NAME).getValue();
>>>   DBCPService connection =
>>> context.getProperty(JDBC_CONNECTION_POOL).asControllerService(
>> DBCPService.class);
>>>   Connection conn = connection.getConnection();
>>> *aqSession = AQDriverManager.createAQSession(connection);*
>>> 
>>> The underlined code above fails as it does not like the connection object
>>> that is returned by the DBCPService.
>>> 
>>> Regards
>>> Anil
>>> 
>>> 
>>> On Tue, Mar 14, 2017 at 2:43 PM, Russell Bateman <r...@windofkeltia.com>
>>> wrote:
>>> 
>>>> Anil,
>>>> 
>>>> Typically, your custom processor should have a property, something like
>>>> 
>>>>   public static final PropertyDescriptor DBCP_SERVICE = new
>>>> PropertyDescriptor.Builder()
>>>>   .name("Database Connection Pooling Service")
>>>>   .description("The Controller Service that is used to obtain
>>>> connection to database")
>>>>   .required(true)
>>>>   .identifiesControllerService(DBCPService.class)
>>>>   .build();
>>>> 
>>>> When your NiFi user sets up the flow, he or she puts a reference to
>> NiFi's
>>>> DBCPConnectionPool in it. In configuring that (a ControllerService, you
>>>> tell it that it's Oracle, location, etc.)
>>>> 
>>>> Then, your onTrigger() code would do something like this:
>>>> 
>>>>   final DBCPService dbcpService = context.getProperty(DBCP_SERVI
>>>> CE).asControllerService(DBCPService.class);
>>>> 
>>>> 
>>>> and later...
>>>> 
>>>> 
>>>>   try (final Connection connection = dbcpService.getConnection())
>>>>   {
>>>>   try (final Statement stmt = connection.createStatement(
>> ResultSet.TYPE_FORWARD_ONLY,
>>>> ResultSet.CONCUR_READ_ONLY))
>>>> 
>>>> etc.
>>>> 
>>>> Does this help?
>>>> 
>>>> Russ
>>>> 
>>>> 
>>>> 
>>>> On 03/14/2017 11:54 AM, Anil Rai wrote:
>>>> 
>>>>> We have a use case to connect to oracle database and subscribe to
>> Advanced
>>>>> Queuing (https://docs.oracle.com/cd/A58617_01/server.804/a58241/
>> ch_aq.htm
>>>>> ).
>>&

Re: Connection Issue

[Oleg Zhurakousky]

Anil

When you say "it does not like the connection object. . .” what do you mean by 
that?
Can you please provide stack trace or some other details?

Cheers
Oleg

> On Mar 14, 2017, at 4:06 PM, Anil Rai  wrote:
> 
> Thanks Russ. Yes, we are doing exactly the same thing.
> 
>driverClass = context.getProperty(DRIVER_CLASS).getValue();
>queueName = context.getProperty(QUEUE_NAME).getValue();
>databaseSchema = context.getProperty(DATABASE_SCHEMA).getValue();
>consumerName = context.getProperty(CONSUMER_NAME).getValue();
>eventName = context.getProperty(EVENT_NAME).getValue();
>DBCPService connection =
> context.getProperty(JDBC_CONNECTION_POOL).asControllerService(DBCPService.class);
>Connection conn = connection.getConnection();
> *aqSession = AQDriverManager.createAQSession(connection);*
> 
> The underlined code above fails as it does not like the connection object
> that is returned by the DBCPService.
> 
> Regards
> Anil
> 
> 
> On Tue, Mar 14, 2017 at 2:43 PM, Russell Bateman 
> wrote:
> 
>> Anil,
>> 
>> Typically, your custom processor should have a property, something like
>> 
>>public static final PropertyDescriptor DBCP_SERVICE = new
>> PropertyDescriptor.Builder()
>>.name("Database Connection Pooling Service")
>>.description("The Controller Service that is used to obtain
>> connection to database")
>>.required(true)
>>.identifiesControllerService(DBCPService.class)
>>.build();
>> 
>> When your NiFi user sets up the flow, he or she puts a reference to NiFi's
>> DBCPConnectionPool in it. In configuring that (a ControllerService, you
>> tell it that it's Oracle, location, etc.)
>> 
>> Then, your onTrigger() code would do something like this:
>> 
>>final DBCPService dbcpService = context.getProperty(DBCP_SERVI
>> CE).asControllerService(DBCPService.class);
>> 
>> 
>> and later...
>> 
>> 
>>try (final Connection connection = dbcpService.getConnection())
>>{
>>try (final Statement stmt = 
>> connection.createStatement(ResultSet.TYPE_FORWARD_ONLY,
>> ResultSet.CONCUR_READ_ONLY))
>> 
>> etc.
>> 
>> Does this help?
>> 
>> Russ
>> 
>> 
>> 
>> On 03/14/2017 11:54 AM, Anil Rai wrote:
>> 
>>> We have a use case to connect to oracle database and subscribe to Advanced
>>> Queuing (https://docs.oracle.com/cd/A58617_01/server.804/a58241/ch_aq.htm
>>> ).
>>> Below is the java snippet to establish this connection from a java client.
>>> We can run this in eclipse and consume message from the advanced queue.
>>> **
>>> Class.forName("oracle.jdbc.driver.OracleDriver");
>>> connection = DriverManager.getConnection("
>>> jdbc:oracle:thin:@xxx-yyy.zzz.com:1521/DB1","user", "pwd");
>>> connection.setAutoCommit(true);
>>> Class.forName("oracle.AQ.AQOracleDriver");
>>> aqSession = AQDriverManager.createAQSession(connection);
>>> System.out.println("AQ Session --->" + aqSession);
>>> 
>>> 
>>> We have created a custom processor in Nifi. This processor is getting the
>>> connection string using getConnection function of Standard DBCP service.
>>> The problem is, the connection object that is retrieved from eclipse
>>> versus
>>> what is returned from DBCP service is different. We have made sure we are
>>> referring to the same jar both in eclipse and Nifi (ojdbc7.jar)
>>> It fails @  aqSession = AQDriverManager.createAQSession(connection);
>>> The connection object that comes from DBCP is not what is expected by
>>> AQDriverManager.
>>> 
>>> Any help is greatly appreciated.
>>> 
>>> Thanks
>>> Anil
>>> 
>>> 
>> 

Re: NiFI XProc Processor

[Oleg Zhurakousky]

Steve

Thank you very much for desire to contribute and such a detailed explanation of 
your contribution.
I left some comments in line [OLEG], so let us know what you think.

Cheers
Oleg

> On Mar 7, 2017, at 10:17 AM, Steve Lawrence  wrote:
> 
> We have developed a NiFi processor that uses XMLCalabash [1] to add
> support for XProc [2] processing. XProc is an XML transformation
> language that defines and XML pipeline, allowing for complex validation,
> transformation, and routing of XML data within the pipeline, using
> existing XML technologies such as RelaxNG, Schematron, XSD Schema,
> XQuery, XSLT, XPath and custom XProc transformations.
> 
> This new processor is mostly straightforward, but we had some questions
> regarding the specific implementation and the handling of non-thread
> safe code. The code is available for viewing here:
> 
> 
> https://opensource.ncsa.illinois.edu/bitbucket/projects/DFDL/repos/nifi-xproc/browse
> 
> In this processor, a property is created to provide an XProc file, which
> defines the pipeline input and output "ports". XML goes into an input
> port, goes through the pipeline, and one or more XML documents exit at
> specified output ports. This NiFi processor maps each output port to a
> dynamic NiFi relationship. It does this mapping in the
> onPropertyModified method when the XProc file property is changed. This
> method also stores the XMLCalabash XRuntime and XPipeline objects (which
> do all the pipeline work) in volatile member variables to be used later
> in onTrigger. The members are saved here to avoid recreating them in
> each call to onTrigger. Is this an acceptable place to do that? It seems
> this normally happens in an @OnScheduled method or in the first call to
> onTrigger, however the objects must be created in onPropertyModified to
> get the output ports, so this does avoid recreating the same objects
> multiple times.
[OLEG] Without getting into more details, both approaches are acceptable. 
However assigning values in onTrigger()in certain cases is more preferable. 
Those cases primarily deal with obtaining references to a remote resource 
(i.e., connection factory, socket etc) and for those cases exception handling 
is much simpler. I can definitely elaborate further if need to and point to a 
few examples where we do that, but it appears that it is not the case for you, 
so your current approach seems acceptable. And as far as multi-threading for 
onTrigger(), such assignments are done in a typical synchronized block with 
null check.

> Also note that the same objects are created in the
> XML_PIPELINE_VALIDATOR but are not saved due to the validator being
> static, so there is already some duplication. Is there a standard way to
> avoid duplication/is this an acceptable way to handle this?

[OLEG] Not fully understand the question, but keep in mind that regardless of 
the amount of threads, there is only one instance of the processor at any given 
time, so any reference held by such instance is essentially a singleton as 
well. Does that help?
> 
> The other concern we have is that the XPipeline and XRuntime objects
> created by XML Calabash are not thread safe. To resolve this issue, the
> processor is annotated with @TriggerSerially. Is this the correct
> solution, or is there a some other preferred method. Perhaps ThreadLocal
> or a thread safe pool of XPipeline objects is preferred?

[OLEG] Definitely not thread local since there is no guarantee that you will 
get the same thread or a particular thread on subsequent invocation. The 
@TriggerSerially is obviously the most defensive way to avoid collisions. That 
said I probably need to better understand the issue. However off the top of my 
head one way of ensuring the correctness for such scenarios is to maintain a 
Map of such objects as an instance variable (like a pool) where key is 
something that would ensure that you always get the correct object.
> 
> 
> Lastly, is this something the devs would be interested in pulling in
> NiFI, and if not, what could be changed to achieve this? The code is
> licensed as Apache v2 and we would be happy to contribute the code to
> NiFi if deemed acceptable.

[OLEG] This is probably the most difficult question to answer since immediate 
answer is we don’t know ;) Only the community can decide. So what I would 
suggest is to raise a JIRA - https://issues.apache.org/jira/browse/NIFI and 
submit a PR for it and see if it gets any traction. Further more we are 
currently working on the concept of the Extension/Artifact Registry to 
accommodate growing request for more NiFi components. 
> 
> Thanks,
> - Steve
> 
> [1] http://xmlcalabash.com/
> [2] https://www.w3.org/TR/xproc/
> 

Re: Hosting API's on Nifi

[Oleg Zhurakousky]

Anil

Aside from opening another port I don’t se how you can overcome this issue. 
HandleHttpRequest essentially starts another web server and this server needs a 
port to listen on.
Further more, there are many other network based Processors that come with NiFi 
that would fall int the same category - “a processors that need to bind to a 
port” to facilitate communication with external systems, so I’d recommend 
bringing this up with your AWS admins.

I know there is nor much help in my reply, but I hope you understand.

Cheers
Oleg

> On Mar 2, 2017, at 2:08 PM, Rai, Anil (GE Digital)  wrote:
> 
> 
> 
> 
> Hello All,
> 
> I  am exposing an API using HandleHttpRequest on my local nifi instance. The 
> HandleHttpRequest processor requires a Listening port that I need to provide. 
> If I enter 80 in that field, the processor fails when it starts saying 
> “unable to initialize the server”. Which is expected as the webserver uses 
> that port to serve the canvas. So if we provide any other randon number then 
> it works fine.
> 
> When I promote the above API on the nifi cluster that is hosted on our AWS 
> farm, then we are unable to invoke this API. As only 80 and 443 are opened on 
> AWS.
> 
> How do we overcome this problem?
> 
> Regards
> Anil
> 

Re: [REMINDER] Please signoff when committing other people's changes

[Oleg Zhurakousky]

Thanks Bryan.

If ‘-s’ is only for showcasing the committer I don’t believe anyone would have 
any issues with it, but my concern at the moment is purely legal, so I am not 
sure who is the right person to answer that, but figured raising the concern is 
the least I can do.

Cheers
Oleg


> On Mar 2, 2017, at 8:16 AM, Bryan Bende <bbe...@gmail.com> wrote:
> 
> The sign-off is so we can easily see who the reviewer/merger was from
> the git history.
> 
> We can always go back to the JIRA or PR and the reviewer/merger should
> have commented there, but its convenient to see it in the git history
> in my opinion.
> 
> Personally, whenever merging someones contribution I use "git am
> --signoff < patchfile" which I guess is equivalent to doing the ammend
> after applying the patch.
> 
> 
> On Thu, Mar 2, 2017 at 8:05 AM, Oleg Zhurakousky
> <ozhurakou...@hortonworks.com> wrote:
>> Andre
>> 
>> Thanks for the reminder. I admit that I did not know that we require it in 
>> the Contributor Guide, so thanks for pointing it out.
>> However, your email did prompt me to look at the purpose and origin of the 
>> ‘-s’ flag and led me to this thread on Stack Overflow - 
>> http://stackoverflow.com/questions/1962094/what-is-the-sign-off-feature-in-git-for.
>> 
>> And I am now wondering if we should require it or even use it in the first 
>> place, since it’s origin, history and purpose appears to have more 
>> “individual” legal implications then showcasing the actual committer.
>> 
>> Thoughts?
>> 
>> Cheers
>> Oleg
>> 
>> On Mar 2, 2017, at 6:35 AM, Andre 
>> <andre-li...@fucs.org<mailto:andre-li...@fucs.org>> wrote:
>> 
>> dev,
>> 
>> May I remind you to ensure we follow the Contributor Guide and use:
>> 
>> git commit --amend -s
>> 
>> when merging commits from your peers?
>> 
>> While git pretty-format can be used to reveal the committer, I am sure that
>> all of us will agree that as an inclusive community we value both the
>> pretty and ugly formats...
>> 
>> So can we give the ugly format the support it deserves and ensure we add
>> the neat Signed-off-by stamp to the commit message?
>> 
>> Cheers
>> 
> 

Re: [REMINDER] Please signoff when committing other people's changes

[Oleg Zhurakousky]

Andre

Thanks for the reminder. I admit that I did not know that we require it in the 
Contributor Guide, so thanks for pointing it out.
However, your email did prompt me to look at the purpose and origin of the ‘-s’ 
flag and led me to this thread on Stack Overflow - 
http://stackoverflow.com/questions/1962094/what-is-the-sign-off-feature-in-git-for.

And I am now wondering if we should require it or even use it in the first 
place, since it’s origin, history and purpose appears to have more “individual” 
legal implications then showcasing the actual committer.

Thoughts?

Cheers
Oleg

On Mar 2, 2017, at 6:35 AM, Andre 
> wrote:

dev,

May I remind you to ensure we follow the Contributor Guide and use:

git commit --amend -s

when merging commits from your peers?

While git pretty-format can be used to reveal the committer, I am sure that
all of us will agree that as an inclusive community we value both the
pretty and ugly formats...

So can we give the ugly format the support it deserves and ensure we add
the neat Signed-off-by stamp to the commit message?

Cheers

Re: session recover behaviour in nifi-jms-processor

[Oleg Zhurakousky]

Sorry, just noticed the typo. Instead of “limit the possibility of a message 
loads. . .” should be "limit the possibility of a message loss…"

Cheers
Oleg
> On Feb 24, 2017, at 9:28 AM, Oleg Zhurakousky <ozhurakou...@hortonworks.com> 
> wrote:
> 
> Dominic
> 
> Great to hear from you!
> As you can see from the inline comment in the code, the recover is there for 
> a reason primarily to ensure or should I say limit the possibility of a 
> message loads in the event of a processor and/or NiFi crash. 
> As you may be aware, in NiFi we do prefer message duplication over message 
> loss. That said, I do see several possibilities for improvement especially 
> for the high traffic scenarios you are describing. One such improvement would 
> be to create a listening container version of ConsumeJMS which has far more 
> control over threading and session caching/sharing.
> 
> Would you mind racing a JIRA issue - 
> https://issues.apache.org/jira/browse/NIFI/ describing everything you just 
> did and we’ll handle it.
> 
> Cheers
> Oleg
> 
>> On Feb 24, 2017, at 9:08 AM, Dominik Benz <dominik.b...@gmail.com> wrote:
>> 
>> Hi,
>> 
>> we're currently using Nifi to consume a relatively high-traffic JMS topic
>> (40-60 messages per second). 
>> 
>> Worked well in principle - however, we then noticed that the the outbound
>> rate (i.e. the number of messages we fetched) of the topic was consistently
>> slightly higher than its inbound rate (i.e. the actual number of messages
>> sent to the topic). This puzzled me, because (being the only subscriber to
>> the topic) I would expect inbound and outbound traffic to be identical
>> (given we can consume fast enough, which we can).
>> 
>> Digging deeper, I found that in 
>> 
>> org.apache.nifi.jms.processors.JMSConsumer
>> 
>> the method "consume" performs a session.recover:
>> 
>> 
>> 
>> session.recover (as written in the comment) basically stops message delivery
>> and re-starts from the last non-acked message. However, I think this leads
>> to the following issue in high-traffic contexts:
>> 
>> 1) several threads perform the JMS session callback in parallel
>> 2) each callback performs a session.recover
>> 3) during high traffic, the situation arises that the ACKs from another
>> thread may not (yet) have arrived at the JMS server
>> 4) this implies that the pointer of session.recover will reconsume the
>> not-yet-acked message from another thread
>> 
>> For verification, I performed so far the following steps:
>> 
>> (a) manual implementation of a simplistic synchronous JMS topic consumer ->
>> inbound/outbound identical as expected
>> (b) patched nifi-jms-processors and commented out session.recover() -
>> inbout/outbound identical as expected
>> 
>> Any thoughts on this? My current impression is that session.recover in its
>> current usage doesn't play well together with the multi-threaded JMS
>> consumers. Or do I have any misconception? 
>> 
>> Thanks & best regards,
>> Dominik
>> 
>> 
>> 
>> --
>> View this message in context: 
>> http://apache-nifi-developer-list.39713.n7.nabble.com/session-recover-behaviour-in-nifi-jms-processor-tp14940.html
>> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
>> 
> 

Re: session recover behaviour in nifi-jms-processor

[Oleg Zhurakousky]

Dominic

Great to hear from you!
As you can see from the inline comment in the code, the recover is there for a 
reason primarily to ensure or should I say limit the possibility of a message 
loads in the event of a processor and/or NiFi crash. 
As you may be aware, in NiFi we do prefer message duplication over message 
loss. That said, I do see several possibilities for improvement especially for 
the high traffic scenarios you are describing. One such improvement would be to 
create a listening container version of ConsumeJMS which has far more control 
over threading and session caching/sharing.

Would you mind racing a JIRA issue - 
https://issues.apache.org/jira/browse/NIFI/ describing everything you just did 
and we’ll handle it.

Cheers
Oleg

> On Feb 24, 2017, at 9:08 AM, Dominik Benz  wrote:
> 
> Hi,
> 
> we're currently using Nifi to consume a relatively high-traffic JMS topic
> (40-60 messages per second). 
> 
> Worked well in principle - however, we then noticed that the the outbound
> rate (i.e. the number of messages we fetched) of the topic was consistently
> slightly higher than its inbound rate (i.e. the actual number of messages
> sent to the topic). This puzzled me, because (being the only subscriber to
> the topic) I would expect inbound and outbound traffic to be identical
> (given we can consume fast enough, which we can).
> 
> Digging deeper, I found that in 
> 
>  org.apache.nifi.jms.processors.JMSConsumer
> 
> the method "consume" performs a session.recover:
> 
> 
> 
> session.recover (as written in the comment) basically stops message delivery
> and re-starts from the last non-acked message. However, I think this leads
> to the following issue in high-traffic contexts:
> 
> 1) several threads perform the JMS session callback in parallel
> 2) each callback performs a session.recover
> 3) during high traffic, the situation arises that the ACKs from another
> thread may not (yet) have arrived at the JMS server
> 4) this implies that the pointer of session.recover will reconsume the
> not-yet-acked message from another thread
> 
> For verification, I performed so far the following steps:
> 
> (a) manual implementation of a simplistic synchronous JMS topic consumer ->
> inbound/outbound identical as expected
> (b) patched nifi-jms-processors and commented out session.recover() -
> inbout/outbound identical as expected
> 
> Any thoughts on this? My current impression is that session.recover in its
> current usage doesn't play well together with the multi-threaded JMS
> consumers. Or do I have any misconception? 
> 
> Thanks & best regards,
>  Dominik
> 
> 
> 
> --
> View this message in context: 
> http://apache-nifi-developer-list.39713.n7.nabble.com/session-recover-behaviour-in-nifi-jms-processor-tp14940.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
> 

Re: [DISCUSS] Scale-out/Object Storage - taming the diversity of processors

[Oleg Zhurakousky]

Just wanted to add one more point which IMHO just as important. . .
Certain “artifacts” (i.e., NARs that depends on libraries which are not ASF 
friendly) may not fit the ASF licensing requirements of genuine Apache NiFi 
distribution, yet add a great value for greater community of NiFi users, so 
having them NOT being part of official NiFi distribution is a value in itself.

Cheers
Oleg

> On Feb 22, 2017, at 12:52 PM, Oleg Zhurakousky <ozhurakou...@hortonworks.com> 
> wrote:
> 
> Adam
> 
> I 100% agree with your comment on "official/sanctioned”. As an external 
> artifact registry such as BinTray for example or GitHub, one can not control 
> what is there, rather how to get it. The final decision is left to the end 
> user.
> Artifacts could be rated and/or Apache NiFi (and/or commercial distributions 
> of NiFi) can “endorse” and/or “un-endorse” certain artifacts and IMHO that is 
> perfectly fine. On top of that a future distribution of NiFi can have 
> configuration to account for the “endorsed/supported” artifacts, yet it 
> should not stop one from downloading and trying something new.
> 
> Cheers
> Oleg
> 
>> On Feb 22, 2017, at 12:43 PM, Adam Lamar <adamond...@gmail.com> wrote:
>> 
>> Hey all,
>> 
>> I can understand Andre's perspective - when I was building the ListS3
>> processor, I mostly just copied the bits that made sense from ListHDFS and
>> ListFile. That worked, but its a poor way to ensure consistency across
>> List* processors.
>> 
>> As a once-in-a-while contributor, I love the idea that community
>> contributions are respected and we're not dropping them, because they solve
>> real needs right now, and it isn't clear another approach would be better.
>> 
>> And I disagree slightly with the notion that an artifact registry will
>> solve the problem - I think it could make it worse, at least from a
>> consistency point of view. Taming _is_ important, which is one reason
>> registry communities have official/sanctioned modules. Quality and
>> interoperability can vary vastly.
>> 
>> By convention, it seems like NiFi already has a handful of well-understood
>> patterns - List, Fetch, Get, Put, etc all mean something specific in
>> processor terms. Is there a reason not to formalize those patterns in the
>> code as well? That would help with processor consistency, and if done
>> right, it may even be easier to write new processors, fix bugs, etc.
>> 
>> For example, ListS3 initially shipped with some bad session commit()
>> behavior, which was obvious once identified, but a generalized
>> AbstractListProcessor (higher level that the one that already exists) could
>> make it easier to avoid this class of bug.
>> 
>> Admittedly this could be a lot of work.
>> 
>> Cheers,
>> Adam
>> 
>> 
>> 
>> On Wed, Feb 22, 2017 at 8:38 AM, Oleg Zhurakousky <
>> ozhurakou...@hortonworks.com> wrote:
>> 
>>> I’ll second Pierre
>>> 
>>> Yes with the current deployment model the amount of processors and the
>>> size of NiFi distribution is a concern simply because it’s growing with
>>> each release. But it should not be the driver to start jamming more
>>> functionality into existing processors which on the surface may look like
>>> related (even if they are).
>>> Basically a processor should never be complex with regard to it being
>>> understood by the end user who is non-technical, so “specialization” is
>>> always takes precedence here since it limits “configuration” and thus
>>> making such processor simpler. It also helps with maintenance and
>>> management of such processor by the developer. Also, having multiple
>>> related processors will promote healthy competition where my MyputHDFS may
>>> for certain cases be better/faster then YourPutHDFS and why not have both?
>>> 
>>> The “artifact registry” (flow, extension, template etc) is the only answer
>>> here since it will remove the “proliferation” and the need for “taming”
>>> anything from the picture. With “artifact registry” one or one million
>>> processors, the NiFi size/state will always remain constant and small.
>>> 
>>> Cheers
>>> Oleg
>>>> On Feb 22, 2017, at 6:05 AM, Pierre Villard <pierre.villard...@gmail.com>
>>> wrote:
>>>> 
>>>> Hey guys,
>>>> 
>>>> Thanks for the thread Andre.
>>>> 
>>>> +1 to James' answer.
>>>> 
>>>> I understand the interest that would provide a single processor to
>>> connect

Re: [DISCUSS] Scale-out/Object Storage - taming the diversity of processors

[Oleg Zhurakousky]

Adam

I 100% agree with your comment on "official/sanctioned”. As an external 
artifact registry such as BinTray for example or GitHub, one can not control 
what is there, rather how to get it. The final decision is left to the end user.
Artifacts could be rated and/or Apache NiFi (and/or commercial distributions of 
NiFi) can “endorse” and/or “un-endorse” certain artifacts and IMHO that is 
perfectly fine. On top of that a future distribution of NiFi can have 
configuration to account for the “endorsed/supported” artifacts, yet it should 
not stop one from downloading and trying something new.

Cheers
Oleg

> On Feb 22, 2017, at 12:43 PM, Adam Lamar <adamond...@gmail.com> wrote:
> 
> Hey all,
> 
> I can understand Andre's perspective - when I was building the ListS3
> processor, I mostly just copied the bits that made sense from ListHDFS and
> ListFile. That worked, but its a poor way to ensure consistency across
> List* processors.
> 
> As a once-in-a-while contributor, I love the idea that community
> contributions are respected and we're not dropping them, because they solve
> real needs right now, and it isn't clear another approach would be better.
> 
> And I disagree slightly with the notion that an artifact registry will
> solve the problem - I think it could make it worse, at least from a
> consistency point of view. Taming _is_ important, which is one reason
> registry communities have official/sanctioned modules. Quality and
> interoperability can vary vastly.
> 
> By convention, it seems like NiFi already has a handful of well-understood
> patterns - List, Fetch, Get, Put, etc all mean something specific in
> processor terms. Is there a reason not to formalize those patterns in the
> code as well? That would help with processor consistency, and if done
> right, it may even be easier to write new processors, fix bugs, etc.
> 
> For example, ListS3 initially shipped with some bad session commit()
> behavior, which was obvious once identified, but a generalized
> AbstractListProcessor (higher level that the one that already exists) could
> make it easier to avoid this class of bug.
> 
> Admittedly this could be a lot of work.
> 
> Cheers,
> Adam
> 
> 
> 
> On Wed, Feb 22, 2017 at 8:38 AM, Oleg Zhurakousky <
> ozhurakou...@hortonworks.com> wrote:
> 
>> I’ll second Pierre
>> 
>> Yes with the current deployment model the amount of processors and the
>> size of NiFi distribution is a concern simply because it’s growing with
>> each release. But it should not be the driver to start jamming more
>> functionality into existing processors which on the surface may look like
>> related (even if they are).
>> Basically a processor should never be complex with regard to it being
>> understood by the end user who is non-technical, so “specialization” is
>> always takes precedence here since it limits “configuration” and thus
>> making such processor simpler. It also helps with maintenance and
>> management of such processor by the developer. Also, having multiple
>> related processors will promote healthy competition where my MyputHDFS may
>> for certain cases be better/faster then YourPutHDFS and why not have both?
>> 
>> The “artifact registry” (flow, extension, template etc) is the only answer
>> here since it will remove the “proliferation” and the need for “taming”
>> anything from the picture. With “artifact registry” one or one million
>> processors, the NiFi size/state will always remain constant and small.
>> 
>> Cheers
>> Oleg
>>> On Feb 22, 2017, at 6:05 AM, Pierre Villard <pierre.villard...@gmail.com>
>> wrote:
>>> 
>>> Hey guys,
>>> 
>>> Thanks for the thread Andre.
>>> 
>>> +1 to James' answer.
>>> 
>>> I understand the interest that would provide a single processor to
>> connect
>>> to all the back ends... and we could document/improve the PutHDFS to ease
>>> such use but I really don't think that it will benefit the user
>> experience.
>>> That may be interesting in some cases for some users but I don't think
>> that
>>> would be a majority.
>>> 
>>> I believe NiFi is great for one reason: you have a lot of specialized
>>> processors that are really easy to use and efficient for what they've
>> been
>>> designed for.
>>> 
>>> Let's ask ourselves the question the other way: with the NiFi registry on
>>> its way, what is the problem having multiple processors for each back
>> end?
>>> I don't really see the issue here. OK we have a lot of processors (but I
>>> believe this is a good point for NiF

Re: [DISCUSS] Scale-out/Object Storage - taming the diversity of processors

[Oleg Zhurakousky]

I’ll second Pierre

Yes with the current deployment model the amount of processors and the size of 
NiFi distribution is a concern simply because it’s growing with each release. 
But it should not be the driver to start jamming more functionality into 
existing processors which on the surface may look like related (even if they 
are).
Basically a processor should never be complex with regard to it being 
understood by the end user who is non-technical, so “specialization” is always 
takes precedence here since it limits “configuration” and thus making such 
processor simpler. It also helps with maintenance and management of such 
processor by the developer. Also, having multiple related processors will 
promote healthy competition where my MyputHDFS may for certain cases be 
better/faster then YourPutHDFS and why not have both?

The “artifact registry” (flow, extension, template etc) is the only answer here 
since it will remove the “proliferation” and the need for “taming” anything 
from the picture. With “artifact registry” one or one million processors, the 
NiFi size/state will always remain constant and small.

Cheers
Oleg
> On Feb 22, 2017, at 6:05 AM, Pierre Villard  
> wrote:
> 
> Hey guys,
> 
> Thanks for the thread Andre.
> 
> +1 to James' answer.
> 
> I understand the interest that would provide a single processor to connect
> to all the back ends... and we could document/improve the PutHDFS to ease
> such use but I really don't think that it will benefit the user experience.
> That may be interesting in some cases for some users but I don't think that
> would be a majority.
> 
> I believe NiFi is great for one reason: you have a lot of specialized
> processors that are really easy to use and efficient for what they've been
> designed for.
> 
> Let's ask ourselves the question the other way: with the NiFi registry on
> its way, what is the problem having multiple processors for each back end?
> I don't really see the issue here. OK we have a lot of processors (but I
> believe this is a good point for NiFi, for user experience, for
> advertising, etc. - maybe we should improve the processor listing though,
> but again, this will be part of the NiFi Registry work), it generates a
> heavy NiFi binary (but that will be solved with the registry), but that's
> all, no?
> 
> Also agree on the positioning aspect: IMO NiFi should not be highly tied to
> the Hadoop ecosystem. There is a lot of users using NiFi with absolutely no
> relation to Hadoop. Not sure that would send the good "signal".
> 
> Pierre
> 
> 
> 
> 
> 2017-02-22 6:50 GMT+01:00 Andre :
> 
>> Andrew,
>> 
>> 
>> On Wed, Feb 22, 2017 at 11:21 AM, Andrew Grande 
>> wrote:
>> 
>>> I am observing one assumption in this thread. For some reason we are
>>> implying all these will be hadoop compatible file systems. They don't
>>> always have an HDFS plugin, nor should they as a mandatory requirement.
>>> 
>> 
>> You are partially correct.
>> 
>> There is a direct assumption in the availability of a HCFS (thanks Matt!)
>> implementation.
>> 
>> This is the case with:
>> 
>> * Windows Azure Blob Storage
>> * Google Cloud Storage Connector
>> * MapR FileSystem (currently done via NAR recompilation / mvn profile)
>> * Alluxio
>> * Isilon (via HDFS)
>> * others
>> 
>> But I would't say this will apply to every other use storage system and in
>> certain cases may not even be necessary (e.g. Isilon scale-out storage may
>> be reached using its native HDFS compatible interfaces).
>> 
>> 
>> Untie completely from the Hadoop nar. This allows for effective minifi
>>> interaction without the weight of hadoop libs for example. Massive size
>>> savings where it matters.
>>> 
>>> 
>> Are you suggesting a use case were MiNiFi agents interact directly with
>> cloud storage, without relying on NiFi hubs to do that?
>> 
>> 
>>> For the deployment, it's easy enough for an admin to either rely on a
>>> standard tar or rpm if the NAR modules are already available in the
>> distro
>>> (well, I won't talk registry till it arrives). Mounting a common
>> directory
>>> on every node or distributing additional jars everywhere, plus configs,
>> and
>>> then keeping it consistent across is something which can be avoided by
>>> simpler packaging.
>>> 
>> 
>> As long the NAR or RPM supports your use-case, which is not the case of
>> people running NiFi with MapR-FS for example. For those, a recompilation is
>> required anyway. A flexible processor may remove the need to recompile (I
>> am currently playing with the classpath implication to MapR users).
>> 
>> Cheers
>> 

Re: [ANNOUNCE] New Apache NiFi PMC Member - James Wing

[Oleg Zhurakousky]

Awesome! Well earned! Congrats!

> On Feb 22, 2017, at 10:09 AM, Joe Witt  wrote:
> 
> congrats james and thanks for your efforts!
> 
> On Wed, Feb 22, 2017 at 10:03 AM, Joe Skora  wrote:
>> Congrats James!
>> 
>> On Wed, Feb 22, 2017 at 9:58 AM, Aldrin Piri  wrote:
>> 
>>> Team,
>>> 
>>> On behalf of the Apache NiFi PMC, I am pleased to announce that James Wing
>>> has accepted the PMC's invitation to join the Apache NiFi PMC.  We
>>> greatly appreciate all of Joe's hard work and generous contributions to
>>> the project. We look forward to his continued involvement in the project.
>>> 
>>> James started out contributing in January 2016 with review and assistance
>>> on all things AWS. After receiving committer status, James embraced the
>>> role and continued to assist in fostering and growing the community. Beyond
>>> code contributions, James is active in the community lists, votes, reviews
>>> and external sites helping solve questions about NiFi.
>>> 
>>> Please join us in congratulating and welcoming James to the Apache NiFi
>>> PMC.
>>> 
>>> Congratulations and welcome, James!
>>> 
> 

Re: [ANNOUNCE] New Apache NiFi Committer Jeff Storck

[Oleg Zhurakousky]

Congrats Jeff!
Well earned!

> On Feb 21, 2017, at 7:55 PM, Scott Aslan  wrote:
> 
> Congrats Jeff! Well deserved!
> 
> On Tue, Feb 21, 2017 at 6:36 PM, Koji Kawamura 
> wrote:
> 
>> Congratulations Jeff!
>> 
>> On Wed, Feb 22, 2017 at 7:52 AM, Andre  wrote:
>>> Welcome aboard Jeff! Well deserved
>>> 
>>> On Wed, Feb 22, 2017 at 6:41 AM, Aldrin Piri  wrote:
>>> 
 On behalf of the Apache NiFI PMC, I am very pleased to announce that
>> Jeff
 Storck has accepted the PMC's invitation to become a committer on the
 Apache NiFi project. We greatly appreciate all of Jeff's hard work and
 generous contributions and look forward to continued involvement in the
 project.
 
 Jeff's contributions include significant efforts toward upgrade and
 migration processes inclusive of flow layout when upgrading from 0.x to
>> 1.x
 and the ZooKeeper migration toolkit.
 
 Congrats, Jeff!
 
>> 
> 
> 
> 
> -- 
> *Scott Aslan = new WebDeveloper(*
> *{"location": {"city": "Saint Cloud","state": "FL",
>"zip": "34771"},"contact": {"email":
> "scottyas...@gmail.com ","linkedin":
> "http://www.linkedin.com/in/scottyaslan
> "}});*

Re: Email processor test failure

[Oleg Zhurakousky]

Chris

ListenSMTP is essentially an email server. This processor essentially allows 
you to send emails to it. So by running it you are starting a simple email 
server. 
That said something is obviously not going well on your end. Any firewalls or 
other specific networking setup that you have that may cause it?
Anyway, you must have some stack traces to share.

Cheers
Oleg

> On Feb 21, 2017, at 1:57 PM, Chris Herrera  wrote:
> 
> Hi All,
> 
> Apologies for apparently going braindead…I’m sure I’m doing something silly…. 
> I am in the process of starting to work on some custom processors and 
> controller services, and I am running into an issue with a few tests in the 
> email processor.
> 
> Specifically org.apache.nifi.processors.email.TestListenSMTP is timing out on 
> validateSuccesfulInteraction… when stepping through it seems as if it is 
> timing out. However, I also see that there is a test smtp server that should 
> be stood up for the test…has anyone run into this before.
> 
> Thanks a lot!
> Chris Herrera
> 
> Tests run: 3, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 48.071 sec 
> <<< FAILURE! - in org.apache.nifi.processors.email.TestListenSMTP
> validateSuccessfulInteraction(org.apache.nifi.processors.email.TestListenSMTP)
>   Time elapsed: 15.04 sec  <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at 
> org.apache.nifi.processors.email.TestListenSMTP.validateSuccessfulInteraction(TestListenSMTP.java:91)
> 
> validateSuccessfulInteractionWithTls(org.apache.nifi.processors.email.TestListenSMTP)
>   Time elapsed: 16.518 sec  <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at 
> org.apache.nifi.processors.email.TestListenSMTP.validateSuccessfulInteractionWithTls(TestListenSMTP.java:157)
> 
> validateTooLargeMessage(org.apache.nifi.processors.email.TestListenSMTP)  
> Time elapsed: 16.513 sec  <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at 
> org.apache.nifi.processors.email.TestListenSMTP.validateTooLargeMessage(TestListenSMTP.java:203)
> 

Re: Creating a custom DatabaseAdapter

[Oleg Zhurakousky]

Stanislav

Sorry to hear you’re having problems. 
Is ether any way we can look at the code (i.e., via GitHub)? Also, when you say 
you don’t see your DatabaseAdapter is not picked up, do you see any messages in 
the logs?

It is hard to determine the issue without more info, but usually symptoms like 
this point to miss-configuration of some type and are usually very easy to fix, 
once diagnosed. 
Let us know what you can do.

Also, a lot of times i helps to just copy another working bundle, then rename 
the artifact(s), build and make sure you can see it being deployed. Now you can 
modify the code any way you like knowing that at the very least you don’t have 
any configuration issues.

Let us know
Cheers
Oleg
> On Feb 17, 2017, at 5:05 AM, Stanislav  wrote:
> 
> Hi,
> 
> I would like to create a custom 
> org.apache.nifi.processors.standard.db.DatabaseAdapter, is this possible?
> I have tried creating new class that implements the interface and specified 
> the full class name in 
> META-INF\services\org.apache.nifi.processors.standard.db.DatabaseAdapter 
> file, and created a nar.
> But this does not appear to be working, the custom processor i have in that 
> nar are picked-up but the DatabaseAdapter isn`t.
> Any tips on how to make it work?
> 
> Best regards,
> Stanislav.
> 

Re: [VOTE] Establish Registry, a sub-project of Apache NiFi

[Oleg Zhurakousky]

+1 Here as well. We desperately need it. 

> On Feb 10, 2017, at 12:11 PM, Jeremy Dyer  wrote:
> 
> +1 non-binding. I like the separation and I see a lot of need for this in
> the community.
> 
> On Fri, Feb 10, 2017 at 12:03 PM, Matt Burgess  wrote:
> 
>> +1 binding
>> 
>> On Fri, Feb 10, 2017 at 11:40 AM, Bryan Bende  wrote:
>>> All,
>>> 
>>> Following a solid discussion for the past few days [1] regarding the
>>> establishment of Registry as a sub-project of Apache NiFi, I'd like to
>>> call a formal vote to record this important community decision and
>>> establish consensus.
>>> 
>>> The scope of this project is to define APIs for interacting with
>>> resources that one or more NiFi instances may be interested in, such
>>> as a flow registry for versioned flows, an extension registry for
>>> extensions, and possibly other configuration resources in the future.
>>> In addition, this project will provide reference implementations of
>>> these registries, with the goal of allowing the community to build a
>>> diverse set of implementations, such as a Git provider for versioned
>>> flows, or a bintray provider for an extension registry.
>>> 
>>> I am a +1 and looking forward to the future work in this area.
>>> 
>>> The vote will be open for 72 hours and be a majority rule vote.
>>> 
>>> [ ] +1 Establish Registry, a subproject of Apache NiFi
>>> [ ]   0 Do not care
>>> [ ]  -1 Do not establish Registry, a subproject of Apache NiFi
>>> 
>>> Thanks,
>>> 
>>> Bryan
>>> 
>>> [1] http://mail-archives.apache.org/mod_mbox/nifi-dev/201702.
>> mbox/%3CCALo_M19euo2LLy0PVWmE70FzeLhQRcCtX6TC%3DqoiBVfn4zFQMA%40mail.
>> gmail.com%3E
>> 

Re: [DISCUSS] Proposal for an Apache NiFi sub-project - NiFi Registry

[Oleg Zhurakousky]

Joe

Versioning is probably the second main driver for the externalization as a 
concept. I believe in it so strongly that I would probably cast -1 if 
versioning was not part of it ;)
And some of the existing ready to use solutions already provide versioning of 
artifacts (Binary, GitHub etc), which sets the expectations, so not having i 
would be sad. . . so sad ;)

Cheers
Oleg
> On Feb 9, 2017, at 8:45 AM, Joe Skora  wrote:
> 
> +1 as well.  Glad to help if needed.
> 
> Do you think this will include support for versioned Processors and
> Controller Services, such that I could have SuperWidgetProcessor 1.0 and
> SuperWidgetProcessor 1.5 on the same flow?
> 
> On Thu, Feb 9, 2017 at 8:42 AM, Matt Gilman  wrote:
> 
>> +1. I really like this idea. Should ease deployments between instances and
>> facilitate a better UX throughout the lifecycle of a dataflow.
>> 
>> Matt
>> 
>> On Thu, Feb 9, 2017 at 7:52 AM, Koji Kawamura 
>> wrote:
>> 
>>> Huge +1! I was excited to read the design documents :)
>>> Agree with flow versioning at ProcessGroup level.
>>> 
>>> I don't know if this is helpful, but here is an experimental project
>>> of mine which tries to achieve the same goal, versioning ProcessGroup.
>>> https://github.com/ijokarumawak/nifi-deploy-process-group
>>> 
>>> It contains logics that will probably need to be implemented such as
>>> checking remaining flow files in the queues around ProcessGroup, or
>>> checking number of input/output ports from/to the ProcessGroup ... etc
>>> 
>>> Hope that helps in some way, and I'd like to help make this come true!
>>> 
>>> Thanks,
>>> Koji
>>> 
>>> On Thu, Feb 9, 2017 at 9:43 PM, Joe Gresock  wrote:
 +1, I've been waiting for this idea since NiFi went open source!
 
 On Thu, Feb 9, 2017 at 4:53 AM, Ricky Saltzer 
>>> wrote:
 
> I'm a big +1 to this proposal. It would solve a huge burden that is
>>> keeping
> NARs up to date in environments where there's alot of teams that share
>>> NARs
> but have separate NiFi deployments and repositories.
> 
> On Feb 8, 2017 7:09 PM, "Peter Wicks (pwicks)" 
>>> wrote:
> 
>> I think a lot of us are facing the same challenges, and this sounds
>>> like
> a
>> step in the right direction.
>> I had actually started to dig into a Flow Configuration plugin that
>>> would
>> use Git branches to copy/sync flows between instances/environments,
>>> and
>> keep them versioned; hadn't gotten very far.
>> 
>> -Original Message-
>> From: Jeremy Dyer [mailto:jdy...@gmail.com]
>> Sent: Wednesday, February 08, 2017 3:54 PM
>> To: dev@nifi.apache.org
>> Subject: Re: [DISCUSS] Proposal for an Apache NiFi sub-project -
>> NiFi
>> Registry
>> 
>> Bryan - I think this is a fantastic idea. I would also think this
>>> would
> be
>> a good place to add a "device registry" as well. It makes much more
>>> sense
>> in my mind to have these efforts in sub projects outside of the
> nifi/minifi
>> core.
>> 
>> On Wed, Feb 8, 2017 at 4:50 PM, Bryan Bende 
>> wrote:
>> 
>>> NiFi Community,
>>> 
>>> I'd like to initiate a discussion around creating a sub-project of
>>> NiFi to encompass the registry capabilities outlined in several of
>>> the
>>> feature proposals on the Wiki [1]. A possible name for this
>>> sub-project is simply "NiFi Registry".
>>> 
>>> Currently there are two feature proposals that call for NiFi to
>>> interact with an external registry:
>>> 
>>> Configuration Management of Flows [2]  - This feature proposal
>> calls
>>> for a flow registry where versioned flows can be published and
>>> consumed, allowing flows to be easily migrated between
>> environments
>>> .
>>> 
>>> Extension Registry [3] - This feature proposal calls for a place
>> to
>>> publish NARs containing extensions, allowing NiFi to decouple
>> itself
>>> from including all of the NARs in the main distribution, and
>>> allowing
>>> better discovery of available extensions.
>>> 
>>> The idea would be to create a NiFi Registry sub-project, with
>>> sub-modules for the various registries. These registries could
>> then
>>> be
>>> packaged and distributed as a single artifact and run as a
>>> complimentary application to NiFi and MiNiFi. NiFi would not
>> require
>>> the registry application, however, a given NiFi could be
>> configured
>>> to
>>> know about one or more flow registries, or one or more extension
>>> registries.
>>> 
>>> Creating a sub-project would allow the registry code to evolve
>>> independently of NiFi and be released on it's own timeline. In
>>> addition, it would make tracking issues/work much clearer through
>> a
>>> separate JIRA.
>>> 
>>> Please 

Re: [DISCUSS] Proposal for an Apache NiFi sub-project - NiFi Registry

[Oleg Zhurakousky]

Bryan

While I am huge +1 for breaking up NiFi into manageable structure, I am still 
unclear as to the scope of this project.

1. As discussed in [3], there are public repositories and services out there 
that have been designed for the exact problem we are facing in NiFi today. 
Those repositories and services (i.e., Bintray) have been embraced by the 
larger OS community of developers and open source projects already. So at the 
very least we need to have some answer/story about why we decided NOT to rely 
on them (if that is the decision).
2. On the flip side, it is perfectly valid to assume that someone or some 
organization will not find solutions such as BinTray attractive or usable in 
their environment and may prefer a custom solution. In that case having 
alternatives may help, but in itself is not a solution since all of the 
available alternatives may also NOT be sufficient. 
So, in other words relying on any existing solution or building multiple may 
not solve the issue I (the customer) may have. This leaves only one option - a 
project that defines a set of strategies (interfaces) that would allow one to 
either integrate with an already existing solution or implement a new custom 
one. Now, we (NiFi community) could contribute integration modules to such 
project based on those pre-defined Registry interfaces. IMHO such approach will 
foster more community collaboration and acceptance by exposing such integration 
model.

Also, regardless of the decision, the bigger work will be overhauling NiFi 
deployment/packaging model and UI interaction since with external registries 
NARs, flows etc., would need to be pulled/removed/updated while NiFi is running 
and that I think is a much bigger work.

In the end I am +1 for the project and super excited that the idea is finally 
starting to get traction, but wanted to contribute with above comments to 
ensure that the project's scope is clearly defined.

Cheers
Oleg
> On Feb 8, 2017, at 4:50 PM, Bryan Bende  wrote:
> 
> NiFi Community,
> 
> I'd like to initiate a discussion around creating a sub-project of
> NiFi to encompass the registry capabilities outlined in several of the
> feature proposals on the Wiki [1]. A possible name for this
> sub-project is simply "NiFi Registry".
> 
> Currently there are two feature proposals that call for NiFi to
> interact with an external registry:
> 
> Configuration Management of Flows [2]  - This feature proposal calls
> for a flow registry where versioned flows can be published and
> consumed, allowing flows to be easily migrated between environments .
> 
> Extension Registry [3] - This feature proposal calls for a place to
> publish NARs containing extensions, allowing NiFi to decouple itself
> from including all of the NARs in the main distribution, and allowing
> better discovery of available extensions.
> 
> The idea would be to create a NiFi Registry sub-project, with
> sub-modules for the various registries. These registries could then be
> packaged and distributed as a single artifact and run as a
> complimentary application to NiFi and MiNiFi. NiFi would not require
> the registry application, however, a given NiFi could be configured to
> know about one or more flow registries, or one or more extension
> registries.
> 
> Creating a sub-project would allow the registry code to evolve
> independently of NiFi and be released on it's own timeline. In
> addition, it would make tracking issues/work much clearer through a
> separate JIRA.
> 
> Please discuss and provide and thoughts or feedback.
> 
> Thanks,
> 
> Bryan
> 
> [1] https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals
> [2] 
> https://cwiki.apache.org/confluence/display/NIFI/Configuration+Management+of+Flows
> [3] 
> https://cwiki.apache.org/confluence/display/NIFI/Extension+Repositories+%28aka+Extension+Registry%29+for+Dynamically-loaded+Extensions
> 

Re: GenerateFlowFile.java

[Oleg Zhurakousky]

Alessio

The onScheduled() is only called once when processor is started. Started means 
scheduled to execute, hence the name of the operation. When the process is 
scheduled its scheduler triggers the execution of the processor and the same 
scheduler controls how often the onTrigger is called. For example, with default 
TimeDrivenScheduler set to 1 sec, the onTrigger() operation will be invoked 
every sec.

So to answer your question in short your use case should be implemented in 
onTrigger(). However onSchedule() is where you could/should obtain the 
reference to your service.

Does that clarify?

Cheers
Oleg
> On Feb 2, 2017, at 4:43 AM, Alessio Palma  
> wrote:
> 
> Hello  all,
> 
> I'm reading the GenerateFlowFile processor, which is started with no input 
> files. I need to write a processor very close to this one,  which accept no 
> files, but when scheduled has to output into the queue the file returned from 
> a service it calls.
> 
> I did not understand how the framework calls this kind of processor.
> There are both onScheduled and onTrigger methods.
> In which order are they called ?
> Do I need both?
> 
> AP

Re: problems with custom controller services, and other comments

[Oleg Zhurakousky]

Michael

Sorry took a while, but we just had one of our guys reporting the same symptoms 
and as Bryan suggested it was actually a problem in the POM, specifically the 
POM file for NAR modules of your custom processor.
Basically once your processor is referencing the service it’s NRA module has to 
declare dependency on such service.
I m pasting an example POM for a NAR module from one of our guys who is 
referencing DBCP service from his custom bundle

http://maven.apache.org/POM/4.0.0; 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
4.0.0

org.apache.nifi
nifi-sql-bundle
1.1.0


nifi-sql-nar
nar



org.apache.nifi
nifi-dbcp-service-nar
nar


org.apache.nifi
nifi-sql-processors
1.1.0




Note the first dependency (wifi-dbcp-service-nar)

Hope that helps

Cheers
Oleg
On Jan 10, 2017, at 1:29 PM, Oleg Zhurakousky 
<ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote:

Michael

That is indeed strange.
Quick question, is there any chance we can look at your code (the custom CS). 
Also, what version of NiFi you are using?
Will be looking into it, but any additional info is helpful

Cheers
Oleg


On Jan 10, 2017, at 1:14 PM, Knapp, Michael 
<michael.kn...@capitalone.com<mailto:michael.kn...@capitalone.com>> wrote:

Devs,

For some reason NIFI is not working with some custom controller services I have 
written.  I wrote new implementations of AWSCredentialsProviderService, that 
aim to work with session tokens.  I am hoping to run NIFI from my local machine 
and to be able to connect with AWS using session tokens.

For both implementations I tried, it fails when I try to create them from the 
web UI.  I created a PutS3Object processor, and configured the “AWS Credentials 
Provider Service” property.  From that property I tried to create a new service 
and selected my custom implementation.  When I click “create” the value for the 
credentials provider service is the ID of the controller service, not its name. 
 While my controller services require several properties to be set, the web UI 
is not letting me set them.  Usually I see an arrow next to the property, which 
allows me to configure a controller service, but I am not getting that now.  I 
looked in the nifi-app logs, and I do not see any exception, I have even set 
the logging to TRACE for all things, and still don’t see any problem in the 
logs.  The PutS3Object processor is not validating because the controller 
service is found to be invalid.  I tried creating a unit test, it seems to work 
for me in tests, but I can’t use TestRunners because that is processor 
oriented, not meant for controller services.  I have a suspicion that spring’s 
aspect oriented programming is somehow fuddling with my service.

Does anybody know what I am doing wrong here?

Other unrelated comments:

1.   The first time you unpack NIFI it takes super long for it to start for 
me, like a half hour or more.  I think you should make it easy for people to 
scale back their NIFI implementation.  Really I would like to start it with 
just the minimum NAR files for it to start, and I can add others that I need.  
Maybe a sub-directory in lib for the essential nars could help people separate 
the essential stuff from the optional nars.  The first time I tried installing 
it, I thought it was broken when really it just was taking forever (over 30 
minutes).  I think that new users will probably abandon NIFI if they can’t get 
it to start quickly out of the box.  Maybe split the optional nars into an 
“extra-lib”, and people can move those into lib as necessary for their goals.

2.   Building NIFI from source takes over an hour for me, really I just 
want to build the bare minimum things to get it to start.  I tried creating 
maven profiles to have it build just the minimum pieces, but this proved to be 
non-trivial as maven does not seem to respect the “modules” tag in profiles, 
and the nifi-assembly project requires all of the optional nars to also be 
built.  Creating this might be too complicated for me.  Has anybody thought 
about supporting a quick/minimal build?

3.   The “nifi-aws-processors” is challenging to use because in one project 
they have defined the interfaces for controller services 
(AWSCredentialsProviderService) and also included the services.  I tried 
creating my own nar with an implementation of AWSCredentialsProviderService, 
but since it depended on “nifi-aws-processors”, my nar was also re-hosting all 
of the AWS processors.  I was facing a lot of classpath issues because of this. 
 I worked around this by using maven shade to shade in the 
“nifi-aws-processors” into my own jar, but excluding the services it provided.  
Then in my nar project I had to exclude the dependency on 
“nifi-aws-processors”.  This was a lot of work on my part when all they needed 
to do was split

Re: [DISCUSS] Run Once scheduling

[Oleg Zhurakousky]

I was just about to suggest the same. 
Run-once would be a bit counter intuitive to the flow processing as a concept. 
Basically think of it this way; Flow or parts of it have only two states - 
RUNNING or STOPPED. In the RUNNING state it processes the data as it arrives 
(every second, every minute or every day etc). Indeed there may be a concern 
that the processor will do a lot of 'dry’ spins if no data is available but 
fortunately NiFi allows you to limit the impact of that by configuring “yield 
duration’. By default it is set to 1 sec, but for your case you may wan to set 
it to 1 hour or so essentially controlling the scheduling of such processor 
between ‘dry’ spins.

That said and just to entertain the idea of Run Once, what do you think should 
be the processor state if it did ran once? Let’s assume it did and somehow was 
stopped. . . then what? The data arrived on the incoming queue, but nothing is 
processed until someone manually goes and re-starts the processor. Right?
I mean from the general workflow standpoint the concern is very valid, but from 
flow processing the fact that NiFi does not support it is actually more of a 
feature rather then lack of functionality.

Thoughts?

Cheers
Oleg

> On Jan 12, 2017, at 1:02 PM, Joe Witt  wrote:
> 
> Naz,
> 
> Why not just leave all the processes running?  If the data only
> arrives periodically that is ok, right?
> 
> Thanks
> Joe
> 
> On Thu, Jan 12, 2017 at 10:54 AM, Irizarry Jr., Nazario  
> wrote:
>> On a project that I am on we have been looking at using NiFi for 
>> orchestrations that are invoked infrequently.  For example, once a month a 
>> new data input product becomes available and then one wants to run it 
>> through a set of processing steps that can be nicely implemented using NiFi 
>> processors.  However, using the interval or cron scheduling for this purpose 
>> begins to get cumbersome after a while with the need to start and manually 
>> stop these occasional flows.
>> 
>> It would be fairly easy to add an additional scheduling option - “Run Once” 
>> for this use case.  The behavior would be that when a processor is set to 
>> run once it automatically stops after it has successfully processed one 
>> input.
>> 
>> What do people think?  We are willing to implement this small enhancement.
>> 
>> Cheers,
>> 
>> Naz Irizarry
>> MITRE Corp.
>> 617-893-0074
>> 
>> 
>> 
> 

Re: problems with custom controller services, and other comments

[Oleg Zhurakousky]

Michael

That is indeed strange. 
Quick question, is there any chance we can look at your code (the custom CS). 
Also, what version of NiFi you are using?
Will be looking into it, but any additional info is helpful 

Cheers
Oleg


> On Jan 10, 2017, at 1:14 PM, Knapp, Michael  
> wrote:
> 
> Devs,
> 
> For some reason NIFI is not working with some custom controller services I 
> have written.  I wrote new implementations of AWSCredentialsProviderService, 
> that aim to work with session tokens.  I am hoping to run NIFI from my local 
> machine and to be able to connect with AWS using session tokens.
> 
> For both implementations I tried, it fails when I try to create them from the 
> web UI.  I created a PutS3Object processor, and configured the “AWS 
> Credentials Provider Service” property.  From that property I tried to create 
> a new service and selected my custom implementation.  When I click “create” 
> the value for the credentials provider service is the ID of the controller 
> service, not its name.  While my controller services require several 
> properties to be set, the web UI is not letting me set them.  Usually I see 
> an arrow next to the property, which allows me to configure a controller 
> service, but I am not getting that now.  I looked in the nifi-app logs, and I 
> do not see any exception, I have even set the logging to TRACE for all 
> things, and still don’t see any problem in the logs.  The PutS3Object 
> processor is not validating because the controller service is found to be 
> invalid.  I tried creating a unit test, it seems to work for me in tests, but 
> I can’t use TestRunners because that is processor oriented, not meant for 
> controller services.  I have a suspicion that spring’s aspect oriented 
> programming is somehow fuddling with my service.
> 
> Does anybody know what I am doing wrong here?
> 
> Other unrelated comments:
> 
> 1.   The first time you unpack NIFI it takes super long for it to start 
> for me, like a half hour or more.  I think you should make it easy for people 
> to scale back their NIFI implementation.  Really I would like to start it 
> with just the minimum NAR files for it to start, and I can add others that I 
> need.  Maybe a sub-directory in lib for the essential nars could help people 
> separate the essential stuff from the optional nars.  The first time I tried 
> installing it, I thought it was broken when really it just was taking forever 
> (over 30 minutes).  I think that new users will probably abandon NIFI if they 
> can’t get it to start quickly out of the box.  Maybe split the optional nars 
> into an “extra-lib”, and people can move those into lib as necessary for 
> their goals.
> 
> 2.   Building NIFI from source takes over an hour for me, really I just 
> want to build the bare minimum things to get it to start.  I tried creating 
> maven profiles to have it build just the minimum pieces, but this proved to 
> be non-trivial as maven does not seem to respect the “modules” tag in 
> profiles, and the nifi-assembly project requires all of the optional nars to 
> also be built.  Creating this might be too complicated for me.  Has anybody 
> thought about supporting a quick/minimal build?
> 
> 3.   The “nifi-aws-processors” is challenging to use because in one 
> project they have defined the interfaces for controller services 
> (AWSCredentialsProviderService) and also included the services.  I tried 
> creating my own nar with an implementation of AWSCredentialsProviderService, 
> but since it depended on “nifi-aws-processors”, my nar was also re-hosting 
> all of the AWS processors.  I was facing a lot of classpath issues because of 
> this.  I worked around this by using maven shade to shade in the 
> “nifi-aws-processors” into my own jar, but excluding the services it 
> provided.  Then in my nar project I had to exclude the dependency on 
> “nifi-aws-processors”.  This was a lot of work on my part when all they 
> needed to do was split that project into api, api-nar, impl, and impl-nar.
> 
> 4.   I think it is very confusing how there is a “Controller Services” 
> for the entire NIFI canvas, and separate ones for individual processor 
> groups.  It seems that processors cannot use global controller services, and 
> I am still uncertain about why I would ever create a global one.  From Nifi 
> settings, I would like to also see controller services in processor groups, 
> and vice versa.  From a processor, I would like to assign controller services 
> that are global in scope, not limited to a processor group.  I think this is 
> something that will confuse and frustrate a lot of new developers, driving 
> them to consider competing products.
> 
> 5.   I think the developer guide needs some clarification on what jars 
> are provided and not.  New developers will be unsure if they should include 
> “nifi-api” as a provided or compile dependency, and same goes for 
> 

Re: NiFi installation issues

[Oleg Zhurakousky]

Pushkar

It appears that something is running on port 8080 so Jetty can not start. You 
can see at the bottom of the log the following (see below)


  1.
2017-01-09 17:24:23,606 WARN [main] org.apache.nifi.web.server.JettyServer 
Failed to start web server... shutting down.
  2.
java.net.BindException: Address already in use
  3.
at sun.nio.ch.Net.bind0(Native Method) ~[na:1.8.0_112]
  4.
at sun.nio.ch.Net.bind(Net.java:433) ~[na:1.8.0_112]
  5.
at sun.nio.ch.Net.bind(Net.java:425) ~[na:1.8.0_112]
  6.
at 
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) 
~[na:1.8.0_112]

On Jan 9, 2017, at 9:37 AM, Pushkara R 
> wrote:

Oleg

I've pasted the app log here http://pastebin.com/YXkEvNZJ
and the bootstrap log here http://pastebin.com/iPJeJvwX.

I built the project using 'maven clean install' (after setting
MAVEN_OPTS="-Xmx2048m -XX:MaxPermSize=128m" because of OutOfMemory
exceptions).
Then I extracted the nifi-1.2.0-snapshot-bin.tar.gz.
Followed by '$ ./bin/nifi.sh start'

I then try to connect to 'http://localhost:8080/nifi/' which gives me the
screen I pasted in my previous mail.

These were the documentation as available in the README.md in the github
clone of the project.

Thanks
Pushkar

Re: NiFi installation issues

[Oleg Zhurakousky]

Pushkar

Any chance you can look and provide relevant logs? You can find them in the 
/log directory.
Also, could you please be more specific as to what directions did you follow. 
Just want to make sure that if there are any issues with documentation it is 
corrected.

Cheers
Oleg

> On Jan 9, 2017, at 8:46 AM, Pushkara R  wrote:
> 
> Hi,
> 
> I'm installing NiFi on my arch linux machine following the instructions in
> https://github.com/apache/nifi.
> 
> After I built the project, I executed the NiFi startup script and tried
> connecting to the local server but I got a 404 error with the following
> message.
> 
> HTTP ERROR: 404
> 
> Problem accessing /nifi/. Reason:
> 
>Not Found
> 
> --
> Powered by Jetty:// 9.3.9.v20160517 
> 
> Could someone please help me debug this?
> 
> Thank You
> Pushkar

Re: [ANNOUNCE] New Apache NiFi Committer Joey Frazee

[Oleg Zhurakousky]

Congrats Joey!

Nice to have you on board!
Oleg

On Jan 3, 2017, at 2:42 PM, Aldrin Piri 
> wrote:

On behalf of the Apache NiFI PMC, I am very pleased to announce that Joey 
Frazee has accepted the PMC's invitation to become a committer on the Apache 
NiFi project. We greatly appreciate all of Joey's hard work and generous 
contributions and look forward to continued involvement in the project.

Joey's contributions including support for HL7, JMS, and EventHub extensions.  
Joey can also be found assisting with the lists, as well as articles and 
repositories based around the NiFi community.

Congrats, Joey!

Re: [ANNOUNCE] New Apache NiFi PMC Member - Joe Skora

[Oleg Zhurakousky]

Congrats Joe!!! Well earned!
Oleg 

Sent from my iPhone

> On Dec 29, 2016, at 14:23, Joe Percivall  wrote:
> 
> Congrats Joe!
> 
>> On Thu, Dec 29, 2016 at 1:11 PM, Tony Kurc  wrote:
>> 
>> Glad to have you on the PMC, Joe!
>> 
>>> On Thu, Dec 29, 2016 at 1:02 PM, Aldrin Piri  wrote:
>>> 
>>> All,
>>> 
>>> On behalf of the Apache NiFi PMC, I am pleased to announce that Joe
>> Skora has
>>> accepted the PMC's invitation to join the Apache NiFi PMC.  Joe has been
>>> with NiFi for quite some time, even before its arrival in the ASF and
>>> became a committer in February.  We are most pleased he brought his
>>> knowledge and supported the community once open sourced and has has
>>> provided continuous and excellent contributions in all facets of the
>>> community.  Of note, Joe was our first community member to carry out a
>>> release without being a PMC member for 0.7.1.
>>> 
>>> Please join us in congratulating and welcoming Joe to the Apache NiFi
>> PMC.
>>> 
>>> Again, congratulations Joe and well deserved!
> 
> 
> 
> -- 
> 
> - - - - - -
> *Joseph Percivall*
> e: joeperciv...@gmail.com

Re: GetKafka and ConvertAvroToJson returns blank

[Oleg Zhurakousky]

Ok, I see the issue after testing it wit your data

You probably did not specify the schema when configuring the ConvertAvroToJson 
processor and because of that you were most likely getting the following error:

[pool-1-thread-1] ERROR org.apache.nifi.processors.avro.ConvertAvroToJSON - 
ConvertAvroToJSON[id=84a6c68c-0064-40a1-9431-80ff35fbc7d3] Failed to convert 
FlowFile[0,147497938581519.mockFlowFile,14B] from Avro to JSON due to 
org.apache.nifi.processor.exception.ProcessException: java.io.IOException: Not 
a data file.; transferring to failure: 
org.apache.nifi.processor.exception.ProcessException: java.io.IOException: Not 
a data file.

So my first question; Do you see the same error in the logs? (just want to 
confirm that you do)

In any event, ConvertAvroToJson has the “Schema” configuration property. Simply 
pass your schema as value to this property (e.g., “{ "namespace": 
"mynamespace.com<http://mynamespace.com/>", "type": "record", "name": 
"TEST_DATA1", "fields": [ {"name": "ENAME", "type": "string"} ] }”)

Cheers
Oleg


On Dec 22, 2016, at 1:49 PM, Oleg Zhurakousky 
<ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote:

Malini

Indeed that is a strange.
Would you mind looking at the logs to see if there are any exceptions/errors 
generated form ConvertAvroToJson? Meanwhile I’ll run some tests with your data.

Cheers
Oleg

On Dec 22, 2016, at 1:37 PM, Malini Shetty 
<malini.she...@gmail.com<mailto:malini.she...@gmail.com>> wrote:

Hi,
GetKafka processor is returning success message and data can be viewed in hex 
format. Next processor called is ConvertAvroToJson is returning blank string, 
while Kafka confluent is able to convert the avro binary data into Json format.

DB2 data is streamed thru CDC to Kafka, schema used in ConvertAvroToJson is


{ "namespace": "mynamespace.com<http://mynamespace.com/>", "type": "record", 
"name": "TEST_DATA1", "fields": [ {"name": "ENAME", "type": "string"} ] }

Can you please help to understand why ConvertAvrotoJson is returning blank 
string?

I have used Putfile processor after GetKafka. Attached is the format and the 
string returned using ConvertAvroToJson is blank.

Regards
Malini Shetty

<10677-hprojectodsdevbinaryavro.zip>

Re: NiFi compilation error - master & 1.1.x branch

[Oleg Zhurakousky]

Andrew

I believe I’ve seen something similar before and it persisted until up’ve 
upgraded JDK. Once on the latest (*_112) all was good, But discussing/testing 
it a bit internally even the older upgrades would do (e.g., *_65)
FWIW, I has the same *_45 version until I started seeing this.

Cheers
Oleg

> On Dec 22, 2016, at 6:49 AM, Andrew Christianson 
>  wrote:
> 
>> What version of Java are using? Oracle or OpenJDK, 8 or 9, etc.? Also
> 
> $ java -version   
>  
> java version "1.8.0_45"   
>   
> 
> Java(TM) SE Runtime Environment (build 1.8.0_45-b14)  
>   
> 
> Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)
> 
>> which branch are you building from? It looks like it's building
> 
> Tried both of these:
> 
> $ git branch -v   
>  
> * master 44c9ea0 NIFI-3236 - SplitJson performance improvements   
>   
> 
>  support/nifi-1.1.x 986e716 NIFI-3188: Added unit test to test corner cases

Re: [VOTE] Release Apache NiFi 1.1.1 (RC1)

[Oleg Zhurakousky]

+1 (non-binding) 

Built and tested on OSX with contrib-check.
Ran several basic flows
Validated Import/Export templates (single node and 3 node cluster)

> On Dec 20, 2016, at 1:49 PM, Joey Frazee  wrote:
> 
> +1 (non-binding)
> 
> - Verified commit hash, checksums and GPG signature
> - Checked root LICENSE and NOTICE
> - Checked version in pom files
> - Ran `mvn -T 2.0C clean install -Pcontrib-check`
> - Tested with PutElasticsearchHttp with/without connection failure (NIFI-3194)
> - Tested with ValidateCsv (NIFI-3175)
> - Tested CSV to Hive data flow with Avro, ORC conversions and PutHiveQL
> 
>> On Dec 20, 2016, at 12:29 PM, James Wing  wrote:
>> 
>> +1 (non-binding). I ran through the release helper -- verified hashes,
>> license/notice/readme files, full build, and tested the resulting binary
>> without issues on JDK 1.8.0_101, Amazon Linux.
>> 
>> Thanks,
>> 
>> James
>> 
>> On Mon, Dec 19, 2016 at 2:35 PM, Joe Percivall <
>> joeperciv...@yahoo.com.invalid> wrote:
>> 
>>> Hello Apache NiFi Community,
>>> 
>>> I am pleased to be calling this vote for the source release of Apache
>>> NiFi, nifi-1.1.1.
>>> 
>>> The source zip, including signatures, digests, etc. can be found at:
>>> https://repository.apache.org/content/repositories/orgapachenifi-1097
>>> 
>>> The Git tag is nifi-1.1.1-RC1
>>> The Git commit hash is a92f2e36ed6be695e4dc6f624f6b3a96e6d1a57c
>>> * https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
>>> a92f2e36ed6be695e4dc6f624f6b3a96e6d1a57c
>>> * https://github.com/apache/nifi/commit/a92f2e36ed6be695e4dc6f624f6b3a
>>> 96e6d1a57c
>>> 
>>> Checksums of nifi-1.1.1-source-release.zip:
>>> MD5: 74955060d8ee295d77a23607ac644a6e
>>> SHA1: 82efc0dc3141d0fad0205b33539e5928da87ad17
>>> SHA256: 25fab8d7abfecf4c0ccef1ed9cd5f0849c829c0741142ed4074bc8dd0781f7d0
>>> 
>>> Release artifacts are signed with the following key:
>>> https://people.apache.org/keys/committer/jpercivall
>>> 
>>> KEYS file available here:
>>> https://dist.apache.org/repos/dist/release/nifi/KEYS
>>> 
>>> 16 issues were closed/resolved for this release:
>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>>> projectId=12316020=12338797
>>> Release note highlights can be found here:
>>> https://cwiki.apache.org/confluence/display/NIFI/
>>> Release+Notes#ReleaseNotes-Version1.1.1
>>> 
>>> The vote will be open for 72 hours.
>>> Please download the release candidate and evaluate the necessary items
>>> including checking hashes, signatures, build from source, and test. Then
>>> please vote:
>>> 
>>> [ ] +1 Release this package as nifi-1.1.1
>>> [ ] +0 no opinion
>>> [ ] -1 Do not release this package because...
>>> 
>>> Thanks!
> 
> 

Re: [ANNOUNCE] New Apache NiFi Committer Jeremy Dyer

[Oleg Zhurakousky]

Congrats Jeremy! Well deserved!
> On Dec 20, 2016, at 8:46 AM, Yolanda Davis  wrote:
> 
> Congratulations Jeremy!
> 
> On Mon, Dec 19, 2016 at 8:23 PM, Aldrin Piri  wrote:
> 
>> On behalf of the Apache NiFI PMC, I am very pleased to announce that Jeremy
>> Dyer has accepted the PMC's invitation to become a committer on the
>> Apache NiFi project. We greatly appreciate all of Jeremy's hard work and
>> generous contributions and look forward to continued involvement in the
>> project.
>> 
>> Jeremy’s contributions include creation a suite of the processors that
>> aided in parsing HTML, build improvement and testing for MiNiFi, as well as
>> many articles and presentations on using NiFi in new and novel ways.
>> 
>> Welcome Jeremy!
>> 
> 
> 
> 
> -- 
> --
> yolanda.m.da...@gmail.com
> @YolandaMDavis

Re: [VOTE] Release Apache NiFi 1.0.1 (RC1)

[Oleg Zhurakousky]

+1 (non-binding)
Used Release Helper to run through steps on OSX with Java 1.8.0_112. Created 
and deployed few simple flows, played with templates etc., all is good.

Oleg

> On Dec 18, 2016, at 10:24 PM, Tony Kurc  wrote:
> 
> +1 (binding)
> 
> Ran through helper, built on ubuntu 14.04 with java 1.8, verified hashes
> and signature. ran simple test flow.
> 
> On Sun, Dec 18, 2016 at 9:15 PM, Bryan Rosander 
> wrote:
> 
>> +1 (non-binding)
>> 
>> - Ran through release helper
>> - Sent data to NiFi via S2S using MiNiFi without proxy, with proxy, with
>> authenticating proxy
>> 
>> On Sun, Dec 18, 2016 at 8:14 PM, Koji Kawamura 
>> wrote:
>> 
>>> +1 (non-binding).
>>> 
>>> - Ran through the release helper
>>> - Ran some basic data flows  (Java 1.8.0_111, Mac OS X 10.11.6)
>>> 
>>> 
>>> 
>>> On Sun, Dec 18, 2016 at 6:28 AM, James Wing  wrote:
 +1 (non-binding).  I ran through the release helper and did some basic
 testing of the built binary (Java 1.8.0_101, AWS Linux).
 
 Thanks for putting this together, Joe.
 
 
 James
 
 On Fri, Dec 16, 2016 at 7:28 AM, Joe Percivall <
 joeperciv...@yahoo.com.invalid> wrote:
 
> I apologize for the improperly formatted message. Hopefully the below
> message is better.
> 
> 
> 
> 
> Hello Apache NiFi Community,
> 
> I am pleased to be calling this vote for the source release of Apache
>>> NiFi,
> nifi-1.0.1.
> 
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1096
> 
> The Git tag is nifi-1.0.1-RC1
> The Git commit hash is 1890f6c522514027ae46f86601f4771f62cadc6d
> * https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> 1890f6c522514027ae46f86601f4771f62cadc6d
> * https://github.com/apache/nifi/commit/
>> 1890f6c522514027ae46f86601f477
> 1f62cadc6d
> 
> Checksums of nifi-1.0.1-source-release.zip:
> MD5: cc7fea9a22c0b48f87dd7152ab83c28c
> SHA1: 88c35d5d3ff9d350473a742cdd8c38204628d343
> SHA256: d9d9628ced5bf3f0f3e0eae7729f4eb507120b072e883e287198fee80fbf
>>> 9d15
> 
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/jpercivall
> 
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
> 
> 6 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12316020=12338865
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/
> Release+Notes#ReleaseNotes-Version1.0.1
> 
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.
>> Then
> please vote:
> 
> [ ] +1 Release this package as nifi-1.0.1
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...
> 
> Thanks!
> 
> 
> 
> On Friday, December 16, 2016 10:23 AM, Joe Percivall <
> joeperciv...@yahoo.com.INVALID> wrote:
> 
> 
> 
> Hello Apache NiFi Community,
> I am pleased to be calling this vote for the source release of Apache
> NiFi,nifi-1.0.1.
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1096
> The Git tag is nifi-1.0.1-RC1The Git commit hash is
> 1890f6c522514027ae46f86601f4771f62cadc6d*
>>> https://git-wip-us.apache.org/
> repos/asf?p=nifi.git;a=commit;h=1890f6c522514027ae46f86601f477
>>> 1f62cadc6d*
> https://github.com/apache/nifi/commit/1890f6c522514027ae46f86601f477
> 1f62cadc6d
> Checksums of nifi-1.0.1-source-release.zip:MD5:
> cc7fea9a22c0b48f87dd7152ab83c28cSHA1: 88c35d5d3ff9d350473a742cdd8c38
>>> 204628d343SHA256:
> d9d9628ced5bf3f0f3e0eae7729f4eb507120b072e883e287198fee80fbf9d15
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/jpercivall
> KEYS file available here:https://dist.apache.org/
> repos/dist/release/nifi/KEYS
> 6 issues were closed/resolved for this release:https://issues.apache.
> org/jira/secure/ReleaseNote.jspa?projectId=12316020&
> version=12338865Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/
> Release+Notes#ReleaseNotes-Version1.0.1
> The vote will be open for 72 hours.Please download the release
>> candidate
> and evaluate the necessary itemsincluding checking hashes, signatures,
> build from source, and test. Thenplease vote:
> [ ] +1 Release this package as nifi-1.0.1[ ] +0 no opinion[ ] -1 Do
>> not
> release this package 

Re: NIFI 1.1.0 build error

[Oleg Zhurakousky]

Alessio

Any chance you have some network connectivity issues when you attempt too build 
or may be a proxy that restricts certain sites?
This error is indeed strange and I can’t recall having it and I’ve tried to do 
the same (as you describe) on my mac and all builds fine. 

Also, just to compare the environments:
olegs-mac:~ foo$ mvn --version
Apache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06; 
2015-04-22T07:57:37-04:00)
Maven home: /usr/local/Cellar/maven/3.3.3/libexec
Java version: 1.8.0_112, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_112.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.11.2", arch: "x86_64", family: "mac"

Cheers
Oleg

> On Dec 15, 2016, at 6:49 AM, Alessio Palma  
> wrote:
> 
> Hello all,
> 
> I downloaded the source code from
> 
> 
> http://apache.panu.it/nifi/1.1.0/nifi-1.1.0-source-release.zip
> 
> 
> then unpacked the archivie and executed:
> 
> 
> mvn dependency:purge-local-repository
> 
> mvn clean install
> 
> 
> and I got this:
> 
> 
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default) 
> on project nifi-standard-processors: Error resolving project artifact: Could 
> not transfer artifact com.martiansoftware:macnificent:pom:0.2.0 from/to 
> org.apche.nifi (http://https://mvnrepository.com/artifact/org.apache.nifi): 
> https: unknown error for project com.martiansoftware:macnificent:jar:0.2.0: 
> Unknown host https: unknown error ->
> 
> 
> 
> any pointers how to fix this ?

Re: Hello

[Oleg Zhurakousky]

Hi

I am not sure I fully understand what ‘data’ you are referring to. Is this the 
usual Log data? If so it is still somewhat unusual requirement to log something 
during some time frame and ignore during others, so that is why I am asking if 
you can clarify what ‘data’ you are referring to.
In any event you also use variety of Unix/Linux tools as well as other OS and 
commercial applications to aggregate, search, manipulate log data. 
For example, let’s say you have a log like this:

07:11:41,118  INFO main server.Server:403 - Started @47542ms
07:17:48,596  INFO main server.JettyServer:832 - NiFi has started. The UI is 
available at the following URLs:
08:01:11,596  INFO main server.JettyServer:834 - 
http://fe80:0:0:0:c07e:7dff:fe92:3ddb%awdl0:8080/nifi
09:18:51,597  INFO main server.JettyServer:834 - 
http://fe80:0:0:0:0:0:0:1%lo0:8080/nifi
09:11:41,597  INFO main server.JettyServer:834 - http://192.168.1.114:8080/nifi
. . .

and you want to extract only logs between 7 and 8 AM, something like this would 
do the trick 'grep "0[7-8]:” my-log.log’, but could be more sophisticated.
Cheers
Oleg 

> On Dec 13, 2016, at 6:08 AM, Vidhyashreemurthy N 
>  wrote:
> 
> Hello,
> 
> Firstly, i thank you for an excellent tool! NiFi has helped me in a lot of
> ways.
> I have started using NiFi just a couple of hours ago. Im asked to perform
> the following task with the next 4 hours. I went through the documentation
> roughly, I didnt find the apt answer. Can you please help me with the
> following issue?
> 
> I have a question about the NiFi operations.
> I want to know how can i set a time frame for data to be logged into log
> file.
> Suppose, in one day, i want the data from 5:00pm to 6:00pm alone to be
> stored into log file. The rest 23 hrs of data should be ignored and not to
> be stored in log file.
> 
> How can i achieve this?

Re: NiFi Spark Processor

[Oleg Zhurakousky]

Shankha

I know it may not always be all that obvious, but have you started the 
OutputPort? Also, have you defined ‘nifi.remote.input.socket.port’ property in 
wifi.properties?
Also, unless you’ve already discovered it there is an excellent blog 
(https://blogs.apache.org/nifi/entry/stream_processing_nifi_and_spark) from 
Mark on this which covers every little detail, so hope that will help.

Cheers
Oleg

On Nov 18, 2016, at 12:29 AM, shankhamajumdar 
> wrote:

Hi joey,

I am trying the POC in different way now.

1. GetFile processor which is getting the files from the local drive.

2. Connecting the processor to the NiFi Output Port.

3. Written a Spark streaming Job which is connecting to the NiFi Output Port
to stream the data.

The Spark Job is able to connect the NiFi Output Port successfully. But not
able to get any data in spark though in the NiFi the data has been queued
from GetFile processor. I have checked the NiFi Data Provenance of the NiFi
Output Port but no data is showing there as well where as GetFile to Output
Port connector is showing the data has been queued.

Regards,
Shankha



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/NiFi-Spark-Processor-tp13899p13913.html
Sent from the Apache NiFi Developer List mailing list archive at 
Nabble.com.

Re: NiFi processor validation

[Oleg Zhurakousky]

Could it also be related to https://issues.apache.org/jira/browse/NIFI-1318?

Cheers
Oleg

On Nov 8, 2016, at 7:52 AM, Joe Witt 
> wrote:

+1 to both of those points:
1) Avoid validating that which it doesn't help (disabled and running)
2) Avoid using web/synchronous threading for any user code

On Tue, Nov 8, 2016 at 7:43 AM, Matt Gilman 
> wrote:
I also agreed these changes make sense. In addition, another approach we
could consider that has been discussed in the past [1] is to perform
component validation asynchronously. This presents its own challenges but
would also be helpful. We should try to avoid calling into user code in any
web thread.

Matt

[1] https://issues.apache.org/jira/browse/NIFI-950

On Mon, Nov 7, 2016 at 6:15 PM, Matt Burgess 
> wrote:

Agreed. Also we validate processors on a timer-based strategy in
FlowController (looks like for snapshotting) and in the web server
(via ControllerFacade), those seem to happen 6-7 times on that
interval (which is like 15-20 seconds). Also we validate all
processors on any change to the canvas (such as moving a processor).
Besides Mike's suggestion, perhaps we should look at a purely
event-driven strategy for validating processors if possible?

Regards,
Matt

On Mon, Nov 7, 2016 at 6:06 PM, Joe Witt 
> wrote:
Makes good sense to me.

On Nov 7, 2016 5:39 PM, "Michael Moser" 
> wrote:

All,

I would like to propose a fundamental change to processor validation
based
on observations in https://issues.apache.org/jira/browse/NIFI-2996. I
would
like to validate processors only when they are in the STOPPED state.

The properties on a processor in the RUNNING state should always be
valid,
else you should not have been able to start the processor. A processor
in
the DISABLED statue doesn't show validation results, so it seems a
waste to
validate its properties.

The reason I'm proposing this change is because the NiFi UI slows down
as
you add more processors and controller services to the graph. Beyond
common
sense expectations that this would be true, it appears that processor
validation is a significant part of the 'cost' on the server when
responding to REST API requests.  Some details from my testing are in
the
JIRA ticket.

Thoughts?

Thanks,
-- Mike

Re: [ANNOUNCE] New Apache NiFi Committer Scott Aslan

[Oleg Zhurakousky]

Congrats Scott! Well earned!

Oleg
> On Nov 4, 2016, at 11:25 AM, Tony Kurc  wrote:
> 
> On behalf of the Apache NiFI PMC, I am very pleased to announce that Scott
> Aslan has accepted the PMC's invitation to become a committer on the Apache
> NiFi project. We greatly appreciate all of Scott's hard work and generous
> contributions to the project. We look forward to his continued involvement
> in the project.
> 
> For those familiar with the UI improvements in 1.x, a lot of the
> implementation was contributed by Scott, so another set of very visible
> contributions! (Sorry for the pun, I couldn't help myself).
> 
> Welcome and congratulations!
> Tony

Re: nifi git commit: This closes #1047

[Oleg Zhurakousky]

Andre

Quick note for the future. . .
I just noticed how this commit appears in the Git logs

===
commit 4acc9ad288cc005127d1709adbb1134f6dab94c6
Author: Andre F de Miranda 
Date:   Thu Nov 3 02:29:11 2016 +1100

This closes #1047
===

As you can see it does not have any description nor the corresponding JIRA. 
For consistency and to simplify traceability it would be nice to have those 
regardless of how trivial the actual issue is.

Cheers
Oleg 


> On Nov 2, 2016, at 11:30 AM, af...@apache.org wrote:
> 
> Repository: nifi
> Updated Branches:
>  refs/heads/master 511f2a0c8 -> 4acc9ad28
> 
> 
> This closes #1047
> 
> 
> Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
> Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/4acc9ad2
> Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/4acc9ad2
> Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/4acc9ad2
> 
> Branch: refs/heads/master
> Commit: 4acc9ad288cc005127d1709adbb1134f6dab94c6
> Parents: 511f2a0
> Author: Andre F de Miranda 
> Authored: Thu Nov 3 02:29:11 2016 +1100
> Committer: Andre F de Miranda 
> Committed: Thu Nov 3 02:29:11 2016 +1100
> 
> --
> 
> --
> 
> 
> 

Re: [ANNOUNCE] New Apache NiFi Committer Bryan Rosander

[Oleg Zhurakousky]

Congrats Bryan!
Glad to have you!
> On Oct 26, 2016, at 1:15 PM, Joe Witt  wrote:
> 
> On behalf of the Apache NiFI PMC, I am very pleased to announce that
> Bryan Rosander has accepted the PMC's invitation to become a committer
> on the Apache NiFi project. We greatly appreciate all of Andre's hard
> work and generous contributions and look forward to continued
> involvement in the project.
> 
> Bryan has contributed to a range of features and discussions such as
> the TLS Toolkit, Site to Site, JSON Conversion, Execute Script,
> Expression Language, MiNiFi, and event log parsing.
> 
> Welcome Bryan!
> 

Re: [ANNOUNCE] New Apache NiFi PMC Member - Matt Burgess

[Oleg Zhurakousky]

Matthew. . . congrats!
Ohh wait, sorry I am actually referring to you Burgess, not that other one that 
is pounding your neighborhood at the moment.
Anyway, stay safe and congrats once again!

Cheers
Oleg

> On Oct 7, 2016, at 11:12 AM, Pierre Villard  
> wrote:
> 
> Congrats Matt!!!
> 
> 2016-10-07 16:39 GMT+02:00 Joe Percivall :
> 
>> Congrats Matt! Very well deserved!
>> - - - - - - Joseph Percivalllinkedin.com/in/Percivalle:
>> joeperciv...@yahoo.com
>> 
>> 
>>On Friday, October 7, 2016 10:37 AM, Bryan Bende 
>> wrote:
>> 
>> 
>> Congrats Matt!
>> 
>> On Fri, Oct 7, 2016 at 10:28 AM, Tony Kurc  wrote:
>> 
>>> On behalf of the Apache NiFI PMC, I am very pleased to announce that Matt
>>> Burgess has accepted the PMC's invitation to join the Apache NiFi PMC. We
>>> greatly appreciate all of Matt's hard work and generous contributions to
>>> the project and all the positive interaction with the community. We look
>>> forward to his continued involvement in the project.
>>> 
>>> Welcome and congratulations!
>>> Tony
>>> 
>> 
>> 
>> 
>> 

Re: Mobile Support?

[Oleg Zhurakousky]

Tamar

NiFi is Java-based product and thus runs in JVM. So any JVM-capable (e.g., 
Android etc) device can run NiFi.
NiFi also has a Minify distribution specifically tailored to pervasive devices 
- https://nifi.apache.org/minifi/

That said, would you be able yo elaborate as to your use-case for running NiFi 
on the devices you describe?

Cheers
Oleg

On Oct 5, 2016, at 8:07 PM, Tamar Alicea 
> wrote:

Afternoon:

Does Nifi can be run in any mobile device (Android, iOS or Windows Phone)

Thanks,
Tamar Alicea

Re: How to Use Nifi Remote Debugger?

[Oleg Zhurakousky]

Also, keep in mind that while remote debugging helps with something that may 
already be running in production, for pure development there is a simpler way 
to run NiFi in debug mode without using remote debugger.
Here is more information - 
https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide#ContributorGuide-RunningNiFiinDebugmode

Cheers
Oleg

On Sep 22, 2016, at 1:00 PM, Russell Bateman 
>
 wrote:

Good for you, Keren!

On 09/22/2016 09:44 AM, Tseytlin, Keren wrote:
Hey All,

Thanks so much for the help! The remote debugger ended up working. Because
my Nifi was on AWS, I needed to do some port tunneling in order to open up
the communication. So I had run a line similar to this:

ssh -i <> -N -L 5500:<>:5500 ec2-user@<>


I documented everything here if anyone is interested:
http://kerent.me/blog/2016/09/22/Remote-Debugging-Nifi-From-EC2.html.

Thanks!

Keren

On 9/21/16, 5:06 PM, "Joe Witt" > 
wrote:

Try
telnet  5500

If that cannot connect then it is almost certainly a firewall issue.

On Wed, Sep 21, 2016 at 4:28 PM, Tseytlin, Keren
> wrote:
Thanks for the answers Russell and Manish.

It appears that it was an issue with the port 8000, when I moved it to
something random then Nifi would start up.

I¹m connecting to an EC2 that I¹ve set up because I need to test some
AWS
communication. Now that I have Nifi running, I¹m trying to connect with
the debugger and getting the following error:

Error running AWS Nifi: Unable to open debugger port
(xx.xxx.xxx.xxx:5500): java.net.ConnectionException ³Operation timed
out²

Does this seem like a networking error? Pretty sure my security groups
have opened up all port communication if coming from my IP.

Keren


On 9/21/16, 4:03 PM, "Russell Bateman"
>
 wrote:

Here's how I've been doing it for the last 6 months. I use IntelliJ, but
I'm also an Eclipse guy.

http://www.javahotchocolate.com/notes/nifi.html#20160323

Hope this helps.

Russ

On 09/21/2016 01:39 PM, Tseytlin, Keren wrote:
Hi All,

I want to set up the remote debugger with Nifi so that I can step
through the code with IntelliJ. I noticed that last year someone else
asked about how to set up remote debugging here ->
https://mail-archives.apache.org/mod_mbox/nifi-dev/201508.mbox/%3C76415
19
c73cd4ad3832b80461b8ce...@mbx080-w2-co-2.exch080.serverpod.net%3E

I followed these steps, and uncommented the line:
java.arg.debug=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,ad
dr
ess=8000. However, when I do that I can't get Nifi to start up properly
anymore. It will say that Nifi is running, but it will attempt to
display an error page (which is some text with html formatting that
doesn't get formatted properly).

Has remote debugging changed in the past year or so since that thread
was posted? Are there any other configs in Nifi that need to be
changed?
Would really appreciate additional detail for remote debugging with
Nifi.

Thanks,
Keren


The information contained in this e-mail is confidential and/or
proprietary to Capital One and/or its affiliates and may only be used
solely in performance of work or services for Capital One. The
information transmitted herewith is intended only for use by the
individual or entity to which it is addressed. If the reader of this
message is not the intended recipient, you are hereby notified that any
review, retransmission, dissemination, distribution, copying or other
use of, or taking of any action in reliance upon this information is
strictly prohibited. If you have received this communication in error,
please contact the sender and delete the material from your computer.



The information contained in this e-mail is confidential and/or
proprietary to Capital One and/or its affiliates and may only be used
solely in performance of work or services for Capital One. The
information transmitted herewith is intended only for use by the
individual or entity to which it is addressed. If the reader of this
message is not the intended recipient, you are hereby notified that any
review, retransmission, dissemination, distribution, copying or other
use of, or taking of any action in reliance upon this information is
strictly prohibited. If you have received this communication in error,
please contact the sender and delete the material from your computer.



The information contained in this e-mail is confidential and/or proprietary to 
Capital One and/or its affiliates and may only be used solely in performance of 
work or services for Capital One. The information transmitted herewith is 
intended only for use by 

Re: [ANNOUNCE] New Apache NiFi Committer Rob Moran

[Oleg Zhurakousky]

Indeed, UI looks pretty awesome now and we all know who’s behind it.
Congrats!

> On Sep 19, 2016, at 6:05 PM, Joe Witt  wrote:
> 
> Pretty cool to see Rob obtain committer status.  His efforts arent always
> obviously tied to a commit but so much of what people see are his concepts
> and ux design.
> 
> Congrats Rob!
> 
> On Sep 19, 2016 5:51 PM, "Matt Burgess"  wrote:
> 
>> Congratulations Rob!! Well deserved!!
>> 
>> -Matt
>> 
>> On Mon, Sep 19, 2016 at 3:08 PM, Tony Kurc  wrote:
>>> On behalf of the Apache NiFI PMC, I am very pleased to announce that Rob
>>> Moran has accepted the PMC's invitation to become a committer on the
>> Apache
>>> NiFi project. We greatly appreciate all of Rob's hard work and generous
>>> contributions to the project. We look forward to his continued
>> involvement
>>> in the project.
>>> 
>>> Among other things, Rob has been responsible for a lot of the look and
>> feel
>>> of the NiFi user interface, to include the awesome logos!
>>> 
>>> Welcome and congratulations!
>>> 
>>> Tony
>> 

Re: Custom Processor

[Oleg Zhurakousky]

Gunjan

If I understand you correctly you are trying to do something that works with 
JMS and SQL in atomic fashion. Possibly (and that is purely a guess) you are 
having some XA transactional concerns. . .
If so, that may be a bit counter-intuitive to micro-services nature of NiFi 
processors where each processor is an independent service with queues in 
between.
If that’s the case I would probably go with custom processor from scratch yet 
utilizing ControllerServices for JMS ConnectionFactory and SQL connection pool.

We can definitely help you with that, but I’ll stop right here just to see if 
you can clarify further

Cheers
Oleg

> On Sep 16, 2016, at 1:19 AM, Gunjan Dave  wrote:
> 
> Hi Oleg,
> Executesql is not the problem, but I am trying to achive something
> different here. Exevutesql was just an example.
> 
> Based on following article,
> https://dzone.com/articles/apache-nifi-not-from-scratch
> 
> i was trying to export classes of my existing application onto nifi custom
> processor. Some classes have multiple database calls and i was thinking of
> how I can do it. Those classes also have mq connection, so extending only
> executesql will not help me.
> 
> is there some other way I can use consumejms or publishjms and nifi
> sqlprocessors in my custom processor?
> 
> I understand that I can create these functionalities by using process
> groups and specific processors on nifi ui, but was just being adventurous
> on using NiFi differently by creating business specific custom processor
> 
> Anyone who has tried such a thing in past and give me some reference?
> 
> THANKS
> Gunjan
> 
> On Thu, Sep 15, 2016, 10:56 PM Gunjan Dave 
> wrote:
> 
>> Hi Nifi Team,
>> I want to create a custom processor in which I want to use the database
>> functionality.
>> Such as, i want to extract some information from database as part of
>> processor. I could Access the database controller service but how do I
>> reuse executesql processor in my custom processor?
>> 

Re: Custom Processor

[Oleg Zhurakousky]

Hi

You can obviously extends the executesql processor, but I am more interested in 
understanding what exactly are you trying to accomplish that executesql doesn’t 
do today. Don’t get me wrong as I am sure you may have a legitimate reason, but 
there may also be missing functionality that we may add to executesql that 
would satisfy your use case.

Anyway, more details would definitely help.
Cheers
Oleg

> On Sep 15, 2016, at 1:27 PM, Gunjan Dave  wrote:
> 
> Hi Nifi Team,
> I want to create a custom processor in which I want to use the database
> functionality.
> Such as, i want to extract some information from database as part of
> processor. I could Access the database controller service but how do I
> reuse executesql processor in my custom processor?

Re: ConsumeJMS and Acknowledgement Mode

[Oleg Zhurakousky]

Good point about lambda. Let me see if I can refactor it a bit.
And sure, please share your patch. Would love to compare.

Cheers
Oleg

> On Sep 15, 2016, at 11:10 AM, McDermott, Chris Kevin (MSDU - 
> STaTS/StorefrontRemote) <chris.mcderm...@hpe.com> wrote:
> 
> Oleg,
> 
> Thanks very much. Yes, I fully agree with you.  
> 
> I saw your PR and noted that it’s for 1.x.  Since it uses lambdas the patch 
> is not going to work for the 0.x branch.  I don’t know what your plan for 0.8 
> is but I have a much less sophisticated patch for the for the 0.x branch. 
> Should I share?
> 
> Chris McDermott
> 
> Remote Business Analytics
> STaTS/StoreFront Remote
> HPE Storage
> Hewlett Packard Enterprise
> Mobile: +1 978-697-5315
> 
> 
> 
> 
> On 9/15/16, 9:45 AM, "Oleg Zhurakousky" <ozhurakou...@hortonworks.com> wrote:
> 
>Chris
> 
>PR was submitted to the JIRA. Basically as I’ve commented in the JIRA I 
> now do not believe we should be exposing any QOS options to the end user and 
> instead implicitly ack messages only after they’ve been consumed by NiFi 
> (i.e., transferred to a success relationship). Basically a classic chain of 
> custody pattern. 
>Let me know if you agree since adding ACK mode options would be a simply 
> change, although at the moment I am reluctant to do so (see my explanation in 
> JIRA)
> 
>Cheers
>Oleg
> 
>> On Sep 14, 2016, at 10:13 AM, McDermott, Chris Kevin (MSDU - 
>> STaTS/StorefrontRemote) <chris.mcderm...@hpe.com> wrote:
>> 
>> Will do.  Thanks!
>> 
>> Chris McDermott
>> 
>> Remote Business Analytics
>> STaTS/StoreFront Remote
>> HPE Storage
>> Hewlett Packard Enterprise
>> Mobile: +1 978-697-5315
>> 
>> 
>> 
>> 
>> On 9/14/16, 10:10 AM, "Oleg Zhurakousky" <ozhurakou...@hortonworks.com> 
>> wrote:
>> 
>>   Chris
>> 
>>   The “AUTO_ACKNOWLEDGE” (actual value 1) is the default. So although it is 
>> not being set explicitly it is set.
>> 
>>   We can definitely expose the other two (client ack and dups ok).
>>   Please raise JIRA.
>>   Cheers
>>   Oleg
>> 
>>   On Sep 14, 2016, at 10:04 AM, McDermott, Chris Kevin (MSDU - 
>> STaTS/StorefrontRemote) 
>> <chris.mcderm...@hpe.com<mailto:chris.mcderm...@hpe.com>> wrote:
>> 
>>   Looking at the code the ACK mode is not being set.
>> 
>> /**
>>   * This method essentially performs initialization of this Processor by
>>   * obtaining an instance of the {@link ConnectionFactory} from the
>>   * {@link JMSConnectionFactoryProvider} (ControllerService) and 
>> performing a
>>   * series of {@link ConnectionFactory} adaptations which eventually 
>> results
>>   * in an instance of the {@link CachingConnectionFactory} used to 
>> construct
>>   * {@link JmsTemplate} used by this Processor.
>>   */
>>  private void buildTargetResource(ProcessContext context) {
>>  if (this.targetResource == null) {
>>  JMSConnectionFactoryProviderDefinition cfProvider = 
>> context.getProperty(CF_SERVICE).asControllerService(JMSConnectionFactoryProviderDefinition.class);
>>  ConnectionFactory connectionFactory = 
>> cfProvider.getConnectionFactory();
>> 
>>  UserCredentialsConnectionFactoryAdapter cfCredentialsAdapter = 
>> new UserCredentialsConnectionFactoryAdapter();
>>  
>> cfCredentialsAdapter.setTargetConnectionFactory(connectionFactory);
>>  
>> cfCredentialsAdapter.setUsername(context.getProperty(USER).getValue());
>>  
>> cfCredentialsAdapter.setPassword(context.getProperty(PASSWORD).getValue());
>> 
>>  this.cachingConnectionFactory = new 
>> CachingConnectionFactory(cfCredentialsAdapter);
>>  
>> this.cachingConnectionFactory.setSessionCacheSize(Integer.parseInt(context.getProperty(SESSION_CACHE_SIZE).getValue()));
>> 
>>  JmsTemplate jmsTemplate = new JmsTemplate();
>>  jmsTemplate.setConnectionFactory(this.cachingConnectionFactory);
>>  
>> jmsTemplate.setDefaultDestinationName(context.getProperty(DESTINATION).getValue());
>>  
>> jmsTemplate.setPubSubDomain(TOPIC.equals(context.getProperty(DESTINATION_TYPE).getValue()));
>> 
>>  // set of properties that may be good candidates for exposure 
>> via configuration
>>  jmsTemplate.setReceiveTimeout(1);
>> 
>>  this.targ

Re: ConsumeJMS and Acknowledgement Mode

[Oleg Zhurakousky]

Chris

PR was submitted to the JIRA. Basically as I’ve commented in the JIRA I now do 
not believe we should be exposing any QOS options to the end user and instead 
implicitly ack messages only after they’ve been consumed by NiFi (i.e., 
transferred to a success relationship). Basically a classic chain of custody 
pattern. 
Let me know if you agree since adding ACK mode options would be a simply 
change, although at the moment I am reluctant to do so (see my explanation in 
JIRA)

Cheers
Oleg

> On Sep 14, 2016, at 10:13 AM, McDermott, Chris Kevin (MSDU - 
> STaTS/StorefrontRemote) <chris.mcderm...@hpe.com> wrote:
> 
> Will do.  Thanks!
> 
> Chris McDermott
> 
> Remote Business Analytics
> STaTS/StoreFront Remote
> HPE Storage
> Hewlett Packard Enterprise
> Mobile: +1 978-697-5315
> 
> 
> 
> 
> On 9/14/16, 10:10 AM, "Oleg Zhurakousky" <ozhurakou...@hortonworks.com> wrote:
> 
>Chris
> 
>The “AUTO_ACKNOWLEDGE” (actual value 1) is the default. So although it is 
> not being set explicitly it is set.
> 
>We can definitely expose the other two (client ack and dups ok).
>Please raise JIRA.
>Cheers
>Oleg
> 
>On Sep 14, 2016, at 10:04 AM, McDermott, Chris Kevin (MSDU - 
> STaTS/StorefrontRemote) 
> <chris.mcderm...@hpe.com<mailto:chris.mcderm...@hpe.com>> wrote:
> 
>Looking at the code the ACK mode is not being set.
> 
>  /**
>* This method essentially performs initialization of this Processor by
>* obtaining an instance of the {@link ConnectionFactory} from the
>* {@link JMSConnectionFactoryProvider} (ControllerService) and 
> performing a
>* series of {@link ConnectionFactory} adaptations which eventually 
> results
>* in an instance of the {@link CachingConnectionFactory} used to 
> construct
>* {@link JmsTemplate} used by this Processor.
>*/
>   private void buildTargetResource(ProcessContext context) {
>   if (this.targetResource == null) {
>   JMSConnectionFactoryProviderDefinition cfProvider = 
> context.getProperty(CF_SERVICE).asControllerService(JMSConnectionFactoryProviderDefinition.class);
>   ConnectionFactory connectionFactory = 
> cfProvider.getConnectionFactory();
> 
>   UserCredentialsConnectionFactoryAdapter cfCredentialsAdapter = 
> new UserCredentialsConnectionFactoryAdapter();
>   
> cfCredentialsAdapter.setTargetConnectionFactory(connectionFactory);
>   
> cfCredentialsAdapter.setUsername(context.getProperty(USER).getValue());
>   
> cfCredentialsAdapter.setPassword(context.getProperty(PASSWORD).getValue());
> 
>   this.cachingConnectionFactory = new 
> CachingConnectionFactory(cfCredentialsAdapter);
>   
> this.cachingConnectionFactory.setSessionCacheSize(Integer.parseInt(context.getProperty(SESSION_CACHE_SIZE).getValue()));
> 
>   JmsTemplate jmsTemplate = new JmsTemplate();
>   jmsTemplate.setConnectionFactory(this.cachingConnectionFactory);
>   
> jmsTemplate.setDefaultDestinationName(context.getProperty(DESTINATION).getValue());
>   
> jmsTemplate.setPubSubDomain(TOPIC.equals(context.getProperty(DESTINATION_TYPE).getValue()));
> 
>   // set of properties that may be good candidates for exposure 
> via configuration
>   jmsTemplate.setReceiveTimeout(1);
> 
>   this.targetResource = 
> this.finishBuildingTargetResource(jmsTemplate);
>   }
> 
>The JavaDoc for JMSTemplate says.
> 
>Default settings for JMS Sessions are "not transacted" and 
> "auto-acknowledge". As defined by the Java EE specification, the transaction 
> and acknowledgement parameters are ignored when a JMS Session is created 
> inside an active transaction, no matter if a JTA transaction or a 
> Spring-managed transaction. To configure them for native JMS usage, specify 
> appropriate values for the "sessionTransacted" and "sessionAcknowledgeMode" 
> bean properties.
> 
>If I understand this correctly it means that ConsumeJMS is always using 
> auto-acknowledge which would explain my lost files on NiFi restart.
> 
>Do you concur?
> 
>Chris McDermott
> 
>Remote Business Analytics
>STaTS/StoreFront Remote
>HPE Storage
>Hewlett Packard Enterprise
>Mobile: +1 978-697-5315
> 
> 
> 
> 
>    On 9/14/16, 9:57 AM, "McDermott, Chris Kevin (MSDU - 
> STaTS/StorefrontRemote)" 
> <chris.mcderm...@hpe.com<mailto:chris.mcderm...@hpe.com>> wrote:
> 
>   Yes.
> 
> 

Re: ConsumeJMS and Acknowledgement Mode

[Oleg Zhurakousky]

Chris

The “AUTO_ACKNOWLEDGE” (actual value 1) is the default. So although it is not 
being set explicitly it is set.

We can definitely expose the other two (client ack and dups ok).
Please raise JIRA.
Cheers
Oleg

On Sep 14, 2016, at 10:04 AM, McDermott, Chris Kevin (MSDU - 
STaTS/StorefrontRemote) 
<chris.mcderm...@hpe.com<mailto:chris.mcderm...@hpe.com>> wrote:

Looking at the code the ACK mode is not being set.

  /**
* This method essentially performs initialization of this Processor by
* obtaining an instance of the {@link ConnectionFactory} from the
* {@link JMSConnectionFactoryProvider} (ControllerService) and performing a
* series of {@link ConnectionFactory} adaptations which eventually results
* in an instance of the {@link CachingConnectionFactory} used to construct
* {@link JmsTemplate} used by this Processor.
*/
   private void buildTargetResource(ProcessContext context) {
   if (this.targetResource == null) {
   JMSConnectionFactoryProviderDefinition cfProvider = 
context.getProperty(CF_SERVICE).asControllerService(JMSConnectionFactoryProviderDefinition.class);
   ConnectionFactory connectionFactory = 
cfProvider.getConnectionFactory();

   UserCredentialsConnectionFactoryAdapter cfCredentialsAdapter = new 
UserCredentialsConnectionFactoryAdapter();
   cfCredentialsAdapter.setTargetConnectionFactory(connectionFactory);
   
cfCredentialsAdapter.setUsername(context.getProperty(USER).getValue());
   
cfCredentialsAdapter.setPassword(context.getProperty(PASSWORD).getValue());

   this.cachingConnectionFactory = new 
CachingConnectionFactory(cfCredentialsAdapter);
   
this.cachingConnectionFactory.setSessionCacheSize(Integer.parseInt(context.getProperty(SESSION_CACHE_SIZE).getValue()));

   JmsTemplate jmsTemplate = new JmsTemplate();
   jmsTemplate.setConnectionFactory(this.cachingConnectionFactory);
   
jmsTemplate.setDefaultDestinationName(context.getProperty(DESTINATION).getValue());
   
jmsTemplate.setPubSubDomain(TOPIC.equals(context.getProperty(DESTINATION_TYPE).getValue()));

   // set of properties that may be good candidates for exposure via 
configuration
   jmsTemplate.setReceiveTimeout(1);

   this.targetResource = this.finishBuildingTargetResource(jmsTemplate);
   }

The JavaDoc for JMSTemplate says.

Default settings for JMS Sessions are "not transacted" and "auto-acknowledge". 
As defined by the Java EE specification, the transaction and acknowledgement 
parameters are ignored when a JMS Session is created inside an active 
transaction, no matter if a JTA transaction or a Spring-managed transaction. To 
configure them for native JMS usage, specify appropriate values for the 
"sessionTransacted" and "sessionAcknowledgeMode" bean properties.

If I understand this correctly it means that ConsumeJMS is always using 
auto-acknowledge which would explain my lost files on NiFi restart.

Do you concur?

Chris McDermott

Remote Business Analytics
STaTS/StoreFront Remote
HPE Storage
Hewlett Packard Enterprise
Mobile: +1 978-697-5315




On 9/14/16, 9:57 AM, "McDermott, Chris Kevin (MSDU - STaTS/StorefrontRemote)" 
<chris.mcderm...@hpe.com<mailto:chris.mcderm...@hpe.com>> wrote:

   Yes.

   Chris McDermott

   Remote Business Analytics
   STaTS/StoreFront Remote
   HPE Storage
   Hewlett Packard Enterprise
   Mobile: +1 978-697-5315




   On 9/14/16, 9:54 AM, "Oleg Zhurakousky" 
<ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote:

   So you’re asking about ConsumeJMS?

On Sep 14, 2016, at 9:49 AM, McDermott, Chris Kevin (MSDU - 
STaTS/StorefrontRemote) 
<chris.mcderm...@hpe.com<mailto:chris.mcderm...@hpe.com>> wrote:

Hi Oleg,

Sorry, I wasn’t more clear.  I should have stated that ConsumeJMS does not 
expose a setting for ACK mode.  Screen shot attached. So my question is which 
ACK mode does it use?

Thanks,

Chris McDermott

Remote Business Analytics
STaTS/StoreFront Remote
HPE Storage
Hewlett Packard Enterprise
Mobile: +1 978-697-5315




On 9/14/16, 9:41 AM, "Oleg Zhurakousky" 
<ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote:

  Chris

  FWIW, the Get/PutJMS* are effectively deprecated processors since they only 
support one JMS provider (ActiiveMQ) and should not be used. Instead we 
recommend Publish/ConsumeJMS pair.

  As to your question, which specific ACK mode you are inquiring about? As the 
doc currently states, the auto-ACK may loose messages. Is that what you are 
using?

  Cheers
  Oleg

  On Sep 14, 2016, at 9:28 AM, McDermott, Chris Kevin (MSDU - 
STaTS/StorefrontRemote) 
<chris.mcderm...@hpe.com<mailto:chris.mcderm...@hpe.com><mailto:chris.mcderm...@hpe.com>>
 wrote:


  Hey folks,

  The GetJMSQueue proce

Re: ConsumeJMS and Acknowledgement Mode

[Oleg Zhurakousky]

Ok, sorry, got confused a bit.

Indeed ConsumeJMS is using auto-ack at the moment, hence can suffer from the 
same problem as GetJMS. 
However, the designed of this new JMS support definitely provides for exposure 
of ACK and other properties to the end user. Please raise JIRA and we’ll take 
care of it.

Cheers
Oleg
> On Sep 14, 2016, at 9:57 AM, McDermott, Chris Kevin (MSDU - 
> STaTS/StorefrontRemote) <chris.mcderm...@hpe.com> wrote:
> 
> Yes.
> 
> Chris McDermott
> 
> Remote Business Analytics
> STaTS/StoreFront Remote
> HPE Storage
> Hewlett Packard Enterprise
> Mobile: +1 978-697-5315
> 
> 
> 
> 
> On 9/14/16, 9:54 AM, "Oleg Zhurakousky" <ozhurakou...@hortonworks.com> wrote:
> 
>So you’re asking about ConsumeJMS?
> 
>> On Sep 14, 2016, at 9:49 AM, McDermott, Chris Kevin (MSDU - 
>> STaTS/StorefrontRemote) <chris.mcderm...@hpe.com> wrote:
>> 
>> Hi Oleg,
>> 
>> Sorry, I wasn’t more clear.  I should have stated that ConsumeJMS does not 
>> expose a setting for ACK mode.  Screen shot attached. So my question is 
>> which ACK mode does it use?
>> 
>> Thanks,
>> 
>> Chris McDermott
>> 
>> Remote Business Analytics
>> STaTS/StoreFront Remote
>> HPE Storage
>> Hewlett Packard Enterprise
>> Mobile: +1 978-697-5315
>> 
>> 
>> 
>> 
>> On 9/14/16, 9:41 AM, "Oleg Zhurakousky" <ozhurakou...@hortonworks.com> wrote:
>> 
>>   Chris
>> 
>>   FWIW, the Get/PutJMS* are effectively deprecated processors since they 
>> only support one JMS provider (ActiiveMQ) and should not be used. Instead we 
>> recommend Publish/ConsumeJMS pair.
>> 
>>   As to your question, which specific ACK mode you are inquiring about? As 
>> the doc currently states, the auto-ACK may loose messages. Is that what you 
>> are using?
>> 
>>   Cheers
>>   Oleg
>> 
>>   On Sep 14, 2016, at 9:28 AM, McDermott, Chris Kevin (MSDU - 
>> STaTS/StorefrontRemote) 
>> <chris.mcderm...@hpe.com<mailto:chris.mcderm...@hpe.com>> wrote:
>> 
>> 
>>   Hey folks,
>> 
>>   The GetJMSQueue processor allows configuration of the Acknowledgment Mode.
>> 
>>   Acknowledgement Mode
>> 
>>   Client Acknowledge
>> 
>>   • Client Acknowledge
>>   • Auto Acknowledge
>> 
>>   The JMS Acknowledgement Mode. Using Auto Acknowledge can cause messages to 
>> be lost on restart of NiFi but may provide better performance than Client 
>> Acknowledge.
>> 
>> 
>>   Can someone speak to how Acknowledgement Mode is handled by the ConsumeJMS 
>> processor?
>> 
>>   The reason I ask is that I seem to be losing messages on NiFi restart.
>> 
>>   Thanks,
>>   Chris McDermott
>> 
>>   Remote Business Analytics
>>   STaTS/StoreFront Remote
>>   HPE Storage
>>   Hewlett Packard Enterprise
>>   Mobile: +1 978-697-5315
>> 
>> 
>> 
>> 
>> 
> 
> 
> 
> 

Re: ConsumeJMS and Acknowledgement Mode

[Oleg Zhurakousky]

So you’re asking about ConsumeJMS?

> On Sep 14, 2016, at 9:49 AM, McDermott, Chris Kevin (MSDU - 
> STaTS/StorefrontRemote) <chris.mcderm...@hpe.com> wrote:
> 
> Hi Oleg,
> 
> Sorry, I wasn’t more clear.  I should have stated that ConsumeJMS does not 
> expose a setting for ACK mode.  Screen shot attached. So my question is which 
> ACK mode does it use?
> 
> Thanks,
> 
> Chris McDermott
> 
> Remote Business Analytics
> STaTS/StoreFront Remote
> HPE Storage
> Hewlett Packard Enterprise
> Mobile: +1 978-697-5315
> 
> 
> 
> 
> On 9/14/16, 9:41 AM, "Oleg Zhurakousky" <ozhurakou...@hortonworks.com> wrote:
> 
>Chris
> 
>FWIW, the Get/PutJMS* are effectively deprecated processors since they 
> only support one JMS provider (ActiiveMQ) and should not be used. Instead we 
> recommend Publish/ConsumeJMS pair.
> 
>As to your question, which specific ACK mode you are inquiring about? As 
> the doc currently states, the auto-ACK may loose messages. Is that what you 
> are using?
> 
>Cheers
>Oleg
> 
>On Sep 14, 2016, at 9:28 AM, McDermott, Chris Kevin (MSDU - 
> STaTS/StorefrontRemote) 
> <chris.mcderm...@hpe.com<mailto:chris.mcderm...@hpe.com>> wrote:
> 
> 
>Hey folks,
> 
>The GetJMSQueue processor allows configuration of the Acknowledgment Mode.
> 
>Acknowledgement Mode
> 
>Client Acknowledge
> 
>• Client Acknowledge
>• Auto Acknowledge
> 
>The JMS Acknowledgement Mode. Using Auto Acknowledge can cause messages to 
> be lost on restart of NiFi but may provide better performance than Client 
> Acknowledge.
> 
> 
>Can someone speak to how Acknowledgement Mode is handled by the ConsumeJMS 
> processor?
> 
>The reason I ask is that I seem to be losing messages on NiFi restart.
> 
>Thanks,
>Chris McDermott
> 
>Remote Business Analytics
>STaTS/StoreFront Remote
>HPE Storage
>Hewlett Packard Enterprise
>Mobile: +1 978-697-5315
> 
> 
> 
> 
> 

Re: ConsumeJMS and Acknowledgement Mode

[Oleg Zhurakousky]

Chris

FWIW, the Get/PutJMS* are effectively deprecated processors since they only 
support one JMS provider (ActiiveMQ) and should not be used. Instead we 
recommend Publish/ConsumeJMS pair.

As to your question, which specific ACK mode you are inquiring about? As the 
doc currently states, the auto-ACK may loose messages. Is that what you are 
using?

Cheers
Oleg

On Sep 14, 2016, at 9:28 AM, McDermott, Chris Kevin (MSDU - 
STaTS/StorefrontRemote) 
> wrote:


Hey folks,

The GetJMSQueue processor allows configuration of the Acknowledgment Mode.

Acknowledgement Mode

Client Acknowledge

• Client Acknowledge
• Auto Acknowledge

The JMS Acknowledgement Mode. Using Auto Acknowledge can cause messages to be 
lost on restart of NiFi but may provide better performance than Client 
Acknowledge.


Can someone speak to how Acknowledgement Mode is handled by the ConsumeJMS 
processor?

The reason I ask is that I seem to be losing messages on NiFi restart.

Thanks,
Chris McDermott

Remote Business Analytics
STaTS/StoreFront Remote
HPE Storage
Hewlett Packard Enterprise
Mobile: +1 978-697-5315

Re: Anyone having issues running nifi-kafka-0-9-processors junits?

[Oleg Zhurakousky]

Ok, I’ll take a look at it.

> On Sep 12, 2016, at 9:20 AM, Joe Skora <jsk...@gmail.com> wrote:
> 
> I've been seeing it for a week for sure, not sure about previously.
> 
> But it is intermittent, a re-run usually succeeds.
> 
> On Mon, Sep 12, 2016 at 8:58 AM, Andre <andre-li...@fucs.org> wrote:
> 
>> Oleg,
>> 
>> Yes, it is intermittent and it only seen it happening when using -T
>> 
>> I would say it happens 1 out of 5 times so we can say it is intermittent.
>> If I recall correctly, Kafka 0.9 is the most common failure but give me a
>> few days and I will try to confirm that.
>> 
>> Cheers
>> 
>> 
>> 
>> On Mon, Sep 12, 2016 at 7:54 AM, Oleg Zhurakousky <
>> ozhurakou...@hortonworks.com> wrote:
>> 
>>> Andre
>>> 
>>> Couple of questions
>>> 1. Since KafkaPublisher class exists in all 3 Kafka processors (0.9 and
>>> 0.10 Kafka clients) under the same package name, which one does it happen
>>> in? I am assuming its intermittent.
>>> 2. How often does it happen?
>>> 
>>> My suspicion is that this test is using embedded Kafka together with NiFi
>>> test mocks and there may be slight possibility of a timing issue under
>> the
>>> load of the build which means the test may need to be tuned. But if this
>>> test fails on 0.10 only it could also mean that we may need to look a
>>> little deeper (since I can’t remember seeng this failure yet) which could
>>> relate to some of the changes in Kafka 10 clients
>>> 
>>> Cheers
>>> Oleg
>>> 
>>>> On Sep 11, 2016, at 8:02 AM, Andre F de Miranda <af...@fucs.org>
>> wrote:
>>>> 
>>>> Hi there,
>>>> 
>>>> is anyone else having frequent errors on some of the Kafka processors?
>>>> 
>>>> Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 22.248
>>> sec
>>>> <<< FAILURE! - in org.apache.nifi.processors.kafka.pubsub.
>>> KafkaPublisherTest
>>>> validateSuccessfulSendAsDelimited(org.apache.nifi.
>>> processors.kafka.pubsub.KafkaPublisherTest)
>>>> Time elapsed: 1.413 sec  <<< ERROR!
>>>> kafka.consumer.ConsumerTimeoutException: null
>>>> at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:69)
>>>> at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:33)
>>>> at kafka.utils.IteratorTemplate.maybeComputeNext(
>>> IteratorTemplate.scala:66)
>>>> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58)
>>>> at kafka.utils.IteratorTemplate.next(IteratorTemplate.scala:38)
>>>> at kafka.consumer.ConsumerIterator.next(ConsumerIterator.scala:46)
>>>> at
>>>> org.apache.nifi.processors.kafka.pubsub.KafkaPublisherTest.
>>> validateSuccessfulSendAsDelimited(KafkaPublisherTest.java:115)
>>>> 
>>>> It seems to happen only (or at least more frequently) when build is
>>>> performed with -T 2.0C
>>>> 
>>>> 
>>>> Cheers
>>> 
>>> 
>> 

Re: Anyone having issues running nifi-kafka-0-9-processors junits?

[Oleg Zhurakousky]

Andre

Couple of questions
1. Since KafkaPublisher class exists in all 3 Kafka processors (0.9 and 0.10 
Kafka clients) under the same package name, which one does it happen in? I am 
assuming its intermittent.
2. How often does it happen?

My suspicion is that this test is using embedded Kafka together with NiFi test 
mocks and there may be slight possibility of a timing issue under the load of 
the build which means the test may need to be tuned. But if this test fails on 
0.10 only it could also mean that we may need to look a little deeper (since I 
can’t remember seeng this failure yet) which could relate to some of the 
changes in Kafka 10 clients

Cheers
Oleg

> On Sep 11, 2016, at 8:02 AM, Andre F de Miranda  wrote:
> 
> Hi there,
> 
> is anyone else having frequent errors on some of the Kafka processors?
> 
> Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 22.248 sec
> <<< FAILURE! - in org.apache.nifi.processors.kafka.pubsub.KafkaPublisherTest
> validateSuccessfulSendAsDelimited(org.apache.nifi.processors.kafka.pubsub.KafkaPublisherTest)
> Time elapsed: 1.413 sec  <<< ERROR!
> kafka.consumer.ConsumerTimeoutException: null
> at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:69)
> at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:33)
> at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66)
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58)
> at kafka.utils.IteratorTemplate.next(IteratorTemplate.scala:38)
> at kafka.consumer.ConsumerIterator.next(ConsumerIterator.scala:46)
> at
> org.apache.nifi.processors.kafka.pubsub.KafkaPublisherTest.validateSuccessfulSendAsDelimited(KafkaPublisherTest.java:115)
> 
> It seems to happen only (or at least more frequently) when build is
> performed with -T 2.0C
> 
> 
> Cheers

Re: Max Kafka message size

[Oleg Zhurakousky]

Chris

The “. . . querying the max.message.bytes. . .” is exactly what I had in mind. 
As you mentioned earlier, that is where the default value came from in the 
first place. So, yes that is what we’re going to link together.

Cheers
Oleg

> On Aug 21, 2016, at 6:15 PM, McDermott, Chris Kevin (MSDU - 
> STaTS/StorefrontRemote) <chris.mcderm...@hpe.com> wrote:
> 
> Thanks for getting back, Oleg. I’d be happy to send demarcated messages; the 
> problem is reassembling them at the other end of the pipe.  I’ve done a lot 
> of searching for techniques to do this.  They all seem to have major draw 
> backs in terms of reliable message delivery or in terms of garbage collection 
> (not in the sense of Java GC, but cleanup of files bounced off of a shared 
> file system.) The nice thing about Kafka is its atomic, it has replicated 
> delivery, and guaranteed GC semantics.   My use case has fairly low 
> throughput requirements (thousands, not millions of TPM) where most messages 
> are fairly small but a few are larger.
> 
> It would be nice if the Kafka client could learn the max message size from 
> Kafka itself by querying the max.message.bytes on the topic, rather than have 
> the flow designer be required to set it on the producer Processors.  For now, 
> though I’d be happy going back to the old behavior where its set on the 
> producer Processors.  On the flip side I am also concerned that the clients 
> (GetKafka and ConsumeKafka) do not expose a max message parameter.  That will 
> be equally problematic.
> 
> Anyway, enough of my blathering.  
> 
> Yours and the communities help is greatly appreciated.
> 
> Thanks,
> 
> Chris McDermott
> 
> Remote Business Analytics
> STaTS/StoreFront Remote
> HPE Storage
> Hewlett Packard Enterprise
> Mobile: +1 978-697-5315
> 
> 
> 
> On 8/21/16, 3:43 PM, "Oleg Zhurakousky" <ozhurakou...@hortonworks.com> wrote:
> 
>Chris
> 
>Sorry for getting late to this
> 
>This indeed was intentional, primarily to follow Kafka’s best practices 
> where Kafka was not designed to be a general data transfer system of 
> arbitrary size but rather “manageable” size. Also, as you know we have 
> ability to demarcate messages essentially allowing you to send a very large 
> FlowFile that will be parsed into chunks where each chunk will end up to be a 
> separate Kafka message.
>That said, we did consider that at some point we may expose it as 
> configuration property and it seems like the time has come.
> 
>Cheers
>Oleg
> 
>> On Aug 20, 2016, at 7:03 PM, McDermott, Chris Kevin (MSDU - 
>> STaTS/StorefrontRemote) <chris.mcderm...@hpe.com> wrote:
>> 
>> Jira is https://issues.apache.org/jira/browse/NIFI-2614.
>> 
>> Thanks,
>> 
>> Chris McDermott
>> 
>> Remote Business Analytics
>> STaTS/StoreFront Remote
>> HPE Storage
>> Hewlett Packard Enterprise
>> Mobile: +1 978-697-5315
>> 
>> 
>> 
>> On 8/20/16, 6:57 PM, "McDermott, Chris Kevin (MSDU - 
>> STaTS/StorefrontRemote)" <chris.mcderm...@hpe.com> wrote:
>> 
>>   I’ll raise a JIRA, Joe.
>> 
>>   Thanks,
>> 
>>   Chris McDermott
>> 
>>   Remote Business Analytics
>>   STaTS/StoreFront Remote
>>   HPE Storage
>>   Hewlett Packard Enterprise
>>   Mobile: +1 978-697-5315
>> 
>> 
>> 
>>   On 8/20/16, 6:52 PM, "Joe Witt" <joe.w...@gmail.com> wrote:
>> 
>>   If no jira is raised sooner I'll raise one and get it sorted.
>> 
>>   On Aug 20, 2016 6:40 PM, "Andrew Psaltis" <psaltis.and...@gmail.com> 
>> wrote:
>> 
>>> Hi Chris,
>>> Sorry for not catching that code path. I am not sure if it is actually a
>>> regression as I took a look at the 1.0.0-BETA code and it matches the
>>> 0.7.0, specifically this comment block:
>>> 
>>> /*
>>> * We're using the default value from Kafka. We are using it to control the
>>> * message size before it goes to to Kafka thus limiting possibility of a
>>> * late failures in Kafka client.
>>> */
>>> 
>>> found at[1] leads me to believe it was intentional and not a regression.
>>> Looking at the 0.6.1 release code it appears that PutKafka used a default
>>> of 5 MB [2].
>>> 
>>> I can speculate on the reasoning behind it, however, I will refrain from
>>> opining on it as I was not involved in any of the conversations related to
>>> the change and enforcement of the 1 MB max.
>>> 
>>> [1]
>>> https://github.com/apache/nif

Re: Max Kafka message size

[Oleg Zhurakousky]

Chris

Sorry for getting late to this

This indeed was intentional, primarily to follow Kafka’s best practices where 
Kafka was not designed to be a general data transfer system of arbitrary size 
but rather “manageable” size. Also, as you know we have ability to demarcate 
messages essentially allowing you to send a very large FlowFile that will be 
parsed into chunks where each chunk will end up to be a separate Kafka message.
That said, we did consider that at some point we may expose it as configuration 
property and it seems like the time has come.

Cheers
Oleg

> On Aug 20, 2016, at 7:03 PM, McDermott, Chris Kevin (MSDU - 
> STaTS/StorefrontRemote)  wrote:
> 
> Jira is https://issues.apache.org/jira/browse/NIFI-2614.
> 
> Thanks,
> 
> Chris McDermott
> 
> Remote Business Analytics
> STaTS/StoreFront Remote
> HPE Storage
> Hewlett Packard Enterprise
> Mobile: +1 978-697-5315
> 
> 
> 
> On 8/20/16, 6:57 PM, "McDermott, Chris Kevin (MSDU - STaTS/StorefrontRemote)" 
>  wrote:
> 
>I’ll raise a JIRA, Joe.
> 
>Thanks,
> 
>Chris McDermott
> 
>Remote Business Analytics
>STaTS/StoreFront Remote
>HPE Storage
>Hewlett Packard Enterprise
>Mobile: +1 978-697-5315
> 
> 
> 
>On 8/20/16, 6:52 PM, "Joe Witt"  wrote:
> 
>If no jira is raised sooner I'll raise one and get it sorted.
> 
>On Aug 20, 2016 6:40 PM, "Andrew Psaltis"  
> wrote:
> 
>> Hi Chris,
>> Sorry for not catching that code path. I am not sure if it is actually a
>> regression as I took a look at the 1.0.0-BETA code and it matches the
>> 0.7.0, specifically this comment block:
>> 
>> /*
>> * We're using the default value from Kafka. We are using it to control the
>> * message size before it goes to to Kafka thus limiting possibility of a
>> * late failures in Kafka client.
>> */
>> 
>> found at[1] leads me to believe it was intentional and not a regression.
>> Looking at the 0.6.1 release code it appears that PutKafka used a default
>> of 5 MB [2].
>> 
>> I can speculate on the reasoning behind it, however, I will refrain from
>> opining on it as I was not involved in any of the conversations related to
>> the change and enforcement of the 1 MB max.
>> 
>> [1]
>> https://github.com/apache/nifi/blob/rel/nifi-1.0.0-BETA-
>> official/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-
>> processors/src/main/java/org/apache/nifi/processors/kafka/
>> PublishingContext.java#L36-L41
>> [2]
>> https://github.com/apache/nifi/blob/rel/nifi-0.6.1/nifi-
>> nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/
>> main/java/org/apache/nifi/processors/kafka/PutKafka.java#L169-L176
>> 
>> Thanks,
>> Andrew
>> 
>> On Sat, Aug 20, 2016 at 6:09 PM, McDermott, Chris Kevin (MSDU -
>> STaTS/StorefrontRemote)  wrote:
>> 
>>> Thanks, Andrew.
>>> 
>>> I’ve set all of the right broker configs to allow larger messages.
>>> Believe me I spent a lot of time banging my head against the wall
>> thinking
>>> that the broker and topic configs were wrong.
>>> 
>>> PublisingKafka uses PublishingContext.  That class has bean property
>>> called maxRequestSize, which defaults to 1048576.  As far as I can tell
>> the
>>> setMaxRequestSize() method is never called (except by some test code.)
>>> KafkaPublisher.publish() calls Max Record Size.getMaxRequestSize() and
>>> passes the result to the constructor for StreamDemarcator.   The publish
>>> method then calls the StreamDemarcator. getNextToken(), which in turns
>>> calls StreamDemarcator.fill() which compares the stream position against
>>> the maxRequestSize and throws the exception with this line.
>>> 
>>> throw new IllegalStateException("Maximum allowed data size of " +
>>> this.maxDataSize + " exceeded.");
>>> 
>>> Which matches what I see in the nifi-app.log file…
>>> 
>>> 2016-08-20 22:03:05,470 ERROR [Timer-Driven Process Thread-8]
>>> o.apache.nifi.processors.kafka.PutKafka
>>> java.lang.IllegalStateException: Maximum allowed data size of 1048576
>>> exceeded.
>>>at org.apache.nifi.stream.io.util.StreamDemarcator.fill(
>> StreamDemarcator.java:153)
>>> ~[nifi-utils-0.7.0.jar:0.7.0]
>>>at org.apache.nifi.stream.io.util.StreamDemarcator.
>>> nextToken(StreamDemarcator.java:105) ~[nifi-utils-0.7.0.jar:0.7.0]
>>>at org.apache.nifi.processors.kafka.KafkaPublisher.publish(
>> KafkaPublisher.java:129)
>>> ~[nifi-kafka-processors-0.7.0.jar:0.7.0]
>>>at org.apache.nifi.processors.kafka.PutKafka$1.process(
>> PutKafka.java:315)
>>> ~[nifi-kafka-processors-0.7.0.jar:0.7.0]
>>>at org.apache.nifi.controller.repository.
>>> StandardProcessSession.read(StandardProcessSession.java:1851)
>>> ~[nifi-framework-core-0.7.0.jar:0.7.0]
>>>at org.apache.nifi.controller.repository.
>>> StandardProcessSession.read(StandardProcessSession.java:1822)
>>> ~[nifi-framework-core-0.7.0.jar:0.7.0]
>>>at 

Re: Pooling connection factory for JMSConnectionFactoryProvider

[Oleg Zhurakousky]

Would you mind raising a JIRA as an improvement request and provide all the 
details and we can switch discussion there. I definitely do not want to ignore 
user feedback and experience, so let's discuss it there.

Cheers
Oleg

> On Aug 16, 2016, at 13:14, Chad Zobrisky <czobri...@gmail.com> wrote:
> 
> Oleg,
> 
> I see that now.  I was thinking at the service level for the caching
> factory to allow more reuse, but am unsure if that would even be a good
> idea.
> 
> Chad
> 
> On Tue, Aug 16, 2016 at 12:18 PM Oleg Zhurakousky <
> ozhurakou...@hortonworks.com> wrote:
> 
>> Actually it is using caching connection factory (see AbstractJMSProcessor):
>> 
>> private volatile CachingConnectionFactory cachingConnectionFactory;
>> 
>> Can you please provide more details what you are doing and what/where you
>> are seeing this?
>> 
>> Cheers
>> Oleg
>> 
>> On Aug 16, 2016, at 12:10 PM, Chad Zobrisky <czobri...@gmail.com> czobri...@gmail.com>> wrote:
>> 
>> Hello,
>> 
>> I ran into a case of many connections from a nifi instance to an ActiveMQ
>> hub and thought about pooled/cached connecitons.  I haven't seen any
>> discussion of adding either a new service for pooled/cached connections or
>> adding it to the current JMSConnectionFactoryProvider, via either
>> ActiveMQ's PooledConnectionFactory or Springs CachingConnectionFactory, for
>> reuse of connections, producers and sessions between processors.
>> 
>> I would like to work on this and was hoping to get suggestions about how to
>> best approach it.
>> 
>> Thanks,
>> Chad
>> 
>> 

Re: Pooling connection factory for JMSConnectionFactoryProvider

[Oleg Zhurakousky]

Actually it is using caching connection factory (see AbstractJMSProcessor):

private volatile CachingConnectionFactory cachingConnectionFactory;

Can you please provide more details what you are doing and what/where you are 
seeing this?

Cheers
Oleg

On Aug 16, 2016, at 12:10 PM, Chad Zobrisky 
> wrote:

Hello,

I ran into a case of many connections from a nifi instance to an ActiveMQ
hub and thought about pooled/cached connecitons.  I haven't seen any
discussion of adding either a new service for pooled/cached connections or
adding it to the current JMSConnectionFactoryProvider, via either
ActiveMQ's PooledConnectionFactory or Springs CachingConnectionFactory, for
reuse of connections, producers and sessions between processors.

I would like to work on this and was hoping to get suggestions about how to
best approach it.

Thanks,
Chad

Re: 0.x Build Breaking on nifi-kafka-pubsub-processors?

[Oleg Zhurakousky]

JFK. Yeah I hope you got the iPhone spell tricks ;)

Sent from my iPhone

> On Aug 6, 2016, at 16:09, Oleg Zhurakousky <ozhurakou...@hortonworks.com> 
> wrote:
> 
> Hmm, I build it before pushing those changes, but now I am thinking may be I 
> forgot to switch JFK to 1.7. I'll look at it tomorrow, thanks for the heads up
> 
> Sent from my iPhone
> 
>> On Aug 6, 2016, at 14:43, James Wing <jvw...@gmail.com> wrote:
>> 
>> It looks like the 0.x build is breaking on nifi-kafka-pubsub-processors.
>> Is this a known issue?
>> (see Travis build https://travis-ci.org/apache/nifi/jobs/150124534, for
>> https://issues.apache.org/jira/browse/NIFI-2423)
>> 
>> [INFO]
>> 
>> [INFO] Building nifi-kafka-pubsub-processors 0.8.0-SNAPSHOT
>> [INFO]
>> 
>> [INFO]
>> [INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-maven) @
>> nifi-kafka-pubsub-processors ---
>> [INFO]
>> [INFO] --- maven-remote-resources-plugin:1.5:process (default) @
>> nifi-kafka-pubsub-processors ---
>> [INFO]
>> [INFO] --- maven-resources-plugin:2.7:resources (default-resources) @
>> nifi-kafka-pubsub-processors ---
>> [INFO] Using 'UTF-8' encoding to copy filtered resources.
>> [INFO] Copying 3 resources
>> [INFO] Copying 3 resources
>> [INFO]
>> [INFO] --- maven-compiler-plugin:3.2:compile (default-compile) @
>> nifi-kafka-pubsub-processors ---
>> [INFO] Changes detected - recompiling the module!
>> [INFO] Compiling 6 source files to
>> /home/travis/build/apache/nifi/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-processors/target/classes
>> [INFO] -
>> [ERROR] COMPILATION ERROR :
>> [INFO] -
>> [ERROR] bootstrap class path not set in conjunction with -source 1.7
>> /home/travis/build/apache/nifi/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-processors/src/main/java/org/apache/nifi/processors/kafka/pubsub/AbstractKafkaProcessor.java:[361,52]
>> error: cannot find symbol
>> [ERROR]   symbol:   method getKeyPassword()
>> location: variable sslContextService of type SSLContextService
>> /home/travis/build/apache/nifi/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-processors/src/main/java/org/apache/nifi/processors/kafka/pubsub/AbstractKafkaProcessor.java:[361,139]
>> error: cannot find symbol
>> [INFO] 2 errors
> 

Re: 0.x Build Breaking on nifi-kafka-pubsub-processors?

[Oleg Zhurakousky]

Hmm, I build it before pushing those changes, but now I am thinking may be I 
forgot to switch JFK to 1.7. I'll look at it tomorrow, thanks for the heads up

Sent from my iPhone

> On Aug 6, 2016, at 14:43, James Wing  wrote:
> 
> It looks like the 0.x build is breaking on nifi-kafka-pubsub-processors.
> Is this a known issue?
> (see Travis build https://travis-ci.org/apache/nifi/jobs/150124534, for
> https://issues.apache.org/jira/browse/NIFI-2423)
> 
> [INFO]
> 
> [INFO] Building nifi-kafka-pubsub-processors 0.8.0-SNAPSHOT
> [INFO]
> 
> [INFO]
> [INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-maven) @
> nifi-kafka-pubsub-processors ---
> [INFO]
> [INFO] --- maven-remote-resources-plugin:1.5:process (default) @
> nifi-kafka-pubsub-processors ---
> [INFO]
> [INFO] --- maven-resources-plugin:2.7:resources (default-resources) @
> nifi-kafka-pubsub-processors ---
> [INFO] Using 'UTF-8' encoding to copy filtered resources.
> [INFO] Copying 3 resources
> [INFO] Copying 3 resources
> [INFO]
> [INFO] --- maven-compiler-plugin:3.2:compile (default-compile) @
> nifi-kafka-pubsub-processors ---
> [INFO] Changes detected - recompiling the module!
> [INFO] Compiling 6 source files to
> /home/travis/build/apache/nifi/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-processors/target/classes
> [INFO] -
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR] bootstrap class path not set in conjunction with -source 1.7
> /home/travis/build/apache/nifi/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-processors/src/main/java/org/apache/nifi/processors/kafka/pubsub/AbstractKafkaProcessor.java:[361,52]
> error: cannot find symbol
> [ERROR]   symbol:   method getKeyPassword()
>  location: variable sslContextService of type SSLContextService
> /home/travis/build/apache/nifi/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-pubsub-processors/src/main/java/org/apache/nifi/processors/kafka/pubsub/AbstractKafkaProcessor.java:[361,139]
> error: cannot find symbol
> [INFO] 2 errors

Re: [ANNOUNCE] New Apache NiFi Committer Yolanda Davis

[Oleg Zhurakousky]

Welcome aboard Yolanda!

Oleg 

> On Jul 30, 2016, at 08:22, Joe Witt  wrote:
> 
> Congrats and thanks Yolanda!
> 
>> On Jul 30, 2016 8:16 AM, "Tony Kurc"  wrote:
>> 
>> On behalf of the Apache NiFI PMC, I am very pleased to announce that
>> Yolanda Davis has accepted the PMC's invitation to become a committer on
>> the Apache NiFi project. We greatly appreciate all of Yolanda's hard work
>> and generous contributions and look forward to continued involvement in the
>> project.
>> 
>> In addition to contributing variable registry support and json conversion,
>> many of you may have heard one of her talks and experienced her enthusiasm
>> firsthand at a myriad of conferences and meetups.
>> 
>> 
>> Tony
>> 

Re: [ANNOUNCE] New Apache NiFi Committer Koji Kawamura

[Oleg Zhurakousky]

Congrats Koji!
Glad to see you on board!

Oleg
On Jul 19, 2016, at 7:45 PM, Andy LoPresto 
> wrote:

Congratulations Koji. Excited for even more great contributions.

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Jul 19, 2016, at 3:58 PM, Matt Burgess 
> wrote:

Congratulations and welcome Koji! Your contributions are greatly
appreciated and I'm looking forward to more in the future!

Regards,
Matt

On Tue, Jul 19, 2016 at 6:55 PM, Tony Kurc 
> wrote:
On behalf of the Apache NiFI PMC, I am very pleased to announce that Koji
Kawamura has accepted the PMC's invitation to become a committer on the
Apache NiFi project. We greatly appreciate all of Koji's hard work and
generous contributions and look forward to continued involvement in the
project.

Koji has been instrumental in couchbase integration, as well improving
site-to-site. Welcome and congratulations!

Tony

Re: ConsumeKafka vs getKafka

[Oleg Zhurakousky]

Sumo

Thank you for pointing this out. Very valid comments and just in time for the 
upcoming 1.0 release. Would you mind to raise a JIRA and we'll address it 
promptly? 

Cheers 
Oleg

> On Jul 17, 2016, at 16:38, Sumanth Chinthagunta  wrote:
> 
> Env: Linux , Kafka 9.0.x, NiFi: 0.7 
> 
> We like to use new ConsumeKafka processor  but found critical limitations 
> compared to old getKafka processor. 
> New ConsumeKafka is not writing critical Kafka attributes  i.e., kafka.key, 
> kafka.offset, kafka.partition etc into flowFile attributes. 
> 
> 
> Old getKafka processor: 
> 
> Standard FlowFile Attributes
> Key: 'entryDate'
>   Value: 'Sun Jul 17 15:17:00 CDT 2016'
> Key: 'lineageStartDate'
>   Value: 'Sun Jul 17 15:17:00 CDT 2016'
> Key: 'fileSize'
>   Value: '183'
> FlowFile Attribute Map Content
> Key: 'filename'
>   Value: '19709945781167274'
> Key: 'kafka.key'
>   Value: '{"database":"test","table":"sc_job","pk.systemid":1}'
> Key: 'kafka.offset'
>   Value: '1184010261'
> Key: 'kafka.partition'
>   Value: '0'
> Key: 'kafka.topic'
>   Value: ‘data'
> Key: 'path'
>   Value: './'
> Key: 'uuid'
>   Value: '244059bb-9ad9-4d74-b1fb-312eee72124a'
> 
> —
> new ConsumeKafka processor : 
> 
> Standard FlowFile Attributes
> Key: 'entryDate'
>   Value: 'Sun Jul 17 15:18:41 CDT 2016'
> Key: 'lineageStartDate'
>   Value: 'Sun Jul 17 15:18:41 CDT 2016'
> Key: 'fileSize'
>   Value: '183'
> FlowFile Attribute Map Content
> Key: 'filename'
>   Value: '19710046870478139'
> Key: 'path'
>   Value: './'
> Key: 'uuid'
>   Value: '349fbeb3-e342-4533-be4c-424793fa5c59’
> 
> Thanks 
> Sumo

Re: GetJMSQueue does not detect dead connections

[Oleg Zhurakousky]

Chris

Thank you for a valuable feedback
I think I do understand the issue and if I were to paraphrase is I would simply 
state that 'Publish/ConsumeJM” does not detect lost connection to the broker’. 
If that’s the case then there are definitely hooks in spring to ensure that is 
handled, otherwise clarify.
In any event, please do raise JIRAs if you believe it is justified and we can 
collaborate there.

Cheers
Oleg

> On Jul 13, 2016, at 2:07 PM, McDermott, Chris Kevin (MSDU - 
> STaTS/StorefrontRemote) <chris.mcderm...@hpe.com> wrote:
> 
> Oleg, 
> 
> I’ve determined that ConsumeJMS has exactly the same problem.  Do a ‘reboot 
> –f now’ on the node running the JMS broker and ConsumeJMS will not notice a 
> problem until the TCP timeout which could be hours, days, or even never.  We 
> could easily fix this in GetJMSQueue, but ConsumeJMS is implemented using the 
> Spring Framework’s JMS library and any fix would probably have to come from 
> the Spring Community.
> 
> I’d like to file a couple of Jiras on this but will wait for your response.
> 
> Thanks,
> 
> Chris McDermott
> 
> Remote Business Analytics
> STaTS/StoreFront Remote
> HPE Storage
> Hewlett Packard Enterprise
> Mobile: +1 978-697-5315
> 
> 
> 
> On 6/16/16, 1:43 PM, "Oleg Zhurakousky" <ozhurakou...@hortonworks.com> wrote:
> 
>> Chris 
>> 
>> Given that we are deprecating Get/PutJMS* in favor of Publish/SubscribeJMS, 
>> I’d suggest start using those once.
>> 
>> Cheers
>> Oleg
>> 
>> 
>>> On Jun 16, 2016, at 1:34 PM, McDermott, Chris Kevin (MSDU - 
>>> STaTS/StorefrontRemote) <chris.mcderm...@hpe.com> wrote:
>>> 
>>> Folks,
>>> 
>>> I’ve been trying to test my GetJMSQueue configuration so that it detects a 
>>> dead broker connection and fails over to an alternate broker.  When I say 
>>> dead connection I mean TCP connection that has not been closed but is no 
>>> longer passing traffic.  In the real world this typically happens when 
>>> broker server crashes and so it does not reset the open connections.  For 
>>> my test case I am using iptables to block traffic.
>>> 
>>> This is the connection URI I am using
>>> 
>>> failover:(tcp://host2:61616,tcp://host1:61616)?randomize=false=3000=3=3=1=0
>>> 
>>> They key parameters here are soTimeout=3 and soWriteTimeout=3
>>> 
>>> These set a 30 second timeout on socket reads and writes.  I’m not sure if 
>>> these are necessary since I believe the JMSConsumer classes specifies its 
>>> own timeout according to the processor configuration.  The important thing 
>>> to note is that when one of these timeouts occurs the AMQ client does not 
>>> close the connection.
>>> 
>>> I believe the deficiency here is that JMSConsumer does not consider the 
>>> possibility that the connection is dead.   The problem with this is that an 
>>> attempt to reconnect and failover to an alternate broker is not made.
>>> 
>>> I think the fix would involve counting the number of sequential empty 
>>> responses on the connection and then closing the connection once that 
>>> number crosses some threshold.  Then subsequent onTrigger() would cause a 
>>> new connection attempt.
>>> 
>>> Thoughts?
>>> 
>>> Chris McDermott
>>> 
>>> Remote Business Analytics
>>> STaTS/StoreFront Remote
>>> HPE Storage
>>> Hewlett Packard Enterprise
>>> Mobile: +1 978-697-5315
>>> 
>>> https://www.storefrontremote.com
>> 
> 

Re: External Repositories (aka Extension Registry) for Dynamically-loaded Extensions

[Oleg Zhurakousky]

Matt, 

Due to “other tasks” I personally haven’t been able yet to follow everything 
that’s going on, nor had I have a chance to look at the POC you and Puspendu 
have created, so I think global update of the page would be too early at the 
moment as it doesn’t really capture all of the concerns and there are certainly 
quite a few vey common concerns that need to be addressed outside of what I was 
able to extract from recent interaction. 
I will do my best to go through all the comments by the end of next week.

Cheers
Oleg

> On Jul 13, 2016, at 1:46 PM, Matt Foley  wrote:
> 
> Hi all,
> I'm about to do a global edit of the design page at 
> https://cwiki.apache.org/confluence/display/NIFI/Extension+Repositories+%28aka+Extension+Registry%29+for+Dynamically-loaded+Extensions
> based on comments by Joe Witt and Aldrin Piri.
> 
> This will include editing text that was written by the folks on the CC: line. 
>  If any of you object up front, please say so, otherwise after the fact 
> there's always Page History :-)
> 
> Among other edits, I intend to unify Sections 3 (Proposed Features) and 5 
> (Use Cases), removing contradictions and reducing the discussion to what we 
> actually intend to implement.  I will add a section on "Possible Future 
> Features", so as to not lose the other ideas.
> 
> I will also break out sections for "Dependency Management and Auto-loading", 
> and "Discovery - Browsing Extensions (and their docs) Without Loading Them".  
> These are two areas that are least well defined, and have already generated 
> the most discussion. 
> 
> I will start this editing tomorrow morning, unless there are '-1's.
> Thanks,
> --Matt
> 
> From: Matt Foley
> Sent: Monday, July 11, 2016 4:20 PM
> To: dev@nifi.apache.org
> Subject: Re: External Repositories (aka Extension Registry) for 
> Dynamically-loaded Extensions
> 
> Thanks, Joe.  I've responded to the issues you raised in Sections 5 and 6.
> Sections 2 and 3 were written by others and I am reticent to edit their text;
> I hope they will accept or respond to your comments there.
> 
> Thanks everyone,
> --Matt
> 
> From: Joe Witt 
> Sent: Monday, July 11, 2016 3:05 PM
> To: dev@nifi.apache.org
> Subject: Re: External Repositories (aka Extension Registry) for 
> Dynamically-loaded Extensions
> 
> Matt, Puspendu,
> 
> I put some thoughts on the wiki today and I think it looks like it is
> coming along nicely.  I believe the slow response cycle right now is
> because there are a few fairly big efforts underway at once including
> the first release of minifi, apache nifi 0.7.0 voting, and closing
> down for an apache nifi 1.0 vote.  Good to keep the discussion going
> and it is great you're pushing this.  It is much needed.
> 
> Joe
> 
> 
> On Mon, Jul 11, 2016 at 3:28 PM, Matt Foley  wrote:
>> Hi all,
>> Looks like this didn't get to the top of anyone's priority list over the 
>> holiday week, since the 
>> Jira, the Wiki 
>> page
>>  and this Mail thread have all been quiet.
>> 
>> If you don't have time to look at the code, that's absolutely okay, but 
>> Sections 5 and 6 of the Wiki 
>> page?
>>  are only about one page of reading total, and we would really appreciate a 
>> couple +1's to say it's okay to proceed with this basic approach. Please? :-)
>> 
>> Thanks,
>> --Matt and Puspendu
>> 
>> From: Matt Foley 
>> Sent: Sunday, July 03, 2016 3:27 PM
>> To: dev@nifi.apache.org
>> Subject: Re: External Repositories (aka Extension Registry) for 
>> Dynamically-loaded Extensions
>> 
>> That would be great. Even though the prototype impl is incomplete, it's 
>> actually about 30%-50% done, if the approach is acceptable to you, the 
>> community. We could easily finish it in a reasonable timeframe for the 
>> release after 1.0, and we're interested in doing so.
>> 
>> Thanks,
>> --Matt
>> 
>>> On Jul 3, 2016, at 3:15 PM, Joe Witt  wrote:
>>> 
>>> Thanks Matt.  Really great to keep having this evolve.  Once we get on
>>> the other side of the 1.0 release we should kick off another roadmap
>>> discussion to recap what has been done in the past six or so months
>>> and also to identify what are some good focus areas for the next 6+
>>> months.
>>> 
 On Sun, Jul 3, 2016 at 9:12 AM, Matt Foley  wrote:
 Hi all,
 
 Puspendu Banerjee and I have been exchanging ideas about how to design the 

Re: Update on 0.7.0 RC2

[Oleg Zhurakousky]

Just as an FYI, the PRs for NIFI-2160 are merged and the issue is resolved.
Hopefully https://issues.apache.org/jira/browse/NIFI-2192 won’t be a pain.

Oleg

On Jul 7, 2016, at 3:03 PM, Oleg Zhurakousky 
<ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote:

As it appears, the spotlight is on me ;)
I’ll merge 2160 once I get to the stopping point with what I am doing now 
(NIFI-826 . . .really close) and that is when I’ll take care of Kafka as well 
(had a nice chat with Mark so there is a plan).

Cheers
Oleg

On Jul 7, 2016, at 2:33 PM, Joe Percivall 
<joeperciv...@yahoo.com.INVALID<mailto:joeperciv...@yahoo.com.invalid>> wrote:

Hello Team,

As the theme with 0.7.0 has been, it's one step forward and one step back. 
There has been some great work and most of the issues previously seen were 
addressed but another blocker has been found.

- "Enabled ControllerServices disabled on restart" NIFI-2160[1] [status] PR 
have been posted by Oleg Zhurakousky and Brandon DeVries gave a +1 on the 
ticket but ambiguous about what is the next step. I commented asking for a 
follow-up.

- "PutKafka results in OOME if sending very large delimited file" NIFI-2192[2] 
[status] New issue discovered by Mark Payne that causes OOME. Oleg Zhurakousky 
is working to address it.

[1] https://issues.apache.org/jira/browse/NIFI-2160
[2] https://issues.apache.org/jira/browse/NIFI-2192

Joe
- - - - - -
Joseph Percivall
linkedin.com/in/Percivall<http://linkedin.com/in/Percivall>
e: joeperciv...@yahoo.com




On Saturday, July 2, 2016 1:02 PM, Joe Percivall <joeperciv...@yahoo.com> wrote:
Hello Team,

There have been two new critical bugs identified by Brandon DeVries and added 
to the 0.7.0 release. The status of the remaining 0.7.0 issues are below.

- "Enabled ControllerServices disabled on restart" NIFI-2160[1] [status] A 
regression due to NIFI-2032 that breaks backwards compatibility. Discussion has 
started between Oleg Zhurakousky and Brandon DeVries.

- "Fingerprint not detecting flow.xml differences" NIFI-2159[2] [status] 
Reported by Brandon. No one has started work on it. NEEDS AN ASSIGNEE

- "Secure clustering returning bad request response" NIFI-2119[3] [status] No 
updates, Andy LoPresto is looking into it.

Again for clarity, NIFI-1974 and NIFI-2089 are listed as 0.7.0 and unresolved 
but have already been merged into 0.7.0. Also due to the reasons highlighted in 
my previous message I will not be tracking NIFI-2120 and it is punted.

[1] https://issues.apache.org/jira/browse/NIFI-2160
[2] https://issues.apache.org/jira/browse/NIFI-2159
[3] https://issues.apache.org/jira/browse/NIFI-2119

Hope everyone has a fun and safe fourth,
Joe
- - - - - -
Joseph Percivall
linkedin.com/in/Percivall
e: joeperciv...@yahoo.com





On Wednesday, June 29, 2016 9:12 AM, Joe Percivall 
<joeperciv...@yahoo.com.INVALID> wrote:
Hello Team,

As most know, the closing of RC1 was due to three issues discovered by Joe 
Witt. A status update for each is below.


- "Secure clustering returning bad request response" NIFI-2119[1] [status] A 
probable source of the problem was discovered by Matt Gilman, NIFI-1753 a 
security upgrade. Andy LoPresto is looking into a fix.

- "source release produces an extraneous file" NIFI-2120[2] [status] I looked 
into how it would be done and I couldn't find any options in the maven release 
plugin that support excluding specific files from the release. I will be 
punting this, unless if someone with more experience with the maven release 
plugin can fix it.

- "Correct license/notice issues" NIFI-2118[3] [status] I addressed the issues 
identified by Joe Witt and the fix was merged by Matt Burgess.

Again I apologize for not finding this issues earlier and will keep you all up 
to date on any changes. Also since RC1 a commit addressing NIFI-1920[4] has 
been merged into the 0.x branch. I will be including it in RC2. It is a simple 
fix that addresses an inadvertent rollback in UnpackContent.

[1] https://issues.apache.org/jira/browse/NIFI-2119
[2] https://issues.apache.org/jira/browse/NIFI-2120
[3] https://issues.apache.org/jira/browse/NIFI-2118
[4] https://issues.apache.org/jira/browse/NIFI-920

Thank you,
Joe
- - - - - -
Joseph Percivall
linkedin.com/in/Percivall
e: joeperciv...@yahoo.com

Re: [ANNOUNCE] New Apache NiFi PMC Member - Andy LoPresto

[Oleg Zhurakousky]

Way to go Andy! Well deserved!

Cheers
Oleg
> On Jul 7, 2016, at 5:09 PM, Matt Burgess  wrote:
> 
> Congratulations Andy! Well deserved!
> 
> 
>> On Jul 7, 2016, at 12:44 PM, Joe Witt  wrote:
>> 
>> On behalf of the Apache NiFI PMC, I am very pleased to announce that
>> Andy LoPresto has accepted the PMC's invitation to join the Apache
>> NiFi PMC. We greatly appreciate all of Andy's hard work and generous
>> contributions to the project with a specific focus on security related
>> elements. We look forward to his continued involvement in the project.
>> 
>> Welcome and congratulations!
> 

Re: Update on 0.7.0 RC2

[Oleg Zhurakousky]

As it appears, the spotlight is on me ;)
I’ll merge 2160 once I get to the stopping point with what I am doing now 
(NIFI-826 . . .really close) and that is when I’ll take care of Kafka as well 
(had a nice chat with Mark so there is a plan). 

Cheers
Oleg

> On Jul 7, 2016, at 2:33 PM, Joe Percivall <joeperciv...@yahoo.com.INVALID> 
> wrote:
> 
> Hello Team,
> 
> As the theme with 0.7.0 has been, it's one step forward and one step back. 
> There has been some great work and most of the issues previously seen were 
> addressed but another blocker has been found.
> 
> - "Enabled ControllerServices disabled on restart" NIFI-2160[1] [status] PR 
> have been posted by Oleg Zhurakousky and Brandon DeVries gave a +1 on the 
> ticket but ambiguous about what is the next step. I commented asking for a 
> follow-up.
> 
> - "PutKafka results in OOME if sending very large delimited file" 
> NIFI-2192[2] [status] New issue discovered by Mark Payne that causes OOME. 
> Oleg Zhurakousky is working to address it.
> 
> [1] https://issues.apache.org/jira/browse/NIFI-2160
> [2] https://issues.apache.org/jira/browse/NIFI-2192
> 
> Joe
> - - - - - - 
> Joseph Percivall
> linkedin.com/in/Percivall
> e: joeperciv...@yahoo.com
> 
> 
> 
> 
> On Saturday, July 2, 2016 1:02 PM, Joe Percivall <joeperciv...@yahoo.com> 
> wrote:
> Hello Team,
> 
> There have been two new critical bugs identified by Brandon DeVries and added 
> to the 0.7.0 release. The status of the remaining 0.7.0 issues are below.
> 
> - "Enabled ControllerServices disabled on restart" NIFI-2160[1] [status] A 
> regression due to NIFI-2032 that breaks backwards compatibility. Discussion 
> has started between Oleg Zhurakousky and Brandon DeVries.
> 
> - "Fingerprint not detecting flow.xml differences" NIFI-2159[2] [status] 
> Reported by Brandon. No one has started work on it. NEEDS AN ASSIGNEE
> 
> - "Secure clustering returning bad request response" NIFI-2119[3] [status] No 
> updates, Andy LoPresto is looking into it.
> 
> Again for clarity, NIFI-1974 and NIFI-2089 are listed as 0.7.0 and unresolved 
> but have already been merged into 0.7.0. Also due to the reasons highlighted 
> in my previous message I will not be tracking NIFI-2120 and it is punted.
> 
> [1] https://issues.apache.org/jira/browse/NIFI-2160
> [2] https://issues.apache.org/jira/browse/NIFI-2159
> [3] https://issues.apache.org/jira/browse/NIFI-2119
> 
> Hope everyone has a fun and safe fourth,
> Joe
> - - - - - - 
> Joseph Percivall
> linkedin.com/in/Percivall
> e: joeperciv...@yahoo.com
> 
> 
> 
> 
> 
> On Wednesday, June 29, 2016 9:12 AM, Joe Percivall 
> <joeperciv...@yahoo.com.INVALID> wrote:
> Hello Team,
> 
> As most know, the closing of RC1 was due to three issues discovered by Joe 
> Witt. A status update for each is below. 
> 
> 
> - "Secure clustering returning bad request response" NIFI-2119[1] [status] A 
> probable source of the problem was discovered by Matt Gilman, NIFI-1753 a 
> security upgrade. Andy LoPresto is looking into a fix.
> 
> - "source release produces an extraneous file" NIFI-2120[2] [status] I looked 
> into how it would be done and I couldn't find any options in the maven 
> release plugin that support excluding specific files from the release. I will 
> be punting this, unless if someone with more experience with the maven 
> release plugin can fix it.
> 
> - "Correct license/notice issues" NIFI-2118[3] [status] I addressed the 
> issues identified by Joe Witt and the fix was merged by Matt Burgess.
> 
> Again I apologize for not finding this issues earlier and will keep you all 
> up to date on any changes. Also since RC1 a commit addressing NIFI-1920[4] 
> has been merged into the 0.x branch. I will be including it in RC2. It is a 
> simple fix that addresses an inadvertent rollback in UnpackContent.
> 
> [1] https://issues.apache.org/jira/browse/NIFI-2119
> [2] https://issues.apache.org/jira/browse/NIFI-2120
> [3] https://issues.apache.org/jira/browse/NIFI-2118
> [4] https://issues.apache.org/jira/browse/NIFI-920
> 
> Thank you,
> Joe
> - - - - - - 
> Joseph Percivall
> linkedin.com/in/Percivall
> e: joeperciv...@yahoo.com
> 

Re: Question about Mock classes in ControllerServiceInitializer

[Oleg Zhurakousky]

Joe

Personally I probably need to look at it more thoroughly to have an intelligent 
answer, but one thing I know for sure (or at least I believe. . .) is that if 
you see any of these Mock* references in the log when you starting NiFi 
instance it’s definitely a bug. 
If that is what you see, I’d suggest to raise JIRA.

Cheers
Oleg

> On Jul 1, 2016, at 7:58 AM, Joe Skora <jsk...@gmail.com> wrote:
> 
> Mark and Oleg, thanks, it makes sense now.
> 
> But, I'm still trying to track down how we have logs containing "... INFO
> [Finalizer] o.a.n.d.mock.MockProcessorLogger Shutting down server".  It
> still doesn't seem like that should occur.
> 
> 
> On Thu, Jun 30, 2016 at 6:30 PM, Oleg Zhurakousky <
> ozhurakou...@hortonworks.com> wrote:
> 
>> Guys
>> 
>> FWIW, there are discussion points on the WIKI that may be relevant to
>> understanding of this issue especially in relation to what Mark just stated
>> 
>> https://cwiki.apache.org/confluence/display/NIFI/Component+documentation+improvements
>> 
>> There are also some points in the Extension Registry (linked from the link
>> above).
>> There is also an open JIRA with some more info:
>> https://issues.apache.org/jira/browse/NIFI-1384
>> 
>> I think in reality if we declare (what is now obvious) a convention that
>> PropertyDescriptors, Relationships etc. all have to be static variables
>> then we would not need to create throw-away instances of class just to get
>> their documentation.
>> Anyway, something to think about. . .
>> 
>> Cheers
>> Oleg
>> 
>> On Jun 30, 2016, at 1:59 PM, Mark Payne <marka...@hotmail.com> marka...@hotmail.com>> wrote:
>> 
>> Joe,
>> 
>> I think the nifi-documentation module is using that to instantiate
>> Processors, Controller Services, etc.
>> so that it can inspect their annotations & call their
>> getPropertyDescriptors() methods, etc. when it generates
>> documentation for the component. Those should not be used for any
>> component that is added to the flow.
>> 
>> 
>> 
>> On Jun 30, 2016, at 1:32 PM, Joe Skora <jsk...@gmail.com> jsk...@gmail.com>> wrote:
>> 
>> Brandon and I have both run into log entries saying something along the
>> lines of "o.a.n.d.mock.MockProcessorLogger Shutting down server".
>> 
>> Checking the code
>> <
>> https://github.com/apache/nifi/blob/release-nifi-0.3.0-rc1/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-documentation/src/main/java/org/apache/nifi/documentation/init/ControllerServiceInitializer.java#L52
>>> ,
>> there are references to the MockProcessorLogger and
>> MockConfigurationContext
>> in the org.apache.nifi.documentation.init.ControllerServiceInitizer,
>> ProcessorInitializer, and ReportingTaskingInitializer classes.
>> 
>> What are we missing?  Why are there Mock framework classes used in regular
>> classes?
>> 
>> 
>> 
>> 

Re: Question about Mock classes in ControllerServiceInitializer

[Oleg Zhurakousky]

Guys

FWIW, there are discussion points on the WIKI that may be relevant to 
understanding of this issue especially in relation to what Mark just stated
https://cwiki.apache.org/confluence/display/NIFI/Component+documentation+improvements

There are also some points in the Extension Registry (linked from the link 
above).
There is also an open JIRA with some more info: 
https://issues.apache.org/jira/browse/NIFI-1384

I think in reality if we declare (what is now obvious) a convention that 
PropertyDescriptors, Relationships etc. all have to be static variables then we 
would not need to create throw-away instances of class just to get their 
documentation.
Anyway, something to think about. . .

Cheers
Oleg

On Jun 30, 2016, at 1:59 PM, Mark Payne 
> wrote:

Joe,

I think the nifi-documentation module is using that to instantiate Processors, 
Controller Services, etc.
so that it can inspect their annotations & call their getPropertyDescriptors() 
methods, etc. when it generates
documentation for the component. Those should not be used for any component 
that is added to the flow.



On Jun 30, 2016, at 1:32 PM, Joe Skora 
> wrote:

Brandon and I have both run into log entries saying something along the
lines of "o.a.n.d.mock.MockProcessorLogger Shutting down server".

Checking the code
,
there are references to the MockProcessorLogger and MockConfigurationContext
in the org.apache.nifi.documentation.init.ControllerServiceInitizer,
ProcessorInitializer, and ReportingTaskingInitializer classes.

What are we missing?  Why are there Mock framework classes used in regular
classes?

Re: ControllerService Enabling and Processor onPropertyModified

[Oleg Zhurakousky]

Indeed this sounds like https://issues.apache.org/jira/browse/NIFI-2032. The PR 
for it has been out https://github.com/apache/nifi/pull/541 so if you get a 
chance to try it please let us know. Hopefully it will be merged soon.

Cheers
Oleg

On Jun 20, 2016, at 4:25 PM, Michael Moser 
> wrote:

Michael,

You may be encountering the bug NIFI-2032 [1] which exists in NiFi 0.6.1.

[1] - https://issues.apache.org/jira/browse/NIFI-2032

-- Mike



On Mon, Jun 20, 2016 at 12:20 PM, Michael D. Coon 

wrote:

All,
  Before I get too deep in submitting Jira tickets, etc. I'm wondering if
this is expected behavior. I'm using NiFi 0.6.1.

  I have a ControllerService that I reference as a service property on my
Processor. The Processor, in turn, uses the ControllerService's internal
configuration state to determine Processor output relationships. But, it
appears that at NiFi startup, I'm given the ControllerService before it is
actually enabled. When I try to invoke the  methods to get the
ControllerService's state, it fails (because it's not enabled).
  Two problems I found:
1) Logging the invocation exception for calling a disabled
ControllerService is being suppressed in this case, which caused this to
take me a full day to track down2) Why would I ever be given a service to
use without it being fully enabled?
I thought I would just block in my Processor's onPropertyModified method
until the ControllerService was enabled; but it looks like the thread
that's actually enabling the service is calling my Processor's
onPropertyModified method and the ControllerService is never enabled until
my Processor's onPropertyModified method is done.
Is this expected behavior? If so, can someone please explain the
assumptions around sending non-enabled ControllerServices to my Processor?

Mike

Re: [DISCUSS] Closing in on the Apache NiFi 0.7.0 Release

[Oleg Zhurakousky]

2009 is all done (including 0.x)

Sent from my iPhone

> On Jun 17, 2016, at 14:16, Joe Percivall <joeperciv...@yahoo.com.INVALID> 
> wrote:
> 
> Team,
> 
> We are continuing to have good progress towards the 0.7.0 release. Tagged as 
> "0.7.0" in Jira and still pending, we currently have 9 tickets that are all 
> patch available. 
> 
> As a note, there seem to be a number of unversioned "patch available" tickets 
> in Jira. I am going to take a look at them to see if any can easily make it 
> into 0.7.0 (or are critical) but unless if they are critical I am not going 
> to hold up the release for them. 
> 
> The current status for those tagged as 0.7.0:
> 
> - "Processors could be started before the Controller Services that they 
> depend on" NIFI-2032[1] [status] Critical bug that was added two days ago. 
> Oleg Zhurakousky posted a PR and needs review.
> 
> 
> - "Allow user to specify file filter regex when unpacking zip/tar archives" 
> NIFI-1568[2] [status] Matt Burgess finished reviewing and is waiting for 
> Ricky Saltzer to address the final comment.
> 
> - "Create PutSlack processor" NIFI-1578 [3] [status] Matt Burgess found one 
> final issue before it is committed. Waiting for Adam Lamar to address.
> 
> -  "Allow empty Content-Type in InvokeHTTP processor" NIFI-1620[4] [status] 
> Pierre Villard addressed Adam Taft's comments is waiting for a final review.
> 
> - "Support Custom Properties in Expression Language" NIFI-1974[5] [status] No 
> Change, waiting for Mark Payne to give a final review.
> 
> - "StandardProcessNode and AbstractConfiguredComponent duplicate instance 
> variable "annotationData"" NIFI-2009[6] [status] Was merged into master but 
> needs to fix merge conflicts to be merged into 0.x
> - "Create a processor to extract WAV file characteristics" NIFI-615[7] 
> [status] Joe Skora posted a new branch addressing the comments but needs to 
> rebase and update PR. Branch needs final review from Joe Percivall (me)
> 
> - "Add SNMP processors" NIFI-1537[8] [status] PR is rebased and Oleg 
> Zhurakousky will be reviewing
> 
> - "Create FlowDebugger processor" NIFI-1829[9] [status] Joe Skora added 
> another commit and rebased. Tony won't be able to finalize review until the 
> weekend, Michael Moser volunteered to finish
> 
> 
> Also NiFi-1850 is marked as 0.7.0 and "Patch Available" in Jira but it was 
> already was merged into 0.x and work is being done to get it into master. I 
> will not reflect it here.
> 
> [1] https://issues.apache.org/jira/browse/NIFI-2032
> [2] https://issues.apache.org/jira/browse/NIFI-1568
> [3] https://issues.apache.org/jira/browse/NIFI-1578
> [4] https://issues.apache.org/jira/browse/NIFI-1620
> [5] https://issues.apache.org/jira/browse/NIFI-1974
> [6] https://issues.apache.org/jira/browse/NIFI-2009
> [7] https://issues.apache.org/jira/browse/NIFI-615
> [8] https://issues.apache.org/jira/browse/NIFI-1537
> [9] https://issues.apache.org/jira/browse/NIFI-1829
> 
> Joe
> - - - - - - 
> Joseph Percivalllinkedin.com/in/Percivall
> e: joeperciv...@yahoo.com
> 
> 
> 
> 
> On Wednesday, June 15, 2016 10:56 AM, Joe Percivall 
> <joeperciv...@yahoo.com.INVALID> wrote:
> Team,
> 
> There was a lot of great progress yesterday, we closed or pushed 6 tickets. 
> Also two tickets were added that are either critical or finishing shortly. 
> The status of the remaining 12 tickets are below:
> 
> - "Corrupted flow file leads to a wedged flow" NIFI-2015[1] Added yesterday. 
> Christopher McDermott, Mark Payne and Oleg Zhurakousky have been discussing 
> the bug but no resolution yet.
> 
> - "Allow user to specify file filter regex when unpacking zip/tar archives" 
> NIFI-1568[2] [status] Matt Burgess is continuing to review
> 
> - "Create PutSlack processor" NIFI-1578 [3] [status] Actively being worked on 
> by contributor and reviewers
> 
> -  "Allow empty Content-Type in InvokeHTTP processor" NIFI-1620[4] [status] 
> No change since yesterday, still waiting for final review by Adam Taft. 
> Commented asking if Adam would like me to finish the review
> 
> - "If unable to write to Content Repository, Process Session should 
> automatically roll itself back" NIFI-1644[5] [status] No progress, commented 
> asking Mark Payne (reporter) to see if it can slide.
> 
> - "Misconfigured MonitorMemory ReportingTask can not be stopped" NIFI-1690[6] 
> [status] Actively being worked on by contributor and reviewer
> 
> - "Support Custom Properties in Expression Language&quo

Re: GetJMSQueue does not detect dead connections

[Oleg Zhurakousky]

Yes, that is documentation bug.

Chris would you mind raising a JIRA or I can do it.

Cheers
Oleg

> On Jun 16, 2016, at 3:15 PM, Joe Witt <joe.w...@gmail.com> wrote:
> 
> Oleg - so to Chris' other comment about docs suggesting the ActiveMQ
> lib path is not needed - is that a doc bug?
> 
> On Thu, Jun 16, 2016 at 3:13 PM, Oleg Zhurakousky
> <ozhurakou...@hortonworks.com> wrote:
>> Chris
>> 
>> That is correct.
>> The idea was to make sure that we can support multiple clients and multiple 
>> vendors since Get/Put only supported AMQP and only one version. The new JMS 
>> support allows you to use any JMS vendor and the only extra work we are 
>> asking you to do is to provide ConnectionFactory JAR(s).
>> 
>> Does that clarify?
>> 
>> Also, yeh tests I was referring to are 
>> https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/test/java/org/apache/nifi/jms/processors
>> 
>> Let me know if you need more help
>> Cheers
>> Oleg
>> On Jun 16, 2016, at 3:08 PM, McDermott, Chris Kevin (MSDU - 
>> STaTS/StorefrontRemote) 
>> <chris.mcderm...@hpe.com<mailto:chris.mcderm...@hpe.com>> wrote:
>> 
>> So does that mean that I cannot use the AMQ client packaged with NiFi but 
>> rather provide my own?
>> 
>> Sorry if I an being obtuse.
>> 
>> Chris McDermott
>> 
>> Remote Business Analytics
>> STaTS/StoreFront Remote
>> HPE Storage
>> Hewlett Packard Enterprise
>> Mobile: +1 978-697-5315
>> 
>> https://www.storefrontremote.com
>> 
>> On 6/16/16, 2:53 PM, "Oleg Zhurakousky" <ozhurakou...@hortonworks.com> wrote:
>> 
>> Yes, you can probably look at the test case for it since it uses embedded 
>> AMQP.
>> 
>> Let m know if you need more help with it.
>> 
>> Cheers
>> Oleg
>> On Jun 16, 2016, at 2:50 PM, McDermott, Chris Kevin (MSDU - 
>> STaTS/StorefrontRemote) <chris.mcderm...@hpe.com> wrote:
>> 
>> Thanks, Oleg.
>> 
>> Do you have an example of how to configure the JMSConnectionFactoryProvider 
>> to work with AMQ?
>> 
>> The documentation says that the MQ Client Libraries path is optional with 
>> org.apache.activemq.ActiveMQConnectionFactory but I am find that is not the 
>> case.
>> 
>> Thanks,
>> 
>> Chris McDermott
>> 
>> Remote Business Analytics
>> STaTS/StoreFront Remote
>> HPE Storage
>> Hewlett Packard Enterprise
>> Mobile: +1 978-697-5315
>> 
>> https://www.storefrontremote.com
>> 
>> On 6/16/16, 1:43 PM, "Oleg Zhurakousky" <ozhurakou...@hortonworks.com> wrote:
>> 
>> Chris
>> 
>> Given that we are deprecating Get/PutJMS* in favor of Publish/SubscribeJMS, 
>> I’d suggest start using those once.
>> 
>> Cheers
>> Oleg
>> 
>> 
>> On Jun 16, 2016, at 1:34 PM, McDermott, Chris Kevin (MSDU - 
>> STaTS/StorefrontRemote) <chris.mcderm...@hpe.com> wrote:
>> 
>> Folks,
>> 
>> I’ve been trying to test my GetJMSQueue configuration so that it detects a 
>> dead broker connection and fails over to an alternate broker.  When I say 
>> dead connection I mean TCP connection that has not been closed but is no 
>> longer passing traffic.  In the real world this typically happens when 
>> broker server crashes and so it does not reset the open connections.  For my 
>> test case I am using iptables to block traffic.
>> 
>> This is the connection URI I am using
>> 
>> failover:(tcp://host2:61616,tcp://host1:61616)?randomize=false=3000=3=3=1=0
>> 
>> They key parameters here are soTimeout=3 and soWriteTimeout=3
>> 
>> These set a 30 second timeout on socket reads and writes.  I’m not sure if 
>> these are necessary since I believe the JMSConsumer classes specifies its 
>> own timeout according to the processor configuration.  The important thing 
>> to note is that when one of these timeouts occurs the AMQ client does not 
>> close the connection.
>> 
>> I believe the deficiency here is that JMSConsumer does not consider the 
>> possibility that the connection is dead.   The problem with this is that an 
>> attempt to reconnect and failover to an alternate broker is not made.
>> 
>> I think the fix would involve counting the number of sequential empty 
>> responses on the connection and then closing the connection once that number 
>> crosses some threshold.  Then subsequent onTrigger() would cause a new 
>> connection attempt.
>> 
>> Thoughts?
>> 
>> Chris McDermott
>> 
>> Remote Business Analytics
>> STaTS/StoreFront Remote
>> HPE Storage
>> Hewlett Packard Enterprise
>> Mobile: +1 978-697-5315
>> 
>> https://www.storefrontremote.com
>> 
>> 
>> 
>> 
>> 
> 

Re: GetJMSQueue does not detect dead connections

[Oleg Zhurakousky]

Chris

That is correct.
The idea was to make sure that we can support multiple clients and multiple 
vendors since Get/Put only supported AMQP and only one version. The new JMS 
support allows you to use any JMS vendor and the only extra work we are asking 
you to do is to provide ConnectionFactory JAR(s).

Does that clarify?

Also, yeh tests I was referring to are 
https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-jms-bundle/nifi-jms-processors/src/test/java/org/apache/nifi/jms/processors

Let me know if you need more help
Cheers
Oleg
On Jun 16, 2016, at 3:08 PM, McDermott, Chris Kevin (MSDU - 
STaTS/StorefrontRemote) 
<chris.mcderm...@hpe.com<mailto:chris.mcderm...@hpe.com>> wrote:

So does that mean that I cannot use the AMQ client packaged with NiFi but 
rather provide my own?

Sorry if I an being obtuse.

Chris McDermott

Remote Business Analytics
STaTS/StoreFront Remote
HPE Storage
Hewlett Packard Enterprise
Mobile: +1 978-697-5315

https://www.storefrontremote.com

On 6/16/16, 2:53 PM, "Oleg Zhurakousky" <ozhurakou...@hortonworks.com> wrote:

Yes, you can probably look at the test case for it since it uses embedded AMQP.

Let m know if you need more help with it.

Cheers
Oleg
On Jun 16, 2016, at 2:50 PM, McDermott, Chris Kevin (MSDU - 
STaTS/StorefrontRemote) <chris.mcderm...@hpe.com> wrote:

Thanks, Oleg.

Do you have an example of how to configure the JMSConnectionFactoryProvider to 
work with AMQ?

The documentation says that the MQ Client Libraries path is optional with 
org.apache.activemq.ActiveMQConnectionFactory but I am find that is not the 
case.

Thanks,

Chris McDermott

Remote Business Analytics
STaTS/StoreFront Remote
HPE Storage
Hewlett Packard Enterprise
Mobile: +1 978-697-5315

https://www.storefrontremote.com

On 6/16/16, 1:43 PM, "Oleg Zhurakousky" <ozhurakou...@hortonworks.com> wrote:

Chris

Given that we are deprecating Get/PutJMS* in favor of Publish/SubscribeJMS, I’d 
suggest start using those once.

Cheers
Oleg


On Jun 16, 2016, at 1:34 PM, McDermott, Chris Kevin (MSDU - 
STaTS/StorefrontRemote) <chris.mcderm...@hpe.com> wrote:

Folks,

I’ve been trying to test my GetJMSQueue configuration so that it detects a dead 
broker connection and fails over to an alternate broker.  When I say dead 
connection I mean TCP connection that has not been closed but is no longer 
passing traffic.  In the real world this typically happens when broker server 
crashes and so it does not reset the open connections.  For my test case I am 
using iptables to block traffic.

This is the connection URI I am using

failover:(tcp://host2:61616,tcp://host1:61616)?randomize=false=3000=3=3=1=0

They key parameters here are soTimeout=3 and soWriteTimeout=3

These set a 30 second timeout on socket reads and writes.  I’m not sure if 
these are necessary since I believe the JMSConsumer classes specifies its own 
timeout according to the processor configuration.  The important thing to note 
is that when one of these timeouts occurs the AMQ client does not close the 
connection.

I believe the deficiency here is that JMSConsumer does not consider the 
possibility that the connection is dead.   The problem with this is that an 
attempt to reconnect and failover to an alternate broker is not made.

I think the fix would involve counting the number of sequential empty responses 
on the connection and then closing the connection once that number crosses some 
threshold.  Then subsequent onTrigger() would cause a new connection attempt.

Thoughts?

Chris McDermott

Remote Business Analytics
STaTS/StoreFront Remote
HPE Storage
Hewlett Packard Enterprise
Mobile: +1 978-697-5315

https://www.storefrontremote.com

Re: GetJMSQueue does not detect dead connections

[Oleg Zhurakousky]

Yes, you can probably look at the test case for it since it uses embedded AMQP.

Let m know if you need more help with it.

Cheers
Oleg
> On Jun 16, 2016, at 2:50 PM, McDermott, Chris Kevin (MSDU - 
> STaTS/StorefrontRemote) <chris.mcderm...@hpe.com> wrote:
> 
> Thanks, Oleg.  
> 
> Do you have an example of how to configure the JMSConnectionFactoryProvider 
> to work with AMQ?
> 
> The documentation says that the MQ Client Libraries path is optional with 
> org.apache.activemq.ActiveMQConnectionFactory but I am find that is not the 
> case.
> 
> Thanks,
> 
> Chris McDermott
> 
> Remote Business Analytics
> STaTS/StoreFront Remote
> HPE Storage
> Hewlett Packard Enterprise
> Mobile: +1 978-697-5315
> 
> https://www.storefrontremote.com
> 
> On 6/16/16, 1:43 PM, "Oleg Zhurakousky" <ozhurakou...@hortonworks.com> wrote:
> 
>> Chris 
>> 
>> Given that we are deprecating Get/PutJMS* in favor of Publish/SubscribeJMS, 
>> I’d suggest start using those once.
>> 
>> Cheers
>> Oleg
>> 
>> 
>>> On Jun 16, 2016, at 1:34 PM, McDermott, Chris Kevin (MSDU - 
>>> STaTS/StorefrontRemote) <chris.mcderm...@hpe.com> wrote:
>>> 
>>> Folks,
>>> 
>>> I’ve been trying to test my GetJMSQueue configuration so that it detects a 
>>> dead broker connection and fails over to an alternate broker.  When I say 
>>> dead connection I mean TCP connection that has not been closed but is no 
>>> longer passing traffic.  In the real world this typically happens when 
>>> broker server crashes and so it does not reset the open connections.  For 
>>> my test case I am using iptables to block traffic.
>>> 
>>> This is the connection URI I am using
>>> 
>>> failover:(tcp://host2:61616,tcp://host1:61616)?randomize=false=3000=3=3=1=0
>>> 
>>> They key parameters here are soTimeout=3 and soWriteTimeout=3
>>> 
>>> These set a 30 second timeout on socket reads and writes.  I’m not sure if 
>>> these are necessary since I believe the JMSConsumer classes specifies its 
>>> own timeout according to the processor configuration.  The important thing 
>>> to note is that when one of these timeouts occurs the AMQ client does not 
>>> close the connection.
>>> 
>>> I believe the deficiency here is that JMSConsumer does not consider the 
>>> possibility that the connection is dead.   The problem with this is that an 
>>> attempt to reconnect and failover to an alternate broker is not made.
>>> 
>>> I think the fix would involve counting the number of sequential empty 
>>> responses on the connection and then closing the connection once that 
>>> number crosses some threshold.  Then subsequent onTrigger() would cause a 
>>> new connection attempt.
>>> 
>>> Thoughts?
>>> 
>>> Chris McDermott
>>> 
>>> Remote Business Analytics
>>> STaTS/StoreFront Remote
>>> HPE Storage
>>> Hewlett Packard Enterprise
>>> Mobile: +1 978-697-5315
>>> 
>>> https://www.storefrontremote.com
>> 
> 

Re: GetJMSQueue does not detect dead connections

[Oleg Zhurakousky]

Chris 

Given that we are deprecating Get/PutJMS* in favor of Publish/SubscribeJMS, I’d 
suggest start using those once.

Cheers
Oleg


> On Jun 16, 2016, at 1:34 PM, McDermott, Chris Kevin (MSDU - 
> STaTS/StorefrontRemote)  wrote:
> 
> Folks,
> 
> I’ve been trying to test my GetJMSQueue configuration so that it detects a 
> dead broker connection and fails over to an alternate broker.  When I say 
> dead connection I mean TCP connection that has not been closed but is no 
> longer passing traffic.  In the real world this typically happens when broker 
> server crashes and so it does not reset the open connections.  For my test 
> case I am using iptables to block traffic.
> 
> This is the connection URI I am using
> 
> failover:(tcp://host2:61616,tcp://host1:61616)?randomize=false=3000=3=3=1=0
> 
> They key parameters here are soTimeout=3 and soWriteTimeout=3
> 
> These set a 30 second timeout on socket reads and writes.  I’m not sure if 
> these are necessary since I believe the JMSConsumer classes specifies its own 
> timeout according to the processor configuration.  The important thing to 
> note is that when one of these timeouts occurs the AMQ client does not close 
> the connection.
> 
> I believe the deficiency here is that JMSConsumer does not consider the 
> possibility that the connection is dead.   The problem with this is that an 
> attempt to reconnect and failover to an alternate broker is not made.
> 
> I think the fix would involve counting the number of sequential empty 
> responses on the connection and then closing the connection once that number 
> crosses some threshold.  Then subsequent onTrigger() would cause a new 
> connection attempt.
> 
> Thoughts?
> 
> Chris McDermott
> 
> Remote Business Analytics
> STaTS/StoreFront Remote
> HPE Storage
> Hewlett Packard Enterprise
> Mobile: +1 978-697-5315
> 
> https://www.storefrontremote.com

Re: unwedgeable flow caused by "Cannot create Provenance Event Record because FlowFile UUID is not set"

[Oleg Zhurakousky]

Thank you Chris

Will be addressed shortly
Oleg

> On Jun 14, 2016, at 9:56 AM, McDermott, Chris Kevin (MSDU - 
> STaTS/StorefrontRemote) <chris.mcderm...@hpe.com> wrote:
> 
> Thanks, Oleg. I’ve created NIFI-2015.
> 
> 
> Regards,
> 
> Chris McDermott
> 
> Remote Business Analytics
> STaTS/StoreFront Remote
> HPE Storage
> Hewlett Packard Enterprise
> Mobile: +1 978-697-5315
> 
> https://www.storefrontremote.com
> 
> On 6/13/16, 3:10 PM, "Oleg Zhurakousky" <ozhurakou...@hortonworks.com> wrote:
> 
>> 2016-06-13 17:33:30,275 ERROR [Timer-Driven Process Thread-4] 
>> o.a.n.p.attributes.UpdateAttribute
>> java.lang.IllegalStateException: Cannot create Provenance Event Record 
>> because FlowFile UUID is not set
>>   at 
>> org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.assertSet(StandardProvenanceEventRecord.java:700)
>>  ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
>>   at 
>> org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.build(StandardProvenanceEventRecord.java:721)
>>  ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
>>   at 
>> org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.build(StandardProvenanceEventRecord.java:412)
>>  ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
>>   at 
>> org.apache.nifi.controller.repository.StandardProcessSession.updateProvenanceRepo(StandardProcessSession.java:634)
>>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
>>   at 
>> org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:295)
>>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
>>   at 
>> org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:283)
>>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
>>   at 
>> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28)
>>  ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
>>   at 
>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1059)
>>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
>>   at 
>> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
>>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
>>   at 
>> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
>>   at 
>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123)
>>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
>>   at 
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
>> [na:1.8.0_45]
>>   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) 
>> [na:1.8.0_45]
>>   at 
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>>  [na:1.8.0_45]
>>   at 
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>>  [na:1.8.0_45]
>>   at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>  [na:1.8.0_45]
>>   at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>  [na:1.8.0_45]
>>   at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> 

Re: Apache NiFi 0.7.0 Release date ?

[Oleg Zhurakousky]

Unfortunately I just raised another relatively trivial JIRA which affects 0.7, 
but I should have it ready by tomorrow.
Otherwise I am +1 to get 0.7 out asap 

Sent from my iPhone

> On Jun 13, 2016, at 16:47, Joe Percivall  
> wrote:
> 
> Looks like time to start finishing up the remaining 17 tickets and getting 
> together a release. I will volunteer to be Release Manager. I'll start 
> another email thread outlining the remaining tickets and their statuses.
> 
> Joe
> - - - - - - Joseph Percivall
> linkedin.com/in/Percivall
> e: joeperciv...@yahoo.com
> 
> 
> 
> 
> On Tuesday, June 7, 2016 11:26 AM, Matt Burgess  wrote:
> There was a whitespace issue in the test HL7 file. Joey moved the
> content into the test class itself and removed the file. I reviewed
> and merged to 0.x and master, so things should be back to normal.
> Sorry for the hiccup.
> 
> Regards,
> Matt
> 
> 
>> On Tue, Jun 7, 2016 at 10:03 AM, Matt Burgess  wrote:
>> I merged an HL7 PR last night, I will take a look. The tests passed
>> for me but maybe I screwed something up with the patch application.
>> 
>>> On Tue, Jun 7, 2016 at 10:00 AM, Andre  wrote:
>>> + 1 on the build errors on HL7.
>>> 
>>> Seems to break things in here as well.
>>> 
>>> Cheers
>>> 
 On Tue, Jun 7, 2016 at 11:58 PM, Ryan H  
 wrote:
 
 I did build the 0.x branch for the latest 0.7.0... It works when you skip
 building the tests.
> 

Re: unwedgeable flow caused by "Cannot create Provenance Event Record because FlowFile UUID is not set"

[Oleg Zhurakousky]

Chris

The fact that your flow is hosed means this is a bug. IMHO corrupted FlowFile 
should not render NiFi unavailable. So please raise the JIRA.

Also, i think the real question here is not how the FF got corrupted but the 
fact that corrupted FF rendered NIFI unavailable and that is the core issue. We 
should probably discuss what should we do with corrupted FF (drop/move/delete 
etc), but I don’t think it should stop the world.

In any event, raise the JIRA and we can discuss it there.

Cheers
Oleg

> On Jun 13, 2016, at 3:01 PM, McDermott, Chris Kevin (MSDU - 
> STaTS/StorefrontRemote)  wrote:
> 
> 2016-06-13 17:33:30,275 ERROR [Timer-Driven Process Thread-4] 
> o.a.n.p.attributes.UpdateAttribute
> java.lang.IllegalStateException: Cannot create Provenance Event Record 
> because FlowFile UUID is not set
>at 
> org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.assertSet(StandardProvenanceEventRecord.java:700)
>  ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
>at 
> org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.build(StandardProvenanceEventRecord.java:721)
>  ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
>at 
> org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.build(StandardProvenanceEventRecord.java:412)
>  ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
>at 
> org.apache.nifi.controller.repository.StandardProcessSession.updateProvenanceRepo(StandardProcessSession.java:634)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
>at 
> org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:295)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
>at 
> org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:283)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
>at 
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28)
>  ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
>at 
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1059)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
>at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
>at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
>at 
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123)
>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
>at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_45]
>at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) 
> [na:1.8.0_45]
>at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>  [na:1.8.0_45]
>at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>  [na:1.8.0_45]
>at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_45]
>at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_45]
>at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> 
> I suspect the error is due to some file corruption that I may have caused 
> through some fault testing (hard reboot.)
> 
> My problem at this point is that it completely wedges the flow. I cannot not 
> examine or empty incoming connection and the processor is not discarding the 
> bad file.  Any suggestions on how to get past this?
> 
> On the file corruption, are there any known issues with NiFi where file 
> corruption could occur when SIGKILLing NiFi? Not that, that is what I did, I 
> was doing a hard reboot, which could make the file system configuration 
> suspect, but if the problem is further up the stack I’d like to be aware of 
> it.
> 
> 
> Thanks,
> 
> Chris McDermott
> 
> Remote Business Analytics
> STaTS/StoreFront Remote
> HPE Storage
> Hewlett Packard Enterprise
> Mobile: +1 978-697-5315
> 
> https://www.storefrontremote.com

Re: [DISCUSS] - Markdown option for documentation artifacts

[Oleg Zhurakousky]

Personally I am ok either way, however the question I have is consistency and 
how different artifacts written via html will be different (look-and-feel) from 
the once created using mark up and then transformed. 

Oleg

> On Jun 7, 2016, at 2:28 PM, Bryan Rosander  wrote:
> 
> Hey all,
> 
> When writing documentation (e.g. the additionalDetails.html for a
> processor) it would be nice to have the option to use Markdown instead of
> html.
> 
> I think Markdown is easier to read and write than raw HTML and for simple
> cases does the job pretty well.  It also has the advantage of being able to
> be translated into other document types easily and it would be rendered by
> default in Github when the file is clicked.
> 
> There is an MIT-licensed Markdown maven plugin (
> https://github.com/walokra/markdown-page-generator-plugin) that seems like
> it might work for translating additionalDetails.md (and others) into an
> equivalent html page.
> 
> Thanks,
> Bryan Rosander

Re: [DISCUSS] Apache NiFi 0.7.0 and 1.0.0

[Oleg Zhurakousky]

in the intent.  I just want us to be clear
>>>>>>> what it is.  It is extra work to ensure each PR is applied to both
>>>>>>> lines and extra work increases contributor and reviewer burden so we
>>>>>>> should be mindful of that as it is a dragging force.  We also need to
>>>>>>> keep in mind that with 1.x we have Java 8 as a minimum and so there
>>>>>>> are cases which will not apply to both and we don't want folks to
>>>>>>> avoid using Java 8 features just so it can apply to both.
>>>>>>> 
>>>>>>> My preference is that we have 0.7 as the last planned feature release
>>>>>>> in 0.x and with that in mind we need to choose to have it be a bit
>>>>>>> before, a bit after, or at the same time as the 1.x release.  I
>>>>>>> personally am comfortable with what I proposed for 0.7 vs 1.0 timing
>>>>>>> but I am fine if the consensus is to release the last 0.x and 1.0 at
>>>>>>> the same time.  Just hoping to avoid needing to have another feature
>>>>>>> release on 0.x after 0.7 other than some special request that might
>>>>>>> come up later (which is also discussed in the support doc).
>>>>>>> 
>>>>>>> I also agree the release process for 1.0 will be significant as it
>>>>>>> will include important new features.  Definitely need folks testing
>>>>>>> out and providing feedback on the features early and often.
>>>>>>> 
>>>>>>> Thanks
>>>>>>> Joe
>>>>>>> 
>>>>>>>> On Tue, May 17, 2016 at 6:20 PM, Michael Moser <moser...@gmail.com>
>>>>>>> wrote:
>>>>>>> 
>>>>>>> The way I read the release support document, I don't think the
>>> feature
>>>>>>> cut-off for the 0.x branch happens when we confirm a release date for
>>>>>> 1.0,
>>>>>>> I think it occurs once we actually release 1.0.  Maybe the cut-off
>>> can
>>>>>>> happen once we declare the first 1.0 release candidate.  I'm sure we
>>>>>> will
>>>>>>> spend significant time doing testing and bug fixes on 1.0 release
>>>>>>> candidates.  If I recall, we spent 2 weeks on 0.6.1 release
>>> candidates.
>>>>>>> 
>>>>>>> -- Mike
>>>>>>> 
>>>>>>> 
>>>>>>> On Tue, May 17, 2016 at 6:04 PM, Joe Witt <joe.w...@gmail.com>
>>> wrote:
>>>>>>> 
>>>>>>> I believe that is right Andy.  The support guide articulates that we
>>>>>>> could do a feature release upon request if there was some specific
>>>>>>> need a community member had but that otherwise the only releases on
>>> an
>>>>>>> older line still supported would be focused on security/data loss
>>> type
>>>>>>> items.
>>>>>>> 
>>>>>>> Thanks
>>>>>>> Joe
>>>>>>> 
>>>>>>> On Tue, May 17, 2016 at 4:58 PM, Andy LoPresto <alopre...@apache.org
>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>> This schedule seems appropriate to me. Once 0.7.0 is released and we
>>>>>>> 
>>>>>>> confirm
>>>>>>> 
>>>>>>> the release date for 1.0, feature development is completely targeted
>>> to
>>>>>>> 
>>>>>>> 1.0,
>>>>>>> 
>>>>>>> correct? Security and data loss bug fixes would still be backported,
>>> but
>>>>>>> 
>>>>>>> new
>>>>>>> 
>>>>>>> features would not.
>>>>>>> 
>>>>>>> Andy LoPresto
>>>>>>> alopre...@apache.org
>>>>>>> alopresto.apa...@gmail.com
>>>>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>>>>>> 
>>>>>>> On May 17, 2016, at 1:19 PM, Joe Witt <joe.w...@gmail.com> wrote:
>>>>>>> 
>>>>>>> Ok - i'm good with an 0.7 release too and think it is a good idea.  I
>>>>>>> am happy to RM th

Re: Nifi Java 1.7 support

[Oleg Zhurakousky]

Yes, that would be with NIFI 1.0.0 

Cheers
Oleg

> On May 31, 2016, at 4:51 PM, M Singh  wrote:
> 
> Hi Folks:
> 
> Just wanted to find out if there is a timeline when Nifi will sunset support 
> for JDK 1.7.
> Thanks
> Mans

Re: [ANNOUNCE] New Apache NiFi Committer Pierre Villard

[Oleg Zhurakousky]

Wow! Finally, very excited!!! Long overdue! Congrats and welcome Pierre!

Oleg 

> On May 25, 2016, at 11:50, Joe Witt  wrote:
> 
> On behalf of the Apache NiFi PMC, I am pleased to announce that Pierre
> Villard has accepted the PMC's invitation to become a committer on the
> Apache NiFi project.  Several months ago Pierre began contributing to
> NiFi in a variety of important ways and quickly expanded those
> contributions to include code, mailing list, release votes, and more.
> We thank him for his efforts thus far and look forward to his
> continued involvement in the project.
> 
> Welcome and congratulations!
> 

Re: NIFI & IBM MQ

[Oleg Zhurakousky]

Ok, seems like this list doesn’t like images (at least for me), so here is text 
version of what was in them

Processor (required properties):
- Destination Name:  (e.g., queue://MYQUEUE)
- Destination Type: QUEUE or TOPIC
- Session Cache size: 1 (read its doc for more details)
- Connection Factory service:  
(e.g., IBMMQ)

Controller Service (e.g., IBMMQ)
- MQ Connection Factory Implementation: com.ibm.mq.jms.MQConnectionFactory
- MQ Client Libraries path: 
- Broker URI:  (e.g., foo.bar:1234)
Then you would have to configure IBM specific properties as Dynamic Properties:
- channel:  (e.g., SYSTEM.ADMIN.SVRCONN)
- queueManeger:  (e.g., FOO)
- transportType:  (make sure you put '1' as a 
value which states that TCP/IP will be used)

You can get more from IBM docs for additional properties which could all be 
provided as dynamic properties

Cheers
Oleg

On May 25, 2016, at 9:41 AM, Oleg Zhurakousky 
<ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote:

Christian

I’d suggest to look at the additional documentation of this new component as it 
provides some level of details, but it would be interesting to get your 
feedback as to what do you think is missing/confusing.

In any event, here are some details. You need to configure ControllerService 
for JMS Connection Factory and Consume and/or PublishJMS processor
Attached are the images showing sample configuration.
Obviously the Destination names, URIs etc would have to be changed to fit your 
environment.

Let me know how it goes.

Cheers
Oleg





On May 25, 2016, at 4:09 AM, christianv 
<christian.vandenhee...@standardbank.co.za<mailto:christian.vandenhee...@standardbank.co.za>>
 wrote:

Tried it which class do I use from IBM ie com.ibm for the setting
JMSConnectionFactoryProvider



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/NIFI-IBM-MQ-tp10651p10679.html
Sent from the Apache NiFi Developer List mailing list archive at 
Nabble.com<http://nabble.com/>.

Re: NIFI & IBM MQ

[Oleg Zhurakousky]

Christian

I’d suggest to look at the additional documentation of this new component as it 
provides some level of details, but it would be interesting to get your 
feedback as to what do you think is missing/confusing.

In any event, here are some details. You need to configure ControllerService 
for JMS Connection Factory and Consume and/or PublishJMS processor
Attached are the images showing sample configuration.
Obviously the Destination names, URIs etc would have to be changed to fit your 
environment.

Let me know how it goes.

Cheers
Oleg


[cid:D06FA6EF-6C92-4B1F-89F9-98437CDDC029@domain.invalid][cid:2FB4D9E7-9763-4155-861E-ED6226FAAD1A@domain.invalid]


On May 25, 2016, at 4:09 AM, christianv 
>
 wrote:

Tried it which class do I use from IBM ie com.ibm for the setting
JMSConnectionFactoryProvider



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/NIFI-IBM-MQ-tp10651p10679.html
Sent from the Apache NiFi Developer List mailing list archive at 
Nabble.com.

Re: NIFI & IBM MQ

[Oleg Zhurakousky]

Christian

Get/PutJMS* processors are effectively deprecated since they only work with 
ActiveMQ.
The new processors that you should use that were specifically developed to 
support multiple JMS providers (tested with IBM MQ, Tibco and ActiveMQ) are 
PublishJMS and ConsumeJMS.

Give it a try and let us know if you need more help.
Cheers
Oleg

> On May 24, 2016, at 5:47 AM, christianv 
>  wrote:
> 
> Hi,
> 
> I am in need of help I am trying to use GetJMSQueue to connect to an IBM MQ
> Queue and I need an example of the property setting I am unable to compile
> the UTI.
> 
> Kind Regards
> 
> 
> 
> --
> View this message in context: 
> http://apache-nifi-developer-list.39713.n7.nabble.com/NIFI-IBM-MQ-tp10651.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
> 

Re: [DISCUSS] Apache NiFi 0.7.0 and 1.0.0

[Oleg Zhurakousky]

Agreed! I would like to see 0.7 within 2-3 weeks as there are a lot of 
improvements and new features/components in it already, and would like to give 
it some miles before 1.0.

Oleg
> On May 17, 2016, at 4:02 PM, James Wing  wrote:
> 
> I'm definitely in favor of releasing 0.7.0, but I don't think we need be
> rigid about the schedule.  If delaying 0.7.0 a few weeks (2-4?) helps pace
> us towards a 1.0 in mid- to late-Summer, that seems reasonable to me.  Do
> we believe that is still a likely target?
> 
> Thanks,
> 
> James
> 
> On Tue, May 17, 2016 at 7:30 AM, Joe Witt  wrote:
> 
>> Team,
>> 
>> Want to start zeroing in on the details of the next releases.  We had
>> a good set of discussions around this back in January and have since
>> been executing along this general path [1].
>> 
>> On the 0.x line the next release would be 0.7.0.  There does appear to
>> be a lot of useful improvements/features/fixes there now and it is
>> time to do a release according to our general 6-8 week approach.
>> However, given all the effort going into 1.x I'd like to get a sense
>> of what the community preference is.
>> 
>> On the 1.0 line the release is coming into focus.  Some things have
>> moved into 1.x and some things look like they'd slide to the right of
>> 1.x as is to be expected.  For example distributed durability (HA
>> Data) looks like a good thing to do post 1.0 given the substantive
>> changes present from the new HA clustering approach and multi-tenant
>> authorization.  I'd also like to dive in and liberally apply Apache
>> Yetus annotations [2] to all the things so we can be really explicit
>> about what parts we can more freely evolve going forward.  We've been
>> a bit awkwardly hamstrung thus far without these so they should help
>> greatly to better convey intent.
>> 
>> For those really interested in things coming in the 1.0 release please
>> take a look through the JIRAs currently there and provide comments on
>> what is important to you, what you'd like to see moved out, in, etc..
>> [3].  At this point there are still a lot of things which will likely
>> need to move out to allow the release to occur in a timely fashion.
>> 
>> Also, keep in mind our stated release line/support model as found here [4].
>> 
>> [1]
>> http://mail-archives.apache.org/mod_mbox/nifi-dev/201601.mbox/%3CCALJK9a4dMw9PyrrihpPwM7DH3R_4v8b%3Dr--LDhK7y5scob-0og%40mail.gmail.com%3E
>> 
>> [2]
>> https://yetus.apache.org/documentation/0.2.1/audience-annotations-apidocs/
>> 
>> [3]
>> https://issues.apache.org/jira/browse/NIFI-1887?jql=fixVersion%20%3D%201.0.0%20AND%20project%20%3D%20NIFI
>> 
>> [4]
>> https://cwiki.apache.org/confluence/display/NIFI/Git+Branching+and+Release+Line+Management
>> 
>> Thanks
>> Joe
>> 

Re: [DISCUSS] nifi meetup/hackathon ground rules?

[Oleg Zhurakousky]

In any event, I think creating JIRA ticket (regardless of how right/wrong it 
may be) would be appropriate in such settings as well as producing a fix and a 
PR/Patch essentially allowing it to be vetted by ASF process. 
On the other hand I also hear Tony’s point about "short turnaround”. Certain 
issues may be very obvious (e.g., NPEs, spelling, misconfiguration etc) and I 
think we need to show a bit more flexibility while fostering community 
participation as I (based on personal experience in previous ventures) strongly 
believe that a bug fixed jointly in such settings and merged right away draws 
more interest from attendees to the specific technology and goes to the heart 
of the community participation/collaboration (everyone present is part of that 
fix - a true community fix). After all, attendees of such event are the 
community and if there is a consensus among all, that should be enough for an 
implied +1. Don’t you agree?

Chers
Oleg

> On May 16, 2016, at 9:38 PM, Joe Witt  wrote:
> 
> Tony: Good point and probably fair game.  It would need to be really
> urgent and really specific I think though.  Otherwise no need to rush.
> 
> Matt: Yeah that is a great idea.
> 
> AdamL: Thanks for offering to setup a zoom.  I'll try to do it and if
> good will send details on meetup invite.  If not I'll ping you.
> 
> Thanks
> Joe
> 
> On Mon, May 16, 2016 at 9:15 PM, Tony Kurc  wrote:
>> Joe - if a bug is discovered during the hackathon and patch developed,
>> would this be an appropriate short turnaroundJIRA/patch/merge type
>> situation?
>> On May 16, 2016 5:31 PM, "Matt Burgess"  wrote:
>> 
>>> One thing that could be done to enable demos while still having
>>> PRs/patches go through the Apache process is to have the Organizer create a
>>> hackathon branch off their fork, and merge in any patches/PRs that are
>>> demo-able, then show the hackathon goodness at the end. Then the regular
>>> process (Jira, review, etc) applies to the Apache branch(es) for inclusion
>>> into the product.
>>> 
>>> 
 On May 16, 2016, at 5:03 PM, Joe Witt  wrote:
 
 Team,
 
 I wanted to shoot out a note to gather input on some rules of
 engagement so to speak for running a nifi hackathon/meetup.  A few of
 us in the DC/MD area have one planned soon [1].
 
 What I'd like to send out to the meetup group are some ground rules
 for how the meetup will operate.  It is important because not everyone
 will be familiar with the Apache Way, it is being hosted in a vendor
 space, and because in general we want to make sure things like this
 can occur more in the future which means we want this to go well!
 
 Key points to make follow but if you have others please share:
 1) Decisions cannot be made in such a setting.  Rather the discussions
 that happen and the ideas and opinions formed in them need to be
 captured on the appropriate feature proposals, JIRAs, mailing-list
 discussions so others can participate.  This includes feature ideas,
 code ideas, roadmap items, etc..
 
 2) We cannot just make up JIRAs, whip up some code, +1 and merge it
 during the meetup.  If something is worthy of a RTC, which is
 basically all things code, then it needs to be given time for folks
 not sitting at the meetup to participate in - that is it should be
 treated like any other contribution.
 
 3) Notes/summary of the meetup should occur and be made available to
 the community.
 
 [1] http://www.meetup.com/ApacheNiFi/events/230804255/
 
 Thanks
 Joe
>>> 
> 

Re: [ANNOUNCE] New Apache NiFi PMC Member - Joe Percivall

[Oleg Zhurakousky]

Congrats Joe!
> On May 16, 2016, at 11:00 PM, Joe Witt  wrote:
> 
> On behalf of the Apache NiFI PMC, I am very pleased to announce that
> Joe Percivall has accepted the PMC's invitation to join the Apache
> NiFi PMC.  We greatly appreciate all of Joe's hard work and generous
> contributions to the project. We look forward to his continued
> involvement in the project.
> 
> Joe demonstrates 'community over code' as-is central to the Apache Way
> in his many wonderful engagements with members of the community be
> they developers building new features or users wanting to better
> understand NiFi.
> 
> Welcome and congratulations!
> 

Re: [discuss] PropertyDescriptor name and displayName attributes

[Oleg Zhurakousky]

I think the the main source of confusion (as it was for me) is the ‘name’ name 
itself. 
Basically the ‘name’ today is really an identifier of the property and 
therefore could/should never change while 'displayName' is volatile. 
Obviously changing “name’ to “id” is out of the question as it would break 
practically everything. But from the documentation perspective may be we should 
consider documenting that ‘name’ corresponds to a unique and perpetual 
identifier of the property that would also be displayed unless override by 
‘displayName’ while such override would only affect the display characteristics 
of the property and not its identification.

Cheers
Oleg

> On May 6, 2016, at 10:25 AM, Joe Witt  wrote:
> 
> Definitely on board with the idea that the 'name' will be the key to a
> resource bundle.  It does imply such names will need to follow
> necessary conventions to be valid resource bundle keys.
> 
> However, in the spirit of always thinking about the developer path to
> productivity I am hopeful we can come up with a nice way to not
> require them to setup a resource bundle.
> 
> The idea being that the following order of operations/thought would exist:
> 
> 1) How can I provide a new property to this processor?
> Answer: Add a property descriptor and set the name.  This name will be
> used to refer to the property descriptor whenever serialized/saving
> the config and it will be rendered through the REST API and thus made
> available as the property name in the UI.
> 
> 2) Oh snap.  I wish I had used a different name because I've found a
> better way to communicate intent to the user.  How do I do this?
> Answer: Go ahead and set displayName.  NiFi will continue to use the
> 'name' for serialization/config saving but will use the displayName
> for what is shown to the user in the UI.
> 
> 3) I would like to support locale sensitive representations of my
> property name.  How can I do this?
> Answer: Add a resource bundle with entries for your property 'name'
> value.  This means the resource bundle needs to exist and your
> property 'name' must adhere to resource bundle key naming requirements
> [1].  If this is supplied and can be looked up then this will be used
> and otherwise will fallback to using displayName value if present and
> otherwise will fallback to using the value of 'name'.
> 
> And in any event we still need to better document/articulate this
> model as the root of this thread was that we hadn't effectively
> communicated the existence of displayName.  I agree this discussion
> ended up getting us to a great place though as we should all strive to
> support internationalization.
> 
> With an approach like this I am onboard.  I think this balances our
> goals of having a simple to use API but also allows those who want to
> support multiple locales to do so cleanly.
> 
> Thanks
> Joe
> 
> [1] https://docs.oracle.com/javase/tutorial/i18n/resbundle/propfile.html
> 
> On Fri, May 6, 2016 at 9:33 AM, Brandon DeVries  wrote:
>> +1.  I like that better.  Deprecate displayName(), and set it
>> "automatically" based on the locale from properties.  The name of the
>> property (which should never change) is the key into the ResourceBundle.
>> 
>> Brandon
>> 
>> 
>> On Fri, May 6, 2016 at 9:24 AM Matt Burgess  wrote:
>> 
>>> Same here. Internationalization is often implemented as properties
>>> files/resources, where you possibly load in a file based on the system
>>> setting for Locale (like processor_names_en_US.properties). If we were
>>> to do internationalization this way (i.e. a non-code based solution,
>>> which is more flexible), then ironically displayName() might/should be
>>> deprecated in favor of using the value of name() as the key in a
>>> properties/lookup file; the corresponding value would be the
>>> appropriate locale-specific "display name".
>>> 
>>> Brandon's links show this approach, I have seen this i18n approach on
>>> other projects/products and it seems to work pretty well.
>>> 
>>> Regards,
>>> Matt
>>> 
>>> On Fri, May 6, 2016 at 9:11 AM, Joe Witt  wrote:
 I share Bryan's perspective.
 
 On Fri, May 6, 2016 at 9:05 AM, Bryan Bende  wrote:
> I might just be resistant to change, but I am still on the fence a
>>> little
> bit...
> 
> In the past the idea has always been you start out with name, and if you
> later need to change what is displayed in the UI, then you add
>>> displayName
> after the fact.
> 
> It sounds like the issue is that a lot of people don't know about
> displayName, so I am totally in favor of increasing awareness through
> documentation,
> but I'm struggling with telling people that they should set displayName
>>> as
> the default behavior.
> 
> For code that is contributed to NiFi, I would expect the PMC/committer
> doing the review/merging to notice if an existing property 

Re: PutKafka-Message Delimiter

[Oleg Zhurakousky]

Is it “\n” as a "new line" or “/n”? If you want to create a message for each 
line then you have to use “new line” and that is CTRL+Enter. If it is “/n”, 
then it’s just a string pattern that will be looked at as any other string.
I am still not following the "no value set” comment. Could you please clarify?

Oleg

> On May 5, 2016, at 8:51 AM, Sourav Gulati <sourav.gul...@impetus.co.in> wrote:
> 
> Oleg,
> 
> Yes it is not a required property.
> 
> As per the documentation, If we do not provide any message delimiter to 
> PutKafka, it writes all  the content of a flowfile as single message to 
> Kafka. My requirement is that I want to write all messages in the flow file 
> as separate messages on Kafka.
> 
> Messages in the flowfile is delimited by "/n" . That is why I was providing 
> "/n" as message delimiter. However, it is not working.
> 
> Regards,
> Sourav Gulati
> 
> -Original Message-
> From: Oleg Zhurakousky [mailto:ozhurakou...@hortonworks.com]
> Sent: Thursday, May 05, 2016 6:08 PM
> To: dev@nifi.apache.org
> Subject: Re: PutKafka-Message Delimiter
> 
> I am not sure what you’re saying since it is not a required property and it 
> uses non empty validator so anything you enter there is a valid value. In 
> other words it can never produce 'no value set’ message.
> I just tested it myself as well to be sure.
> Could you provide more details as to what exactly you are doing?
> 
> Cheers
> Oleg
>> On May 5, 2016, at 7:31 AM, Sourav Gulati <sourav.gul...@impetus.co.in> 
>> wrote:
>> 
>> Could you please provide me that documentation content?
>> 
>> Regards,
>> Sourav Gulati
>> 
>> -Original Message-
>> From: Sourav Gulati
>> Sent: Thursday, May 05, 2016 5:01 PM
>> To: dev@nifi.apache.org
>> Subject: RE: PutKafka-Message Delimiter
>> 
>> If I press " CTRL+Enter" on the value table of "Message delimiter" key. It 
>> says "no value set".
>> 
>> 
>> 
>> Regards,
>> Sourav Gulati
>> 
>> -Original Message-
>> From: Oleg Zhurakousky [mailto:ozhurakou...@hortonworks.com]
>> Sent: Thursday, May 05, 2016 4:54 PM
>> To: dev@nifi.apache.org
>> Subject: Re: PutKafka-Message Delimiter
>> 
>> Yes, we've also updated documentation as well for the upcoming release
>> 
>> Sent from my iPhone
>> 
>>> On May 5, 2016, at 07:22, Sourav Gulati <sourav.gul...@impetus.co.in> wrote:
>>> 
>>> Should I press CTRL+Enter on value tab of "Message delimiter" key?
>>> 
>>> Regards,
>>> Sourav Gulati
>>> 
>>> -Original Message-
>>> From: Oleg Zhurakousky [mailto:ozhurakou...@hortonworks.com]
>>> Sent: Thursday, May 05, 2016 4:41 PM
>>> To: dev@nifi.apache.org
>>> Subject: Re: PutKafka-Message Delimiter
>>> 
>>> Please use Ctrl+Enter.
>>> The '\n' is interpreted literally as a string (not new line).
>>> Let me know how it goes
>>> Cheers
>>> Oleg
>>> 
>>>> On May 5, 2016, at 7:01 AM, Sourav Gulati <sourav.gul...@impetus.co.in> 
>>>> wrote:
>>>> 
>>>> Hi All,
>>>> 
>>>> I am tried setting message delimiter in put kafka to /n . But it is not 
>>>> working. Please guide.
>>>> 
>>>> Regards,
>>>> Sourav Gulati
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> NOTE: This message may contain information that is confidential, 
>>>> proprietary, privileged or otherwise protected by law. The message is 
>>>> intended solely for the named addressee. If received in error, please 
>>>> destroy and notify the sender. Any use of this email is prohibited when 
>>>> received in error. Impetus does not represent, warrant and/or guarantee, 
>>>> that the integrity of this communication has been maintained nor that the 
>>>> communication is free of errors, virus, interception or interference.
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> NOTE: This message may contain information that is confidential, 
>>> proprietary, privileged or otherwise protected by law. The message is 
>>> intended solely for the named addressee. If received in error, please 
>>> destroy and 

Re: PutKafka-Message Delimiter

[Oleg Zhurakousky]

I am not sure what you’re saying since it is not a required property and it 
uses non empty validator so anything you enter there is a valid value. In other 
words it can never produce 'no value set’ message.
I just tested it myself as well to be sure.
Could you provide more details as to what exactly you are doing?

Cheers
Oleg
> On May 5, 2016, at 7:31 AM, Sourav Gulati <sourav.gul...@impetus.co.in> wrote:
> 
> Could you please provide me that documentation content?
> 
> Regards,
> Sourav Gulati
> 
> -Original Message-
> From: Sourav Gulati
> Sent: Thursday, May 05, 2016 5:01 PM
> To: dev@nifi.apache.org
> Subject: RE: PutKafka-Message Delimiter
> 
> If I press " CTRL+Enter" on the value table of "Message delimiter" key. It 
> says "no value set".
> 
> 
> 
> Regards,
> Sourav Gulati
> 
> -Original Message-
> From: Oleg Zhurakousky [mailto:ozhurakou...@hortonworks.com]
> Sent: Thursday, May 05, 2016 4:54 PM
> To: dev@nifi.apache.org
> Subject: Re: PutKafka-Message Delimiter
> 
> Yes, we've also updated documentation as well for the upcoming release
> 
> Sent from my iPhone
> 
>> On May 5, 2016, at 07:22, Sourav Gulati <sourav.gul...@impetus.co.in> wrote:
>> 
>> Should I press CTRL+Enter on value tab of "Message delimiter" key?
>> 
>> Regards,
>> Sourav Gulati
>> 
>> -Original Message-
>> From: Oleg Zhurakousky [mailto:ozhurakou...@hortonworks.com]
>> Sent: Thursday, May 05, 2016 4:41 PM
>> To: dev@nifi.apache.org
>> Subject: Re: PutKafka-Message Delimiter
>> 
>> Please use Ctrl+Enter.
>> The '\n' is interpreted literally as a string (not new line).
>> Let me know how it goes
>> Cheers
>> Oleg
>> 
>>> On May 5, 2016, at 7:01 AM, Sourav Gulati <sourav.gul...@impetus.co.in> 
>>> wrote:
>>> 
>>> Hi All,
>>> 
>>> I am tried setting message delimiter in put kafka to /n . But it is not 
>>> working. Please guide.
>>> 
>>> Regards,
>>> Sourav Gulati
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> NOTE: This message may contain information that is confidential, 
>>> proprietary, privileged or otherwise protected by law. The message is 
>>> intended solely for the named addressee. If received in error, please 
>>> destroy and notify the sender. Any use of this email is prohibited when 
>>> received in error. Impetus does not represent, warrant and/or guarantee, 
>>> that the integrity of this communication has been maintained nor that the 
>>> communication is free of errors, virus, interception or interference.
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> NOTE: This message may contain information that is confidential, 
>> proprietary, privileged or otherwise protected by law. The message is 
>> intended solely for the named addressee. If received in error, please 
>> destroy and notify the sender. Any use of this email is prohibited when 
>> received in error. Impetus does not represent, warrant and/or guarantee, 
>> that the integrity of this communication has been maintained nor that the 
>> communication is free of errors, virus, interception or interference.
> 
> 
> 
> 
> 
> 
> 
> 
> NOTE: This message may contain information that is confidential, proprietary, 
> privileged or otherwise protected by law. The message is intended solely for 
> the named addressee. If received in error, please destroy and notify the 
> sender. Any use of this email is prohibited when received in error. Impetus 
> does not represent, warrant and/or guarantee, that the integrity of this 
> communication has been maintained nor that the communication is free of 
> errors, virus, interception or interference.
>