Re: Support for Elastic Search in Future releases

[Matthew Burgess]

Shweta,

There is a Jira case for Elasticsearch processors:

https://issues.apache.org/jira/browse/NIFI-1275


I plan to work on these (with other folks in the NiFi community if interested) 
very soon.

Regards,
Matt



On 12/18/15, 2:17 AM, "shweta"  wrote:

>Hi,
>
>I wanted to know if there are any plans to have custom processors supporting 
>Data ingestion/egestion 
>for Elastic search just like there is on for SOLR.
>
>Thanks,
>Shweta
>
>
>
>--
>View this message in context: 
>http://apache-nifi-developer-list.39713.n7.nabble.com/Support-for-Elastic-Search-in-Future-releases-tp5849.html
>Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Support for Elastic Search in Future releases

[Igor Kravzov]

Shweta, I am also looking forward for ES support. In the mean time I
inject data using PostHTTP processor and can try to help you if you have
any questions. I even managed to create index-per-day because http
processor URL property supports scripting language and I can format current
date  as a string.
The only problem I have now is assembling data for bulk indexing
Doing something wrong..
On Dec 18, 2015 6:48 AM, "Matthew Burgess"  wrote:

> Shweta,
>
> There is a Jira case for Elasticsearch processors:
>
> https://issues.apache.org/jira/browse/NIFI-1275
>
>
> I plan to work on these (with other folks in the NiFi community if
> interested) very soon.
>
> Regards,
> Matt
>
>
>
> On 12/18/15, 2:17 AM, "shweta"  wrote:
>
> >Hi,
> >
> >I wanted to know if there are any plans to have custom processors
> supporting
> >Data ingestion/egestion
> >for Elastic search just like there is on for SOLR.
> >
> >Thanks,
> >Shweta
> >
> >
> >
> >--
> >View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/Support-for-Elastic-Search-in-Future-releases-tp5849.html
> >Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>
>

Re: Cluster Setup

[Bryan Bende]

Hello,

You can run the NCM on the same machine as a node, but they still have to
be two separate processes. You would create two copies of the directory
where you extracted nifi and set one to be the manager and one to be the
node, you'll also have to give them different nifi.web.http.port values.

For the second error, you could try stopping the instance that is the node
and remove the flow.xml.gz that is under conf, and then start it again to
have it pull the flow from the NCM.

-Bryan

On Fri, Dec 18, 2015 at 10:50 AM, plj  wrote:

> Howdy,
>
>   I'm trying to set up a cluster for the 1st time.  I 1st tried to setup a
> NCM and a node on the same machine.  In nifi.properties I set:
> nifi.web.http.port=8081
> nifi.cluster.is.node=true
> nifi.cluster.node.address=
> nifi.cluster.node.protocol.port=8083
> nifi.cluster.is.manager=true
> nifi.cluster.manager.address=
> nifi.cluster.manager.protocol.port=8082
>
> I got the error:
> 2015-12-18 11:14:56,827 ERROR [NiFi logging handler] org.apache.nifi.StdErr
> Failed to start web server: ...
> nested exception is java.lang.IllegalStateException: Application may be
> configured as a cluster manager or a node, but not both.
>
> What did I do wrong?  The admin guild says :
> "it is also perfectly fine to install the NCM and one of the nodes on the
> same server, as the NCM is very lightweight".
>
> So I decided I could figure that out later and set
> nifi.cluster.is.node=false
> I started my NCM and then started a node on another machine.  I got the
> following error.
>
> 2015-12-18 11:06:45,112 INFO [Handle Controller Startup Failure Message
> from
> [id=24b57ab2-1853-4814-8c31-4e467d3364e3, apiAddress=localhost,
> apiPort=8081, socketAddress=localhost, socketPort=8083]]
> o.a.n.c.manager.impl.WebClusterManager Node Event:
> [id=24b57ab2-1853-4814-8c31-4e467d3364e3, apiAddress=localhost,
> apiPort=8081, socketAddress=localhost, socketPort=8083] -- 'Node could not
> join cluster because it failed to start up properly. Setting node to
> Disconnected. Node reported the following error: Failed to connect node to
> cluster because local flow is different than cluster flow.'
>
> How do I get the cluster flow and the local flow to be the same?
>
> thank you
>
>
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/Cluster-Setup-tp5853.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Cluster Start Error: Apache NiFi is already running, listening to Bootstrap on port

[Sumanth Chinthagunta]

Hi 
I am following clustering instructions as per the link below:  NCM, Node1 in 
Server1 and Node2 on Server2.
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#clustering
 


I created two copies of Nifi 0.4.0 folders on Server1 and started NCM, then 
tried to start Node1 and got the following error. 
Error: 
org.apache.nifi.bootstrap.Command Apache NiFi is already running, listening to 
Bootstrap on port 43010
 
Am I missing any steps?
Thanks 
Sumo

Re: Cluster Start Error: Apache NiFi is already running, listening to Bootstrap on port

[Corey Flowers]

If you have a node and cluster on the same box you must be running on
a different ports. Also, if you are not using the default locations of
the repositories, you should set up the repos under different storage
locations. For instance, on ours, the ncms, keep the default locations
and the nodes are changed to larger storage file systems. Part of this
is that the ncm, doesn't actually move data, so the defaults should
suffice. Let me know if you need more info.

Oh and make sure the site to site ports are different too.

Later!

Sent from my iPhone

> On Dec 18, 2015, at 1:00 PM, Sumanth Chinthagunta  wrote:
>
>
> Hi
> I am following clustering instructions as per the link below:  NCM, Node1 in 
> Server1 and Node2 on Server2.
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#clustering
>  
> 
>
> I created two copies of Nifi 0.4.0 folders on Server1 and started NCM, then 
> tried to start Node1 and got the following error.
> Error:
> org.apache.nifi.bootstrap.Command Apache NiFi is already running, listening 
> to Bootstrap on port 43010
>
> Am I missing any steps?
> Thanks
> Sumo
>

Cluster Setup

[plj]

Howdy,

  I'm trying to set up a cluster for the 1st time.  I 1st tried to setup a
NCM and a node on the same machine.  In nifi.properties I set:
nifi.web.http.port=8081
nifi.cluster.is.node=true
nifi.cluster.node.address=
nifi.cluster.node.protocol.port=8083
nifi.cluster.is.manager=true
nifi.cluster.manager.address=
nifi.cluster.manager.protocol.port=8082

I got the error:
2015-12-18 11:14:56,827 ERROR [NiFi logging handler] org.apache.nifi.StdErr
Failed to start web server: ...
nested exception is java.lang.IllegalStateException: Application may be
configured as a cluster manager or a node, but not both.

What did I do wrong?  The admin guild says :
"it is also perfectly fine to install the NCM and one of the nodes on the
same server, as the NCM is very lightweight".

So I decided I could figure that out later and set
nifi.cluster.is.node=false
I started my NCM and then started a node on another machine.  I got the
following error.

2015-12-18 11:06:45,112 INFO [Handle Controller Startup Failure Message from
[id=24b57ab2-1853-4814-8c31-4e467d3364e3, apiAddress=localhost,
apiPort=8081, socketAddress=localhost, socketPort=8083]]
o.a.n.c.manager.impl.WebClusterManager Node Event:
[id=24b57ab2-1853-4814-8c31-4e467d3364e3, apiAddress=localhost,
apiPort=8081, socketAddress=localhost, socketPort=8083] -- 'Node could not
join cluster because it failed to start up properly. Setting node to
Disconnected. Node reported the following error: Failed to connect node to
cluster because local flow is different than cluster flow.'

How do I get the cluster flow and the local flow to be the same?

thank you





--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Cluster-Setup-tp5853.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Testing handling of static class methods

[Matt Burgess]

You could move the one static call into an instance method of PutJMS, and use 
Mockito.spy() to get a partial mock of the processor, then use when() to 
override the instance method in the test. Not sure if that's how it's done in 
other places but it's worked for me in the past.

Regards,
Matt

Sent from my iPhone

> On Dec 18, 2015, at 3:20 PM, Joe Skora  wrote:
> 
> For unit testing, one problem I've run into is overriding the returns from
> static class methods.
> 
> For instance, PutJMS contains this code:
> 
> try {
>>wrappedProducer = JmsFactory.createMessageProducer(context, true);
>>logger.info("Connected to JMS server {}",
>>new Object[]{context.getProperty(URL).getValue()});
>> } catch (final JMSException e) {
>>logger.error("Failed to connect to JMS Server due to {}", new
>> Object[]{e});
>>session.transfer(flowFiles, REL_FAILURE);
>>context.yield();
>>return;
>> }
> 
> where JmsFactory.createmessageProducer call being defined as
> 
> public static WrappedMessageProducer createMessageProducer(...
> 
> which presents a problem since it can't be easily overridden for a unit
> test.  Exercising the
> 
> How you handle this problem?
> 
> Regards,
> Joe

Re: Cluster Setup

[plj]

Thank you for your help.  I deleted the flow.xml.gz .  So now I have the NCM
on machine 'b' and a node on machine 'c'.  I start up a and then start up b. 
In b's logs I see logs of heartbeats:
2015-12-18 14:41:05,807 INFO [Process NCM Request-10]
o.a.n.c.p.impl.SocketProtocolListener Finished processing request
790c06c7-6857-4a90-9d4f-76ec99159369 (type=HEARTBEAT, length=3250 bytes) in
2 millis

On machine c is also see heartbeats:
2015-12-18 14:39:55,741 INFO [Clustering Tasks Thread-3]
org.apache.nifi.cluster.heartbeat Heartbeat created at 2015-12-18
14:39:55,664 and sent at 2015-12-18 14:39:55,741; send took 6 millis

Everything seems fine.

Then I try to view NiFi on the  NCM which I set the web to:
nifi.web.http.port=8089

http://machine-b.mitre.org:8089/nifi/

I see:
An unexpected error has occurred

No nodes were able to process this request.


I don't see anything in the logs to suggest what went wrong.

Thoughts?






--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Cluster-Setup-tp5853p5860.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Testing handling of static class methods

[Joe Skora]

For unit testing, one problem I've run into is overriding the returns from
static class methods.

For instance, PutJMS contains this code:

try {
> wrappedProducer = JmsFactory.createMessageProducer(context, true);
> logger.info("Connected to JMS server {}",
> new Object[]{context.getProperty(URL).getValue()});
> } catch (final JMSException e) {
> logger.error("Failed to connect to JMS Server due to {}", new
> Object[]{e});
> session.transfer(flowFiles, REL_FAILURE);
> context.yield();
> return;
> }
>

where JmsFactory.createmessageProducer call being defined as

public static WrappedMessageProducer createMessageProducer(...
>

which presents a problem since it can't be easily overridden for a unit
test.  Exercising the

How you handle this problem?

Regards,
Joe

Re: Cluster Start Error: Apache NiFi is already running, listening to Bootstrap on port

[Matt Gilman]

Sumo,

It appears that while getting all the nodes stood up, there is no node that
is assigned the primary role. NiFi will be in safe mode until a primary is
established. If you go to the cluster table (accessed from the cluster icon
in the upper right hand corner) you should be able to assign the primary
role to one of the nodes in your cluster. This is done by clicking the
ribbon icon in the row for the desired node.

The primary node is responsible for running processors that are configured
with the scheduling strategy of 'Primary Node Only'. When scheduled to run,
these will only execute on that Node. This is useful if your using certain
protocols that would be troublesome if multiple nodes executed concurrently
(like [S]FTP for instance).

Matt

On Fri, Dec 18, 2015 at 7:26 PM, Sumanth Chinthagunta 
wrote:

> Issue 1:
> in my NiFi installation I granted only read access to NIFI_HOME/lib and I
> got this error while trying to start NiFi.
>
> This got resolved when I gave read+write access to NIFI_HOME/lib. this
> makes me wonder why NiFi start process needs write access to lib folder!
>
> INFO [main] org.apache.nifi.BootstrapListener Successfully initiated
> communication with Bootstrap
> WARN [main] org.apache.nifi.nar.NarUnpacker Unable to load NAR library
> bundles due to java.io.IOException: /software/nifi-node1/./lib directory
> does not have read/write privilege Will proceed without loading any further
> Nar bundles
> ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to
> java.lang.IllegalStateException: Unable to find the framework NAR
> ClassLoader.
> java.lang.IllegalStateException: Unable to find the framework NAR
> ClassLoader.
> at org.apache.nifi.NiFi.(NiFi.java:116)
> ~[nifi-runtime-0.4.0.jar:0.4.0]
> at org.apache.nifi.NiFi.main(NiFi.java:227)
> ~[nifi-runtime-0.4.0.jar:0.4.0]
> INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web
> server...
> INFO [Thread-1] org.apache.nifi.NiFi Jetty web server shutdown completed
> (nicely or otherwise).
>
>
> Issue 2:
> Server 1 : NCM , node1
> Server 2 : node2
>
> Case 1: I started NiFi NCM on server 1 and  then node2 on server 2. when
> you access admin UI, you will see error even  node2 and NCM are
> communicating.
> Case 2:  I started NiFi NCM and then node1 on server 1. Admin UI works as
> expected.
> My workaround was, to set nifi.web.http.host to hostname instead of
> leaving to default ‘localhost’  on Server2. this should be documented.
>
> Issue 3:
> After above issues are resolved, now I see both node1 and node2 connected
> to NCM. when I try to add any processors or process group, I am getting
> following error in the browser UI:
>
> Cluster is unable to service request to change flow: Received a mutable
> request [PUT --
> http://xyz:8090/nifi-api/controller/process-groups/77afb88b-5f7d-45d7-a1f4-f9e9269b489b/processors/bceff62d-435d-4c47-a907-bb2a26cd0e56]
> <
> http://xyz:8090/nifi-api/controller/process-groups/77afb88b-5f7d-45d7-a1f4-f9e9269b489b/processors/bceff62d-435d-4c47-a907-bb2a26cd0e56]>
> while in safe mode
>
> please let me know if you find anything wrong with my setup.
>
> Thanks
> Sumo
>
> > On Dec 18, 2015, at 11:37 AM, Sumanth Chinthagunta 
> wrote:
> >
> > Thanks Corey and Mark for quick response.
> >
> > You are right, I was sharing some folders using symbolic links between
> NCM and Node1. After removing sharing for bin folder, it works fine :)
> >
> > NCM/Node1
> > bin -> /software/nifi/latest/bin/
> > conf
> > content_repository
> > database_repository
> > docs -> /software/nifi/latest/docs/
> > flowfile_repository
> > lib -> /software/nifi/latest/lib/
> > logs
> > provenance_repository
> > work
> >
> > Thanks
> > Sumo
> >
> >> On Dec 18, 2015, at 10:05 AM, Mark Payne  wrote:
> >>
> >> Sumo,
> >>
> >> When you say "I created two copies of Nifi 0.4.0 folders on Server1"
> does that mean that you made a copy of the first directory, or
> >> that you untarred the .tar.gz again?
> >>
> >> It looks like the same nifi.pid file is in the bin/ directory of both
> instances. You should be able to delete the nifi.pid file from the node
> >> and then start it up.
> >>
> >> Thanks
> >> -Mark
> >>
> >>
> >>> On Dec 18, 2015, at 12:59 PM, Sumanth Chinthagunta 
> wrote:
> >>>
> >>>
> >>> Hi
> >>> I am following clustering instructions as per the link below:  NCM,
> Node1 in Server1 and Node2 on Server2.
> >>>
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#clustering
> <
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#clustering
> >
> >>>
> >>> I created two copies of Nifi 0.4.0 folders on Server1 and started NCM,
> then tried to start Node1 and got the following error.
> >>> Error:
> >>> org.apache.nifi.bootstrap.Command Apache NiFi is already running,
> listening to Bootstrap on port 43010
> >>>
> >>> Am I missing any steps?
> >>> Thanks
> >>> Sumo
> >>>
> >>
> 

Re: Cluster Start Error: Apache NiFi is already running, listening to Bootstrap on port

[Sumanth Chinthagunta]

got it. now my cluster is all set :) 
 Thanks 

> On Dec 18, 2015, at 5:04 PM, Matt Gilman  wrote:
> 
> Sumo,
> 
> It appears that while getting all the nodes stood up, there is no node that
> is assigned the primary role. NiFi will be in safe mode until a primary is
> established. If you go to the cluster table (accessed from the cluster icon
> in the upper right hand corner) you should be able to assign the primary
> role to one of the nodes in your cluster. This is done by clicking the
> ribbon icon in the row for the desired node.
> 
> The primary node is responsible for running processors that are configured
> with the scheduling strategy of 'Primary Node Only'. When scheduled to run,
> these will only execute on that Node. This is useful if your using certain
> protocols that would be troublesome if multiple nodes executed concurrently
> (like [S]FTP for instance).
> 
> Matt
> 
> On Fri, Dec 18, 2015 at 7:26 PM, Sumanth Chinthagunta  >
> wrote:
> 
>> Issue 1:
>> in my NiFi installation I granted only read access to NIFI_HOME/lib and I
>> got this error while trying to start NiFi.
>> 
>> This got resolved when I gave read+write access to NIFI_HOME/lib. this
>> makes me wonder why NiFi start process needs write access to lib folder!
>> 
>> INFO [main] org.apache.nifi.BootstrapListener Successfully initiated
>> communication with Bootstrap
>> WARN [main] org.apache.nifi.nar.NarUnpacker Unable to load NAR library
>> bundles due to java.io.IOException: /software/nifi-node1/./lib directory
>> does not have read/write privilege Will proceed without loading any further
>> Nar bundles
>> ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to
>> java.lang.IllegalStateException: Unable to find the framework NAR
>> ClassLoader.
>> java.lang.IllegalStateException: Unable to find the framework NAR
>> ClassLoader.
>>at org.apache.nifi.NiFi.(NiFi.java:116)
>> ~[nifi-runtime-0.4.0.jar:0.4.0]
>>at org.apache.nifi.NiFi.main(NiFi.java:227)
>> ~[nifi-runtime-0.4.0.jar:0.4.0]
>> INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web
>> server...
>> INFO [Thread-1] org.apache.nifi.NiFi Jetty web server shutdown completed
>> (nicely or otherwise).
>> 
>> 
>> Issue 2:
>> Server 1 : NCM , node1
>> Server 2 : node2
>> 
>> Case 1: I started NiFi NCM on server 1 and  then node2 on server 2. when
>> you access admin UI, you will see error even  node2 and NCM are
>> communicating.
>> Case 2:  I started NiFi NCM and then node1 on server 1. Admin UI works as
>> expected.
>> My workaround was, to set nifi.web.http.host to hostname instead of
>> leaving to default ‘localhost’  on Server2. this should be documented.
>> 
>> Issue 3:
>> After above issues are resolved, now I see both node1 and node2 connected
>> to NCM. when I try to add any processors or process group, I am getting
>> following error in the browser UI:
>> 
>> Cluster is unable to service request to change flow: Received a mutable
>> request [PUT --
>> http://xyz:8090/nifi-api/controller/process-groups/77afb88b-5f7d-45d7-a1f4-f9e9269b489b/processors/bceff62d-435d-4c47-a907-bb2a26cd0e56]
>> <
>> http://xyz:8090/nifi-api/controller/process-groups/77afb88b-5f7d-45d7-a1f4-f9e9269b489b/processors/bceff62d-435d-4c47-a907-bb2a26cd0e56]
>>  
>> >
>> while in safe mode
>> 
>> please let me know if you find anything wrong with my setup.
>> 
>> Thanks
>> Sumo
>> 
>>> On Dec 18, 2015, at 11:37 AM, Sumanth Chinthagunta 
>> wrote:
>>> 
>>> Thanks Corey and Mark for quick response.
>>> 
>>> You are right, I was sharing some folders using symbolic links between
>> NCM and Node1. After removing sharing for bin folder, it works fine :)
>>> 
>>> NCM/Node1
>>> bin -> /software/nifi/latest/bin/
>>> conf
>>> content_repository
>>> database_repository
>>> docs -> /software/nifi/latest/docs/
>>> flowfile_repository
>>> lib -> /software/nifi/latest/lib/
>>> logs
>>> provenance_repository
>>> work
>>> 
>>> Thanks
>>> Sumo
>>> 
 On Dec 18, 2015, at 10:05 AM, Mark Payne  wrote:
 
 Sumo,
 
 When you say "I created two copies of Nifi 0.4.0 folders on Server1"
>> does that mean that you made a copy of the first directory, or
 that you untarred the .tar.gz again?
 
 It looks like the same nifi.pid file is in the bin/ directory of both
>> instances. You should be able to delete the nifi.pid file from the node
 and then start it up.
 
 Thanks
 -Mark
 
 
> On Dec 18, 2015, at 12:59 PM, Sumanth Chinthagunta 
>> wrote:
> 
> 
> Hi
> I am following clustering instructions as per the link below:  NCM,
>> Node1 in Server1 and Node2 on Server2.
> 
>> 

Re: Support for Elastic Search in Future releases

[Joe Witt]

Shweta,

Please subscribe to the mailing list so that the community may
continue to see your messages without requiring moderator support each
time.  Please use the subscribe link found here
https://nifi.apache.org/mailing_lists.html

Thanks
Joe

On Fri, Dec 18, 2015 at 2:17 AM, shweta  wrote:
> Hi,
>
> I wanted to know if there are any plans to have custom processors supporting
> Data ingestion/egestion
> for Elastic search just like there is on for SOLR.
>
> Thanks,
> Shweta
>
>
>
> --
> View this message in context: 
> http://apache-nifi-developer-list.39713.n7.nabble.com/Support-for-Elastic-Search-in-Future-releases-tp5849.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Cluster Start Error: Apache NiFi is already running, listening to Bootstrap on port

[Sumanth Chinthagunta]

Issue 1:
in my NiFi installation I granted only read access to NIFI_HOME/lib and I got 
this error while trying to start NiFi.

This got resolved when I gave read+write access to NIFI_HOME/lib. this makes me 
wonder why NiFi start process needs write access to lib folder!

INFO [main] org.apache.nifi.BootstrapListener Successfully initiated 
communication with Bootstrap
WARN [main] org.apache.nifi.nar.NarUnpacker Unable to load NAR library bundles 
due to java.io.IOException: /software/nifi-node1/./lib directory does not have 
read/write privilege Will proceed without loading any further Nar bundles
ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to 
java.lang.IllegalStateException: Unable to find the framework NAR ClassLoader.
java.lang.IllegalStateException: Unable to find the framework NAR ClassLoader.
at org.apache.nifi.NiFi.(NiFi.java:116) 
~[nifi-runtime-0.4.0.jar:0.4.0]
at org.apache.nifi.NiFi.main(NiFi.java:227) 
~[nifi-runtime-0.4.0.jar:0.4.0]
INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...
INFO [Thread-1] org.apache.nifi.NiFi Jetty web server shutdown completed 
(nicely or otherwise).


Issue 2:
Server 1 : NCM , node1
Server 2 : node2

Case 1: I started NiFi NCM on server 1 and  then node2 on server 2. when you 
access admin UI, you will see error even  node2 and NCM are communicating. 
Case 2:  I started NiFi NCM and then node1 on server 1. Admin UI works as 
expected. 
My workaround was, to set nifi.web.http.host to hostname instead of leaving to 
default ‘localhost’  on Server2. this should be documented. 

Issue 3:
After above issues are resolved, now I see both node1 and node2 connected to 
NCM. when I try to add any processors or process group, I am getting following 
error in the browser UI: 

Cluster is unable to service request to change flow: Received a mutable request 
[PUT -- 
http://xyz:8090/nifi-api/controller/process-groups/77afb88b-5f7d-45d7-a1f4-f9e9269b489b/processors/bceff62d-435d-4c47-a907-bb2a26cd0e56]
 

 while in safe mode

please let me know if you find anything wrong with my setup.

Thanks 
Sumo

> On Dec 18, 2015, at 11:37 AM, Sumanth Chinthagunta  wrote:
> 
> Thanks Corey and Mark for quick response.
> 
> You are right, I was sharing some folders using symbolic links between  NCM 
> and Node1. After removing sharing for bin folder, it works fine :)
> 
> NCM/Node1
> bin -> /software/nifi/latest/bin/
> conf
> content_repository
> database_repository
> docs -> /software/nifi/latest/docs/
> flowfile_repository
> lib -> /software/nifi/latest/lib/
> logs
> provenance_repository
> work
> 
> Thanks
> Sumo
> 
>> On Dec 18, 2015, at 10:05 AM, Mark Payne  wrote:
>> 
>> Sumo,
>> 
>> When you say "I created two copies of Nifi 0.4.0 folders on Server1" does 
>> that mean that you made a copy of the first directory, or
>> that you untarred the .tar.gz again?
>> 
>> It looks like the same nifi.pid file is in the bin/ directory of both 
>> instances. You should be able to delete the nifi.pid file from the node
>> and then start it up.
>> 
>> Thanks
>> -Mark
>> 
>> 
>>> On Dec 18, 2015, at 12:59 PM, Sumanth Chinthagunta  
>>> wrote:
>>> 
>>> 
>>> Hi 
>>> I am following clustering instructions as per the link below:  NCM, Node1 
>>> in Server1 and Node2 on Server2.
>>> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#clustering
>>>  
>>> 
>>> 
>>> I created two copies of Nifi 0.4.0 folders on Server1 and started NCM, then 
>>> tried to start Node1 and got the following error. 
>>> Error: 
>>> org.apache.nifi.bootstrap.Command Apache NiFi is already running, listening 
>>> to Bootstrap on port 43010
>>> 
>>> Am I missing any steps?
>>> Thanks 
>>> Sumo
>>> 
>> 
> 

Re: Testing handling of static class methods

[Matthew Burgess]

Definitely a topic ripe for debate :) In my view there’s a whole spectrum, with 
one side being the case Oleg describes, where the existing encapsulation is not 
compromised solely for the sake of testing. On the far side is pure 
design-by-contract. For example, the case could be made that the JMS processor 
should not be so tightly coupled to a particular client, and certainly not to a 
class but rather to an interface. Another upside for moving the client call to 
a protected method is not just for testing but so that child classes can 
override, which is not an encapsulation thing but inheritance. That might not 
be useful in this particular case, but if we’re talking OO in general then it 
applies.

Since Bryan has cited precedence for the inner class stuff in NiFi, I tend 
towards that as a consistent approach. Then again, to quote my close friend 
Oscar Wilde ;) "Consistency is the last refuge of the unimaginative" lol

Cheers!
Matt



On 12/18/15, 5:54 PM, "Oleg Zhurakousky"  wrote:

>Personally I am with Joe on this one
>
>Exposing visibility on the method just for testing is very dangerous as it 
>breaks encapsulation. There are different expectations and consideration on 
>things that are made private, protected and public. Yes, all of that is 
>meaningless when one uses reflection, but that’s a whole other discussion. 
>Relaxing visibility implies advertisement that something is up for grabs. And 
>in fact in Joe’s case while his intentions were noble, the fallout could be 
>anything but (see my comments here https://github.com/apache/nifi/pull/145).
>
>Just my opinion
>
>Cheers
>Oleg
>
>On Dec 18, 2015, at 5:42 PM, Joe Skora 
>> wrote:
>
>Wrapping createMessageProducer() in an instance method is a good
>suggestion, but it seems overkill just to enable testing.  Prompted by
>Oleg's suggestion, I got around instance variable visibility with
>Reflection, which is nice because it doesn't require "private" be changed
>to "protected" in the class under test and doesn't require an inner test
>class.  But, that doesn't appear to be possible for static methods, so
>wrapping with class methods may be the only choice.  Hopefully, I've missed
>something.
>
>
>On Fri, Dec 18, 2015 at 3:58 PM, Bryan Bende 
>> wrote:
>
>If you get it into a protected instance method, you can also make an inner
>class in your test, something like TestablePutJMS extends PutJMS, and
>overrides that method to return a mock or whatever you want. That is a
>common pattern in a lot of the processor tests.
>
>On Fri, Dec 18, 2015 at 3:44 PM, Matt Burgess 
>> wrote:
>
>You could move the one static call into an instance method of PutJMS, and
>use Mockito.spy() to get a partial mock of the processor, then use when()
>to override the instance method in the test. Not sure if that's how it's
>done in other places but it's worked for me in the past.
>
>Regards,
>Matt
>
>Sent from my iPhone
>
>On Dec 18, 2015, at 3:20 PM, Joe Skora 
>> wrote:
>
>For unit testing, one problem I've run into is overriding the returns
>from
>static class methods.
>
>For instance, PutJMS contains this code:
>
>try {
>  wrappedProducer = JmsFactory.createMessageProducer(context, true);
>  logger.info("Connected to JMS server {}",
>  new Object[]{context.getProperty(URL).getValue()});
>} catch (final JMSException e) {
>  logger.error("Failed to connect to JMS Server due to {}", new
>Object[]{e});
>  session.transfer(flowFiles, REL_FAILURE);
>  context.yield();
>  return;
>}
>
>where JmsFactory.createmessageProducer call being defined as
>
>public static WrappedMessageProducer createMessageProducer(...
>
>which presents a problem since it can't be easily overridden for a unit
>test.  Exercising the
>
>How you handle this problem?
>
>Regards,
>Joe
>
>
>

Re: Testing Custom Validators

[Devin Fisher]

Thanks, I'm guessing that subject is not to0 important to the validation
workflow but provides useful info for the user in the context of the
property that they are setting.

The validator I'm writing is not using the context so it should be fine. If
I need to do more I guess I'll figure out how the mock the rest then.

Devin

On Fri, Dec 18, 2015 at 1:16 PM, Mark Payne  wrote:

> Devin,
>
> Mockito should be find for most cases. If you validator itself is calling
> into the ValidationContext, for instance to create a new PropertyValue
> object, then you may need to mock out some methods. Otherwise, it should
> be fine.
>
> When validate() is called, it is given the 'input' (which is the value to
> valid) and the 'subject' (which is a description of what is being
> validated).
> So generally, the 'subject' is the name of the Property that is being
> validated.
>
> Thanks
> -Mark
>
> > On Dec 18, 2015, at 3:11 PM, Devin Fisher <
> devin.fis...@perfectsearchcorp.com> wrote:
> >
> > I'm trying to create some tests for some Validators that I'm creating.
> But
> > I can't figure out an easy way to create MockValidationContext. I don't
> > want to create the whole environment that I would for Processors.
> >
> > I looked
> > at
> nifi/nifi-commons/nifi-processor-utilities/src/test/java/org/apache/nifi/processor/util/TestStandardValidators.java
> > and it looks that it just uses Mockito to mock it.
> >
> > So, is that the recommended way to do it. Will that allow me to control
> the
> > ValidationContext?
> >
> > Also, not sure if asking two questions in one email is allowed, what is
> the
> > subject parameter in the validate method?
> >
> > Devin
>
>

Re: Testing handling of static class methods

[Joe Skora]

Wrapping createMessageProducer() in an instance method is a good
suggestion, but it seems overkill just to enable testing.  Prompted by
Oleg's suggestion, I got around instance variable visibility with
Reflection, which is nice because it doesn't require "private" be changed
to "protected" in the class under test and doesn't require an inner test
class.  But, that doesn't appear to be possible for static methods, so
wrapping with class methods may be the only choice.  Hopefully, I've missed
something.


On Fri, Dec 18, 2015 at 3:58 PM, Bryan Bende  wrote:

> If you get it into a protected instance method, you can also make an inner
> class in your test, something like TestablePutJMS extends PutJMS, and
> overrides that method to return a mock or whatever you want. That is a
> common pattern in a lot of the processor tests.
>
> On Fri, Dec 18, 2015 at 3:44 PM, Matt Burgess  wrote:
>
> > You could move the one static call into an instance method of PutJMS, and
> > use Mockito.spy() to get a partial mock of the processor, then use when()
> > to override the instance method in the test. Not sure if that's how it's
> > done in other places but it's worked for me in the past.
> >
> > Regards,
> > Matt
> >
> > Sent from my iPhone
> >
> > > On Dec 18, 2015, at 3:20 PM, Joe Skora  wrote:
> > >
> > > For unit testing, one problem I've run into is overriding the returns
> > from
> > > static class methods.
> > >
> > > For instance, PutJMS contains this code:
> > >
> > > try {
> > >>wrappedProducer = JmsFactory.createMessageProducer(context, true);
> > >>logger.info("Connected to JMS server {}",
> > >>new Object[]{context.getProperty(URL).getValue()});
> > >> } catch (final JMSException e) {
> > >>logger.error("Failed to connect to JMS Server due to {}", new
> > >> Object[]{e});
> > >>session.transfer(flowFiles, REL_FAILURE);
> > >>context.yield();
> > >>return;
> > >> }
> > >
> > > where JmsFactory.createmessageProducer call being defined as
> > >
> > > public static WrappedMessageProducer createMessageProducer(...
> > >
> > > which presents a problem since it can't be easily overridden for a unit
> > > test.  Exercising the
> > >
> > > How you handle this problem?
> > >
> > > Regards,
> > > Joe
> >
>

Re: Testing handling of static class methods

[Bryan Bende]

If you get it into a protected instance method, you can also make an inner
class in your test, something like TestablePutJMS extends PutJMS, and
overrides that method to return a mock or whatever you want. That is a
common pattern in a lot of the processor tests.

On Fri, Dec 18, 2015 at 3:44 PM, Matt Burgess  wrote:

> You could move the one static call into an instance method of PutJMS, and
> use Mockito.spy() to get a partial mock of the processor, then use when()
> to override the instance method in the test. Not sure if that's how it's
> done in other places but it's worked for me in the past.
>
> Regards,
> Matt
>
> Sent from my iPhone
>
> > On Dec 18, 2015, at 3:20 PM, Joe Skora  wrote:
> >
> > For unit testing, one problem I've run into is overriding the returns
> from
> > static class methods.
> >
> > For instance, PutJMS contains this code:
> >
> > try {
> >>wrappedProducer = JmsFactory.createMessageProducer(context, true);
> >>logger.info("Connected to JMS server {}",
> >>new Object[]{context.getProperty(URL).getValue()});
> >> } catch (final JMSException e) {
> >>logger.error("Failed to connect to JMS Server due to {}", new
> >> Object[]{e});
> >>session.transfer(flowFiles, REL_FAILURE);
> >>context.yield();
> >>return;
> >> }
> >
> > where JmsFactory.createmessageProducer call being defined as
> >
> > public static WrappedMessageProducer createMessageProducer(...
> >
> > which presents a problem since it can't be easily overridden for a unit
> > test.  Exercising the
> >
> > How you handle this problem?
> >
> > Regards,
> > Joe
>

Re: Testing handling of static class methods

[Oleg Zhurakousky]

Personally I am with Joe on this one

Exposing visibility on the method just for testing is very dangerous as it 
breaks encapsulation. There are different expectations and consideration on 
things that are made private, protected and public. Yes, all of that is 
meaningless when one uses reflection, but that’s a whole other discussion. 
Relaxing visibility implies advertisement that something is up for grabs. And 
in fact in Joe’s case while his intentions were noble, the fallout could be 
anything but (see my comments here https://github.com/apache/nifi/pull/145).

Just my opinion

Cheers
Oleg

On Dec 18, 2015, at 5:42 PM, Joe Skora 
> wrote:

Wrapping createMessageProducer() in an instance method is a good
suggestion, but it seems overkill just to enable testing.  Prompted by
Oleg's suggestion, I got around instance variable visibility with
Reflection, which is nice because it doesn't require "private" be changed
to "protected" in the class under test and doesn't require an inner test
class.  But, that doesn't appear to be possible for static methods, so
wrapping with class methods may be the only choice.  Hopefully, I've missed
something.


On Fri, Dec 18, 2015 at 3:58 PM, Bryan Bende 
> wrote:

If you get it into a protected instance method, you can also make an inner
class in your test, something like TestablePutJMS extends PutJMS, and
overrides that method to return a mock or whatever you want. That is a
common pattern in a lot of the processor tests.

On Fri, Dec 18, 2015 at 3:44 PM, Matt Burgess 
> wrote:

You could move the one static call into an instance method of PutJMS, and
use Mockito.spy() to get a partial mock of the processor, then use when()
to override the instance method in the test. Not sure if that's how it's
done in other places but it's worked for me in the past.

Regards,
Matt

Sent from my iPhone

On Dec 18, 2015, at 3:20 PM, Joe Skora 
> wrote:

For unit testing, one problem I've run into is overriding the returns
from
static class methods.

For instance, PutJMS contains this code:

try {
  wrappedProducer = JmsFactory.createMessageProducer(context, true);
  logger.info("Connected to JMS server {}",
  new Object[]{context.getProperty(URL).getValue()});
} catch (final JMSException e) {
  logger.error("Failed to connect to JMS Server due to {}", new
Object[]{e});
  session.transfer(flowFiles, REL_FAILURE);
  context.yield();
  return;
}

where JmsFactory.createmessageProducer call being defined as

public static WrappedMessageProducer createMessageProducer(...

which presents a problem since it can't be easily overridden for a unit
test.  Exercising the

How you handle this problem?

Regards,
Joe

Re: Testing handling of static class methods

[Sean Busbey]

You could use PowerMock to either call private static methods directly
or to mock them out.

https://github.com/jayway/powermock/wiki/MockStatic

https://github.com/jayway/powermock/wiki/MockPrivate

https://github.com/jayway/powermock/wiki/BypassEncapsulation#invoking-a-private-method

On Fri, Dec 18, 2015 at 4:42 PM, Joe Skora  wrote:
> Wrapping createMessageProducer() in an instance method is a good
> suggestion, but it seems overkill just to enable testing.  Prompted by
> Oleg's suggestion, I got around instance variable visibility with
> Reflection, which is nice because it doesn't require "private" be changed
> to "protected" in the class under test and doesn't require an inner test
> class.  But, that doesn't appear to be possible for static methods, so
> wrapping with class methods may be the only choice.  Hopefully, I've missed
> something.
>
>
> On Fri, Dec 18, 2015 at 3:58 PM, Bryan Bende  wrote:
>
>> If you get it into a protected instance method, you can also make an inner
>> class in your test, something like TestablePutJMS extends PutJMS, and
>> overrides that method to return a mock or whatever you want. That is a
>> common pattern in a lot of the processor tests.
>>
>> On Fri, Dec 18, 2015 at 3:44 PM, Matt Burgess  wrote:
>>
>> > You could move the one static call into an instance method of PutJMS, and
>> > use Mockito.spy() to get a partial mock of the processor, then use when()
>> > to override the instance method in the test. Not sure if that's how it's
>> > done in other places but it's worked for me in the past.
>> >
>> > Regards,
>> > Matt
>> >
>> > Sent from my iPhone
>> >
>> > > On Dec 18, 2015, at 3:20 PM, Joe Skora  wrote:
>> > >
>> > > For unit testing, one problem I've run into is overriding the returns
>> > from
>> > > static class methods.
>> > >
>> > > For instance, PutJMS contains this code:
>> > >
>> > > try {
>> > >>wrappedProducer = JmsFactory.createMessageProducer(context, true);
>> > >>logger.info("Connected to JMS server {}",
>> > >>new Object[]{context.getProperty(URL).getValue()});
>> > >> } catch (final JMSException e) {
>> > >>logger.error("Failed to connect to JMS Server due to {}", new
>> > >> Object[]{e});
>> > >>session.transfer(flowFiles, REL_FAILURE);
>> > >>context.yield();
>> > >>return;
>> > >> }
>> > >
>> > > where JmsFactory.createmessageProducer call being defined as
>> > >
>> > > public static WrappedMessageProducer createMessageProducer(...
>> > >
>> > > which presents a problem since it can't be easily overridden for a unit
>> > > test.  Exercising the
>> > >
>> > > How you handle this problem?
>> > >
>> > > Regards,
>> > > Joe
>> >
>>

Re: Subscribe me to Apache NiFi

[Joe Witt]

Please send an email to dev-subscr...@nifi.apache.org

On Sat, Dec 19, 2015 at 1:35 AM, Deepak Dixit  wrote:
> Kindly add to dev mailing list.
>
> --
> From:
>
> Deepak D Dixit
> deepakdixit2...@gmail.com
> +919028507537

Re: discuss nifi 0.4.1

[Joe Witt]

Sean,

Yeah i don't disagree with that point.  The caveat being it was only a
change to that client not a change to support the new client API and
the behavior with existing clients old and new verified.

I'd prefer to stick with 0.4.1 and if you still think it is best to
actually just revert that commit and apply it toward 0.5.0.

What do you think?

Thanks
Joe

On Fri, Dec 18, 2015 at 11:08 PM, Sean Busbey  wrote:
> Can we update to 0.5.0 instead? The kafka client change isn't
> something I'd expect in a patch release.
>
> On Fri, Dec 18, 2015 at 9:54 PM, Joe Witt  wrote:
>> ok - so master is presently on 041 and it does indeed appear to be all
>> incremental friendly fixes.  So looks like we can just use the normal
>> process.  As excited as I was to use cherry-pick doesn't look like it
>> is needed.
>>
>> The bugs fixed on 041 so far are all nice cleanup items and things
>> which have been problematic for quite a while.  However, there are a
>> few site-to-site issues that would create some pretty annoying issues
>> for users so best to eliminate them.  And big thanks to Matt Clarke
>> for finding and reporting them!
>>
>> Gonna prep an RC.
>>
>> Thanks
>> Joe
>>
>> On Thu, Dec 17, 2015 at 7:53 PM, Tony Kurc  wrote:
>>> I have no objection to "because we should be able to do this well!" as a
>>> reason.
>>>
>>> On Thu, Dec 17, 2015 at 7:45 PM, Oleg Zhurakousky <
>>> ozhurakou...@hortonworks.com> wrote:
>>>
 Generally RCs are used to address that level of validation, so in the end
 I still think it's a more of a culture one chooses. One common  example;
 x.x.1+ = maintenance, x.1+.0 = minor features + bugs and 1+.0.0 = major
 features.

 In any event IMHO the ability to quickly release maintenance releases is
 very important  as it showcases our attention to quality

 Sent from my iPhone

 > On Dec 17, 2015, at 19:32, Tony Kurc  wrote:
 >
 > I'm not sure I understand "more validation" reasoning - won't features at
 > the end have very little validation?
 >
 >> On Thu, Dec 17, 2015 at 7:26 PM, Ryan Blue  wrote:
 >>
 >> Another reason to release 0.4.1 is to allow the additions that warrant
 >> 0.5.0 to have more validation before release. With a regular release
 cycle,
 >> features can go in at the beginning to have more time for catching bugs
 in
 >> them. I also agree with what Sean said below.
 >>
 >> rb
 >>
 >>> On 12/17/2015 04:00 PM, Sean Busbey wrote:
 >>>
 >>> On Thu, Dec 17, 2015 at 5:50 PM, Tony Kurc  wrote:
 >>>
 >>> s/features/buxfixes/
 
  On Thu, Dec 17, 2015 at 6:50 PM, Tony Kurc  wrote:
 
  Is there a reason to not just cut a 0.5.0 instead of grafting 0.5.0
 > features onto 0.4.1?
 >>> This is a good question.
 >>>
 >>> Some downstream users might have different change processes for a
 patch vs
 >>> minor release, so making a 0.4.1 that fixes what we determine to be a
 >>> substantial gap in the 0.4 line would be nice for them.
 >>>
 >>> While we might be a young project now, it would be good to already have
 >>> the
 >>> habit practiced for when we have more users in enterprise settings.
 >>>
 >>> On the other hand, 0.4.0 just happened, so a release in 3 days would
 >>> minimize the number of "stuck on 0.4.0" folks.
 >>
 >> --
 >> Ryan Blue
 >> Software Engineer
 >> Cloudera, Inc.
 >>

Re: discuss nifi 0.4.1

[Sean Busbey]

with the Kafka client change backed out for 0.5.0, I'm good to go with
0.4.1 on the rest of the changes.

On Fri, Dec 18, 2015 at 10:11 PM, Joe Witt  wrote:
> Sean,
>
> Yeah i don't disagree with that point.  The caveat being it was only a
> change to that client not a change to support the new client API and
> the behavior with existing clients old and new verified.
>
> I'd prefer to stick with 0.4.1 and if you still think it is best to
> actually just revert that commit and apply it toward 0.5.0.
>
> What do you think?
>
> Thanks
> Joe
>
> On Fri, Dec 18, 2015 at 11:08 PM, Sean Busbey  wrote:
>> Can we update to 0.5.0 instead? The kafka client change isn't
>> something I'd expect in a patch release.
>>
>> On Fri, Dec 18, 2015 at 9:54 PM, Joe Witt  wrote:
>>> ok - so master is presently on 041 and it does indeed appear to be all
>>> incremental friendly fixes.  So looks like we can just use the normal
>>> process.  As excited as I was to use cherry-pick doesn't look like it
>>> is needed.
>>>
>>> The bugs fixed on 041 so far are all nice cleanup items and things
>>> which have been problematic for quite a while.  However, there are a
>>> few site-to-site issues that would create some pretty annoying issues
>>> for users so best to eliminate them.  And big thanks to Matt Clarke
>>> for finding and reporting them!
>>>
>>> Gonna prep an RC.
>>>
>>> Thanks
>>> Joe
>>>
>>> On Thu, Dec 17, 2015 at 7:53 PM, Tony Kurc  wrote:
 I have no objection to "because we should be able to do this well!" as a
 reason.

 On Thu, Dec 17, 2015 at 7:45 PM, Oleg Zhurakousky <
 ozhurakou...@hortonworks.com> wrote:

> Generally RCs are used to address that level of validation, so in the end
> I still think it's a more of a culture one chooses. One common  example;
> x.x.1+ = maintenance, x.1+.0 = minor features + bugs and 1+.0.0 = major
> features.
>
> In any event IMHO the ability to quickly release maintenance releases is
> very important  as it showcases our attention to quality
>
> Sent from my iPhone
>
> > On Dec 17, 2015, at 19:32, Tony Kurc  wrote:
> >
> > I'm not sure I understand "more validation" reasoning - won't features 
> > at
> > the end have very little validation?
> >
> >> On Thu, Dec 17, 2015 at 7:26 PM, Ryan Blue  wrote:
> >>
> >> Another reason to release 0.4.1 is to allow the additions that warrant
> >> 0.5.0 to have more validation before release. With a regular release
> cycle,
> >> features can go in at the beginning to have more time for catching bugs
> in
> >> them. I also agree with what Sean said below.
> >>
> >> rb
> >>
> >>> On 12/17/2015 04:00 PM, Sean Busbey wrote:
> >>>
> >>> On Thu, Dec 17, 2015 at 5:50 PM, Tony Kurc  wrote:
> >>>
> >>> s/features/buxfixes/
> 
>  On Thu, Dec 17, 2015 at 6:50 PM, Tony Kurc  wrote:
> 
>  Is there a reason to not just cut a 0.5.0 instead of grafting 0.5.0
> > features onto 0.4.1?
> >>> This is a good question.
> >>>
> >>> Some downstream users might have different change processes for a
> patch vs
> >>> minor release, so making a 0.4.1 that fixes what we determine to be a
> >>> substantial gap in the 0.4 line would be nice for them.
> >>>
> >>> While we might be a young project now, it would be good to already 
> >>> have
> >>> the
> >>> habit practiced for when we have more users in enterprise settings.
> >>>
> >>> On the other hand, 0.4.0 just happened, so a release in 3 days would
> >>> minimize the number of "stuck on 0.4.0" folks.
> >>
> >> --
> >> Ryan Blue
> >> Software Engineer
> >> Cloudera, Inc.
> >>
>



-- 
Sean

Re: discuss nifi 0.4.1

[Joe Witt]

rgr that - will revert, save patch on ticket, plus this means someone
can get Oleg's kafka tester patch reviewed there too.

thanks

On Fri, Dec 18, 2015 at 11:26 PM, Sean Busbey  wrote:
> with the Kafka client change backed out for 0.5.0, I'm good to go with
> 0.4.1 on the rest of the changes.
>
> On Fri, Dec 18, 2015 at 10:11 PM, Joe Witt  wrote:
>> Sean,
>>
>> Yeah i don't disagree with that point.  The caveat being it was only a
>> change to that client not a change to support the new client API and
>> the behavior with existing clients old and new verified.
>>
>> I'd prefer to stick with 0.4.1 and if you still think it is best to
>> actually just revert that commit and apply it toward 0.5.0.
>>
>> What do you think?
>>
>> Thanks
>> Joe
>>
>> On Fri, Dec 18, 2015 at 11:08 PM, Sean Busbey  wrote:
>>> Can we update to 0.5.0 instead? The kafka client change isn't
>>> something I'd expect in a patch release.
>>>
>>> On Fri, Dec 18, 2015 at 9:54 PM, Joe Witt  wrote:
 ok - so master is presently on 041 and it does indeed appear to be all
 incremental friendly fixes.  So looks like we can just use the normal
 process.  As excited as I was to use cherry-pick doesn't look like it
 is needed.

 The bugs fixed on 041 so far are all nice cleanup items and things
 which have been problematic for quite a while.  However, there are a
 few site-to-site issues that would create some pretty annoying issues
 for users so best to eliminate them.  And big thanks to Matt Clarke
 for finding and reporting them!

 Gonna prep an RC.

 Thanks
 Joe

 On Thu, Dec 17, 2015 at 7:53 PM, Tony Kurc  wrote:
> I have no objection to "because we should be able to do this well!" as a
> reason.
>
> On Thu, Dec 17, 2015 at 7:45 PM, Oleg Zhurakousky <
> ozhurakou...@hortonworks.com> wrote:
>
>> Generally RCs are used to address that level of validation, so in the end
>> I still think it's a more of a culture one chooses. One common  example;
>> x.x.1+ = maintenance, x.1+.0 = minor features + bugs and 1+.0.0 = major
>> features.
>>
>> In any event IMHO the ability to quickly release maintenance releases is
>> very important  as it showcases our attention to quality
>>
>> Sent from my iPhone
>>
>> > On Dec 17, 2015, at 19:32, Tony Kurc  wrote:
>> >
>> > I'm not sure I understand "more validation" reasoning - won't features 
>> > at
>> > the end have very little validation?
>> >
>> >> On Thu, Dec 17, 2015 at 7:26 PM, Ryan Blue  wrote:
>> >>
>> >> Another reason to release 0.4.1 is to allow the additions that warrant
>> >> 0.5.0 to have more validation before release. With a regular release
>> cycle,
>> >> features can go in at the beginning to have more time for catching 
>> >> bugs
>> in
>> >> them. I also agree with what Sean said below.
>> >>
>> >> rb
>> >>
>> >>> On 12/17/2015 04:00 PM, Sean Busbey wrote:
>> >>>
>> >>> On Thu, Dec 17, 2015 at 5:50 PM, Tony Kurc  wrote:
>> >>>
>> >>> s/features/buxfixes/
>> 
>>  On Thu, Dec 17, 2015 at 6:50 PM, Tony Kurc  wrote:
>> 
>>  Is there a reason to not just cut a 0.5.0 instead of grafting 0.5.0
>> > features onto 0.4.1?
>> >>> This is a good question.
>> >>>
>> >>> Some downstream users might have different change processes for a
>> patch vs
>> >>> minor release, so making a 0.4.1 that fixes what we determine to be a
>> >>> substantial gap in the 0.4 line would be nice for them.
>> >>>
>> >>> While we might be a young project now, it would be good to already 
>> >>> have
>> >>> the
>> >>> habit practiced for when we have more users in enterprise settings.
>> >>>
>> >>> On the other hand, 0.4.0 just happened, so a release in 3 days would
>> >>> minimize the number of "stuck on 0.4.0" folks.
>> >>
>> >> --
>> >> Ryan Blue
>> >> Software Engineer
>> >> Cloudera, Inc.
>> >>
>>
>
>
>
> --
> Sean

Re: Cluster Setup

[Matthew Clarke]

I believe you are having issues related to hostname resolution on your
nodes and NCM. By leaving the varies host properties in your NiFi
properties file blank, they are resolving to localhost. Each stem thinks
localhost is itself. Try filling in the properties either with FQDNs that
each node can successfully resolve or populate these host properties with
IPs.  So the rest are multiple host properties to populate. One under
site-to-site properties, one under web properties, and a couple under
cluster properties.
On Dec 18, 2015 3:00 PM, "plj"  wrote:

> Thank you for your help.  I deleted the flow.xml.gz .  So now I have the
> NCM
> on machine 'b' and a node on machine 'c'.  I start up a and then start up
> b.
> In b's logs I see logs of heartbeats:
> 2015-12-18 14:41:05,807 INFO [Process NCM Request-10]
> o.a.n.c.p.impl.SocketProtocolListener Finished processing request
> 790c06c7-6857-4a90-9d4f-76ec99159369 (type=HEARTBEAT, length=3250 bytes) in
> 2 millis
>
> On machine c is also see heartbeats:
> 2015-12-18 14:39:55,741 INFO [Clustering Tasks Thread-3]
> org.apache.nifi.cluster.heartbeat Heartbeat created at 2015-12-18
> 14:39:55,664 and sent at 2015-12-18 14:39:55,741; send took 6 millis
>
> Everything seems fine.
>
> Then I try to view NiFi on the  NCM which I set the web to:
> nifi.web.http.port=8089
>
> http://machine-b.mitre.org:8089/nifi/
>
> I see:
> An unexpected error has occurred
>
> No nodes were able to process this request.
>
>
> I don't see anything in the logs to suggest what went wrong.
>
> Thoughts?
>
>
>
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/Cluster-Setup-tp5853p5860.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: discuss nifi 0.4.1

[Joe Witt]

ok - so master is presently on 041 and it does indeed appear to be all
incremental friendly fixes.  So looks like we can just use the normal
process.  As excited as I was to use cherry-pick doesn't look like it
is needed.

The bugs fixed on 041 so far are all nice cleanup items and things
which have been problematic for quite a while.  However, there are a
few site-to-site issues that would create some pretty annoying issues
for users so best to eliminate them.  And big thanks to Matt Clarke
for finding and reporting them!

Gonna prep an RC.

Thanks
Joe

On Thu, Dec 17, 2015 at 7:53 PM, Tony Kurc  wrote:
> I have no objection to "because we should be able to do this well!" as a
> reason.
>
> On Thu, Dec 17, 2015 at 7:45 PM, Oleg Zhurakousky <
> ozhurakou...@hortonworks.com> wrote:
>
>> Generally RCs are used to address that level of validation, so in the end
>> I still think it's a more of a culture one chooses. One common  example;
>> x.x.1+ = maintenance, x.1+.0 = minor features + bugs and 1+.0.0 = major
>> features.
>>
>> In any event IMHO the ability to quickly release maintenance releases is
>> very important  as it showcases our attention to quality
>>
>> Sent from my iPhone
>>
>> > On Dec 17, 2015, at 19:32, Tony Kurc  wrote:
>> >
>> > I'm not sure I understand "more validation" reasoning - won't features at
>> > the end have very little validation?
>> >
>> >> On Thu, Dec 17, 2015 at 7:26 PM, Ryan Blue  wrote:
>> >>
>> >> Another reason to release 0.4.1 is to allow the additions that warrant
>> >> 0.5.0 to have more validation before release. With a regular release
>> cycle,
>> >> features can go in at the beginning to have more time for catching bugs
>> in
>> >> them. I also agree with what Sean said below.
>> >>
>> >> rb
>> >>
>> >>> On 12/17/2015 04:00 PM, Sean Busbey wrote:
>> >>>
>> >>> On Thu, Dec 17, 2015 at 5:50 PM, Tony Kurc  wrote:
>> >>>
>> >>> s/features/buxfixes/
>> 
>>  On Thu, Dec 17, 2015 at 6:50 PM, Tony Kurc  wrote:
>> 
>>  Is there a reason to not just cut a 0.5.0 instead of grafting 0.5.0
>> > features onto 0.4.1?
>> >>> This is a good question.
>> >>>
>> >>> Some downstream users might have different change processes for a
>> patch vs
>> >>> minor release, so making a 0.4.1 that fixes what we determine to be a
>> >>> substantial gap in the 0.4 line would be nice for them.
>> >>>
>> >>> While we might be a young project now, it would be good to already have
>> >>> the
>> >>> habit practiced for when we have more users in enterprise settings.
>> >>>
>> >>> On the other hand, 0.4.0 just happened, so a release in 3 days would
>> >>> minimize the number of "stuck on 0.4.0" folks.
>> >>
>> >> --
>> >> Ryan Blue
>> >> Software Engineer
>> >> Cloudera, Inc.
>> >>
>>

Re: discuss nifi 0.4.1

[Sean Busbey]

Can we update to 0.5.0 instead? The kafka client change isn't
something I'd expect in a patch release.

On Fri, Dec 18, 2015 at 9:54 PM, Joe Witt  wrote:
> ok - so master is presently on 041 and it does indeed appear to be all
> incremental friendly fixes.  So looks like we can just use the normal
> process.  As excited as I was to use cherry-pick doesn't look like it
> is needed.
>
> The bugs fixed on 041 so far are all nice cleanup items and things
> which have been problematic for quite a while.  However, there are a
> few site-to-site issues that would create some pretty annoying issues
> for users so best to eliminate them.  And big thanks to Matt Clarke
> for finding and reporting them!
>
> Gonna prep an RC.
>
> Thanks
> Joe
>
> On Thu, Dec 17, 2015 at 7:53 PM, Tony Kurc  wrote:
>> I have no objection to "because we should be able to do this well!" as a
>> reason.
>>
>> On Thu, Dec 17, 2015 at 7:45 PM, Oleg Zhurakousky <
>> ozhurakou...@hortonworks.com> wrote:
>>
>>> Generally RCs are used to address that level of validation, so in the end
>>> I still think it's a more of a culture one chooses. One common  example;
>>> x.x.1+ = maintenance, x.1+.0 = minor features + bugs and 1+.0.0 = major
>>> features.
>>>
>>> In any event IMHO the ability to quickly release maintenance releases is
>>> very important  as it showcases our attention to quality
>>>
>>> Sent from my iPhone
>>>
>>> > On Dec 17, 2015, at 19:32, Tony Kurc  wrote:
>>> >
>>> > I'm not sure I understand "more validation" reasoning - won't features at
>>> > the end have very little validation?
>>> >
>>> >> On Thu, Dec 17, 2015 at 7:26 PM, Ryan Blue  wrote:
>>> >>
>>> >> Another reason to release 0.4.1 is to allow the additions that warrant
>>> >> 0.5.0 to have more validation before release. With a regular release
>>> cycle,
>>> >> features can go in at the beginning to have more time for catching bugs
>>> in
>>> >> them. I also agree with what Sean said below.
>>> >>
>>> >> rb
>>> >>
>>> >>> On 12/17/2015 04:00 PM, Sean Busbey wrote:
>>> >>>
>>> >>> On Thu, Dec 17, 2015 at 5:50 PM, Tony Kurc  wrote:
>>> >>>
>>> >>> s/features/buxfixes/
>>> 
>>>  On Thu, Dec 17, 2015 at 6:50 PM, Tony Kurc  wrote:
>>> 
>>>  Is there a reason to not just cut a 0.5.0 instead of grafting 0.5.0
>>> > features onto 0.4.1?
>>> >>> This is a good question.
>>> >>>
>>> >>> Some downstream users might have different change processes for a
>>> patch vs
>>> >>> minor release, so making a 0.4.1 that fixes what we determine to be a
>>> >>> substantial gap in the 0.4 line would be nice for them.
>>> >>>
>>> >>> While we might be a young project now, it would be good to already have
>>> >>> the
>>> >>> habit practiced for when we have more users in enterprise settings.
>>> >>>
>>> >>> On the other hand, 0.4.0 just happened, so a release in 3 days would
>>> >>> minimize the number of "stuck on 0.4.0" folks.
>>> >>
>>> >> --
>>> >> Ryan Blue
>>> >> Software Engineer
>>> >> Cloudera, Inc.
>>> >>
>>>

Subscribe me to Apache NiFi

[Deepak Dixit]

Kindly add to dev mailing list.

-- 
From:

Deepak D Dixit
deepakdixit2...@gmail.com
+919028507537