Re: How to lock getfile upto putfile write into same file?

[prabhu Mahendran]

I have running multiple instances of PutFile writing to same file that
leads fail in Strange ways.

I have tried MergeContent processor but it not yields same results for all
time.

For one time it correctly merges files according to the filename but
sometimes it could not merge.

In my use case i have to store the contents in 10 files(number may be
changed).A filename is specified in every flow files.

I couldn't  merge the files with same filename in some times.And also i
cannot use expression language in "Minimum Number of entries" attribute if
i have set that property directly it can work.



On Thu, May 18, 2017 at 2:09 PM, Joe Witt  wrote:

> PutFile while writing automatically writes with a dot prepended and
> once the write is complete it removes the dot.
>
> If you have multiple instances of PutFile writing to the same file
> with the intent of appending to that file it will like fail in strange
> ways.
>
> I would recommend you simply build up the complete thing you want to
> write out to a file using MergeContent and send the merged results to
> PutFile.  However, it should not be the case that you have PutFile and
> GetFile sharing a given file in the same NiFi as you can simply
> connect the processors to move the data without have to leave the
> confines of NiFi using file IO.
>
> Thanks
>
> On Thu, May 18, 2017 at 3:06 AM, prabhu Mahendran
>  wrote:
> > Ok. I will append the '.' before Putfile using UpdateAttribute. Is this
> > name(without '.') will automatically changed by Putfile when its done?
> >
> >
> >
> > Since I am appending each rows from html content into proper csv using
> > Putfile processor. Is this works fine? How putfile knows complete html
> > flowfiles has been moved.
> >
> >
> > On Tue, May 16, 2017 at 7:53 PM, Juan Sequeiros 
> wrote:
> >>
> >> Prabhu,
> >>
> >> PutFile should pre-pend files with a "." while it is writing to it and
> >> then in the end will copy the "." file to its "filename" value.
> >>
> >> On GetFile side make sure that "ignore hidden files" is set to true.
> >>
> >>
> >>
> >> On Tue, May 16, 2017 at 1:29 AM prabhu Mahendran <
> prabhuu161...@gmail.com>
> >> wrote:
> >>>
> >>> I have scheduled getfile processor to 0 sec to track the local folder.
> >>>
> >>> Issue I have faced: PutFile is appending few flowfiles into single
> file.
> >>> Getfile has been configured with keepsourcefile as false. So getfile is
> >>> fetching partial content before putfile writes into local location.
> >>>
> >>> How to handle this issue? Can we lock the file till putfile/custom
> >>> processor completely writes the file and remove lock once completed??
> >
> >
>

How to process files sequentially?

[prabhu Mahendran]

I have approximately 1000 files in local drive.I need to move that files
into SQL Server accordingly one after another.

Since local drive having files like file1.csv,file2.csv,..upto file1000.csv.I
am sure that number of files in local drive may change dynamically.

I can able to created template for move that files into SQL Server.But i
have to process the file2 when file 1 has been completely moved into SQL
Server.

Is this possible in NiFi without using Wait\Notify processor?

can anyone please guide me to solve this?

Re: Nifi Cluster fails to disconnect node when node was killed

[Joe Witt]

Neil,

Want to make sure I understand what you're saying.  What are stating
is a single point of failure?

Thanks
Joe

On Thu, May 18, 2017 at 5:27 PM, Neil Derraugh
 wrote:
> Thanks for the insight Matt.
>
> It's a disaster recovery issue.  It's not something I plan on doing on
> purpose.  It seems it is a single point of failure unfortunately.  I can see
> no other way to resolve the issue other than to blow everything away and
> start a new cluster.
>
> On Thu, May 18, 2017 at 2:49 PM, Matt Gilman 
> wrote:
>>
>> Neil,
>>
>> Disconnecting a node prior to removal is the correct process. It appears
>> that the check was lost going from 0.x to 1.x. Folks reported this JIRA [1]
>> indicating that deleting a connected node did not work. This process does
>> not work because the node needs to be disconnected first. The JIRA was
>> addressed by restoring the check that a node is disconnected prior to
>> deletion.
>>
>> Hopefully the JIRA I filed earlier today [2] will address the phantom node
>> you were seeing. Until then, can you update your workaround to disconnect
>> the node in question prior to deletion?
>>
>> Thanks
>>
>> Matt
>>
>> [1] https://issues.apache.org/jira/browse/NIFI-3295
>> [2] https://issues.apache.org/jira/browse/NIFI-3933
>>
>> On Thu, May 18, 2017 at 12:29 PM, Neil Derraugh
>>  wrote:
>>>
>>> Pretty sure this is the problem I was describing in the "Phantom Node"
>>> thread recently.
>>>
>>> If I kill non-primary nodes the cluster remains healthy despite the lost
>>> nodes.  The terminated nodes end up with a DISCONNECTED status.
>>>
>>> If I kill the primary it winds up with a CONNECTED status, but a new
>>> primary/cluster coordinator gets elected too.
>>>
>>> Additionally it seems in 1.2.0 that the REST API no longer support
>>> deleting a node in a CONNECTED state (Cannot remove Node with ID
>>> 1780fde7-c2f4-469c-9884-fe843eac5b73 because it is not disconnected, current
>>> state = CONNECTED).  So right now I don't have a workaround and have to kill
>>> all the nodes and start over.
>>>
>>> On Thu, May 18, 2017 at 11:20 AM, Mark Payne 
>>> wrote:

 Hello,

 Just looking through this thread now. I believe that I understand the
 problem. I have updated the JIRA with details about what I think is the
 problem and a potential remedy for the problem.

 Thanks
 -Mark

 > On May 18, 2017, at 9:49 AM, Matt Gilman 
 > wrote:
 >
 > Thanks for the additional details. They will be helpful when working
 > the JIRA. All nodes, including the coordinator, heartbeat to the active
 > coordinator. This means that the coordinator effectively heartbeats to
 > itself. It appears, based on your log messages, that this is not 
 > happening.
 > Because no heartbeats were receive from any node, the lack of heartbeats
 > from the terminated node is not considered.
 >
 > Matt
 >
 > Sent from my iPhone
 >
 >> On May 18, 2017, at 8:30 AM, ddewaele  wrote:
 >>
 >> Found something interesting in the centos-b debug logging
 >>
 >> after centos-a (the coordinator) is killed centos-b takes over.
 >> Notice how
 >> it "Will not disconnect any nodes due to lack of heartbeat" and how
 >> it still
 >> sees centos-a as connected despite the fact that there are no
 >> heartbeats
 >> anymore.
 >>
 >> 2017-05-18 12:41:38,010 INFO [Leader Election Notification Thread-2]
 >> o.apache.nifi.controller.FlowController This node elected Active
 >> Cluster
 >> Coordinator
 >> 2017-05-18 12:41:38,010 DEBUG [Leader Election Notification Thread-2]
 >> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor Purging old heartbeats
 >> 2017-05-18 12:41:38,014 INFO [Leader Election Notification Thread-1]
 >> o.apache.nifi.controller.FlowController This node has been elected
 >> Primary
 >> Node
 >> 2017-05-18 12:41:38,353 DEBUG [Heartbeat Monitor Thread-1]
 >> o.a.n.c.c.h.AbstractHeartbeatMonitor Received no new heartbeats. Will
 >> not
 >> disconnect any nodes due to lack of heartbeat
 >> 2017-05-18 12:41:41,336 DEBUG [Process Cluster Protocol Request-3]
 >> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor Received new heartbeat
 >> from
 >> centos-b:8080
 >> 2017-05-18 12:41:41,337 DEBUG [Process Cluster Protocol Request-3]
 >> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor
 >>
 >> Calculated diff between current cluster status and node cluster
 >> status as
 >> follows:
 >> Node: [NodeConnectionStatus[nodeId=centos-b:8080, state=CONNECTED,
 >> updateId=45], NodeConnectionStatus[nodeId=centos-a:8080,
 >> state=CONNECTED,
 >> updateId=42]]
 >> Self: [NodeConnectionStatus[nodeId=centos-b:8080, state=CONNECTED,
 >> updateId=45], 

Re: how to set version for NIFI customized Processor?

[尹文才]

Thanks Bryan and Joe, I managed to set the specific version for my
processor with the properties.

2017-05-18 20:50 GMT+08:00 Bryan Bende :

> As Joe mentioned, the default behavior is to use the Maven group,
> artifact, and version, which will happen by default if you build your
> NAR with the latest NAR Maven plugin (version 1.2.0 at this time).
>
> If you prefer to set different values than the Maven fields, you can
> override them by specifying the following properties in your NAR's
> pom.xml:
>
> 
>org.apache.nifi.overridden
>nifi-overridden-test-nar
>2.0
>  
>
> Again, you only need to do this if you want your NAR to show up in
> NiFi differently than your Maven project.
>
>
> On Thu, May 18, 2017 at 8:05 AM, Joe Witt  wrote:
> > Just rebuild it with the latest nifi nar maven plugin and it will get
> > the version info at that time.
> >
> > thanks
> >
> > On Thu, May 18, 2017 at 4:20 AM, 尹文才  wrote:
> >> Thanks Joe, is it possible to set a specific version for a customized
> NIFI
> >> processor and show the version in that processor selection dialog?
> >>
> >> 2017-05-18 10:42 GMT+08:00 Joe Witt :
> >>>
> >>> Hello
> >>>
> >>> They will remain unversioned until they are built with the latest nar
> >>> plugin.  This is described briefly in the migration guidance [1].
> >>>
> >>> For anything built with the older nifi nar plugin the resulting nar
> >>> will not have suffiicient bundle data for the framework to have it
> >>> support component versioning so that is why it shows as unversioned.
> >>>
> >>> [1] https://cwiki.apache.org/confluence/display/NIFI/
> Migration+Guidance
> >>>
> >>> Thanks
> >>> Joe
> >>>
> >>> On Wed, May 17, 2017 at 10:37 PM, 尹文才  wrote:
> >>> > Hi guys, I have installed NIFI 1.2 and have played with it for a
> while.
> >>> > One thing I noticed for 1.2 is that when I placed my previously
> written
> >>> > customized Processor
> >>> > into 1.2, I could see in the Processor select dialog there's a
> Version
> >>> > field
> >>> > for each processor
> >>> > and the value for my processor is unversioned. So does anyone know
> how
> >>> > to
> >>> > set version for
> >>> > customized processor? Thanks.
> >>
> >>
>

Re: Nifi Cluster fails to disconnect node when node was killed

[Neil Derraugh]

Thanks for the insight Matt.

It's a disaster recovery issue.  It's not something I plan on doing on
purpose.  It seems it is a single point of failure unfortunately.  I can
see no other way to resolve the issue other than to blow everything away
and start a new cluster.

On Thu, May 18, 2017 at 2:49 PM, Matt Gilman 
wrote:

> Neil,
>
> Disconnecting a node prior to removal is the correct process. It appears
> that the check was lost going from 0.x to 1.x. Folks reported this JIRA [1]
> indicating that deleting a connected node did not work. This process does
> not work because the node needs to be disconnected first. The JIRA was
> addressed by restoring the check that a node is disconnected prior to
> deletion.
>
> Hopefully the JIRA I filed earlier today [2] will address the phantom node
> you were seeing. Until then, can you update your workaround to disconnect
> the node in question prior to deletion?
>
> Thanks
>
> Matt
>
> [1] https://issues.apache.org/jira/browse/NIFI-3295
> [2] https://issues.apache.org/jira/browse/NIFI-3933
>
> On Thu, May 18, 2017 at 12:29 PM, Neil Derraugh  intellifylearning.com> wrote:
>
>> Pretty sure this is the problem I was describing in the "Phantom Node"
>> thread recently.
>>
>> If I kill non-primary nodes the cluster remains healthy despite the lost
>> nodes.  The terminated nodes end up with a DISCONNECTED status.
>>
>> If I kill the primary it winds up with a CONNECTED status, but a new
>> primary/cluster coordinator gets elected too.
>>
>> Additionally it seems in 1.2.0 that the REST API no longer support
>> deleting a node in a CONNECTED state (Cannot remove Node with ID
>> 1780fde7-c2f4-469c-9884-fe843eac5b73 because it is not disconnected,
>> current state = CONNECTED).  So right now I don't have a workaround and
>> have to kill all the nodes and start over.
>>
>> On Thu, May 18, 2017 at 11:20 AM, Mark Payne 
>> wrote:
>>
>>> Hello,
>>>
>>> Just looking through this thread now. I believe that I understand the
>>> problem. I have updated the JIRA with details about what I think is the
>>> problem and a potential remedy for the problem.
>>>
>>> Thanks
>>> -Mark
>>>
>>> > On May 18, 2017, at 9:49 AM, Matt Gilman 
>>> wrote:
>>> >
>>> > Thanks for the additional details. They will be helpful when working
>>> the JIRA. All nodes, including the coordinator, heartbeat to the active
>>> coordinator. This means that the coordinator effectively heartbeats to
>>> itself. It appears, based on your log messages, that this is not happening.
>>> Because no heartbeats were receive from any node, the lack of heartbeats
>>> from the terminated node is not considered.
>>> >
>>> > Matt
>>> >
>>> > Sent from my iPhone
>>> >
>>> >> On May 18, 2017, at 8:30 AM, ddewaele  wrote:
>>> >>
>>> >> Found something interesting in the centos-b debug logging
>>> >>
>>> >> after centos-a (the coordinator) is killed centos-b takes over.
>>> Notice how
>>> >> it "Will not disconnect any nodes due to lack of heartbeat" and how
>>> it still
>>> >> sees centos-a as connected despite the fact that there are no
>>> heartbeats
>>> >> anymore.
>>> >>
>>> >> 2017-05-18 12:41:38,010 INFO [Leader Election Notification Thread-2]
>>> >> o.apache.nifi.controller.FlowController This node elected Active
>>> Cluster
>>> >> Coordinator
>>> >> 2017-05-18 12:41:38,010 DEBUG [Leader Election Notification Thread-2]
>>> >> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor Purging old heartbeats
>>> >> 2017-05-18 12:41:38,014 INFO [Leader Election Notification Thread-1]
>>> >> o.apache.nifi.controller.FlowController This node has been elected
>>> Primary
>>> >> Node
>>> >> 2017-05-18 12:41:38,353 DEBUG [Heartbeat Monitor Thread-1]
>>> >> o.a.n.c.c.h.AbstractHeartbeatMonitor Received no new heartbeats.
>>> Will not
>>> >> disconnect any nodes due to lack of heartbeat
>>> >> 2017-05-18 12:41:41,336 DEBUG [Process Cluster Protocol Request-3]
>>> >> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor Received new heartbeat
>>> from
>>> >> centos-b:8080
>>> >> 2017-05-18 12:41:41,337 DEBUG [Process Cluster Protocol Request-3]
>>> >> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor
>>> >>
>>> >> Calculated diff between current cluster status and node cluster
>>> status as
>>> >> follows:
>>> >> Node: [NodeConnectionStatus[nodeId=centos-b:8080, state=CONNECTED,
>>> >> updateId=45], NodeConnectionStatus[nodeId=centos-a:8080,
>>> state=CONNECTED,
>>> >> updateId=42]]
>>> >> Self: [NodeConnectionStatus[nodeId=centos-b:8080, state=CONNECTED,
>>> >> updateId=45], NodeConnectionStatus[nodeId=centos-a:8080,
>>> state=CONNECTED,
>>> >> updateId=42]]
>>> >> Difference: []
>>> >>
>>> >>
>>> >> 2017-05-18 12:41:41,337 INFO [Process Cluster Protocol Request-3]
>>> >> o.a.n.c.p.impl.SocketProtocolListener Finished processing request
>>> >> 410e7db5-8bb0-4f97-8ee8-fc8647c54959 (type=HEARTBEAT, length=2341
>>> bytes)
>>> >> from centos-b:8080 in 3 

Re: Nifi Cluster fails to disconnect node when node was killed

[Matt Gilman]

Neil,

Disconnecting a node prior to removal is the correct process. It appears
that the check was lost going from 0.x to 1.x. Folks reported this JIRA [1]
indicating that deleting a connected node did not work. This process does
not work because the node needs to be disconnected first. The JIRA was
addressed by restoring the check that a node is disconnected prior to
deletion.

Hopefully the JIRA I filed earlier today [2] will address the phantom node
you were seeing. Until then, can you update your workaround to disconnect
the node in question prior to deletion?

Thanks

Matt

[1] https://issues.apache.org/jira/browse/NIFI-3295
[2] https://issues.apache.org/jira/browse/NIFI-3933

On Thu, May 18, 2017 at 12:29 PM, Neil Derraugh <
neil.derra...@intellifylearning.com> wrote:

> Pretty sure this is the problem I was describing in the "Phantom Node"
> thread recently.
>
> If I kill non-primary nodes the cluster remains healthy despite the lost
> nodes.  The terminated nodes end up with a DISCONNECTED status.
>
> If I kill the primary it winds up with a CONNECTED status, but a new
> primary/cluster coordinator gets elected too.
>
> Additionally it seems in 1.2.0 that the REST API no longer support
> deleting a node in a CONNECTED state (Cannot remove Node with ID
> 1780fde7-c2f4-469c-9884-fe843eac5b73 because it is not disconnected,
> current state = CONNECTED).  So right now I don't have a workaround and
> have to kill all the nodes and start over.
>
> On Thu, May 18, 2017 at 11:20 AM, Mark Payne  wrote:
>
>> Hello,
>>
>> Just looking through this thread now. I believe that I understand the
>> problem. I have updated the JIRA with details about what I think is the
>> problem and a potential remedy for the problem.
>>
>> Thanks
>> -Mark
>>
>> > On May 18, 2017, at 9:49 AM, Matt Gilman 
>> wrote:
>> >
>> > Thanks for the additional details. They will be helpful when working
>> the JIRA. All nodes, including the coordinator, heartbeat to the active
>> coordinator. This means that the coordinator effectively heartbeats to
>> itself. It appears, based on your log messages, that this is not happening.
>> Because no heartbeats were receive from any node, the lack of heartbeats
>> from the terminated node is not considered.
>> >
>> > Matt
>> >
>> > Sent from my iPhone
>> >
>> >> On May 18, 2017, at 8:30 AM, ddewaele  wrote:
>> >>
>> >> Found something interesting in the centos-b debug logging
>> >>
>> >> after centos-a (the coordinator) is killed centos-b takes over. Notice
>> how
>> >> it "Will not disconnect any nodes due to lack of heartbeat" and how it
>> still
>> >> sees centos-a as connected despite the fact that there are no
>> heartbeats
>> >> anymore.
>> >>
>> >> 2017-05-18 12:41:38,010 INFO [Leader Election Notification Thread-2]
>> >> o.apache.nifi.controller.FlowController This node elected Active
>> Cluster
>> >> Coordinator
>> >> 2017-05-18 12:41:38,010 DEBUG [Leader Election Notification Thread-2]
>> >> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor Purging old heartbeats
>> >> 2017-05-18 12:41:38,014 INFO [Leader Election Notification Thread-1]
>> >> o.apache.nifi.controller.FlowController This node has been elected
>> Primary
>> >> Node
>> >> 2017-05-18 12:41:38,353 DEBUG [Heartbeat Monitor Thread-1]
>> >> o.a.n.c.c.h.AbstractHeartbeatMonitor Received no new heartbeats. Will
>> not
>> >> disconnect any nodes due to lack of heartbeat
>> >> 2017-05-18 12:41:41,336 DEBUG [Process Cluster Protocol Request-3]
>> >> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor Received new heartbeat
>> from
>> >> centos-b:8080
>> >> 2017-05-18 12:41:41,337 DEBUG [Process Cluster Protocol Request-3]
>> >> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor
>> >>
>> >> Calculated diff between current cluster status and node cluster status
>> as
>> >> follows:
>> >> Node: [NodeConnectionStatus[nodeId=centos-b:8080, state=CONNECTED,
>> >> updateId=45], NodeConnectionStatus[nodeId=centos-a:8080,
>> state=CONNECTED,
>> >> updateId=42]]
>> >> Self: [NodeConnectionStatus[nodeId=centos-b:8080, state=CONNECTED,
>> >> updateId=45], NodeConnectionStatus[nodeId=centos-a:8080,
>> state=CONNECTED,
>> >> updateId=42]]
>> >> Difference: []
>> >>
>> >>
>> >> 2017-05-18 12:41:41,337 INFO [Process Cluster Protocol Request-3]
>> >> o.a.n.c.p.impl.SocketProtocolListener Finished processing request
>> >> 410e7db5-8bb0-4f97-8ee8-fc8647c54959 (type=HEARTBEAT, length=2341
>> bytes)
>> >> from centos-b:8080 in 3 millis
>> >> 2017-05-18 12:41:41,339 INFO [Clustering Tasks Thread-2]
>> >> o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2017-05-18
>> >> 12:41:41,330 and sent to centos-b:10001 at 2017-05-18 12:41:41,339;
>> send
>> >> took 8 millis
>> >> 2017-05-18 12:41:43,354 INFO [Heartbeat Monitor Thread-1]
>> >> o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 1 heartbeats
>> in
>> >> 93276 nanos
>> >> 2017-05-18 12:41:46,346 DEBUG [Process Cluster Protocol Request-4]
>> 

Re: Nifi Cluster fails to disconnect node when node was killed

[Neil Derraugh]

Pretty sure this is the problem I was describing in the "Phantom Node"
thread recently.

If I kill non-primary nodes the cluster remains healthy despite the lost
nodes.  The terminated nodes end up with a DISCONNECTED status.

If I kill the primary it winds up with a CONNECTED status, but a new
primary/cluster coordinator gets elected too.

Additionally it seems in 1.2.0 that the REST API no longer support deleting
a node in a CONNECTED state (Cannot remove Node with ID
1780fde7-c2f4-469c-9884-fe843eac5b73 because it is not disconnected,
current state = CONNECTED).  So right now I don't have a workaround and
have to kill all the nodes and start over.

On Thu, May 18, 2017 at 11:20 AM, Mark Payne  wrote:

> Hello,
>
> Just looking through this thread now. I believe that I understand the
> problem. I have updated the JIRA with details about what I think is the
> problem and a potential remedy for the problem.
>
> Thanks
> -Mark
>
> > On May 18, 2017, at 9:49 AM, Matt Gilman 
> wrote:
> >
> > Thanks for the additional details. They will be helpful when working the
> JIRA. All nodes, including the coordinator, heartbeat to the active
> coordinator. This means that the coordinator effectively heartbeats to
> itself. It appears, based on your log messages, that this is not happening.
> Because no heartbeats were receive from any node, the lack of heartbeats
> from the terminated node is not considered.
> >
> > Matt
> >
> > Sent from my iPhone
> >
> >> On May 18, 2017, at 8:30 AM, ddewaele  wrote:
> >>
> >> Found something interesting in the centos-b debug logging
> >>
> >> after centos-a (the coordinator) is killed centos-b takes over. Notice
> how
> >> it "Will not disconnect any nodes due to lack of heartbeat" and how it
> still
> >> sees centos-a as connected despite the fact that there are no heartbeats
> >> anymore.
> >>
> >> 2017-05-18 12:41:38,010 INFO [Leader Election Notification Thread-2]
> >> o.apache.nifi.controller.FlowController This node elected Active
> Cluster
> >> Coordinator
> >> 2017-05-18 12:41:38,010 DEBUG [Leader Election Notification Thread-2]
> >> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor Purging old heartbeats
> >> 2017-05-18 12:41:38,014 INFO [Leader Election Notification Thread-1]
> >> o.apache.nifi.controller.FlowController This node has been elected
> Primary
> >> Node
> >> 2017-05-18 12:41:38,353 DEBUG [Heartbeat Monitor Thread-1]
> >> o.a.n.c.c.h.AbstractHeartbeatMonitor Received no new heartbeats. Will
> not
> >> disconnect any nodes due to lack of heartbeat
> >> 2017-05-18 12:41:41,336 DEBUG [Process Cluster Protocol Request-3]
> >> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor Received new heartbeat from
> >> centos-b:8080
> >> 2017-05-18 12:41:41,337 DEBUG [Process Cluster Protocol Request-3]
> >> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor
> >>
> >> Calculated diff between current cluster status and node cluster status
> as
> >> follows:
> >> Node: [NodeConnectionStatus[nodeId=centos-b:8080, state=CONNECTED,
> >> updateId=45], NodeConnectionStatus[nodeId=centos-a:8080,
> state=CONNECTED,
> >> updateId=42]]
> >> Self: [NodeConnectionStatus[nodeId=centos-b:8080, state=CONNECTED,
> >> updateId=45], NodeConnectionStatus[nodeId=centos-a:8080,
> state=CONNECTED,
> >> updateId=42]]
> >> Difference: []
> >>
> >>
> >> 2017-05-18 12:41:41,337 INFO [Process Cluster Protocol Request-3]
> >> o.a.n.c.p.impl.SocketProtocolListener Finished processing request
> >> 410e7db5-8bb0-4f97-8ee8-fc8647c54959 (type=HEARTBEAT, length=2341
> bytes)
> >> from centos-b:8080 in 3 millis
> >> 2017-05-18 12:41:41,339 INFO [Clustering Tasks Thread-2]
> >> o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2017-05-18
> >> 12:41:41,330 and sent to centos-b:10001 at 2017-05-18 12:41:41,339; send
> >> took 8 millis
> >> 2017-05-18 12:41:43,354 INFO [Heartbeat Monitor Thread-1]
> >> o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 1 heartbeats
> in
> >> 93276 nanos
> >> 2017-05-18 12:41:46,346 DEBUG [Process Cluster Protocol Request-4]
> >> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor Received new heartbeat from
> >> centos-b:8080
> >> 2017-05-18 12:41:46,346 DEBUG [Process Cluster Protocol Request-4]
> >> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor
> >>
> >> Calculated diff between current cluster status and node cluster status
> as
> >> follows:
> >> Node: [NodeConnectionStatus[nodeId=centos-b:8080, state=CONNECTED,
> >> updateId=45], NodeConnectionStatus[nodeId=centos-a:8080,
> state=CONNECTED,
> >> updateId=42]]
> >> Self: [NodeConnectionStatus[nodeId=centos-b:8080, state=CONNECTED,
> >> updateId=45], NodeConnectionStatus[nodeId=centos-a:8080,
> state=CONNECTED,
> >> updateId=42]]
> >> Difference: []
> >>
> >>
> >>
> >>
> >> --
> >> View this message in context: http://apache-nifi-users-list.
> 2361937.n4.nabble.com/Nifi-Cluster-fails-to-disconnect-
> node-when-node-was-killed-tp1942p1950.html
> >> Sent from the Apache NiFi 

Re: [EXT] Parsing Email Attachments

[Nick Carenza]

@pwicks no luck on escaping \n. Thanks for the suggestion.

I even tried hardcoding the boundary value, in case it had something to do
with using nifi expressions in regex but that didn't work either.

On Wed, May 17, 2017 at 10:54 PM, Peter Wicks (pwicks) 
wrote:

> Nick,
>
>
>
> Try escaping your \n’s, see if that helps.
>
>
>
> (?s)(.*\\n\\n${boundary}\\nContent-Type: text\/plain;
> charset="UTF-8"\\n\\n)(.*?)(\\n\\n${boundary}.*)
>
>
>
> *From:* Nick Carenza [mailto:nick.care...@thecontrolgroup.com]
> *Sent:* Thursday, May 18, 2017 11:27 AM
> *To:* users@nifi.apache.org
> *Subject:* [EXT] Parsing Email Attachments
>
>
>
> Hey Nifi-ers,
>
>
>
> I haven't been having any luck trying to parse email after consuming them
> with pop3.
>
>
>
> I am composing a simple message with gmail with just plain text and it
> comes out like this (with many headers removed):
>
>
>
> Delivered-To: sl...@company.com
>
> Return-Path: 
>
> MIME-Version: 1.0
>
> Received: by 0.0.0.0 with HTTP; Tue, 16 May 2017 17:54:04 -0700 (PDT)
>
> From: User 
>
> Date: Tue, 16 May 2017 17:54:04 -0700
>
> Subject: test subject
>
> To: em...@company.com
>
> Content-Type: multipart/alternative; boundary="
> f403045f83d499711a054fadb980"
>
>
>
> --f403045f83d499711a054fadb980
>
> Content-Type: text/plain; charset="UTF-8"
>
>
>
> test email body
>
>
>
> --f403045f83d499711a054fadb980
>
> Content-Type: text/html; charset="UTF-8"
>
>
>
> test email body
>
>
>
> --f403045f83d499711a054fadb980--
>
>
>
> I just want the email body and ExtractEmailAttachments doesn't seem to
> extract the parts between the boundaries like I hoped it would.
>
>
>
> So instead I use ExtractEmailHeaders and additionally extract the
> Content-Type header which I then retrieve just the boundary value with an
> UpdateAttribute processor configure like:
>
>
>
> boundary: ${email.headers.content-type:substringAfter('boundary="'):
> substringBefore('"'):prepend('--')}
>
>
>
> Then I wrote a sweet regex for ReplaceText to clean this up:
>
>
>
> (?s)(.*\n\n${boundary}\nContent-Type: text\/plain;
> charset="UTF-8"\n\n)(.*?)(\n\n${boundary}.*)
>
>
>
> [image: Inline image 1]
>
>
>
> ... but even though this works in regex testers and sublimetext, it seems
> to have no effect in my flow.
>
>
>
> Anyone have any insight on this?
>
>
>
> Thanks,
>
> Nick
>

Re: Nifi Cluster fails to disconnect node when node was killed

[Mark Payne]

Hello,

Just looking through this thread now. I believe that I understand the problem. 
I have updated the JIRA with details about what I think is the problem and a 
potential remedy for the problem.

Thanks
-Mark

> On May 18, 2017, at 9:49 AM, Matt Gilman  wrote:
> 
> Thanks for the additional details. They will be helpful when working the 
> JIRA. All nodes, including the coordinator, heartbeat to the active 
> coordinator. This means that the coordinator effectively heartbeats to 
> itself. It appears, based on your log messages, that this is not happening. 
> Because no heartbeats were receive from any node, the lack of heartbeats from 
> the terminated node is not considered.
> 
> Matt
> 
> Sent from my iPhone
> 
>> On May 18, 2017, at 8:30 AM, ddewaele  wrote:
>> 
>> Found something interesting in the centos-b debug logging 
>> 
>> after centos-a (the coordinator) is killed centos-b takes over. Notice how
>> it "Will not disconnect any nodes due to lack of heartbeat" and how it still
>> sees centos-a as connected despite the fact that there are no heartbeats
>> anymore.
>> 
>> 2017-05-18 12:41:38,010 INFO [Leader Election Notification Thread-2]
>> o.apache.nifi.controller.FlowController This node elected Active Cluster
>> Coordinator
>> 2017-05-18 12:41:38,010 DEBUG [Leader Election Notification Thread-2]
>> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor Purging old heartbeats
>> 2017-05-18 12:41:38,014 INFO [Leader Election Notification Thread-1]
>> o.apache.nifi.controller.FlowController This node has been elected Primary
>> Node
>> 2017-05-18 12:41:38,353 DEBUG [Heartbeat Monitor Thread-1]
>> o.a.n.c.c.h.AbstractHeartbeatMonitor Received no new heartbeats. Will not
>> disconnect any nodes due to lack of heartbeat
>> 2017-05-18 12:41:41,336 DEBUG [Process Cluster Protocol Request-3]
>> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor Received new heartbeat from
>> centos-b:8080
>> 2017-05-18 12:41:41,337 DEBUG [Process Cluster Protocol Request-3]
>> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor 
>> 
>> Calculated diff between current cluster status and node cluster status as
>> follows:
>> Node: [NodeConnectionStatus[nodeId=centos-b:8080, state=CONNECTED,
>> updateId=45], NodeConnectionStatus[nodeId=centos-a:8080, state=CONNECTED,
>> updateId=42]]
>> Self: [NodeConnectionStatus[nodeId=centos-b:8080, state=CONNECTED,
>> updateId=45], NodeConnectionStatus[nodeId=centos-a:8080, state=CONNECTED,
>> updateId=42]]
>> Difference: []
>> 
>> 
>> 2017-05-18 12:41:41,337 INFO [Process Cluster Protocol Request-3]
>> o.a.n.c.p.impl.SocketProtocolListener Finished processing request
>> 410e7db5-8bb0-4f97-8ee8-fc8647c54959 (type=HEARTBEAT, length=2341 bytes)
>> from centos-b:8080 in 3 millis
>> 2017-05-18 12:41:41,339 INFO [Clustering Tasks Thread-2]
>> o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2017-05-18
>> 12:41:41,330 and sent to centos-b:10001 at 2017-05-18 12:41:41,339; send
>> took 8 millis
>> 2017-05-18 12:41:43,354 INFO [Heartbeat Monitor Thread-1]
>> o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 1 heartbeats in
>> 93276 nanos
>> 2017-05-18 12:41:46,346 DEBUG [Process Cluster Protocol Request-4]
>> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor Received new heartbeat from
>> centos-b:8080
>> 2017-05-18 12:41:46,346 DEBUG [Process Cluster Protocol Request-4]
>> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor 
>> 
>> Calculated diff between current cluster status and node cluster status as
>> follows:
>> Node: [NodeConnectionStatus[nodeId=centos-b:8080, state=CONNECTED,
>> updateId=45], NodeConnectionStatus[nodeId=centos-a:8080, state=CONNECTED,
>> updateId=42]]
>> Self: [NodeConnectionStatus[nodeId=centos-b:8080, state=CONNECTED,
>> updateId=45], NodeConnectionStatus[nodeId=centos-a:8080, state=CONNECTED,
>> updateId=42]]
>> Difference: []
>> 
>> 
>> 
>> 
>> --
>> View this message in context: 
>> http://apache-nifi-users-list.2361937.n4.nabble.com/Nifi-Cluster-fails-to-disconnect-node-when-node-was-killed-tp1942p1950.html
>> Sent from the Apache NiFi Users List mailing list archive at Nabble.com.

Re: Archive selected "feeds"

[Juan Sequeiros]

Thank you.

I've created this JIRA feature request:

https://issues.apache.org/jira/browse/NIFI-3934

On Wed, May 17, 2017 at 12:51 PM Joe Witt  wrote:

> The concept of being able to provide hints to the underlying content
> repository about which content is more important to keep longer than
> others is a good one.  I'd recommend filing a JIRA for the ask and
> providing your proposed concept as one to consider.
>
> Definitely see something like this being useful but I suspect we'll
> want to consider alternative approaches like having such hints about
> retention be placed on the processors that create/modify/touch the
> data because when you consider that most flows are graphs and
> non-linear the decisions about retention become more complex.
>
> Thanks
>
> On Wed, May 17, 2017 at 12:40 PM, Juan Sequeiros 
> wrote:
> > All,
> >
> > On latest NIFI can we archive only specific FlowFiles based on an
> attribute?
> > I can put in a feature request if not.
> >
> > For instance:
> >
> > archive.max.retention.period=12 hours
> > archive.max.retention.attributes.period=5 days
> >
> archive.retention.attributes=${some.special.attribute.for.my.longer.retention}
> >
> > Then everything is archived for 12 hours unless the flowFile is set with
> an
> > attribute that would be configured through
> > archive.max.retention.attributes.period
> >
> > And have archive.max.usage.percentage=50% ( supersede everything else )
> >
> > Thank you
>

Re: Nifi Cluster fails to disconnect node when node was killed

[ddewaele]

Hi,
 
Just wanted to point out that the newly appointed coordinator (centos-b)
does end up sending heartbeats to itself as you described. 

2017-05-18 12:41:41,336 DEBUG [Process Cluster Protocol Request-3]
o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor Received new heartbeat from
centos-b:8080

It seems heartbeats are purged when a new coordinator is selected.

https://github.com/apache/nifi/blob/b73ba7f8d4f6319881c26b8faad121ceb12041ab/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/heartbeat/ClusterProtocolHeartbeatMonitor.java#L136

And disconnecting nodes can only be done based on existing heartbeats.

https://github.com/apache/nifi/blob/d838f61291d2582592754a37314911b701c6891b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/heartbeat/AbstractHeartbeatMonitor.java#L162

As the centos-a heartbeats were purged, centos-a never gets disconnected.




--
View this message in context: 
http://apache-nifi-users-list.2361937.n4.nabble.com/Nifi-Cluster-fails-to-disconnect-node-when-node-was-killed-tp1942p1954.html
Sent from the Apache NiFi Users List mailing list archive at Nabble.com.

Re: Nifi Cluster fails to disconnect node when node was killed

[Matt Gilman]

Thanks for the additional details. They will be helpful when working the JIRA. 
All nodes, including the coordinator, heartbeat to the active coordinator. This 
means that the coordinator effectively heartbeats to itself. It appears, based 
on your log messages, that this is not happening. Because no heartbeats were 
receive from any node, the lack of heartbeats from the terminated node is not 
considered.

Matt

Sent from my iPhone

> On May 18, 2017, at 8:30 AM, ddewaele  wrote:
> 
> Found something interesting in the centos-b debug logging 
> 
> after centos-a (the coordinator) is killed centos-b takes over. Notice how
> it "Will not disconnect any nodes due to lack of heartbeat" and how it still
> sees centos-a as connected despite the fact that there are no heartbeats
> anymore.
> 
> 2017-05-18 12:41:38,010 INFO [Leader Election Notification Thread-2]
> o.apache.nifi.controller.FlowController This node elected Active Cluster
> Coordinator
> 2017-05-18 12:41:38,010 DEBUG [Leader Election Notification Thread-2]
> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor Purging old heartbeats
> 2017-05-18 12:41:38,014 INFO [Leader Election Notification Thread-1]
> o.apache.nifi.controller.FlowController This node has been elected Primary
> Node
> 2017-05-18 12:41:38,353 DEBUG [Heartbeat Monitor Thread-1]
> o.a.n.c.c.h.AbstractHeartbeatMonitor Received no new heartbeats. Will not
> disconnect any nodes due to lack of heartbeat
> 2017-05-18 12:41:41,336 DEBUG [Process Cluster Protocol Request-3]
> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor Received new heartbeat from
> centos-b:8080
> 2017-05-18 12:41:41,337 DEBUG [Process Cluster Protocol Request-3]
> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor 
> 
> Calculated diff between current cluster status and node cluster status as
> follows:
> Node: [NodeConnectionStatus[nodeId=centos-b:8080, state=CONNECTED,
> updateId=45], NodeConnectionStatus[nodeId=centos-a:8080, state=CONNECTED,
> updateId=42]]
> Self: [NodeConnectionStatus[nodeId=centos-b:8080, state=CONNECTED,
> updateId=45], NodeConnectionStatus[nodeId=centos-a:8080, state=CONNECTED,
> updateId=42]]
> Difference: []
> 
> 
> 2017-05-18 12:41:41,337 INFO [Process Cluster Protocol Request-3]
> o.a.n.c.p.impl.SocketProtocolListener Finished processing request
> 410e7db5-8bb0-4f97-8ee8-fc8647c54959 (type=HEARTBEAT, length=2341 bytes)
> from centos-b:8080 in 3 millis
> 2017-05-18 12:41:41,339 INFO [Clustering Tasks Thread-2]
> o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2017-05-18
> 12:41:41,330 and sent to centos-b:10001 at 2017-05-18 12:41:41,339; send
> took 8 millis
> 2017-05-18 12:41:43,354 INFO [Heartbeat Monitor Thread-1]
> o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 1 heartbeats in
> 93276 nanos
> 2017-05-18 12:41:46,346 DEBUG [Process Cluster Protocol Request-4]
> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor Received new heartbeat from
> centos-b:8080
> 2017-05-18 12:41:46,346 DEBUG [Process Cluster Protocol Request-4]
> o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor 
> 
> Calculated diff between current cluster status and node cluster status as
> follows:
> Node: [NodeConnectionStatus[nodeId=centos-b:8080, state=CONNECTED,
> updateId=45], NodeConnectionStatus[nodeId=centos-a:8080, state=CONNECTED,
> updateId=42]]
> Self: [NodeConnectionStatus[nodeId=centos-b:8080, state=CONNECTED,
> updateId=45], NodeConnectionStatus[nodeId=centos-a:8080, state=CONNECTED,
> updateId=42]]
> Difference: []
> 
> 
> 
> 
> --
> View this message in context: 
> http://apache-nifi-users-list.2361937.n4.nabble.com/Nifi-Cluster-fails-to-disconnect-node-when-node-was-killed-tp1942p1950.html
> Sent from the Apache NiFi Users List mailing list archive at Nabble.com.

Re: how to set version for NIFI customized Processor?

[Bryan Bende]

As Joe mentioned, the default behavior is to use the Maven group,
artifact, and version, which will happen by default if you build your
NAR with the latest NAR Maven plugin (version 1.2.0 at this time).

If you prefer to set different values than the Maven fields, you can
override them by specifying the following properties in your NAR's
pom.xml:


   org.apache.nifi.overridden
   nifi-overridden-test-nar
   2.0
 

Again, you only need to do this if you want your NAR to show up in
NiFi differently than your Maven project.


On Thu, May 18, 2017 at 8:05 AM, Joe Witt  wrote:
> Just rebuild it with the latest nifi nar maven plugin and it will get
> the version info at that time.
>
> thanks
>
> On Thu, May 18, 2017 at 4:20 AM, 尹文才  wrote:
>> Thanks Joe, is it possible to set a specific version for a customized NIFI
>> processor and show the version in that processor selection dialog?
>>
>> 2017-05-18 10:42 GMT+08:00 Joe Witt :
>>>
>>> Hello
>>>
>>> They will remain unversioned until they are built with the latest nar
>>> plugin.  This is described briefly in the migration guidance [1].
>>>
>>> For anything built with the older nifi nar plugin the resulting nar
>>> will not have suffiicient bundle data for the framework to have it
>>> support component versioning so that is why it shows as unversioned.
>>>
>>> [1] https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance
>>>
>>> Thanks
>>> Joe
>>>
>>> On Wed, May 17, 2017 at 10:37 PM, 尹文才  wrote:
>>> > Hi guys, I have installed NIFI 1.2 and have played with it for a while.
>>> > One thing I noticed for 1.2 is that when I placed my previously written
>>> > customized Processor
>>> > into 1.2, I could see in the Processor select dialog there's a Version
>>> > field
>>> > for each processor
>>> > and the value for my processor is unversioned. So does anyone know how
>>> > to
>>> > set version for
>>> > customized processor? Thanks.
>>
>>

Re: Nifi Cluster fails to disconnect node when node was killed

[ddewaele]

Found something interesting in the centos-b debug logging 

after centos-a (the coordinator) is killed centos-b takes over. Notice how
it "Will not disconnect any nodes due to lack of heartbeat" and how it still
sees centos-a as connected despite the fact that there are no heartbeats
anymore.

2017-05-18 12:41:38,010 INFO [Leader Election Notification Thread-2]
o.apache.nifi.controller.FlowController This node elected Active Cluster
Coordinator
2017-05-18 12:41:38,010 DEBUG [Leader Election Notification Thread-2]
o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor Purging old heartbeats
2017-05-18 12:41:38,014 INFO [Leader Election Notification Thread-1]
o.apache.nifi.controller.FlowController This node has been elected Primary
Node
2017-05-18 12:41:38,353 DEBUG [Heartbeat Monitor Thread-1]
o.a.n.c.c.h.AbstractHeartbeatMonitor Received no new heartbeats. Will not
disconnect any nodes due to lack of heartbeat
2017-05-18 12:41:41,336 DEBUG [Process Cluster Protocol Request-3]
o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor Received new heartbeat from
centos-b:8080
2017-05-18 12:41:41,337 DEBUG [Process Cluster Protocol Request-3]
o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor 

Calculated diff between current cluster status and node cluster status as
follows:
Node: [NodeConnectionStatus[nodeId=centos-b:8080, state=CONNECTED,
updateId=45], NodeConnectionStatus[nodeId=centos-a:8080, state=CONNECTED,
updateId=42]]
Self: [NodeConnectionStatus[nodeId=centos-b:8080, state=CONNECTED,
updateId=45], NodeConnectionStatus[nodeId=centos-a:8080, state=CONNECTED,
updateId=42]]
Difference: []


2017-05-18 12:41:41,337 INFO [Process Cluster Protocol Request-3]
o.a.n.c.p.impl.SocketProtocolListener Finished processing request
410e7db5-8bb0-4f97-8ee8-fc8647c54959 (type=HEARTBEAT, length=2341 bytes)
from centos-b:8080 in 3 millis
2017-05-18 12:41:41,339 INFO [Clustering Tasks Thread-2]
o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2017-05-18
12:41:41,330 and sent to centos-b:10001 at 2017-05-18 12:41:41,339; send
took 8 millis
2017-05-18 12:41:43,354 INFO [Heartbeat Monitor Thread-1]
o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 1 heartbeats in
93276 nanos
2017-05-18 12:41:46,346 DEBUG [Process Cluster Protocol Request-4]
o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor Received new heartbeat from
centos-b:8080
2017-05-18 12:41:46,346 DEBUG [Process Cluster Protocol Request-4]
o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor 

Calculated diff between current cluster status and node cluster status as
follows:
Node: [NodeConnectionStatus[nodeId=centos-b:8080, state=CONNECTED,
updateId=45], NodeConnectionStatus[nodeId=centos-a:8080, state=CONNECTED,
updateId=42]]
Self: [NodeConnectionStatus[nodeId=centos-b:8080, state=CONNECTED,
updateId=45], NodeConnectionStatus[nodeId=centos-a:8080, state=CONNECTED,
updateId=42]]
Difference: []




--
View this message in context: 
http://apache-nifi-users-list.2361937.n4.nabble.com/Nifi-Cluster-fails-to-disconnect-node-when-node-was-killed-tp1942p1950.html
Sent from the Apache NiFi Users List mailing list archive at Nabble.com.

Re: Nifi Cluster fails to disconnect node when node was killed

[Matt Gilman]

Thank you for the confirmation. I've filed this JIRA [1] to track the issue.

Matt

[1] https://issues.apache.org/jira/browse/NIFI-3933

On Thu, May 18, 2017 at 8:21 AM, ddewaele  wrote:

> Thanks for the response.
>
> When killing a non-coordinator node, it does take 8 * 5 seconds before I
> see
> this :
>
> nifi-app.log:2017-05-18 12:04:29,644 INFO [Heartbeat Monitor Thread-1]
> o.a.n.c.c.node.NodeClusterCoordinator Status of centos-b:8080 changed from
> NodeConnectionStatus[nodeId=centos-b:8080, state=CONNECTED, updateId=26]
> to
> NodeConnectionStatus[nodeId=centos-b:8080, state=DISCONNECTED, Disconnect
> Code=Lack of Heartbeat, Disconnect Reason=Have not received a heartbeat
> from
> node in 43 seconds, updateId=27]
>
> When killing the coordinator node, the newly appointed coordinator doesn't
> seem to detect the heartbeat timeout.
>
> I'll see if I can enable the debug logging.
>
> My Nifi runs inside a KVM. KVM includes 3 seperate VMs. External zookeeper
> (replicated mode) running on the 3 VMs, and 2 VMs used for NiFi nodes.
>
> I have the same issue in a dockerized environment
>
>
>
>
>
>
> --
> View this message in context: http://apache-nifi-users-list.
> 2361937.n4.nabble.com/Nifi-Cluster-fails-to-disconnect-
> node-when-node-was-killed-tp1942p1948.html
> Sent from the Apache NiFi Users List mailing list archive at Nabble.com.
>

Re: Nifi Cluster fails to disconnect node when node was killed

[ddewaele]

Thanks for the response. 

When killing a non-coordinator node, it does take 8 * 5 seconds before I see
this :

nifi-app.log:2017-05-18 12:04:29,644 INFO [Heartbeat Monitor Thread-1]
o.a.n.c.c.node.NodeClusterCoordinator Status of centos-b:8080 changed from
NodeConnectionStatus[nodeId=centos-b:8080, state=CONNECTED, updateId=26] to
NodeConnectionStatus[nodeId=centos-b:8080, state=DISCONNECTED, Disconnect
Code=Lack of Heartbeat, Disconnect Reason=Have not received a heartbeat from
node in 43 seconds, updateId=27]

When killing the coordinator node, the newly appointed coordinator doesn't
seem to detect the heartbeat timeout.

I'll see if I can enable the debug logging.

My Nifi runs inside a KVM. KVM includes 3 seperate VMs. External zookeeper
(replicated mode) running on the 3 VMs, and 2 VMs used for NiFi nodes.

I have the same issue in a dockerized environment






--
View this message in context: 
http://apache-nifi-users-list.2361937.n4.nabble.com/Nifi-Cluster-fails-to-disconnect-node-when-node-was-killed-tp1942p1948.html
Sent from the Apache NiFi Users List mailing list archive at Nabble.com.

Re: Nifi Cluster fails to disconnect node when node was killed

[ddewaele]

I can reproduce the issue by killing the java processes associated with the
cluster coordinator node.

The NiFi UI will not be accessible anymore until that particular node is
brought up again, or until the node entry is removed from the cluster (via
the REST API).

Killing non-coordinator nodes does result in nifi detected heartbeat loss
and flagging it as DISCONNECTED.




--
View this message in context: 
http://apache-nifi-users-list.2361937.n4.nabble.com/Nifi-Cluster-fails-to-disconnect-node-when-node-was-killed-tp1942p1947.html
Sent from the Apache NiFi Users List mailing list archive at Nabble.com.

Re: Nifi Cluster fails to disconnect node when node was killed

[Matt Gilman]

Hi,

Once the new coordinator was elected, it is responsible for disconnecting
nodes due to lack of heartbeat. It will wait 8 times the
configured nifi.cluster.protocol.heartbeat.interval before the node
is disconnected. Can you confirm that this amount of time has elapsed?

Did you see any messages containing "Have not received a heartbeat from
node in" or "Failed to remove heartbeat for" during this time? Can you
describe your environment a little more? Are you running an external or
embedded zookeeper?

Can you enabled debug level logging for this package?
org.apache.nifi.cluster.coordination.heartbeat

Thanks

Matt

On Thu, May 18, 2017 at 7:30 AM, ddewaele  wrote:

> Hi,
>
> I have a NiFi cluster up and running and I'm testing various failover
> scenarios.
>
> I have 2 nodes in the cluster :
>
> - centos-a : Coordinator node / primary
> - centos-b : Cluster node
>
> I noticed in 1 of the scenarios when I killed the Cluster Coordinator node,
> that the following happened :
>
> centos-b couldn't contact the coordinator anymore and became the new
> coordinator / primary node. (as expected) :
>
> Failed to send heartbeat due to:
> org.apache.nifi.cluster.protocol.ProtocolException: Failed to send message
> to Cluster Coordinator due to: java.net.ConnectException: Connection
> refused
> (Connection refused)
> This node has been elected Leader for Role 'Primary Node'
> This node has been elected Leader for Role 'Cluster Coordinator'
>
> When attempting to access the UI on centos-b, I got the following error :
>
> 2017-05-18 11:18:49,368 WARN [Replicate Request Thread-2]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET
> /nifi-api/flow/current-user to centos-a:8080 due to {}
>
> If my understanding is correct, NiFi will try to replicate to connected
> nodes in the cluster. Here, centos-a was killed a while back and should
> have
> been disconnected, but as far as NiFi was concerned it was still connected.
>
> As a result I cannot access the UI anymore (due to the replication error),
> but I can lookup the cluster info via the REST API. And sure enough, it
> still sees centos-a as being CONNECTED.
>
> {
> "cluster": {
> "generated": "11:20:13 UTC",
> "nodes": [
> {
> "activeThreadCount": 0,
> "address": "centos-b",
> "apiPort": 8080,
> "events": [
> {
> "category": "INFO",
> "message": "Node Status changed from CONNECTING to
> CONNECTED",
> "timestamp": "05/18/2017 11:17:31 UTC"
> },
> {
> "category": "INFO",
> "message": "Node Status changed from [Unknown Node]
> to CONNECTING",
> "timestamp": "05/18/2017 11:17:27 UTC"
> }
> ],
> "heartbeat": "05/18/2017 11:20:09 UTC",
> "nodeId": "a5bce78d-23ea-4435-a0dd-4b731459f1b9",
> "nodeStartTime": "05/18/2017 11:17:25 UTC",
> "queued": "8,492 / 13.22 MB",
> "roles": [
> "Primary Node",
> "Cluster Coordinator"
> ],
> "status": "CONNECTED"
> },
> {
> "address": "centos-a",
> "apiPort": 8080,
> "events": [],
> "nodeId": "b89e8418-4b7f-4743-bdf4-4a08a92f3892",
> "roles": [],
> "status": "CONNECTED"
> }
> ]
> }
> }
>
> When centos-a was brought back online, i noticed the following status
> change
> :
>
> Status of centos-a:8080 changed from
> NodeConnectionStatus[nodeId=centos-a:8080, state=CONNECTED, updateId=15]
> to
> NodeConnectionStatus[nodeId=centos-a:8080, state=CONNECTING, updateId=19]
>
> So it went from connected -> connecting.
>
> It clearly missed the disconnected step here.
>
> When shutting down the centos-a node using nifi.sh stop, it goes into the
> DISCONNECTED state :
>
> Status of centos-a:8080 changed from
> NodeConnectionStatus[nodeId=centos-a:8080, state=CONNECTED, updateId=12]
> to
> NodeConnectionStatus[nodeId=centos-a:8
> 080, state=DISCONNECTED, Disconnect Code=Node was Shutdown, Disconnect
> Reason=Node was Shutdown, updateId=13]
>
> How can I debug this further, and can somebody provide some additional
> insights ? I have seen nodes getting disconnected due to missing heartbeats
>
> tatus of centos-a:8080 changed from
> NodeConnectionStatus[nodeId=centos-a:8080, state=CONNECTED, updateId=10]
> to
> NodeConnectionStatus[nodeId=centos-a:8080, state=DISCONNECTED, Disconnect
> Code=Lack of Heartbeat, Disconnect Reason=Have not received a heartbeat
> from
> node in 41 seconds, updateId=11]
>
> But sometimes it doesn't seem to detect this, and NiFi keeps on thinking it
> is 

Re: is it possible for a NIFI processor to be both timer driven and event driven?

[Joe Witt]

Hello

It is certainly possible to have a process act as a listening daemon
and you can see this in various Listen processors.  Basically once
they are scheduled they open up some port under some protocol and
listen for incoming data.  The custom processor you're mentioning can
spawn a thread to do this server/port and it can still use its
scheduled to do other tasks but be careful about the design here so
that processors isn't doing too many things.

THanks

On Thu, May 18, 2017 at 4:19 AM, 尹文才  wrote:
> Hi guys, actually I have 2 questions, what I need to do is like this:
> I have a Java program outside NIFI and inside NIFI I have a Customized
> processor inside NIFI,
> I need to send some data from my Java program to my NIFI processor and the
> processor is
> timer-driven by now. The processor needs to do work when the time is due and
> it also needs to
> work when the Java program sends data to it and the processor's work time is
> not due at the time.
> Is this possible? Thanks

Re: How to lock getfile upto putfile write into same file?

[Joe Witt]

PutFile while writing automatically writes with a dot prepended and
once the write is complete it removes the dot.

If you have multiple instances of PutFile writing to the same file
with the intent of appending to that file it will like fail in strange
ways.

I would recommend you simply build up the complete thing you want to
write out to a file using MergeContent and send the merged results to
PutFile.  However, it should not be the case that you have PutFile and
GetFile sharing a given file in the same NiFi as you can simply
connect the processors to move the data without have to leave the
confines of NiFi using file IO.

Thanks

On Thu, May 18, 2017 at 3:06 AM, prabhu Mahendran
 wrote:
> Ok. I will append the '.' before Putfile using UpdateAttribute. Is this
> name(without '.') will automatically changed by Putfile when its done?
>
>
>
> Since I am appending each rows from html content into proper csv using
> Putfile processor. Is this works fine? How putfile knows complete html
> flowfiles has been moved.
>
>
> On Tue, May 16, 2017 at 7:53 PM, Juan Sequeiros  wrote:
>>
>> Prabhu,
>>
>> PutFile should pre-pend files with a "." while it is writing to it and
>> then in the end will copy the "." file to its "filename" value.
>>
>> On GetFile side make sure that "ignore hidden files" is set to true.
>>
>>
>>
>> On Tue, May 16, 2017 at 1:29 AM prabhu Mahendran 
>> wrote:
>>>
>>> I have scheduled getfile processor to 0 sec to track the local folder.
>>>
>>> Issue I have faced: PutFile is appending few flowfiles into single file.
>>> Getfile has been configured with keepsourcefile as false. So getfile is
>>> fetching partial content before putfile writes into local location.
>>>
>>> How to handle this issue? Can we lock the file till putfile/custom
>>> processor completely writes the file and remove lock once completed??
>
>

Re: how to set version for NIFI customized Processor?

[Joe Witt]

Just rebuild it with the latest nifi nar maven plugin and it will get
the version info at that time.

thanks

On Thu, May 18, 2017 at 4:20 AM, 尹文才  wrote:
> Thanks Joe, is it possible to set a specific version for a customized NIFI
> processor and show the version in that processor selection dialog?
>
> 2017-05-18 10:42 GMT+08:00 Joe Witt :
>>
>> Hello
>>
>> They will remain unversioned until they are built with the latest nar
>> plugin.  This is described briefly in the migration guidance [1].
>>
>> For anything built with the older nifi nar plugin the resulting nar
>> will not have suffiicient bundle data for the framework to have it
>> support component versioning so that is why it shows as unversioned.
>>
>> [1] https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance
>>
>> Thanks
>> Joe
>>
>> On Wed, May 17, 2017 at 10:37 PM, 尹文才  wrote:
>> > Hi guys, I have installed NIFI 1.2 and have played with it for a while.
>> > One thing I noticed for 1.2 is that when I placed my previously written
>> > customized Processor
>> > into 1.2, I could see in the Processor select dialog there's a Version
>> > field
>> > for each processor
>> > and the value for my processor is unversioned. So does anyone know how
>> > to
>> > set version for
>> > customized processor? Thanks.
>
>

Nifi Cluster fails to disconnect node when node was killed

[ddewaele]

Hi,

I have a NiFi cluster up and running and I'm testing various failover
scenarios.

I have 2 nodes in the cluster :

- centos-a : Coordinator node / primary
- centos-b : Cluster node

I noticed in 1 of the scenarios when I killed the Cluster Coordinator node,
that the following happened :

centos-b couldn't contact the coordinator anymore and became the new
coordinator / primary node. (as expected) :

Failed to send heartbeat due to:
org.apache.nifi.cluster.protocol.ProtocolException: Failed to send message
to Cluster Coordinator due to: java.net.ConnectException: Connection refused
(Connection refused)
This node has been elected Leader for Role 'Primary Node'
This node has been elected Leader for Role 'Cluster Coordinator'

When attempting to access the UI on centos-b, I got the following error :

2017-05-18 11:18:49,368 WARN [Replicate Request Thread-2]
o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request GET
/nifi-api/flow/current-user to centos-a:8080 due to {}

If my understanding is correct, NiFi will try to replicate to connected
nodes in the cluster. Here, centos-a was killed a while back and should have
been disconnected, but as far as NiFi was concerned it was still connected.

As a result I cannot access the UI anymore (due to the replication error),
but I can lookup the cluster info via the REST API. And sure enough, it
still sees centos-a as being CONNECTED.

{
"cluster": {
"generated": "11:20:13 UTC",
"nodes": [
{
"activeThreadCount": 0,
"address": "centos-b",
"apiPort": 8080,
"events": [
{
"category": "INFO",
"message": "Node Status changed from CONNECTING to
CONNECTED",
"timestamp": "05/18/2017 11:17:31 UTC"
},
{
"category": "INFO",
"message": "Node Status changed from [Unknown Node]
to CONNECTING",
"timestamp": "05/18/2017 11:17:27 UTC"
}
],
"heartbeat": "05/18/2017 11:20:09 UTC",
"nodeId": "a5bce78d-23ea-4435-a0dd-4b731459f1b9",
"nodeStartTime": "05/18/2017 11:17:25 UTC",
"queued": "8,492 / 13.22 MB",
"roles": [
"Primary Node",
"Cluster Coordinator"
],
"status": "CONNECTED"
},
{
"address": "centos-a",
"apiPort": 8080,
"events": [],
"nodeId": "b89e8418-4b7f-4743-bdf4-4a08a92f3892",
"roles": [],
"status": "CONNECTED"
}
]
}
}

When centos-a was brought back online, i noticed the following status change
:

Status of centos-a:8080 changed from
NodeConnectionStatus[nodeId=centos-a:8080, state=CONNECTED, updateId=15] to
NodeConnectionStatus[nodeId=centos-a:8080, state=CONNECTING, updateId=19]

So it went from connected -> connecting.

It clearly missed the disconnected step here.

When shutting down the centos-a node using nifi.sh stop, it goes into the
DISCONNECTED state :

Status of centos-a:8080 changed from
NodeConnectionStatus[nodeId=centos-a:8080, state=CONNECTED, updateId=12] to
NodeConnectionStatus[nodeId=centos-a:8
080, state=DISCONNECTED, Disconnect Code=Node was Shutdown, Disconnect
Reason=Node was Shutdown, updateId=13]

How can I debug this further, and can somebody provide some additional
insights ? I have seen nodes getting disconnected due to missing heartbeats

tatus of centos-a:8080 changed from
NodeConnectionStatus[nodeId=centos-a:8080, state=CONNECTED, updateId=10] to
NodeConnectionStatus[nodeId=centos-a:8080, state=DISCONNECTED, Disconnect
Code=Lack of Heartbeat, Disconnect Reason=Have not received a heartbeat from
node in 41 seconds, updateId=11]

But sometimes it doesn't seem to detect this, and NiFi keeps on thinking it
is CONNECTED, despite not having received heartbeats in ages.

Any ideas ?



--
View this message in context: 
http://apache-nifi-users-list.2361937.n4.nabble.com/Nifi-Cluster-fails-to-disconnect-node-when-node-was-killed-tp1942.html
Sent from the Apache NiFi Users List mailing list archive at Nabble.com.

Re: how to set version for NIFI customized Processor?

[尹文才]

Thanks Joe, is it possible to set a specific version for a customized NIFI
processor and show the version in that processor selection dialog?

2017-05-18 10:42 GMT+08:00 Joe Witt :

> Hello
>
> They will remain unversioned until they are built with the latest nar
> plugin.  This is described briefly in the migration guidance [1].
>
> For anything built with the older nifi nar plugin the resulting nar
> will not have suffiicient bundle data for the framework to have it
> support component versioning so that is why it shows as unversioned.
>
> [1] https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance
>
> Thanks
> Joe
>
> On Wed, May 17, 2017 at 10:37 PM, 尹文才  wrote:
> > Hi guys, I have installed NIFI 1.2 and have played with it for a while.
> > One thing I noticed for 1.2 is that when I placed my previously written
> > customized Processor
> > into 1.2, I could see in the Processor select dialog there's a Version
> field
> > for each processor
> > and the value for my processor is unversioned. So does anyone know how to
> > set version for
> > customized processor? Thanks.
>

is it possible for a NIFI processor to be both timer driven and event driven?

[尹文才]

Hi guys, actually I have 2 questions, what I need to do is like this:
I have a Java program outside NIFI and inside NIFI I have a Customized
processor inside NIFI,
I need to send some data from my Java program to my NIFI processor and the
processor is
timer-driven by now. The processor needs to do work when the time is due
and it also needs to
work when the Java program sends data to it and the processor's work time
is not due at the time.
Is this possible? Thanks

Re: Kafka - Hadoop Kerberos issue

[Arnaud G]

mail was sent before completion:

Security Protocol: SASL_SSL
Kerberos Service Name:  myuser (please note that its mandatory to put
something here or the process fail)
SSL Context Service: KafkaSSLcontext
sasl.jaas.config: org.apache.kafka.common.security.plain.PlainSaslServer
require username="kauser" password="X";
sasl.mechanism: PLAIN

This configuration is working fine we can consume the topics without any
issue.

4) Here is the configuration of the PutHDFS :
 Hadoop Configuration Ressources:
/etc/hadoop/conf/hdfs-site.xml;/etc/hadoop/conf/core-site.xml
 Kerberos Principal: us...@realm.com
 Kerberos Keytab: /etc/security/keytabs/user1.keytab


Additional classpath is empty.

Please note that the Kafka is an external instance (and using SASL/PLAIN)
and not using the same Kerberos configuration than HDFS. I suppose that if
we were using the same Kerberos domain everything will be working fine.

A.


On Thu, May 18, 2017 at 9:22 AM, Arnaud G  wrote:

> Hi again,
>
> Here are more details:
>
> 1) We are redoing the test on a test cluster which is plain 1.2 vanilla
> (upgraded from 1.1 but without any additional JAR, bootstrap is the package
> version.
> 2) The org.apache.kafka.common.security.plain.PlainSaslServer is coming
> from our JAAS configuration and form understanding is part of the standard
> 0.10 client library
> 3) Here is the configuration of our ConsumeKafka_0_10:
>
>
>
>
> On Wed, May 17, 2017 at 6:50 PM, Bryan Rosander 
> wrote:
>
>> Hey guys,
>>
>> I also used a flow that has PublishKafka_0_10 and ConsumeKafka_0_10 as
>> well as PutHDFS and ListHDFS all talking to a secured cluster.  I've tried
>> various configurations of running vs stopped processors including
>> restarting nifi in each configuration and can't seem to reproduce.
>>
>> It's strange that you're getting org.apache.kafka.common.securi
>> ty.plain.PlainSaslServer$PlainSaslServerFactory in the list of Sasl
>> service factories.
>>
>> When I put a breakpoint at org.apache.hadoop.security.Sas
>> lRpcServer$FastSaslServerFactory.(SaslRpcServer.java:380) I see
>> several factory instances but the closest to that kafka class
>> is org.apache.hadoop.security.SaslPlainServer$SaslPlainServerFactory.
>>
>> Thanks,
>> Bryan Rosander
>>
>> On Wed, May 17, 2017 at 12:28 PM, Bryan Bende  wrote:
>>
>>> Hi Arnaud,
>>>
>>> I created a flow that had PublishKafka_0_10 running with dynamic JAAS
>>> properties, and also PutHDFS and ListHDFS talking to a kerberized
>>> HDFS, and I wasn't able to reproduce the above error. I also stopped
>>> and started all of the processors in different orders, but they
>>> continued to work.
>>>
>>> Is there any pattern for you that causes the problem? For example,
>>> does it always happen if you start the Kafka processors first and then
>>> the HDFS processors, and not the other way around?
>>>
>>> Also, can you confirm that your PutHDFS processor is not using the
>>> "Additional Classpath Resources" property, and that there were no
>>> additional JARs added to NiFi's lib directory?
>>>
>>> Just want to ensure nothing else is on the classpath of the PutHDFS
>>> processor besides what would be expected.
>>>
>>> Thanks,
>>>
>>> Bryan
>>>
>>>
>>> On Wed, May 17, 2017 at 10:24 AM, Arnaud G 
>>> wrote:
>>> > Hi,
>>> >
>>> > We didn't fully tested it in 1.1 as we were waiting for the dynamic
>>> load of
>>> > JAAS in the new Kafka consumer/producer processor. We expected that
>>> loading
>>> > a JAAS when starting Nifi may had impact on all Kerberos processes.
>>> >
>>> > Thanks,
>>> >
>>> > Arnaud
>>> >
>>> > On Wed, May 17, 2017 at 2:31 PM, Bryan Bende  wrote:
>>> >>
>>> >> Hello,
>>> >>
>>> >> Thanks for reporting this, we definitely want to figure out what is
>>> >> going on here.
>>> >>
>>> >> Was this flow working fine before using Apache NiFi 1.1.x (or some
>>> >> earlier version) and then stopped working when upgrading to 1.2.0?
>>> >>
>>> >> Thanks,
>>> >>
>>> >> Bryan
>>> >>
>>> >>
>>> >> On Wed, May 17, 2017 at 4:49 AM, Arnaud G 
>>> wrote:
>>> >> > Hi!
>>> >> >
>>> >> > We are currently facing an issue and we have not yet find a
>>> potential
>>> >> > solution.
>>> >> >
>>> >> > We are running 1.2 and we have are facing an interaction between the
>>> >> > Kafka
>>> >> > 10.2 and PutHDFS processors.
>>> >> >
>>> >> > We are connecting to an external Kafka server that require
>>> SASL/PLAIN
>>> >> > authentication. To enable this we are using the new Kafka processor
>>> with
>>> >> > a
>>> >> > JAAS configuration in the processor to authenticate, and this is
>>> working
>>> >> > fine.
>>> >> >
>>> >> > However on an unrelated flow the PutHDFS processor stopped to work
>>> as
>>> >> > the
>>> >> > processor is now using the JAAS configuration and ignoring the
>>> Kerberos
>>> >> > configuration of the processor. (the setup are completely unrelated)

Re: Kafka - Hadoop Kerberos issue

[Arnaud G]

Hi again,

Here are more details:

1) We are redoing the test on a test cluster which is plain 1.2 vanilla
(upgraded from 1.1 but without any additional JAR, bootstrap is the package
version.
2) The org.apache.kafka.common.security.plain.PlainSaslServer is coming
from our JAAS configuration and form understanding is part of the standard
0.10 client library
3) Here is the configuration of our ConsumeKafka_0_10:




On Wed, May 17, 2017 at 6:50 PM, Bryan Rosander 
wrote:

> Hey guys,
>
> I also used a flow that has PublishKafka_0_10 and ConsumeKafka_0_10 as
> well as PutHDFS and ListHDFS all talking to a secured cluster.  I've tried
> various configurations of running vs stopped processors including
> restarting nifi in each configuration and can't seem to reproduce.
>
> It's strange that you're getting org.apache.kafka.common.security.plain.
> PlainSaslServer$PlainSaslServerFactory in the list of Sasl service
> factories.
>
> When I put a breakpoint at org.apache.hadoop.security.SaslRpcServer$
> FastSaslServerFactory.(SaslRpcServer.java:380) I see several
> factory instances but the closest to that kafka class
> is org.apache.hadoop.security.SaslPlainServer$SaslPlainServerFactory.
>
> Thanks,
> Bryan Rosander
>
> On Wed, May 17, 2017 at 12:28 PM, Bryan Bende  wrote:
>
>> Hi Arnaud,
>>
>> I created a flow that had PublishKafka_0_10 running with dynamic JAAS
>> properties, and also PutHDFS and ListHDFS talking to a kerberized
>> HDFS, and I wasn't able to reproduce the above error. I also stopped
>> and started all of the processors in different orders, but they
>> continued to work.
>>
>> Is there any pattern for you that causes the problem? For example,
>> does it always happen if you start the Kafka processors first and then
>> the HDFS processors, and not the other way around?
>>
>> Also, can you confirm that your PutHDFS processor is not using the
>> "Additional Classpath Resources" property, and that there were no
>> additional JARs added to NiFi's lib directory?
>>
>> Just want to ensure nothing else is on the classpath of the PutHDFS
>> processor besides what would be expected.
>>
>> Thanks,
>>
>> Bryan
>>
>>
>> On Wed, May 17, 2017 at 10:24 AM, Arnaud G  wrote:
>> > Hi,
>> >
>> > We didn't fully tested it in 1.1 as we were waiting for the dynamic
>> load of
>> > JAAS in the new Kafka consumer/producer processor. We expected that
>> loading
>> > a JAAS when starting Nifi may had impact on all Kerberos processes.
>> >
>> > Thanks,
>> >
>> > Arnaud
>> >
>> > On Wed, May 17, 2017 at 2:31 PM, Bryan Bende  wrote:
>> >>
>> >> Hello,
>> >>
>> >> Thanks for reporting this, we definitely want to figure out what is
>> >> going on here.
>> >>
>> >> Was this flow working fine before using Apache NiFi 1.1.x (or some
>> >> earlier version) and then stopped working when upgrading to 1.2.0?
>> >>
>> >> Thanks,
>> >>
>> >> Bryan
>> >>
>> >>
>> >> On Wed, May 17, 2017 at 4:49 AM, Arnaud G 
>> wrote:
>> >> > Hi!
>> >> >
>> >> > We are currently facing an issue and we have not yet find a potential
>> >> > solution.
>> >> >
>> >> > We are running 1.2 and we have are facing an interaction between the
>> >> > Kafka
>> >> > 10.2 and PutHDFS processors.
>> >> >
>> >> > We are connecting to an external Kafka server that require SASL/PLAIN
>> >> > authentication. To enable this we are using the new Kafka processor
>> with
>> >> > a
>> >> > JAAS configuration in the processor to authenticate, and this is
>> working
>> >> > fine.
>> >> >
>> >> > However on an unrelated flow the PutHDFS processor stopped to work as
>> >> > the
>> >> > processor is now using the JAAS configuration and ignoring the
>> Kerberos
>> >> > configuration of the processor. (the setup are completely unrelated)
>> >> >
>> >> > We tried a lot of configuration but we have not yet find a way to
>> allow
>> >> > Nifi
>> >> > to authenticate to both Kafka 10.2 and HDFS at the same time, it's
>> >> > either
>> >> > one or the other.
>> >> >
>> >> > Are we missing something? Is there a way to ensure that both
>> processor
>> >> > keep
>> >> > their own configuration?
>> >> >
>> >> > Thanks!
>> >> >
>> >> >
>> >> > For reference PutHDFS failure log:
>> >> >
>> >> > 2017-05-16 15:27:01,199 ERROR [StandardProcessScheduler Thread-8]
>> >> > o.apache.nifi.processors.hadoop.PutHDFS
>> >> > PutHDFS[id=b6464c58-fee9-341c-8428-932218f7d239]
>> >> > PutHDFS[id=b6464c58-fee9-341c-8428-932218f7d239] failed to invoke
>> >> > @OnScheduled method due to java.lang.RuntimeException: Failed while
>> >> > executing one of processor's OnScheduled task.; processor will not be
>> >> > scheduled to run for 30 seconds: java.lang.RuntimeException: Failed
>> >> > while
>> >> > executing one of processor's OnScheduled task.
>> >> >
>> >> > java.lang.RuntimeException: Failed while executing one of processor's
>> >> > OnScheduled task.
>> >> >
>> >> > at
>> >> >
>> >> > 

Re: How to lock getfile upto putfile write into same file?

[prabhu Mahendran]

Ok. I will append the '.' before Putfile using UpdateAttribute. Is this
name(without '.') will automatically changed by Putfile when its done?



Since I am appending each rows from html content into proper csv using
Putfile processor. Is this works fine? How putfile knows complete html
flowfiles has been moved.

On Tue, May 16, 2017 at 7:53 PM, Juan Sequeiros  wrote:

> Prabhu,
>
> PutFile should pre-pend files with a "." while it is writing to it and
> then in the end will copy the "." file to its "filename" value.
>
> On GetFile side make sure that "ignore hidden files" is set to true.
>
>
>
> On Tue, May 16, 2017 at 1:29 AM prabhu Mahendran 
> wrote:
>
>> I have scheduled getfile processor to 0 sec to track the local folder.
>>
>> Issue I have faced: PutFile is appending few flowfiles into single file.
>> Getfile has been configured with keepsourcefile as false. So getfile is
>> fetching partial content before putfile writes into local location.
>>
>> How to handle this issue? Can we lock the file till putfile/custom
>> processor completely writes the file and remove lock once completed??
>>
>