I have running multiple instances of PutFile writing to same file that
leads fail in Strange ways.
I have tried MergeContent processor but it not yields same results for all
time.
For one time it correctly merges files according to the filename but
sometimes it could not merge.
In my use case i
I have approximately 1000 files in local drive.I need to move that files
into SQL Server accordingly one after another.
Since local drive having files like file1.csv,file2.csv,..upto file1000.csv.I
am sure that number of files in local drive may change dynamically.
I can able to created template
Neil,
Want to make sure I understand what you're saying. What are stating
is a single point of failure?
Thanks
Joe
On Thu, May 18, 2017 at 5:27 PM, Neil Derraugh
wrote:
> Thanks for the insight Matt.
>
> It's a disaster recovery issue. It's not something I plan on doing on
> purpose. It seem
Thanks Bryan and Joe, I managed to set the specific version for my
processor with the properties.
2017-05-18 20:50 GMT+08:00 Bryan Bende :
> As Joe mentioned, the default behavior is to use the Maven group,
> artifact, and version, which will happen by default if you build your
> NAR with the lat
Thanks for the insight Matt.
It's a disaster recovery issue. It's not something I plan on doing on
purpose. It seems it is a single point of failure unfortunately. I can
see no other way to resolve the issue other than to blow everything away
and start a new cluster.
On Thu, May 18, 2017 at 2:
Neil,
Disconnecting a node prior to removal is the correct process. It appears
that the check was lost going from 0.x to 1.x. Folks reported this JIRA [1]
indicating that deleting a connected node did not work. This process does
not work because the node needs to be disconnected first. The JIRA wa
Pretty sure this is the problem I was describing in the "Phantom Node"
thread recently.
If I kill non-primary nodes the cluster remains healthy despite the lost
nodes. The terminated nodes end up with a DISCONNECTED status.
If I kill the primary it winds up with a CONNECTED status, but a new
pri
@pwicks no luck on escaping \n. Thanks for the suggestion.
I even tried hardcoding the boundary value, in case it had something to do
with using nifi expressions in regex but that didn't work either.
On Wed, May 17, 2017 at 10:54 PM, Peter Wicks (pwicks)
wrote:
> Nick,
>
>
>
> Try escaping your
Hello,
Just looking through this thread now. I believe that I understand the problem.
I have updated the JIRA with details about what I think is the problem and a
potential remedy for the problem.
Thanks
-Mark
> On May 18, 2017, at 9:49 AM, Matt Gilman wrote:
>
> Thanks for the additional de
Thank you.
I've created this JIRA feature request:
https://issues.apache.org/jira/browse/NIFI-3934
On Wed, May 17, 2017 at 12:51 PM Joe Witt wrote:
> The concept of being able to provide hints to the underlying content
> repository about which content is more important to keep longer than
> ot
Hi,
Just wanted to point out that the newly appointed coordinator (centos-b)
does end up sending heartbeats to itself as you described.
2017-05-18 12:41:41,336 DEBUG [Process Cluster Protocol Request-3]
o.a.n.c.c.h.ClusterProtocolHeartbeatMonitor Received new heartbeat from
centos-b:8080
It se
Thanks for the additional details. They will be helpful when working the JIRA.
All nodes, including the coordinator, heartbeat to the active coordinator. This
means that the coordinator effectively heartbeats to itself. It appears, based
on your log messages, that this is not happening. Because
Hi Arnaud,
Thanks for following up and providing the additional information.
I can in fact recreate the problem now, and it is specific to using
the PlainSaslServer provided by Kafka. Once you have started the Kafka
processor with that configuration, the PlainSaslServer is being
registered as a p
As Joe mentioned, the default behavior is to use the Maven group,
artifact, and version, which will happen by default if you build your
NAR with the latest NAR Maven plugin (version 1.2.0 at this time).
If you prefer to set different values than the Maven fields, you can
override them by specifyin
Found something interesting in the centos-b debug logging
after centos-a (the coordinator) is killed centos-b takes over. Notice how
it "Will not disconnect any nodes due to lack of heartbeat" and how it still
sees centos-a as connected despite the fact that there are no heartbeats
anymore.
Thank you for the confirmation. I've filed this JIRA [1] to track the issue.
Matt
[1] https://issues.apache.org/jira/browse/NIFI-3933
On Thu, May 18, 2017 at 8:21 AM, ddewaele wrote:
> Thanks for the response.
>
> When killing a non-coordinator node, it does take 8 * 5 seconds before I
> see
>
Thanks for the response.
When killing a non-coordinator node, it does take 8 * 5 seconds before I see
this :
nifi-app.log:2017-05-18 12:04:29,644 INFO [Heartbeat Monitor Thread-1]
o.a.n.c.c.node.NodeClusterCoordinator Status of centos-b:8080 changed from
NodeConnectionStatus[nodeId=centos-b:8080
I can reproduce the issue by killing the java processes associated with the
cluster coordinator node.
The NiFi UI will not be accessible anymore until that particular node is
brought up again, or until the node entry is removed from the cluster (via
the REST API).
Killing non-coordinator nodes do
Hi,
Once the new coordinator was elected, it is responsible for disconnecting
nodes due to lack of heartbeat. It will wait 8 times the
configured nifi.cluster.protocol.heartbeat.interval before the node
is disconnected. Can you confirm that this amount of time has elapsed?
Did you see any message
Hello
It is certainly possible to have a process act as a listening daemon
and you can see this in various Listen processors. Basically once
they are scheduled they open up some port under some protocol and
listen for incoming data. The custom processor you're mentioning can
spawn a thread to do
PutFile while writing automatically writes with a dot prepended and
once the write is complete it removes the dot.
If you have multiple instances of PutFile writing to the same file
with the intent of appending to that file it will like fail in strange
ways.
I would recommend you simply build up
Just rebuild it with the latest nifi nar maven plugin and it will get
the version info at that time.
thanks
On Thu, May 18, 2017 at 4:20 AM, 尹文才 wrote:
> Thanks Joe, is it possible to set a specific version for a customized NIFI
> processor and show the version in that processor selection dialog
Hi,
I have a NiFi cluster up and running and I'm testing various failover
scenarios.
I have 2 nodes in the cluster :
- centos-a : Coordinator node / primary
- centos-b : Cluster node
I noticed in 1 of the scenarios when I killed the Cluster Coordinator node,
that the following happened :
cento
Thanks Joe, is it possible to set a specific version for a customized NIFI
processor and show the version in that processor selection dialog?
2017-05-18 10:42 GMT+08:00 Joe Witt :
> Hello
>
> They will remain unversioned until they are built with the latest nar
> plugin. This is described briefl
Hi guys, actually I have 2 questions, what I need to do is like this:
I have a Java program outside NIFI and inside NIFI I have a Customized
processor inside NIFI,
I need to send some data from my Java program to my NIFI processor and the
processor is
timer-driven by now. The processor needs to do
mail was sent before completion:
Security Protocol: SASL_SSL
Kerberos Service Name: myuser (please note that its mandatory to put
something here or the process fail)
SSL Context Service: KafkaSSLcontext
sasl.jaas.config: org.apache.kafka.common.security.plain.PlainSaslServer
requi
Hi again,
Here are more details:
1) We are redoing the test on a test cluster which is plain 1.2 vanilla
(upgraded from 1.1 but without any additional JAR, bootstrap is the package
version.
2) The org.apache.kafka.common.security.plain.PlainSaslServer is coming
from our JAAS configuration and for
Ok. I will append the '.' before Putfile using UpdateAttribute. Is this
name(without '.') will automatically changed by Putfile when its done?
Since I am appending each rows from html content into proper csv using
Putfile processor. Is this works fine? How putfile knows complete html
flowfiles h
28 matches
Mail list logo