Thanks Andy/Joe/Juan for the helpful replies. I have some follow-up
questions...
(1)
I'm not quite sure how a flowfile-level audit trail can be used for
governance. Is it expected that some system will process the info as
follows?
* for each provenance event
* if (is_bad_record(event)) then
Thanks Joe, Juan,
Perhaps it would be useful to be able to generate provenance events for
a _sample_ of flowfiles? eg every Nth flowfile created by a "data
ingress" (GET* or LISTEN*) processor gets tracked? Or maybe better:
every flowfile gets tracked with a probability of N, to ensure that
s
Good morning. I have established an initial single-threaded NiFi server
instance for my customers. It works well, but I anticipate increasing usage
as groups learn more about it. I also want to move beyond our single
threaded-ness.
I would like to take the next step in the evolution of our NiFi ca
Jim,
I would offer you a few bits of advice. First, NiFi relies on ZooKeeper to
coordinate which node is responsible
to act as the Cluster Coordinator and which node should be the Primary Node.
NiFi does allow you to start
and embedded ZooKeeper, but for production use, it is recommended that yo
These are invaluable insights Mark. Thank you very much. We'll tune
accordingly, and employ an external Zookeeper running on boxes other than
our NiFi server cluster boxes.
On Thu, Apr 20, 2017 at 8:52 AM, Mark Payne wrote:
> Jim,
>
> I would offer you a few bits of advice. First, NiFi relies on
Hello ,
I want to add a "\n "at the end of my flow ( it's a record in
fact) before sending the record to a Put HDFS processor in append mode
I tried to put the ReplaceText Processor in the flow with the following
config
Search Value (?s)(^.*$)
Replacement Value \\n
Charac
Can you try something like this Phil:
${uuid:append('\n')}
Jim
On Thu, Apr 20, 2017 at 9:25 AM, wrote:
> Hello ,
>
> I want to add a “\n “at the end of my flow ( it’s a record ….in
> fact) before sending the record to a Put HDFS processor in append mode
>
> I tried to put the Replac
☺ Thanks a lot James it works nicely now ……
From: James McMahon [mailto:jsmcmah...@gmail.com]
Sent: jeudi 20 avril 2017 15:36
To: users@nifi.apache.org
Subject: Re: appending \n to a flow
Can you try something like this Phil:
${uuid:append('\n')}
Jim
On Thu, Apr 20, 2017 at 9:25 AM,
mailto:p
You're most welcome sir. Cheers! Jim
On Thu, Apr 20, 2017 at 9:44 AM, wrote:
> J Thanks a lot James it works nicely now ……
>
>
>
> *From:* James McMahon [mailto:jsmcmah...@gmail.com]
> *Sent:* jeudi 20 avril 2017 15:36
> *To:* users@nifi.apache.org
> *Subject:* Re: appending \n to a flow
>
>
>
Phil,
For your configuration of ReplaceText, I think you need a couple of changes.
Firstly, you don't
want to append a newline to the end of every line - you want to append it to
the end of the FlowFile.
So you'd want "Evaluation Mode" set to "Entire Text". Secondly, NiFi is not
going to substi
Hello,
I am following the steps described in
https://community.hortonworks.com/articles/90355/collect-data-from-opc-ua-protocol.html
to connect to an OPC UA server from MiNiFi.
I built https://github.com/wadesalazar/NIFI-OPCUA from source code using Maven.
However, after adding the nar file to
Hi all,
MiNiFi's WholeConfigDifferentiator along with the PullHttpChangeIngestor is
working well for me. I see that the differentiator is configurable and
additional implementations can be provided. Are there any examples of
circumstances in which the using WholeConfigDifferentiator would not be t
Thanks Mark ..
I understand the difference of records and flow of records and how to manage
them with the “Entire text “ you mention ☺
From: Mark Payne [mailto:marka...@hotmail.com]
Sent: jeudi 20 avril 2017 15:48
To: users@nifi.apache.org
Subject: Re: appending \n to a flow
Phil,
For your c
Hello Jeff,
Glad to hear the WholeConfigDifferentiator is working well. While it is
configurable, that is currently the only implementation. Is there a
specific place that suggests there are currently more implementations? The
documentation lists it as the only one[1]. When I wrote it up I impleme
Thanks Joe. It does look like a misconfiguration on my end. I started with
a new topic and everything works fine.
On Wed, Apr 19, 2017 at 4:45 PM, Joe Witt wrote:
> Hello
>
> UnpackContent is for archive formats. gzip/snappy/lz4 aren't archive
> formats but are compression types. You want Comp
Hi Joe,
No, the documentation is clear that is currently only available
implementation. Since it was designed to be extendable, I was wondering if
there were any example cases in which using the WholeConfigDifferentiator might
not be the ideal differentiator and I should provide my own implementat
Hey Jeff,
Just to hone in a bit more, MiNiFi would likely consume from some of the
work surrounding the Command and Control [1] efforts.
There is some initial work under way which should serve as a nice
foundation to start exploring a more full featured and richer
implementation. Certainly plan
Thanks for that info, Aldrin! I will be glad to look over it.
Jeff
On Thu, Apr 20, 2017 at 12:09 PM, Aldrin Piri wrote:
> Hey Jeff,
>
> Just to hone in a bit more, MiNiFi would likely consume from some of the
> work surrounding the Command and Control [1] efforts.
>
> There is some initial work
BTW, your NiFi instance is not single-threaded, it's a single node. It
still runs multiple worker threads in the flow.
Andrew
On Thu, Apr 20, 2017, 7:01 AM James McMahon wrote:
> Good morning. I have established an initial single-threaded NiFi server
> instance for my customers. It works well,
19 matches
Mail list logo
