Hey Mans,
To load balance and send the FlowFiles to a processing group or processor in
the same cluster you will still set up an RPG but you will just point the RPG
to the current cluster's NCM.
The NiFi Cluster Manager (NCM) is in charge of splitting the incoming FlowFiles
among the nodes.
Hey Charlie,
Sorry no one has followed up with you yet. One way I see around
ConvertCharacterSet not supporting expression language is to route on attribute
(assuming the character set is extracted to be an attribute) to different
ConvertCharacterSet processors depending on the input character
:56 PM, M Singh <mans2si...@yahoo.com> wrote:
Joe:
No problem and thanks for your support in fixing the issue.
Mans
On Monday, November 9, 2015 12:35 PM, Joe Percivall <joeperciv...@yahoo.com>
wrote:
Mans,
I did a bit of testing using my refactor and I understand what the
is up,
Joe
- - - - - -
Joseph Percivall
linkedin.com/in/Percivall
e: joeperciv...@yahoo.com
On Monday, November 9, 2015 2:05 PM, Joe Percivall <joeperciv...@yahoo.com>
wrote:
Hey Mans,
Thanks for creating the ticket. First, have you tried the patch I supplied in
1086 [1]? I changed the unde
Hey Chris,
I feel your "Ugg" and I'd be happy to knock it out. I'm doing a bunch of work
with InvokeHttp already so I should be able to get this done pretty quick.
Joe- - - - - -
Joseph Percivall
linkedin.com/in/Percivall
e: joeperciv...@yahoo.com
On Monday, November 9, 2015 3:23 PM,
linkedin.com/in/Percivall
e: joeperciv...@yahoo.com
On Monday, November 9, 2015 3:26 PM, Joe Percivall <joeperciv...@yahoo.com>
wrote:
Hey Chris,
I feel your "Ugg" and I'd be happy to knock it out. I'm doing a bunch of work
with InvokeHttp already so I should be able to get
Hey Krish,
Welcome to NiFi! Storm is how I got my start with data flow as well and NiFi
has been awesome. NiFi's "processors" are equivalent to Storm's "spouts" and
"bolts".
NiFi can certainly handle the logging/auditing use case among others. With
NiFi's data provenance you can see exactly how
ot; along with the request using CURL for example. I
tried to set that on the "Attributes to Send" property in the InvokeHTTP
processor, but it did not work. It always returns XML.
Thanks,
Indus
On Fri, Dec 4, 2015 at 2:49 PM, Joe Percivall <joeperciv...@yahoo.com> wrote:
Hello Indus,
>
Hello Indus,
Are you receiving a specific error when you attempt to receive a JSON response?
I can configure an InvokeHTTP processor to GET "http://api.randomuser.me/; by
just setting the URL and it properly receives back a JSON response.
Joe
- - - - - -
Joseph Percivall
on' is invalid because 'Accept' is
not a supported property."
Did I do something wrong?
Thanks,
Indus
On Fri, Dec 4, 2015 at 3:12 PM, Joe Percivall <joeperciv...@yahoo.com> wrote:
I see, as for the request I'm guessing you mean you pass a header with key
"Accept" and val
is the comments example. i have many people who commented the post, i need
to do an HTTP request for each one. So i need to invoke not one HTTP request
but many after each retrieval of one post comment.
Kindly.
On 1 December 2015 at 17:07, Joe Percivall <joeperciv...@yahoo.com> wrote:
He
Hey Chandu,
You messed up bit in the config of your ReplaceText processor. You should be
running "Line-by-line" mode so each line is treated separately.
Also your search value is a bit messed up. It was trying to use capture groups
in it ("$2"), it wasn't successfully handling an initial date
Does anyone have any more opinions on this? - - - - - - Joseph
Percivalllinkedin.com/in/Percivalle: joeperciv...@yahoo.com
On Saturday, December 5, 2015 6:54 AM, Nigel Jones
wrote:
I have no experience of nabble but I'm also open to any similar suggestions? .
Hello Kacem,
There are multiple different ways to get information into NiFi. ListenHTTP
would be the easiest way to start an HTTP Server that is used to receive
FlowFiles from remote sources. Assuming your PHP code only acts as a router,
ListenHTTP should allow you to bypass the PHP code
>>request. The thing is i receive them with random names, i'll try to find out
>>how i can parse them to rename them as JSON files instead of POST fields.
>>Thanks for your help.
>>
>>
>>On 29 December 2015 at 16:50, Joe Percivall <joeperciv...@yahoo.com
Hello James,
I'm actually working on repo that includes many different templates. There are
a few that have InvokeHttp in them (can find the exact ones using the excel doc
at the top level) but this is a simple one that uses InvokeHttp:
I accidentally replied to just James with one of my responses. Forwarding the
thread back.
- - - - - -
Joseph Percivall
linkedin.com/in/Percivall
e: joeperciv...@yahoo.com
On Friday, December 18, 2015 5:20 PM, Joe Percivall <joeperciv...@yahoo.com>
wrote:
Following this post[1] on s
used that template, but changed the
remote URL to point to my server (https://www.google.com -> http://localhost).
James
-----Original Message-
From: Joe Percivall [mailto:joeperciv...@yahoo.com]
Sent: Friday, December 18, 2015 1:25 PM
To: users@nifi.apache.org
Subject: [RMX:NL] Re: [RMX:NL] Re: I
Hello Charlie,
I was looking back through and saw this wasn't totally resolved yet.
Couple questions. First, what system are you using? There are a couple of
options for the stream command depending on what you're using. Also are you
able to get new commands (using yum or brew)?
The key
Hey Tony,
I can't speak to the best practice for this controller service but just wanted
to follow up to see if this user got their question answered.
Joe
- - - - - -
Joseph Percivall
linkedin.com/in/Percivall
e: joeperciv...@yahoo.com
On Monday, November 23, 2015 3:32 PM, Tony Kurc
tant now. I
want to see how Nifi can be useful here like encrpyting only few fields
before processing.
Is it possible to create a new flow file with those fields, route and
encrypt that flow file?
Thanks,
Naveen
On 11/30/15, 11:28 AM, "Joe Percivall" <joeperciv...@yahoo.com>
Hello Kacem,
For your first question you can set up a relatively simple flow to achieve
this. Assuming the first retrieval is a GET method call you can do: GetHTTP ->
ExtractText -> InvokeHTTP. What this does is, it first gets the JSON files from
the social network using GetHTTP. Then with
Hello Subhash,
Sorry no one has gotten back to you sooner but what kind of examples are you
looking for? Are you looking for examples of how they are configured, a
template or just cool applications?
Joe
- - - - - -
Joseph Percivall
linkedin.com/in/Percivall
e: joeperciv...@yahoo.com
Hello Adam,
Are you still seeing high cpu usage?
Sorry no has gotten back to you sooner, we are all working very hard to get
0.4.0 out.
Joe
- - - - - -
Joseph Percivall
linkedin.com/in/Percivall
e: joeperciv...@yahoo.com
On Friday, November 13, 2015 4:10 PM, Adam Lamar
Hello Sudeep,
We are currently lacking a "GetDistributedMapCache" processor that corresponds
to the "PutDistributedMapCache". I created a ticket[1] and will be working on
it today. If you have any comments, configuration suggestions, etc. please let
me know or comment on the ticket.
[1]
For number one, you can also use RouteText[1] with the matching strategy
"Satisfies Expression". Then as a dynamic property use this expression
"${lineNo:le(10)}". This will route first 10 lines to the "matched"
relationship (assuming "Route to each matching Property Name" is not selected).
file and not to the one i waned too.
On Mon, Jun 13, 2016 at 10:40 AM, Joe Percivall <joeperciv...@yahoo.com> wrote:
Hello,
>
>In order to use ReplaceText[1] to solely append a value to the end of then
>entire text then change the "Replacement Strategy" to "Append"
Hello,
In order to use ReplaceText[1] to solely append a value to the end of then
entire text then change the "Replacement Strategy" to "Append" and leave
"Evaluation Mode" as "Entire Text". This will take whatever is the
"Replacement Value" and append it as a literal(without interpreting
e:
Yes that's exactly correct.
> On Jun 13, 2016, at 11:14 AM, Joe Percivall <joeperciv...@yahoo.com> wrote:
>
> Sorry I got a bit confused, in your original question you said that you
> wanted to append the value and I took it that you just wanted to append the
> value to
Hello Stephane,
Just to be sure I have your use-case correct, you are ingesting a continuous
stream of lat/lon information for various devices. Every 1 second you want to
take the information from the previous second and write out just the most
recent lat/lon of each device.
An important
Hello Ralph,
A common way to replace the contents of a FlowFile with specific text is to use
the ReplaceText processor. The default search query will match the entire
content and replace it with whatever your replacement value is. So if you set
the replacement value (complete with expression
to build the code for only a particular processor? Just curious
if we can build and deploy a particular processor in an existing NiFi
environment.
On Wed, Jan 13, 2016 at 9:33 PM, sudeep mishra <sudeepshekh...@gmail.com> wrote:
Thanks Joe. I will try out the patch.
On Wed, Jan 13, 2016 a
ecific configuration as of now as I am still exploring NiFi.
Though I think it would be helpful to let user store and retrieve the cache
values in different formats json, avro etc.
Thanks & Regards,
Sudeep
On Tue, Jan 12, 2016 at 9:15 PM, Joe Percivall <joeperciv...@yahoo.com> wrot
Hello Roland,
Were you able to achieve any success incorporating the JSON sorting into a
custom processor?
Joe
- - - - - -
Joseph Percivall
linkedin.com/in/Percivall
e: joeperciv...@yahoo.com
On Monday, January 11, 2016 2:03 AM, 彭光裕 wrote:
Hello Joe,
Thanks
Hello Lars and Sudeep,
I created a Jira ticket for this issue:
https://issues.apache.org/jira/browse/NIFI-1407
We can continue the conversation there instead two different email threads.
Joe - - - - - - Joseph Percivalllinkedin.com/in/Percivalle:
joeperciv...@yahoo.com
On Thursday,
Hello Madhu,
If you're looking for a template to show how to create a dynamic property for
RouteOnAttribute to use, I'd suggest checking out this template[1]. It is a
simple template that checks to see if the an attribute matches 'NiFi'.
Also provenance can be a very powerful debugging tool.
Hello Andrew,
I believe I was running into something similar before and my problem was
actually with the curl command itself. When I added "--data-binary" to the
command it worked.
Relevant SO question:
http://stackoverflow.com/questions/9134003/binary-data-posting-with-curl
Joe
- - - - - -
Hello Obaid,
Sorry no one has gotten back to you sooner, many of the developers are working
diligently to get 0.5.0 done.
I don't know too much about loading SQL tables but when you say "local file
system filename", is this a file that exits on the target system? If so you may
just be able to
Hello Sudeep,
How precise do you need the date/time to be? What you could do is add an
UpdateAttribute processor[1] after ingesting which uses the Expression language
functions "now" [2] and "format" [3] to add the date/time down to the
millisecond.
There would of course be a bit of error
ld see an AttributeToJson object
but it seems to be appending the information in flow file content or attribute.
What is a good way to capture only attributes and send it to MongoDb?
On Tue, Feb 2, 2016 at 8:42 PM, Joe Percivall <joeperciv...@yahoo.com> wrote:
Hello Sudeep,
>
>How precise do you
Hello Richard,
This is actually came up a couple days ago and is a known bug [1]. In another
ticket[2] I added the option to InvokeHttp expose the option and my patch is
pending. [1] https://issues.apache.org/jira/browse/NIFI-1396[2]
https://issues.apache.org/jira/browse/NIFI-1405
Joe- - - - -
Hello Lars,
You are correct that the WAL is different from swapping.
Swapping is used when a single connection queue grows to be very large. A chunk
of the FlowFiles are then swapped out of JVM memory and written to disk. Where
they are stored until they are swapped back in for processing. The
Hello Chris,
The EvaluateJsonPath processor has the property "Destination" which gives you
the option to send it either to the FlowFile content or a FlowFile attribute.
Selecting "flowfile-attribute" will place the value in the "kafka.key"
attribute of the FlowFile. You can find documentation
Hello,
ListFile is a source processor so this behavior is expected. It supports
expression language so that it can be configured to utilize certains methods.
For example, some people may want to get a list of files from a rotating
directory that gets created every hour. To do that they would
Hello Claudio,
Your use-case actually could leverage a couple of recently added features to
create a really cool open-source processor. The two key features that were
added are State Management and the ability to reference processor specific
variables in expression language. You can take a
On Wednesday, March 2, 2016 11:19 AM, Joe Percivall <joeperciv...@yahoo.com>
wrote:
Hello Claudio,
Your use-case actually could leverage a couple of recently added features to
create a really cool open-source processor. The two key features that were
added are State Management and the a
Hello,
Welcome to NiFi!
I just tried running an ExecuteStreamCommand processor with the properties you
have (I created a script and modified the paths to point to a folder that
exists) and two things jump out. One, the Max attribute length must take an
integer. If you set it to be a path the
atch file (there is no new file called foo.txt on my desktop).
>
> Have you any idea what I am doing wrong? I am pretty sure should be an easy
> fix. Please fell free to make any comment or suggestion regarding to my case.
>
> Thanks in advance.
>
>> El 2 mar 2016, a la
Hello Sumo,
Check out the "Replacement Strategy" property of the "ReplaceText"
processor[1]. Setting this to "prepend", an "Evaluation Mode" of "Entire text"
and a character set of UTF-8 you should be able to accomplish this.
Hello Igor,
I got your template working by using the below replacement string and changing
the "Replacement Strategy" to "Always Replace". I've attached a template that
works for me.
{"test":"${teststr:replaceAll('"','"')}"}
The backslashes are a bit weird because they escape characters
Hello Stéphane,
Just adding on to Matt's and Andy's answers, Andy mentioned Provenance[1] for
replaying events but I also find it very useful for debugging processors/flows
as well. Data Provenance is a core feature of NiFi and it allows you to see
exactly what the FlowFile looked like
Hello
The Apache NiFi team would like to announce the release of Apache NiFi
1.0.0-BETA.
The upcoming 1.0.0 release will mark the culmination of a lot of work over
the last few months with many new framework level features being added.
This Beta release was cut to give our Apache NiFi users a
Hey Stephane,
Currently working with Decimals in NiFi is like putting a square peg in a round
hole. I haven't tried to do it much but for a simple use-case like yours I
believe there are two options.
1: Use Expression Language and a Regex to move the decimal2: (Not 100% sure
this works) Use
Loader.loadClass(ClassLoader.java:424) ~[na:1.8.0_72]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_72]
... 11 common frames omitted
On Tue, Jun 28, 2016 at 2:16 PM, Joe Percivall <joeperciv...@yahoo.com> wrote:
Okhttp has elected to not have built-in support for Digest Auth[
>> Ismael
>>
>>
>>
>> --
>> View this message in context: http://apache-nifi-users-list.
>> 2361937.n4.nabble.com/How-to-use-Rest-API-ressource-data-tra
>> nsfer-tp666.html
>> Sent from the Apache NiFi Users List mailing list archive at Nabble.com.
>>
>
>
--
*Joe Percivall*
linkedin.com/in/Percivall
e: jperciv...@apache.com
far as i can
>>>> tell doesn't have access to that flowfile content.
>>>>
>>>> Without having to put another processor in between monitor activity and
>>>> put slack, is there a way to use the flowfile content directly from
>>>> expression language?
>>>>
>>>> If not does anyone else think this would be really useful to make
>>>> processors like these more compatible?
>>>>
>>>
>>>
>>
>
--
*Joe Percivall*
linkedin.com/in/Percivall
e: jperciv...@apache.com
- Moving users list to BCC
Hello Gunjan,
This seems like a good potential idea. The proper place to submit the
suggestion is through the Apache NiFi Jira[1]. It can more easily be discussed
and worked on there.
[1] https://issues.apache.org/jira/browse/NIFI
Suggestions/ideas from users are
Hello Adam,
Sorry no one has responded yet.
Taking a look at the stack trace, I think you are running into NIFI-2087[1].
This was addressed in 1.0.0.
[1] https://issues.apache.org/jira/browse/NIFI-2087
Joe
- - - - - -
Joseph Percivall
linkedin.com/in/Percivall
e: joeperciv...@yahoo.com
Hello Keren,
The "decompress" mode is an option of the CompressContent[1] processor and
should solve your use-case.
[1]
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.CompressContent/index.html
Joe
- - - - - -
Joseph Percivall
Hello John,
Ah sorry, that mention of "Non-stable" is my fault. The current 1.0.0 release
is stable. That note was for the 1.0.0-BETA release and I forgot to remove it
when changing over to 1.0.0.
It have been removed.
Thank you for brining this up!
Joe
- - - - - -
Joseph Percivall
Hello Sai,
I'm gonna paraphrase what I think your use-case is first, let me know if this
is wrong. You want to keep track of the number of logs coming in and every hour
you want to document how many came in in that hour. Currently NiFi doesn't
handle this type of "stateful" event processing
Nov 3, 2016 at 2:39 PM, Joe Percivall <joeperciv...@yahoo.com> wrote:
Hello,
>
>
>Sorry you are having problems with PublishMQTT. What MQTT broker are you
>trying to hit? and could you pass along what configuration you have set
>(including scheduling tab)? I'd like to tr
Hello Panos,
With the 1.0.0 update, security policies were added to process groups and
controller level features. This means that there is now a difference between
creating a Controller Service (CS) which is scoped to the controller (ie. for
Reporting tasks) and creating a CS which is scoped to
Hello,
Sorry you are having problems with PublishMQTT. What MQTT broker are you trying
to hit? and could you pass along what configuration you have set (including
scheduling tab)? I'd like to try and reproduce if possible.
I'm guessing there is nothing more to that stacktrace? I ask (hoping that
Hey Juan,
I believe you need to escape the escape, so in order to do a double quote it
would be \\" as the property value.
Joe
- - - - - -
Joseph Percivall
linkedin.com/in/Percivall
e: joeperciv...@yahoo.com
On Friday, December 16, 2016 2:05 PM, Juan Sequeiros
Hello,
I don't see any text between "... test error :" and "(on CentOS it works
fine)". Could you try reformatting and resending?
Joe
On Thu, Dec 29, 2016 at 5:26 AM, ddewaele wrote:
> When I try to create a local build of nifi on my mac I always get the
> following test
Hello Kevin,
Thanks for emailing the users list. What problems exactly are you running
into?
The repository.apache.org is the Apache repo that holds the artifacts after
they are released by the Release Manager. I released 1.1.1 in the same way
as 0.7.1, 1.0.0 and 1.0.1 so I'm not sure where the
ticket for this, or post this on the mailinglist ?
On Tue, Dec 20, 2016 at 5:03 PM, Joe Percivall <joeperciv...@yahoo.com> wrote:
> I am running on a Mac version 10.10.5. I pulled down the 1.1.0 zip, created a
> flow with a GenerateFF -> ReplaceText -> LogAttribute (one for success an
This question actually gets back to a discussion on the user entering literal
vs. escaped text. In the NiFi UI the user inputs the text into the box and then
it is converted into a Java String which gets automatically escaped in order to
pass along the string as the user wrote it (so a
, December 21, 2016, 9:04:32 AM EST, Joe Percivall
<joeperciv...@yahoo.com> wrote:
Totally didn't realized I accidentally moved this off of the mailing list.
Brining it back.
--
I just realized as well that NiFi is only using Quartz for the CronExpression.
I think this bug
, 2016, 2:06:44 PM EST, Joe Percivall
<joeperciv...@yahoo.com> wrote:
I have created a JIRA here[1] for the issue.
If you need a fix/workaround now (and are using the latest version), then you
may want to utilize the newly added ability to use state with UpdateAttribute.
You can set up
Hello NiFi Users,
Thanks to a contribution by Andre de Miranda we have a new "Powered By
Apache NiFi" section on our webpage[1]. It would be great to expand this
list out with many more companies and organizations that are using Apache
NiFi.
If you'd like to be added to this list please feel
well for me. I see that the differentiator is configurable and
> additional implementations can be provided. Are there any examples of
> circumstances in which the using WholeConfigDifferentiator would not be the
> best choice?
>
> Thanks,
> Jeff
>
--
*Joe Percivall*
linkedin.com/in/Percivall
e: jperciv...@apache.com
e to be set to a
> small number of flowfiles to keep the source from flooding the destination.
> Has anyone come across this problem before? I am running 12 instances of
> NiFi on version 1.1.1 on Ubuntu 14.04.
>
>
>
> Regards,
>
>
>
> Kevin
>
>
>
--
*Joe Percivall*
e: joeperciv...@gmail.com
al level - a lightweight NiFi server to
> accelerate delivery of priority one content and email alerts, a priority
> two suite of NiFi servers for standard flowfile volume, a priority three
> suite of servers to handle long-term bulk processing, etc etc?
>
> Thanks in advance for your help. -Jim
>
--
*Joe Percivall*
linkedin.com/in/Percivall
e: jperciv...@apache.com
uot;, has anyone used them in practice?
[1]
https://community.hortonworks.com/articles/7882/hdfnifi-best-practices-for-setting-up-a-high-perfo.html
[2]
https://developers.redhat.com/blog/2017/04/04/openjdk-and-containers/#more-433899
[3]
https://blog.csanchez.org/2017/05/31/running-a-jvm-in-a-conta
l/2980
> [2] https://github.com/apache/nifi/pull/2983
>
>
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
>
--
*Joe Percivall*
linkedin.com/in/Percivall
e: jperciv...@apache.com
/questions/49854237/is-xxmaxramfraction-1-safe-for-production-in-a-containered-environment
>
>
> On Thu, Aug 30, 2018 at 7:54 PM Joe Percivall
> wrote:
>
>> Hey everyone,
>>
>> I was recently searching for a best practice guide for running a
>> production instance o
re already supported via
> JsonQueryElasticSearch. If you didn't know that, please try it out and drop
> some feedback on what is needed to make it work for your use cases.)
>
> Thanks,
>
> Mike
>
--
*Joe Percivall*
linkedin.com/in/Percivall
e: jperciv...@apache.com
the input JSON
> document and saving the information inside a Flow attribute.
> >
> > What can I do about this? I don't like monkey patching.. is there any
> other way to get around this?
> >
> > Thanks
>
--
*Joe Percivall*
linkedin.com/in/Percivall
e: jperciv...@apache.com
80 matches
Mail list logo