t; AFAIK, you'll have to create a custom version of GetKafka that enables
> adjustments to fetch.message.max.bytes.
>
>
>
> On Thu, Oct 13, 2016 at 1:22 PM, Igor Kravzov <igork.ine...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I am getting the following exceptio
Hi,
I am getting the following exception in nifi-0.6.1:
kafka.common.MessageSizeTooLargeException: Found a message larger than the
maximum fetch size of this consumer. Increase the fetch size, or decrease
the maximum message size the broker will allow.
What is the max size? How can I increase
s a warn worthy event or if it should be ignored; the configuration
> you would likely desire in this case.
>
> Let us know if this does not work out as anticipated.
>
>
>
>
> On Mon, Aug 8, 2016 at 2:47 PM, Igor Kravzov <igork.ine...@gmail.com>
> wrote:
>
>>
Hi,
In my workflow I need to identify retweets. According to Twitter API
"Native retweets" are tweet objects that contain a child "retweeted_status"
node that contains the original tweet being retweeted. Identify them by the
presence or absence of that node.
I am using EvaluateJsonPath
ark
>
>
>
> On Jun 29, 2016, at 3:37 PM, Igor Kravzov <igork.ine...@gmail.com> wrote:
>
> How can I convert tweet date/time stamp in format "Wed Jun 29 19:04:20
> + 2016" to this format "2016-06-29 15:04:41" using date functions?
>
> Thanks in advance.
>
>
>
How can I convert tweet date/time stamp in format "Wed Jun 29 19:04:20
+ 2016" to this format "2016-06-29 15:04:41" using date functions?
Thanks in advance.
Great. thanks.
On Wed, Jun 29, 2016 at 3:18 PM, Mark Payne <marka...@hotmail.com> wrote:
> Igor ,
>
> You can use a comma-separated list of e-mail names.
>
> Thanks
> -Mark
>
> > On Jun 29, 2016, at 3:16 PM, Igor Kravzov <igork.ine...@gmail.com>
>
Hi,
I have a date which comes in format like this
one 2016-05-17T18:10:44-04:00. -04:00 is timezone.
What format should I set in toDate method to properly convert to a date?
essure
> and make sure I don't pile up too many flow files and fill up the disk.
>
> Thanks,
>
> James
>
> On Thu, Jun 9, 2016 at 2:35 PM, Igor Kravzov <igork.ine...@gmail.com>
> wrote:
>
>> Guys, what is the actual role of funnel?
>>
>
>
Guys, what is the actual role of funnel?
set to
> an attribute populated with the MMdd-formatteddate so it bins files by
> day.
>
> Thanks,
>
> James
>
>
> On Thu, Jun 9, 2016 at 7:37 AM, Igor Kravzov <igork.ine...@gmail.com>
> wrote:
>
>> Hi,
>>
>> In my workflow data is com
Hi,
In my workflow data is coming form Kafka Topic in JSON format and saved ot
HDFS directory by day.
HDFS directory path constructed as /mypath/${now():format("MMdd")}
JSON file contains attribute {... "adddate":"2016-06-07 17:06:17" }
I want to avoid over-spill of a previous date files
Sorry, I found the issue. Client ID and group name properties contained
spaces..
On Wed, Jun 8, 2016 at 6:39 PM, Oleg Zhurakousky <
ozhurakou...@hortonworks.com> wrote:
> I am not sure I understand the question
>
> Sent from my iPhone
>
> > On Jun 8, 2016, at 18:23,
ing; when it's
> converted to an attribute by EvaluateJsonPath it should be converted to a
> String. If this is indeed the distinction please let us know, it sounds
> like it could be a bug.
>
> Regards,
> Matt
>
> On Jun 7, 2016, at 7:30 PM, Igor Kravzov <igork.ine.
use "id" as the Identifier Attribute value in
> PutElasticsearch.
>
> I admit this is one of the more confusing properties of the
> Elasticsearch processors and I sometimes doubt the "wisdom" behind my
> choice :-P Please let me know if this works for you, or if you
Hi,
I have JSON file something like this
{"id":160889137,"url":"
http://twitter.com/tchiagoolimpio/statuses/740301352253825024
","pagetype":"twitter")
and want ti use id as identifier in ES but getting the error bellow:
2016-06-07 18:16:40,363 ERROR [Timer-Driven Process Thread-3]
the of the Max Bin Age.
>
> Thanks
> -Markk
>
> > On May 31, 2016, at 12:07 PM, Igor Kravzov <igork.ine...@gmail.com>
> wrote:
> >
> > There are 2 configuration properties: Maximum Group Size and Maximum
> Number of entries.
> > Are these mutually exclus
There are 2 configuration properties: Maximum Group Size and Maximum Number
of entries.
Are these mutually exclusive? I want to create a file to store in HDFS but
limit size at 64MB as HDFS block (or should I go bigger?).
Max Bin Age property
Since content can be in different length and and not
rofile a specific portion of your data flow:
>
> start_time = ${now():toNumber()}
>
> data flow of some duration
>
> elapsed_time = ${now():toNumber():minus(${start_time})}
>
>
> -Lee
>
>
>
>
> On Fri, May 27, 2016 at 2:21 PM, Igor Kravzov <igork.ine...@
> I'll look in more details once back on line
>
> Oleg
>
> Sent from my iPhone
>
> On May 26, 2016, at 17:20, Igor Kravzov <igork.ine...@gmail.com> wrote:
>
> Why I am getting exception bellow
>
>
> 2016-05-26 17:13:36,776 INFO
> [d217bceb-d58b-4031-9143-
(in terms of
> sequence of events)
>
> Thanks
> Oleg
>
> > On May 26, 2016, at 4:12 PM, Igor Kravzov <igork.ine...@gmail.com>
> wrote:
> >
> > Why I am getting warning bellow?
> >
> >
> >
> >
>
>
through and you
> should be rockin'.
>
> Thanks
> Joe
>
> On Thu, May 19, 2016 at 5:37 PM, Igor Kravzov <igork.ine...@gmail.com>
> wrote:
> > If I want to use PutHDFS processor does it mean that NiFi instance should
> > run on the same machine as hadoop in
operties file you can - specifically around cluster
> configuration?
>
> Thanks
> Joe
>
> On Thu, May 19, 2016 at 3:00 PM, Igor Kravzov <igork.ine...@gmail.com>
> wrote:
> > NiFi 0.6.1
> > I followed simple cluster setup guide. But when i start NCM it shuts do
nting throttling from your web service.
>
>
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ControlRate/index.html
>
> Thanks,
>
> James
>
> On Wed, May 18, 2016 at 11:31 AM, Igor Kravzov <igork.ine...@gmail.com>
> wrot
ategy of 'filename' and point at a file containing precisely the
>> bytes you want.
>>
>> Thanks
>> Joe
>>
>> On Thu, May 12, 2016 at 3:49 PM, Igor Kravzov <igork.ine...@gmail.com>
>> wrote:
>> > Joe, If I put \n or '\n' the processor add
quot;Delimiter Strategy" to "text" and put a value for
> the demarcator of \n.
>
> That should get you there I think.
>
> [1]
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.MergeContent/index.html
>
> Thanks
> Joe
>
&g
Looks like am missing something. How to configure SplitJson to split array
like bellow to individual JSON files. Basically split on "root" of array.
[{
"id":1,
"data":"data1"
},
{
"id":2,
"data":"data2"
}
]
Result should be to separate files
{
"id":1,
Hi,
I am about to build following workflow:
1. Reads file (search keyword on a new line) form a disk (GetFile PR)
2. Split file (SplitText PR)
3. Assign splits line to an attribute (UpdateAttribute, how???)
4. Pass attribute to HTTP query (InvokeHTTP PR)
So, how to assign split line to an
ot;ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
>
> ./logs/nifi-bootstrap_%d.log
>
> 5
>
>
>
>
>
> On Fri, May 6, 2016 at 9:40 AM, Igor Kravzov <igork.ine...@gmail.com>
> wrote:
>
>> Is there a configuration option to produce daily application log file
>> instead of one big file?
>>
>
>
Is there a way to monitor a particular processor or a relationship?
For example I would like to get a notification if a processor started to
throw errors. Or to be notified if a particular number of flowfiles goes
to "fail" relationship. Or entire node is down.
-483
>
> Thanks
> Joe
>
> On Sun, May 1, 2016 at 2:36 PM, Igor Kravzov <igork.ine...@gmail.com>
> wrote:
> > Thanks Aldrin for the repose.
> > What didn't fully understand from documentation: is automatic fail-over
> > implemented? I would rather configure e
ditional questions or if things are unclear.
>
> --aldrin
>
>
>
> On Sun, May 1, 2016 at 1:20 PM, Igor Kravzov <igork.ine...@gmail.com>
> wrote:
>
>> If I understand correctly in cluster mode the same dataflow runs on all
>> the notes.
>> So l
If I understand correctly in cluster mode the same dataflow runs on all the
notes.
So let's say I have a simple dataflow with GetTwitter and PutHDFS
processors. And one NCM + 2 nodes.
Does it actually that mean the GetTwitter will be called independently and
potentially simultaneously on each node
lookup keys). NiFi's DistributedCache
> facility might be a good fit for such data.
>
> Andrew
>
> From: Igor Kravzov <igork.ine...@gmail.com>
> Reply-To: "users@nifi.apache.org" <users@nifi.apache.org>
> Date: Monday, April 25, 2016 at 9:58 PM
> To: "us
it
> can lead needing to repeat them multiple times (in this case 4).
>
> Hope this helps,
> Joe
> - - - - - -
> Joseph Percivall
> linkedin.com/in/Percivall
> e: joeperciv...@yahoo.com
>
>
>
> On Tuesday, April 26, 2016 6:10 PM, Igor Kravzov <igork.ine...@gmail
tiple times (in this case 4).
>
> Hope this helps,
> Joe
> - - - - - -
> Joseph Percivall
> linkedin.com/in/Percivall
> e: joeperciv...@yahoo.com
>
>
>
> On Tuesday, April 26, 2016 6:10 PM, Igor Kravzov <igork.ine...@gmail.com>
> wrote:
>
>
>
> Atta
Attached please find the test template. NiFi 0.6.1
I am trying to replace " with \" in a text. So "Here "we" go" should
become \"Here \"we\" go\"
The call is in ReplaceText
processor: {"test":"${teststr:replace('"','\\"')}"}
teststr cerated in UpdateAttribute.
>From some reason unable to make
ntity/user objects. Then you'll get a flow file per entity/user,
> meaning you'll get a separate ES doc for each entity and user,
> stored/indexed/categorized by its type.
>
> Does this help solve your use case? If not please let me know, I'm
> happy to help work through thi
Is there a way to refer to original FlowFile after AttributesToJSON call?
Destination set to flowfile-content.
Or is there a way to produce a FlowFile from an attribute?
I need to extract some properties from Twitter JSON, convert to a new JSON,
and after to extract another property from
if you are willing to build NiFi yourself to test it out.
>
> Cheers,
> Chris
>
> [1] https://issues.apache.org/jira/browse/NIFI-1660
>
>
> From: Igor Kravzov <igork.ine...@gmail.com<mailto:igork.ine...@gmail.com>>
> Reply-To: "users@nifi.apache.org<mail
nal functionality.
>
> Thanks!
> Aldrin
>
>
>
> On Mon, Apr 25, 2016 at 10:33 AM, Igor Kravzov <igork.ine...@gmail.com>
> wrote:
>
>> Hi Aldrin,
>>
>>
>> Attached please find the template. In this workflow I want to pull
>> "entities
a ticket (
> https://issues.apache.org/jira/browse/NIFI) and providing a template of
> your flow that is causing the issue?
>
> Thanks!
>
> On Mon, Apr 25, 2016 at 10:09 AM, Igor Kravzov <igork.ine...@gmail.com>
> wrote:
>
>> Thanks Pierre. It worked. Looks
ate of
> your flow that is causing the issue?
>
> Thanks!
>
> On Mon, Apr 25, 2016 at 10:09 AM, Igor Kravzov <igork.ine...@gmail.com>
> wrote:
>
>> Thanks Pierre. It worked. Looks like I was doing something wrong inside
>> my workflow.
>> Would not
te example attached.
>
> HTH
> Pierre
>
>
> 2016-04-24 20:12 GMT+02:00 Igor Kravzov <igork.ine...@gmail.com>:
>
>> I am not that good in regex. What would be the proper configuration to do
>> the following;
>>
>>1. Remove backslash from text.
I am not that good in regex. What would be the proper configuration to do
the following;
1. Remove backslash from text.
2. Replace "{ with {
3. replace }" with }
Basically I need to clean escaped JSON.
Like before:
case, you may try to add a ReplaceText processor at the end
> of your flow to use regular expressions and remove the backslash characters.
>
> Let me know if it does not help.
> Pierre
>
> 2016-04-24 0:24 GMT+02:00 Igor Kravzov <igork.ine...@gmail.com>:
>
&g
Hi,
I am using EvaluateJsonPath to extract some properties from tweet JSON
returned by GetTwitter processor and later convert to another JSON using
AttributesToJSON processor. One of the properties I want to extract is
"entities" which is structure of arrays by itself.
Unfortunately
; On Thu, Apr 21, 2016 at 5:24 PM, Igor Kravzov <igork.ine...@gmail.com>
> wrote:
> > Thanks guys. I think it will work.
> > One thing: merged file comes out without extension. How do I add
> extension
> > to a merged file?
> >
> > On Thu, Apr 21, 2016 at 4:42
valid array of JSON documents, in MergeContent change
>> the Delimiter Strategy to "Text" which means it will use whatever values
>> you type directly into Header, Footer, Demarcator, and then specify [ ] ,
>> respectively as the values.
>>
>> That will get you s
ng,
> displayname string,
> msg string,
> fulltext string
> )
> row format delimited fields terminated by "|"
> location "/tmp/tweets_staging";
>
> -Bryan
>
>
> On Thu, Apr 21, 2016 at 1:52 PM, Igor Kravzov <igork.ine...@gmail.com>
> wrote:
50 matches
Mail list logo
