Re: NPE MergeContent processor

[Conrad Crampton]

Hi Mark,
That is really good news. As to whether it sounds familiar – I try so many 
things when I was upgrading from 0.6.1 to 1.0.0 I couldn’t say whether this was 
indeed the cause. It may have been – I think though whilst the outcome is 
probably the same, the exact cause may have been due the aforementioned 
‘titting about’ ☺
Anyway, I was thinking of splitting my cluster in two anyway to make better use 
of the syslog ingestion (I think that will give me better throughput as I am 
seeing the syslog ingestion as a bottleneck with repeated warnings over full 
buffer), at which point I will delete the provenance repository anyway which 
will get rid of this won’t it? I’m assuming I can delete all repositories and 
just leave the flowfile.xml to have the same starting point for the workflows?
Anyway, thanks again for pursuing this and once again I am incredibly impressed 
with the reaction to issues/ bugs etc. in this community.
Regards
Conrad

From: Mark Payne 
Reply-To: "users@nifi.apache.org" 
Date: Tuesday, 15 November 2016 at 18:09
To: "users@nifi.apache.org" 
Subject: Re: NPE MergeContent processor


Conrad,



Good news - I have been able to replicate the issue and track down the problem. 
I created a JIRA to address it - 
https://issues.apache.org/jira/browse/NIFI-3040.

I have a PR up to address the issue. It looks like the problem is due to 
Replaying a FlowFile from Provenance and then restarting NiFi before the 
replayed FlowFile

has completed processing. Does that sound familiar?



In the case of MergeContent you'd see a NullPointerException. In other cases it 
will generally just complain because the UUID is null.



The issue has to do with the FlowFile not being properly persisted when a 
REPLAY event occurs. So if you still have the FlowFile that is causing this, 
you'd have

to manually remove it from its queue to address the issue, but the issue 
shouldn't happen any more after this fix makes its way in.



Sorry that this has been you, but thanks for working with us to give us all we 
needed to investigate. And thanks for being patient as we've diagnosed and dug 
in.



Cheers

-Mark


From: Oleg Zhurakousky 
Sent: Friday, November 11, 2016 2:07 PM
To: users@nifi.apache.org
Subject: Re: NPE MergeContent processor

Sorry, I should have been more clear.
I’ve spent considerable amount of time slicing and dicing this thing and while 
I am still validating few possibilities, this is more likely to due to FlowFile 
being rehydrated from the corrupted repo with missing UUID and when such file’s 
ID ends up to be in a parent/child of ProvenanceEventRecord we get this issue.
Basically FlowFile must never exist without UUID similar to the way provenance 
event record where existence if UUID is validated during the call to build(). 
We should definitely do the same in a builder for FlowFile and even though it 
will not eliminate the issue it may help to pin point its origin.

I’ll raise  the corresponding JIRA to improve FlowFile validation.

Cheers
Oleg

> On Nov 11, 2016, at 3:00 PM, Joe Witt  wrote:
>
> that said even if it is due to crashes or even disk full cases we
> should figure out what happened and make it not possible.  We must
> always work to eliminate the possibility of corruption causing events
> and work to recover well in the face of corruption...
>
> On Fri, Nov 11, 2016 at 2:57 PM, Oleg Zhurakousky
>  wrote:
>> Conrad
>>
>> Is it possible that you may be dealing with corrupted repositories (swap,
>> flow file etc.) due to your upgrades or may be even possible crashes?
>>
>> Cheers
>> Oleg
>>
>> On Nov 11, 2016, at 3:11 AM, Conrad Crampton 
>> wrote:
>>
>> Hi,
>> This is the flow. The incoming flow is basically a syslog message which is
>> parsed, enriched then saved to HDFS
>> 1.   Parse (extracttext)
>> 2.   Assign matching parts to attributes
>> 3.   Enrich ip address location
>> 4.   Assign attributes with geoenrichment
>> 5.   Execute python script to parse useragent
>> 6.   Create json from attributes
>> 7.   Convert to avro (all strings)
>> 8.   Convert to target avro schema (had to do 7 & 8 due to bug(?) where
>> couldn’t go directly from json to avro with integers/longs)
>> 9.   Merge into bins (see props below)
>> 10.   Append ‘.avro’ to filenames (for reading in Spark subsequently)
>> 11.   Save to HDFS
>>
>> Does this help at all?
>> If you need anything else just shout.
>> Regards
>> Conrad
>>
>> 
>>
>>
>> 
>> additional out of shot
>> · compression level : 1
>> · Keep Path : false
>>
>>
>> From: Oleg Zhurakousky 
>> Reply-To: "users@nifi.apache.org" 
>> Date: Thursday, 10 November 2016 at 18:40
>> To: "users@nifi.apache.org" 
>> 

Re: NiFi 1.0.0 canvas background

[Matt Gilman]

Russ,

There is no way to change the background color. You'd have to fork the
codebase and update the style for the canvas-container. I believe
previously this was implemented with a repeating background image and is
now implemented with CSS. I also thought the change was subtle.

There has been discussion about implementing an align feature but it's not
something that's been done yet. There is a JIRA that was created awhile
back [1].

Matt

[1] https://issues.apache.org/jira/browse/NIFI-96


On Tue, Nov 15, 2016 at 3:14 PM, Russell Bateman <
russell.bate...@perfectsearchcorp.com> wrote:

> Now that I'm working in the 1.x world, there's no way I can restore the
> nice, light grid from 0.x to 1.x in place of the heavy, dull "fabric"
> background of the canvas in the new version, is there?
>
> I've Googled and prowled around the interface with no success.
>
> Also, wasn't there a way to align rectangles? I thought I remembered
> seeing that in 0.x, but couldn't find how to do it there or in 1.x. Maybe I
> was having a LibreOffice Draw-inspired dream or something.
>
> Thanks,
>
> Russ
>

Re: NiFi 1.0.0 canvas background

[Andrew Grande]

Hi Russ,

Alignment or any snapping to a grid wasn't there before, but that would be
a very welcome feature, I agree.

Regarding the background, I personally didn't notice a change, isn't it the
same?

Andrew

On Tue, Nov 15, 2016, 3:14 PM Russell Bateman <
russell.bate...@perfectsearchcorp.com> wrote:

> Now that I'm working in the 1.x world, there's no way I can restore the
> nice, light grid from 0.x to 1.x in place of the heavy, dull "fabric"
> background of the canvas in the new version, is there?
>
> I've Googled and prowled around the interface with no success.
>
> Also, wasn't there a way to align rectangles? I thought I remembered
> seeing that in 0.x, but couldn't find how to do it there or in 1.x. Maybe I
> was having a LibreOffice Draw-inspired dream or something.
>
> Thanks,
>
> Russ
>

NiFi 1.0.0 canvas background

[Russell Bateman]

Now that I'm working in the 1.x world, there's no way I can restore the 
nice, light grid from 0.x to 1.x in place of the heavy, dull "fabric" 
background of the canvas in the new version, is there?


I've Googled and prowled around the interface with no success.

Also, wasn't there a way to align rectangles? I thought I remembered 
seeing that in 0.x, but couldn't find how to do it there or in 1.x. 
Maybe I was having a LibreOffice Draw-inspired dream or something.


Thanks,

Russ

Re: NPE MergeContent processor

[Mark Payne]

Conrad,


Good news - I have been able to replicate the issue and track down the problem. 
I created a JIRA to address it - 
https://issues.apache.org/jira/browse/NIFI-3040.

I have a PR up to address the issue. It looks like the problem is due to 
Replaying a FlowFile from Provenance and then restarting NiFi before the 
replayed FlowFile

has completed processing. Does that sound familiar?


In the case of MergeContent you'd see a NullPointerException. In other cases it 
will generally just complain because the UUID is null.


The issue has to do with the FlowFile not being properly persisted when a 
REPLAY event occurs. So if you still have the FlowFile that is causing this, 
you'd have

to manually remove it from its queue to address the issue, but the issue 
shouldn't happen any more after this fix makes its way in.


Sorry that this has been you, but thanks for working with us to give us all we 
needed to investigate. And thanks for being patient as we've diagnosed and dug 
in.


Cheers

-Mark



From: Oleg Zhurakousky 
Sent: Friday, November 11, 2016 2:07 PM
To: users@nifi.apache.org
Subject: Re: NPE MergeContent processor

Sorry, I should have been more clear.
I’ve spent considerable amount of time slicing and dicing this thing and while 
I am still validating few possibilities, this is more likely to due to FlowFile 
being rehydrated from the corrupted repo with missing UUID and when such file’s 
ID ends up to be in a parent/child of ProvenanceEventRecord we get this issue.
Basically FlowFile must never exist without UUID similar to the way provenance 
event record where existence if UUID is validated during the call to build(). 
We should definitely do the same in a builder for FlowFile and even though it 
will not eliminate the issue it may help to pin point its origin.

I’ll raise  the corresponding JIRA to improve FlowFile validation.

Cheers
Oleg

> On Nov 11, 2016, at 3:00 PM, Joe Witt  wrote:
>
> that said even if it is due to crashes or even disk full cases we
> should figure out what happened and make it not possible.  We must
> always work to eliminate the possibility of corruption causing events
> and work to recover well in the face of corruption...
>
> On Fri, Nov 11, 2016 at 2:57 PM, Oleg Zhurakousky
>  wrote:
>> Conrad
>>
>> Is it possible that you may be dealing with corrupted repositories (swap,
>> flow file etc.) due to your upgrades or may be even possible crashes?
>>
>> Cheers
>> Oleg
>>
>> On Nov 11, 2016, at 3:11 AM, Conrad Crampton 
>> wrote:
>>
>> Hi,
>> This is the flow. The incoming flow is basically a syslog message which is
>> parsed, enriched then saved to HDFS
>> 1.   Parse (extracttext)
>> 2.   Assign matching parts to attributes
>> 3.   Enrich ip address location
>> 4.   Assign attributes with geoenrichment
>> 5.   Execute python script to parse useragent
>> 6.   Create json from attributes
>> 7.   Convert to avro (all strings)
>> 8.   Convert to target avro schema (had to do 7 & 8 due to bug(?) where
>> couldn’t go directly from json to avro with integers/longs)
>> 9.   Merge into bins (see props below)
>> 10.   Append ‘.avro’ to filenames (for reading in Spark subsequently)
>> 11.   Save to HDFS
>>
>> Does this help at all?
>> If you need anything else just shout.
>> Regards
>> Conrad
>>
>> 
>>
>>
>> 
>> additional out of shot
>> · compression level : 1
>> · Keep Path : false
>>
>>
>> From: Oleg Zhurakousky 
>> Reply-To: "users@nifi.apache.org" 
>> Date: Thursday, 10 November 2016 at 18:40
>> To: "users@nifi.apache.org" 
>> Subject: Re: NPE MergeContent processor
>>
>> Conrad
>>
>> Any chance you an provide a bit more info about your flow?
>> I was able to find a condition when something like this can happen, but it
>> would have to be with some legacy NiFi distribution, so it’s a bit puzzling,
>> but i really want o see if we can close the loop on this.
>> In any event I think it is safe to raise JIRA on this one
>>
>> Cheers
>> Oleg
>>
>>
>> On Nov 10, 2016, at 10:06 AM, Conrad Crampton 
>> wrote:
>>
>> Hi,
>> The processor continues to write (to HDFS – the next processor in flow) and
>> doesn’t block any others coming into this processor (MergeContent), so not
>> quite the same observed behaviour as NIFI-2015.
>> If there is anything else you would like me to do to help with this more
>> than happy to help.
>> Regards
>> Conrad
>>
>> From: Bryan Bende 
>> Reply-To: "users@nifi.apache.org" 
>> Date: Thursday, 10 November 2016 at 14:59
>> To: "users@nifi.apache.org" 
>> Subject: Re: NPE MergeContent processor
>>
>> Conrad,
>>
>> Thanks for reporting this. I wonder if this is also related to:
>>
>> 

Re: Delay Processor

[Andrew Grande]

Joe,

It's good to know some thinking went into this feature before. Basically,
I'm trying to put a spotlight on these 2 areas:

   1. Making it, potentially, more generic than a retry loop. E.g. enhance
   the ControlRate processor.
   2. Making these policies more explicit, so a user wouldn't have to
   second-guess on where to go to configure behavior. Here I don't have a
   strong opinion on whether a dedicated (or enhanced) processor or changes
   into standard processor configuration screens will be the way.

Andrew

Re: Delay Processor

[Andrew Grande]

Oleg,

I'll break my response in 2 threads. I understand your use case of 'delay
until X', but frankly would design it differently if we're talking about
long-term transactions or schedules. It may involve systems external to
NiFi. Anyway, I'd like to keep this use case out of scope for the delay
processor, at least for the immediate discussion.

Now jumping to another thread..

RE: REST provenance api search options

[philippe.gibert]

Thanks a lot Matt
It works nicely !

From: Matt Gilman [mailto:matt.c.gil...@gmail.com]
Sent: mardi 15 novembre 2016 15:08
To: users@nifi.apache.org
Subject: Re: REST provenance api search options

Philippe,

Here's an example command for initiating a provenance search:

curl 'http://localhost:8080/nifi-api/provenance' -H 'Content-Type: 
application/json' -H 'Accept: application/json, text/javascript, */*; q=0.01' 
--data-binary 
'{"provenance":{"request":{"maxResults":1000,"startDate":"11/15/2016 00:00:00 
EST","endDate":"11/15/2016 23:59:59 
EST","searchTerms":{"FlowFileUUID":"","Filename":"","ProcessorID":""' --compressed

The available searchable fields are defined in your nifi.properties file under 
the following properties:

nifi.provenance.repository.indexed.fields
nifi.provenance.repository.indexed.attributes

Because the searches can be long running they are performed asynchronously. 
These means that the curl command above creates the search request but does not 
wait for it to complete. Instead, you'll need to get the uuid to the search 
request to continue to GET it until the search completes. Once completed, you 
should DELETE the search request. Open up the Dev Tools in your browser to seen 
this sequence of requests in action.

Let me know if you have any more questions. Thanks.

Matt


On Tue, Nov 15, 2016 at 2:47 AM, 
> wrote:

Hello,

My SW context : nifi 1.0.0/ubuntu

 I am trying to use the provenance search options . I have the Id of my 
processor ie (ProcessorID ) but it’s not very clear for me how to fill the 
searchableFields.

Is something  as the following to be right  ? and in this case where do i put 
the ProcessorID? in the field or  in the Id ? and what for label?

---

curl -i -X PUT -H 'Content-Type: application/json' -d 
'{"searchableFields":[{"id":"ProcessorID","field":"processorId","label":"Component
 ID","type":"STRING"}]}

' http://localhost:8080/nifi-api/provenance/search-options’

--

​ please can you clarify ? with an example ?

Best regards

phil



_

Ce message et ses pieces jointes peuvent contenir des informations 
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce 
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages 
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou 
falsifie. Merci.

This message and its attachments may contain confidential or privileged 
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete 
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been 
modified, changed or falsified.
Thank you.

Re: REST APIs provenance-events API

[Matt Gilman]

Philippe,

Yes, you are correct. These are the endpoints to access flowfile content
before and after a given provenance event. Specifically, these map to the
View/Download buttons in the Content tab for the Input and Output Claim in
the provenance event details dialog.

Matt

On Tue, Nov 15, 2016 at 8:56 AM,  wrote:

> Hello, My SW context : nifi 1.0.0/ubuntu
>
> I would like to know if both following REST API calls :
>
> GET /provenance-events/{id}/content/input
>
> GET /provenance-events/{id}/content/output
>
>
>
> Are related to getting in the response the  flowfile content before and
> after an event happened on a processor .
>
>
>
> My Use case is :
>
> run a dataflow and inspect on every involved  processor in the DF  the
> flow-content data before and after some  events  ?
>
>
>
> Best regards
>
> Philippe
>
>
>
>
>
> [image: Description : Description : Description :
> cid:image002.gif@01CDFAED.D49218F0] 
>
> Philippe Gibert
>
> Ingénieur R
>
> FT/IMT/OLPS/BIZZ/INFSVC/ITS4B
>
> tél. +33 4 92 94 53 70
>
> mob. +33 6 73 41 11 18
>
> philippe.gib...@orange.com
>
>
>
>
>
> _
>
> Ce message et ses pieces jointes peuvent contenir des informations 
> confidentielles ou privilegiees et ne doivent donc
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu 
> ce message par erreur, veuillez le signaler
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages 
> electroniques etant susceptibles d'alteration,
> Orange decline toute responsabilite si ce message a ete altere, deforme ou 
> falsifie. Merci.
>
> This message and its attachments may contain confidential or privileged 
> information that may be protected by law;
> they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete 
> this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been 
> modified, changed or falsified.
> Thank you.
>
>

Re: Delay Processor

[Joe Witt]

The concept of flowfile penalization exists to support many of the
requirements for delay.  This is something which can be set, by a
component (processor), on a flow file for some given period of time
and which will be honored on the connection it is placed on such that
things without this penalization will be higher priority.  It was
built for the typical failure loops.  This is appropriate when it is
something about that 'flowfile' and the thing the processor it trying
to interact with that is the problem and therefore simply marking that
flowfile as problematic for a bit is likely to allow whatever the
condition is to go away without just making the problem worse.

There is also the concept of process yielding.  A component
(processor) can signal that it should yield which means it will not be
triggered again at its normally scheduled time but rather will yield
until its yield period is up.  This makes sense when there is
something about the state of that processor and its configuration
relative to whatever it is trying to do/interacting with which is
expected to be able to go away or be resolved on its own.

Mark Payne and I have talked in the past about the notion of the
framework automatically tracking flowfiles that appear to be looping
and doing exponential backoff on them.  This would be done just as you
suggested which is via flow file attributes.

Is your case/concern more like the 'something is wrong with this flow
file vs environment' or more like 'something is wrong with this
process vs environment'?



On Tue, Nov 15, 2016 at 8:50 AM, Andrew Grande  wrote:
> Hi,
>
> I'd lime to check where discussions are on this, ir propose the new
> component otherwise.
>
> Use case: make delay strategies explicit, easier to use. E.g. think of a
> failure retry loop.
>
> Currently, ControlRate is somewhat related, but can be improved. E.g.
> introduce delay strategies a la prioritizers on the connection?
>
> Thinking out loud, something like a exponential backoff strategy could be
> kept stateless by adding a number of housekeeping attributes to the FF,
> which eliminates the need for any state in the processor itself.
>
> I'll stop here to see if any ideas were captured prior abd what community
> thinks of it.
>
> Andrew

Re: REST provenance api search options

[Matt Gilman]

Philippe,

Here's an example command for initiating a provenance search:

curl 'http://localhost:8080/nifi-api/provenance' -H 'Content-Type:
application/json' -H 'Accept: application/json, text/javascript, */*;
q=0.01' --data-binary
'{"provenance":{"request":{"maxResults":1000,"startDate":"11/15/2016
00:00:00 EST","endDate":"11/15/2016 23:59:59
EST","searchTerms":{"FlowFileUUID":"","Filename":"","ProcessorID":""' --compressed

The available searchable fields are defined in your nifi.properties file
under the following properties:

nifi.provenance.repository.indexed.fields
nifi.provenance.repository.indexed.attributes

Because the searches can be long running they are performed asynchronously.
These means that the curl command above creates the search request but does
not wait for it to complete. Instead, you'll need to get the uuid to the
search request to continue to GET it until the search completes. Once
completed, you should DELETE the search request. Open up the Dev Tools in
your browser to seen this sequence of requests in action.

Let me know if you have any more questions. Thanks.

Matt


On Tue, Nov 15, 2016 at 2:47 AM,  wrote:

> Hello,
>
> My SW context : nifi 1.0.0/ubuntu
>
>  I am trying to use the provenance search options . I have the Id of my
> processor ie (ProcessorID ) but it’s not very clear for me how to fill the
> searchableFields.
>
> Is something  as the following to be right  ? and in this case where do i
> put the ProcessorID? in the field or  in the Id ? and what for label?
>
> ---
>
> curl -i -X PUT -H 'Content-Type: application/json' -d
> '{"searchableFields":[{"id":"ProcessorID","field":"processorId","label":"Component
> ID","type":"STRING"}]}
>
> ' http://localhost:8080/nifi-api/provenance/search-options’
>
> --
>
> ​ please can you clarify ? with an example ?
>
> Best regards
>
> phil
>
>
>
>
>
> [image: Description : Description : Description :
> cid:image002.gif@01CDFAED.D49218F0] 
>
> Philippe Gibert
>
> Ingénieur R
>
> FT/IMT/OLPS/BIZZ/INFSVC/ITS4B
>
> tél. +33 4 92 94 53 70
>
> mob. +33 6 73 41 11 18
>
> philippe.gib...@orange.com
>
>
>
>
>
> _
>
> Ce message et ses pieces jointes peuvent contenir des informations 
> confidentielles ou privilegiees et ne doivent donc
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu 
> ce message par erreur, veuillez le signaler
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages 
> electroniques etant susceptibles d'alteration,
> Orange decline toute responsabilite si ce message a ete altere, deforme ou 
> falsifie. Merci.
>
> This message and its attachments may contain confidential or privileged 
> information that may be protected by law;
> they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete 
> this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been 
> modified, changed or falsified.
> Thank you.
>
>

Re: Delay Processor

[Oleg Zhurakousky]

I am +1 on this as I’ve seen many cases in the field where it is applicable and 
as you mention exponential back off is one of the common one. That said, I am 
wondering if that has to be a processor at all? Actually let me answer my own 
question. There are definitely cases where it has to be a processor. Those are 
true delay with intention requirements (i.e., Compute Andrew’s greetings -> 
Delay until his b-day -> Send Greetings). 
The exponential back off is a bit different since it almost aligns with circuit 
breaker and re-tries. Currently we simply retry by resubmitting the flow file 
with fixed yield. However one may argue that if something failed the first 
time, it is very likely that it is going to fail again and again. It may also 
succeed, but one may argue that it has a higher chance of succeeding after 
certain delay which increases on subsequent failures until we may choose to 
consider it a failed cause and stop resubmitting it.

So in summary I see the two-part requirement;  A processor and enhancement to 
the core-framework’s retry logic.

Thoughts?
Cheers
Oleg


> On Nov 15, 2016, at 8:50 AM, Andrew Grande  wrote:
> 
> Hi,
> 
> I'd lime to check where discussions are on this, ir propose the new component 
> otherwise.
> 
> Use case: make delay strategies explicit, easier to use. E.g. think of a 
> failure retry loop.
> 
> Currently, ControlRate is somewhat related, but can be improved. E.g. 
> introduce delay strategies a la prioritizers on the connection?
> 
> Thinking out loud, something like a exponential backoff strategy could be 
> kept stateless by adding a number of housekeeping attributes to the FF, which 
> eliminates the need for any state in the processor itself.
> 
> I'll stop here to see if any ideas were captured prior abd what community 
> thinks of it.
> 
> Andrew

REST APIs provenance-events API

[philippe.gibert]

Hello, My SW context : nifi 1.0.0/ubuntu
I would like to know if both following REST API calls :
GET /provenance-events/{id}/content/input
GET /provenance-events/{id}/content/output

Are related to getting in the response the  flowfile content before and after 
an event happened on a processor .

My Use case is :
run a dataflow and inspect on every involved  processor in the DF  the 
flow-content data before and after some  events  ?

Best regards
Philippe


[Description : Description : Description : 
cid:image002.gif@01CDFAED.D49218F0]
Philippe Gibert
Ingénieur R
FT/IMT/OLPS/BIZZ/INFSVC/ITS4B
tél. +33 4 92 94 53 70
mob. +33 6 73 41 11 18
philippe.gib...@orange.com



_

Ce message et ses pieces jointes peuvent contenir des informations 
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce 
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages 
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou 
falsifie. Merci.

This message and its attachments may contain confidential or privileged 
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete 
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been 
modified, changed or falsified.
Thank you.

Delay Processor

[Andrew Grande]

Hi,

I'd lime to check where discussions are on this, ir propose the new
component otherwise.

Use case: make delay strategies explicit, easier to use. E.g. think of a
failure retry loop.

Currently, ControlRate is somewhat related, but can be improved. E.g.
introduce delay strategies a la prioritizers on the connection?

Thinking out loud, something like a exponential backoff strategy could be
kept stateless by adding a number of housekeeping attributes to the FF,
which eliminates the need for any state in the processor itself.

I'll stop here to see if any ideas were captured prior abd what community
thinks of it.

Andrew

Re: Nifi- PutEmail processor issue

[Joe Witt]

Sravani,

The flow you describe makes sense.  So now lets focus on the PutEmail
processor.  Can you please clarify if when it fails to connect the
flowfile is transferred to the 'failure' relationship or the 'success'
relationship?  If it flows to the 'failure' relationship then as
Conrad points out you can just simply have failure loop back to that
processor so it will continue to retry until it actually succeeds in
transferring.  If it flows to the 'success' relationship even though
the email send fails then we need to look into it further and the
stack trace oleg requests will be helpful.

Thanks
Joe

On Tue, Nov 15, 2016 at 7:47 AM, Oleg Zhurakousky
 wrote:
> Sravani
>
> Would you be able to provide a full stack trace of the connection exception.
> Also, while I assume you are providing the correct connection properties
> (i.e., host,port etc) I would still recommend to check the they are correct,
> but in any event the full stack trace would definitely help and you cn find
> it the the wifi app logs.
>
> Cheers
> Oleg
>
> On Nov 15, 2016, at 4:07 AM, Gadiputi, Sravani
>  wrote:
>
> Thank you for your reply.
>
> My requirement is , I just try to send/copy the 3 different files from
> source to destination through Nifi, and these jobs runs weekly once.
> So I wanted to know which file is successfully moved through email.
> In this process, I have configured putemail for each flow. There are hardly
> 3 notifications only.
> Though files have been moved to destination, we could not receive the
> notifications properly and giving the below error.
>
> Please suggest.
>
> Thanks,
> Sravani
>
>
> From: Jeff [mailto:jtsw...@gmail.com]
> Sent: Tuesday, November 15, 2016 1:25 PM
> To: users@nifi.apache.org
> Subject: Re: Nifi- PutEmail processor issue
>
> Hello Sravani,
>
> Could it be possible that the SMTP server you're using is denying
> connections due to the volume of emails your flow might be sending?  How
> many emails are sent per flow file, and how many emails do you estimate are
> sent per minute?
>
> If this is the case, you can modify your flow to aggregate flowfiles with a
> processor like MergeContent so that you can send emails that resemble a
> digest, rather than a separate email for each flowfile that moves through
> your flow.
>
> On Mon, Nov 14, 2016 at 11:59 PM Gadiputi, Sravani
>  wrote:
>
>
> Hi,
>
> I have used PutEmail processor in my project to send email notification for
> successful/failure copying of a files.
> Each file flow having corresponding PutEmail to send  email notification to
> respective recipients.
>
> Here the issue is, sometimes email notification will send to respective
> recipients successfully  for successful/failure job.
> But sometimes for any one specific job email notification will not be send
> to recipients though job is successful, due to  below error.
>
> Error:
>
> Could not connect to SMTP host
> Java.net.ConnectException: Connection timed out
>
> Could you please suggest me how we can overcome this error.
>
>
> Thanks,
> Sravani
>
> This message contains information that may be privileged or confidential and
> is the property of the Capgemini Group. It is intended only for the person
> to whom it is addressed. If you are not the intended recipient, you are not
> authorized to read, print, retain, copy, disseminate, distribute, or use
> this message or any part thereof. If you receive this message in error,
> please notify the sender immediately and delete all copies of this message.
>
>

Re: Nifi- PutEmail processor issue

[Oleg Zhurakousky]

Sravani

Would you be able to provide a full stack trace of the connection exception.
Also, while I assume you are providing the correct connection properties (i.e., 
host,port etc) I would still recommend to check the they are correct, but in 
any event the full stack trace would definitely help and you cn find it the the 
wifi app logs.

Cheers
Oleg

On Nov 15, 2016, at 4:07 AM, Gadiputi, Sravani 
> wrote:

Thank you for your reply.

My requirement is , I just try to send/copy the 3 different files from source 
to destination through Nifi, and these jobs runs weekly once.
So I wanted to know which file is successfully moved through email.
In this process, I have configured putemail for each flow. There are hardly 3 
notifications only.
Though files have been moved to destination, we could not receive the 
notifications properly and giving the below error.

Please suggest.

Thanks,
Sravani


From: Jeff [mailto:jtsw...@gmail.com]
Sent: Tuesday, November 15, 2016 1:25 PM
To: users@nifi.apache.org
Subject: Re: Nifi- PutEmail processor issue

Hello Sravani,

Could it be possible that the SMTP server you're using is denying connections 
due to the volume of emails your flow might be sending?  How many emails are 
sent per flow file, and how many emails do you estimate are sent per minute?

If this is the case, you can modify your flow to aggregate flowfiles with a 
processor like MergeContent so that you can send emails that resemble a digest, 
rather than a separate email for each flowfile that moves through your flow.

On Mon, Nov 14, 2016 at 11:59 PM Gadiputi, Sravani 
> wrote:

Hi,

I have used PutEmail processor in my project to send email notification for 
successful/failure copying of a files.
Each file flow having corresponding PutEmail to send  email notification to 
respective recipients.

Here the issue is, sometimes email notification will send to respective 
recipients successfully  for successful/failure job.
But sometimes for any one specific job email notification will not be send to 
recipients though job is successful, due to  below error.

Error:

Could not connect to SMTP host
Java.net.ConnectException: Connection timed out

Could you please suggest me how we can overcome this error.


Thanks,
Sravani

This message contains information that may be privileged or confidential and is 
the property of the Capgemini Group. It is intended only for the person to whom 
it is addressed. If you are not the intended recipient, you are not authorized 
to read, print, retain, copy, disseminate, distribute, or use this message or 
any part thereof. If you receive this message in error, please notify the 
sender immediately and delete all copies of this message.

RE: Nifi- PutEmail processor issue

[Gadiputi, Sravani]

Thank you for your reply.

My requirement is , I just try to send/copy the 3 different files from source 
to destination through Nifi, and these jobs runs weekly once.
So I wanted to know which file is successfully moved through email.
In this process, I have configured putemail for each flow. There are hardly 3 
notifications only.
Though files have been moved to destination, we could not receive the 
notifications properly and giving the below error.

Please suggest.

Thanks,
Sravani


From: Jeff [mailto:jtsw...@gmail.com]
Sent: Tuesday, November 15, 2016 1:25 PM
To: users@nifi.apache.org
Subject: Re: Nifi- PutEmail processor issue

Hello Sravani,

Could it be possible that the SMTP server you're using is denying connections 
due to the volume of emails your flow might be sending?  How many emails are 
sent per flow file, and how many emails do you estimate are sent per minute?

If this is the case, you can modify your flow to aggregate flowfiles with a 
processor like MergeContent so that you can send emails that resemble a digest, 
rather than a separate email for each flowfile that moves through your flow.

On Mon, Nov 14, 2016 at 11:59 PM Gadiputi, Sravani 
> wrote:

Hi,

I have used PutEmail processor in my project to send email notification for 
successful/failure copying of a files.
Each file flow having corresponding PutEmail to send  email notification to 
respective recipients.

Here the issue is, sometimes email notification will send to respective 
recipients successfully  for successful/failure job.
But sometimes for any one specific job email notification will not be send to 
recipients though job is successful, due to  below error.

Error:

Could not connect to SMTP host
Java.net.ConnectException: Connection timed out

Could you please suggest me how we can overcome this error.


Thanks,
Sravani

This message contains information that may be privileged or confidential and is 
the property of the Capgemini Group. It is intended only for the person to whom 
it is addressed. If you are not the intended recipient, you are not authorized 
to read, print, retain, copy, disseminate, distribute, or use this message or 
any part thereof. If you receive this message in error, please notify the 
sender immediately and delete all copies of this message.