Re: Whitelisting Proxy Host values in a Container Environment?

[Andy LoPresto]

No, ignoring the port could lead to the same malicious host injection in the 
scenario where multiple services are hosted on the same machine (i.e. nifi-1 at 
nifi.apache.org:8443  and nifi-2 at 
nifi.apache.org:8444). Is the port for each different than what is set 
nifi.web.https.port? That value should be automatically included. 


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Oct 19, 2018, at 12:13 AM, Jon Logan  wrote:
> 
> Thanks Andy. Would a solution involving only checking the hostname and not 
> the port be considered? That's where our heartache is coming from. The 
> documentation hints that this is supported by referencing the format 
> host[:port] but the port is not currently optional (unless using 80/443 I 
> suppose).
> 
> On Thu, Oct 18, 2018 at 4:58 AM Andy LoPresto  > wrote:
> Jon, 
> 
> Sorry to hear that securing NiFi is proving difficult for you. 
> 
> The reason this feature was introduced is because prior to this check being 
> enforced, a malicious Host header in requests could poison the server and 
> return untrusted resources. Please see the Admin Guide [1] section on Proxy 
> Configuration for more details. When not running behind a proxy, this feature 
> is usually very simple to configure — the hostname of the NiFi service is 
> automatically included in the whitelist, so requests are handled 
> successfully. The original feature request (for proxy whitelisting) is in 
> NIFI-4761 [2] (unit tests [3]) if you want more detail on why it was 
> implemented, or see this article [4] for more details on host header attacks. 
> 
> If your dynamic hostnames fall in a list that is assigned on demand, you 
> could provide the full list (comma-separated) as the whitelisted host value 
> in all nodes, and this would allow each to be verified. If not, you could 
> request a feature to allow regex-parsing of a pattern for that value as well, 
> but I am opposed to this as it usually introduces more problems than it 
> solves. 
> 
> I guess you could request an explicit disabling of this header check via 
> nifi.properties, but I would certainly encourage you to find a solution that 
> maintains the header check if possible. 
> 
> [1] 
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#proxy_configuration
>  
> 
> [2] https://issues.apache.org/jira/browse/NIFI-4761 
> 
> [3] https://github.com/apache/nifi/pull/2279/files 
> 
> [4] https://dzone.com/articles/what-is-a-host-header-attack 
> 
> 
> 
> 
> 
> Andy LoPresto
> alopre...@apache.org 
> alopresto.apa...@gmail.com 
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> 
>> On Oct 15, 2018, at 11:06 PM, Peter Wilcsinszky > > wrote:
>> 
>> Jon,
>> 
>> as for the port-forward you can bind NiFi to lo(required by the 
>> port-forward)+eth0 (I beleive eth0 is not provider dependent):
>> nifi.web.https.network.interface.lo=lo
>> nifi.web.https.network.interface.eth0=eth0
>> 
>> Peter
>> 
>> On Mon, Oct 15, 2018 at 3:46 PM Jon Logan > > wrote:
>> Thanks Peter. I briefly was playing around with connecting to localhost via 
>> the port-forwarding late last week but was having issues where Nifi was 
>> internally trying to connect to 0.0.0.0 and failing...I'll toy with it some 
>> more this morning (I had set the https listen property as 0.0.0.0 so that 
>> it'd bind to localhost). Or I think the NodePort option will work -- but 
>> will require mucking with the yaml file every deployment to avoid port 
>> clashing between deployments. Mildly inconvenient, but certainly is doable.
>> 
>> If this feature were disabled, certificates would still work -- in this 
>> case, we know our entrypoint, but do not know the port. We could have certs 
>> dynamically issued including the entrypoint hostname into the cert as an 
>> alternate name. Our issue hasn't been knowing the hostname in advance, but 
>> knowing the port, which isn't part of the certificate entry. We haven't made 
>> it to that point yet, but it should work, and is on our roadmap.
>> 
>> Again, thanks a lot -- securing this is turning out to be a lot more 
>> complicated than originally anticipated.
>> 
>> 
>> 
>> 
>> On Mon, Oct 15, 2018 at 3:57 AM Peter Wilcsinszky 
>> mailto:peterwilcsins...@gmail.com>> wrote:
>> Hey,
>> 
>> I can't tell about the original intent and motivations, but this is the Jira 
>> that introduced this check [1]. 
>> 
>> What I can tell is mutual TLS is not the only option to authenticate against 
>> NiFi. You can 

Re: [EXT] ReplaceText cannot consume messages if Regex does not match

[Juan Pablo Gardella]

Exactly, I know it is not an issue in the replace text code, but it happens
inside it. If we are using the ReplaceText in multiple places, it increases
the flow design complexity. We need to evaluate all expressions before
sending to the processor, to be sure will not fail in ReplaceText
processor.

Notice it's impossible if you have to process content dynamically. I would
happy to file a ticket and the patch as I mention.

Juan



On Thu, 18 Oct 2018 at 13:18 Shawn Weeks  wrote:

> I understand the issue now, I’m not sure that a failure of ReplaceText is
> the best place to catch this though.  The reason I’m not sure it’s the best
> place is what happens if there are multiple failures because you had
> multiple expressions, just having them all routed to the same failure
> wouldn’t help you make decisions on what to do with a single attribute.
> Perhaps a better solution would be to use a RouteOnAttribute to check if
> the attributes match a certain pattern before sending them to ReplaceText.
> A possible expression could be
> “${actualSettlementDate:matches('[0-9]{2}/[0-9]{2}/[0-9]{4}')}” however
> that would not catch things that look like dates but aren’t valid.
>
>
>
> Thanks
>
> Shawn Weeks
>
>
>
> *From:* Juan Pablo Gardella 
>
> *Sent:* Thursday, October 18, 2018 11:03 AM
>
>
> *To:* users@nifi.apache.org
> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
> not match
>
>
>
> At *search value*:(?s)(^.*$)
>
>
>
> At *Replacement value*:
>
>
> *
> ${actualSettlementDate:toDate('MM/dd/'):format("-MM-dd'T'00:00:00.000")}*
>
>
>
> The actualSettlementDate is a flowfile attribute. The problem is the
> replacement value is evaluated inside the processor and the *toDate *method
> fails.
>
>
>
> Hope it's clear now.
>
>
>
>
>
> On Thu, 18 Oct 2018 at 12:51 Shawn Weeks 
> wrote:
>
> I’m still trying to understand your actual issue, can your provide a
> screenshot of the ReplaceText config like the attached, I need to see
> exactly where you’re putting the expression. A template would also be
> really helpful.
>
>
>
> Thanks
>
> Shawn Weeks
>
>
>
> *From:* Juan Pablo Gardella 
>
> *Sent:* Thursday, October 18, 2018 10:45 AM
>
>
> *To:* users@nifi.apache.org
> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
> not match
>
>
>
> At ReplaceText
> processor
> we have:
>
>
>
> [image: image.png]
>
> As you can see, only if *StackOverflowError *is raised during the
> evaluation, the flowfile is send to failure relationship. I would like to
> update the code to use Exception or NifiExpressionFailedException (if it
> exits).
>
>
>
> Juan
>
>
>
> On Thu, 18 Oct 2018 at 12:33 Shawn Weeks 
> wrote:
>
> What processor are you defining your expression in? I also may be
> misunderstanding the problem because I don’t see any regular expressions
> anywhere. Can you create a sample workflow showing your issue so I can take
> a look at it.
>
>
>
> Thanks
>
> Shawn Weeks
>
>
>
> *From:* Juan Pablo Gardella 
> *Sent:* Thursday, October 18, 2018 10:27 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
> not match
>
>
>
> No, it's not a valid date. I would like if it an error happens, I would
> like to throw the flowfile to failure and continue.
>
>
>
> On Thu, 18 Oct 2018 at 12:19 Shawn Weeks 
> wrote:
>
> Any expression language syntax has to be correct or the processor won’t
> run. I’m not sure there is any way to work around that except to explicitly
> check that the value you are trying to evaluate is valid. Is the attribute
> “tradeDate” coming from the contents of a flow file or is it defined
> somewhere else. Can you ensure it is a valid date in that format before
> hand?
>
>
>
> Thanks
>
> Shawn Weeks
>
>
>
> *From:* Juan Pablo Gardella 
>
> *Sent:* Thursday, October 18, 2018 10:13 AM
>
>
> *To:* users@nifi.apache.org
> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
> not match
>
>
>
> Hi, the error is not in the processor itself. It's in the expression used
> against flowfile attributes. For example inside the text, I have:
>
>
>
>
> ${tradeDate:toDate('MM/dd/'):format("-MM-dd'T'00:00:00.000")}
>
> And that is the root issue. If it's unable to convert it, the flow cannot
> be consumed. How can I evaluate attributes in a non-blocker way?
>
>
>
> Juan
>
>
>
> On Thu, 18 Oct 2018 at 12:07 Shawn Weeks 
> wrote:
>
> Where is your expression? That’s not the entire configuration for that
> processor.
>
>
>
> Thanks
>
> Shawn Weeks
>
>
>
> *From:* Juan Pablo Gardella 
> *Sent:* Thursday, October 18, 2018 10:03 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
> not match
>
>
>
> Configuration:
>
> Replacement Strategy: Always replace
>
> 

RE: [EXT] ReplaceText cannot consume messages if Regex does not match

[Shawn Weeks]

I understand the issue now, I’m not sure that a failure of ReplaceText is the 
best place to catch this though.  The reason I’m not sure it’s the best place 
is what happens if there are multiple failures because you had multiple 
expressions, just having them all routed to the same failure wouldn’t help you 
make decisions on what to do with a single attribute. Perhaps a better solution 
would be to use a RouteOnAttribute to check if the attributes match a certain 
pattern before sending them to ReplaceText. A possible expression could be 
“${actualSettlementDate:matches('[0-9]{2}/[0-9]{2}/[0-9]{4}')}” however that 
would not catch things that look like dates but aren’t valid.

Thanks
Shawn Weeks

From: Juan Pablo Gardella 
Sent: Thursday, October 18, 2018 11:03 AM
To: users@nifi.apache.org
Subject: Re: [EXT] ReplaceText cannot consume messages if Regex does not match

At search value:(?s)(^.*$)

At Replacement value:

${actualSettlementDate:toDate('MM/dd/'):format("-MM-dd'T'00:00:00.000")}

The actualSettlementDate is a flowfile attribute. The problem is the 
replacement value is evaluated inside the processor and the toDate method fails.

Hope it's clear now.


On Thu, 18 Oct 2018 at 12:51 Shawn Weeks 
mailto:swe...@weeksconsulting.us>> wrote:
I’m still trying to understand your actual issue, can your provide a screenshot 
of the ReplaceText config like the attached, I need to see exactly where you’re 
putting the expression. A template would also be really helpful.

Thanks
Shawn Weeks

From: Juan Pablo Gardella 
mailto:gardellajuanpa...@gmail.com>>
Sent: Thursday, October 18, 2018 10:45 AM

To: users@nifi.apache.org
Subject: Re: [EXT] ReplaceText cannot consume messages if Regex does not match

At ReplaceText 

 processor we have:

[image.png]
As you can see, only if StackOverflowError is raised during the evaluation, the 
flowfile is send to failure relationship. I would like to update the code to 
use Exception or NifiExpressionFailedException (if it exits).

Juan

On Thu, 18 Oct 2018 at 12:33 Shawn Weeks 
mailto:swe...@weeksconsulting.us>> wrote:
What processor are you defining your expression in? I also may be 
misunderstanding the problem because I don’t see any regular expressions 
anywhere. Can you create a sample workflow showing your issue so I can take a 
look at it.

Thanks
Shawn Weeks

From: Juan Pablo Gardella 
mailto:gardellajuanpa...@gmail.com>>
Sent: Thursday, October 18, 2018 10:27 AM
To: users@nifi.apache.org
Subject: Re: [EXT] ReplaceText cannot consume messages if Regex does not match

No, it's not a valid date. I would like if it an error happens, I would like to 
throw the flowfile to failure and continue.

On Thu, 18 Oct 2018 at 12:19 Shawn Weeks 
mailto:swe...@weeksconsulting.us>> wrote:
Any expression language syntax has to be correct or the processor won’t run. 
I’m not sure there is any way to work around that except to explicitly check 
that the value you are trying to evaluate is valid. Is the attribute 
“tradeDate” coming from the contents of a flow file or is it defined somewhere 
else. Can you ensure it is a valid date in that format before hand?

Thanks
Shawn Weeks

From: Juan Pablo Gardella 
mailto:gardellajuanpa...@gmail.com>>
Sent: Thursday, October 18, 2018 10:13 AM

To: users@nifi.apache.org
Subject: Re: [EXT] ReplaceText cannot consume messages if Regex does not match

Hi, the error is not in the processor itself. It's in the expression used 
against flowfile attributes. For example inside the text, I have:

${tradeDate:toDate('MM/dd/'):format("-MM-dd'T'00:00:00.000")}
And that is the root issue. If it's unable to convert it, the flow cannot be 
consumed. How can I evaluate attributes in a non-blocker way?

Juan

On Thu, 18 Oct 2018 at 12:07 Shawn Weeks 
mailto:swe...@weeksconsulting.us>> wrote:
Where is your expression? That’s not the entire configuration for that 
processor.

Thanks
Shawn Weeks

From: Juan Pablo Gardella 
mailto:gardellajuanpa...@gmail.com>>
Sent: Thursday, October 18, 2018 10:03 AM
To: users@nifi.apache.org
Subject: Re: [EXT] ReplaceText cannot consume messages if Regex does not match

Configuration:
Replacement Strategy: Always replace
EvaluationMode: Entire text


On Thu, 18 Oct 2018 at 12:01 Juan Pablo Gardella 
mailto:gardellajuanpa...@gmail.com>> wrote:
Hortonworks nifi based on 1.5.0:

Configuration:
Thanks

On Thu, 18 Oct 2018 at 11:56 Peter Wicks (pwicks) 
mailto:pwi...@micron.com>> wrote:
Hi Juan,

What version of NiFi are you running on?
What mode are you running ReplaceText in, all text or line by line?
Other settings that might be important? What’s your RegEx look like (if your 
able to share).

--Peter

Re: [EXT] ReplaceText cannot consume messages if Regex does not match

[Jeff]

Is the actualSettlementDate attribute's value in the "MM/dd/" format,
with no other text in front of the date?  For instance, "10/18/2018
12:30:00" is parsable by the "MM/dd/" format, but "12:30:00 10/18/2018"
is not.

On Thu, Oct 18, 2018 at 12:02 PM Juan Pablo Gardella <
gardellajuanpa...@gmail.com> wrote:

> At *search value*:(?s)(^.*$)
>
> At *Replacement value*:
>
> * encoding="UTF-8"?>${actualSettlementDate:toDate('MM/dd/'):format("-MM-dd'T'00:00:00.000")}*
>
> The actualSettlementDate is a flowfile attribute. The problem is the
> replacement value is evaluated inside the processor and the *toDate *method
> fails.
>
> Hope it's clear now.
>
>
> On Thu, 18 Oct 2018 at 12:51 Shawn Weeks 
> wrote:
>
>> I’m still trying to understand your actual issue, can your provide a
>> screenshot of the ReplaceText config like the attached, I need to see
>> exactly where you’re putting the expression. A template would also be
>> really helpful.
>>
>>
>>
>> Thanks
>>
>> Shawn Weeks
>>
>>
>>
>> *From:* Juan Pablo Gardella 
>>
>> *Sent:* Thursday, October 18, 2018 10:45 AM
>>
>>
>> *To:* users@nifi.apache.org
>> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
>> not match
>>
>>
>>
>> At ReplaceText
>> processor
>> we have:
>>
>>
>>
>> [image: image.png]
>>
>> As you can see, only if *StackOverflowError *is raised during the
>> evaluation, the flowfile is send to failure relationship. I would like to
>> update the code to use Exception or NifiExpressionFailedException (if it
>> exits).
>>
>>
>>
>> Juan
>>
>>
>>
>> On Thu, 18 Oct 2018 at 12:33 Shawn Weeks 
>> wrote:
>>
>> What processor are you defining your expression in? I also may be
>> misunderstanding the problem because I don’t see any regular expressions
>> anywhere. Can you create a sample workflow showing your issue so I can take
>> a look at it.
>>
>>
>>
>> Thanks
>>
>> Shawn Weeks
>>
>>
>>
>> *From:* Juan Pablo Gardella 
>> *Sent:* Thursday, October 18, 2018 10:27 AM
>> *To:* users@nifi.apache.org
>> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
>> not match
>>
>>
>>
>> No, it's not a valid date. I would like if it an error happens, I would
>> like to throw the flowfile to failure and continue.
>>
>>
>>
>> On Thu, 18 Oct 2018 at 12:19 Shawn Weeks 
>> wrote:
>>
>> Any expression language syntax has to be correct or the processor won’t
>> run. I’m not sure there is any way to work around that except to explicitly
>> check that the value you are trying to evaluate is valid. Is the attribute
>> “tradeDate” coming from the contents of a flow file or is it defined
>> somewhere else. Can you ensure it is a valid date in that format before
>> hand?
>>
>>
>>
>> Thanks
>>
>> Shawn Weeks
>>
>>
>>
>> *From:* Juan Pablo Gardella 
>>
>> *Sent:* Thursday, October 18, 2018 10:13 AM
>>
>>
>> *To:* users@nifi.apache.org
>> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
>> not match
>>
>>
>>
>> Hi, the error is not in the processor itself. It's in the expression used
>> against flowfile attributes. For example inside the text, I have:
>>
>>
>>
>>
>> ${tradeDate:toDate('MM/dd/'):format("-MM-dd'T'00:00:00.000")}
>>
>> And that is the root issue. If it's unable to convert it, the flow cannot
>> be consumed. How can I evaluate attributes in a non-blocker way?
>>
>>
>>
>> Juan
>>
>>
>>
>> On Thu, 18 Oct 2018 at 12:07 Shawn Weeks 
>> wrote:
>>
>> Where is your expression? That’s not the entire configuration for that
>> processor.
>>
>>
>>
>> Thanks
>>
>> Shawn Weeks
>>
>>
>>
>> *From:* Juan Pablo Gardella 
>> *Sent:* Thursday, October 18, 2018 10:03 AM
>> *To:* users@nifi.apache.org
>> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
>> not match
>>
>>
>>
>> Configuration:
>>
>> Replacement Strategy: Always replace
>>
>> EvaluationMode: Entire text
>>
>>
>>
>>
>>
>> On Thu, 18 Oct 2018 at 12:01 Juan Pablo Gardella <
>> gardellajuanpa...@gmail.com> wrote:
>>
>> Hortonworks nifi based on 1.5.0:
>>
>>
>>
>> Configuration:
>>
>> Thanks
>>
>>
>>
>> On Thu, 18 Oct 2018 at 11:56 Peter Wicks (pwicks) 
>> wrote:
>>
>> Hi Juan,
>>
>>
>>
>> What version of NiFi are you running on?
>>
>> What mode are you running ReplaceText in, all text or line by line?
>>
>> Other settings that might be important? What’s your RegEx look like (if
>> your able to share).
>>
>>
>>
>> --Peter
>>
>>
>>
>>
>>
>> *From:* Juan Pablo Gardella [mailto:gardellajuanpa...@gmail.com]
>> *Sent:* Thursday, October 18, 2018 8:53 AM
>> *To:* users@nifi.apache.org
>> *Subject:* [EXT] ReplaceText cannot consume messages if Regex does not
>> match
>>
>>
>>
>> Hi all,
>>
>>
>>
>> I'm seeing that ReplaceText is not able to consume messages that does not
>> match regex. It keeps all the messages in the input queue instead of
>> 

Re: question about ConsumeKafka metrics for incoming data

[Dominique De Vito]

> The value should be close to how many bytes have been consumed from
Kafka.

Very good hint. Thanks Jeff.

And thanks Bryan too !

Dominique


Le jeu. 18 oct. 2018 à 17:52, Jeff  a écrit :

> Dominique,
>
> The "bytes in" value represents how many bytes have been received from an
> incoming connection.  Since ConsumeKafka is directly consuming data from
> Kafka (a "source" processor), and not from a connection from an upstream
> processor, it won't show a value for "bytes in".  If you want to see how
> much data ConsumeKafka has sent downstream, you can look at "bytes out".
> The value should be close to how many bytes have been consumed from Kafka.
> I'm not familiar with the Kafka processors, but I assume the messages are
> encrypted/compressed.
>
> On Thu, Oct 18, 2018 at 11:25 AM Dominique De Vito 
> wrote:
>
>> Hi,
>>
>> While running a ConsumeKafka processor, I noticed there are metrics for
>> incoming data, but there was no value (like "bytes in") provided in the UI.
>>
>> And as far as I have digged into the ConsumeKafka processor, I didn't
>> found any code related to feeding the metrics related to the incoming data.
>>
>> Is it true that ConsumeKafka processor provides no value for the metrics
>> related to the incoming data ?
>>
>> Is it the same for all processors dealing with data coming from outside
>> Nifi ?
>>
>> Thanks.
>>
>> Regards,
>> Dominique
>>
>>

Re: [EXT] ReplaceText cannot consume messages if Regex does not match

[Juan Pablo Gardella]

At *search value*:(?s)(^.*$)

At *Replacement value*:

*${actualSettlementDate:toDate('MM/dd/'):format("-MM-dd'T'00:00:00.000")}*

The actualSettlementDate is a flowfile attribute. The problem is the
replacement value is evaluated inside the processor and the *toDate *method
fails.

Hope it's clear now.


On Thu, 18 Oct 2018 at 12:51 Shawn Weeks  wrote:

> I’m still trying to understand your actual issue, can your provide a
> screenshot of the ReplaceText config like the attached, I need to see
> exactly where you’re putting the expression. A template would also be
> really helpful.
>
>
>
> Thanks
>
> Shawn Weeks
>
>
>
> *From:* Juan Pablo Gardella 
>
> *Sent:* Thursday, October 18, 2018 10:45 AM
>
>
> *To:* users@nifi.apache.org
> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
> not match
>
>
>
> At ReplaceText
> processor
> we have:
>
>
>
> [image: image.png]
>
> As you can see, only if *StackOverflowError *is raised during the
> evaluation, the flowfile is send to failure relationship. I would like to
> update the code to use Exception or NifiExpressionFailedException (if it
> exits).
>
>
>
> Juan
>
>
>
> On Thu, 18 Oct 2018 at 12:33 Shawn Weeks 
> wrote:
>
> What processor are you defining your expression in? I also may be
> misunderstanding the problem because I don’t see any regular expressions
> anywhere. Can you create a sample workflow showing your issue so I can take
> a look at it.
>
>
>
> Thanks
>
> Shawn Weeks
>
>
>
> *From:* Juan Pablo Gardella 
> *Sent:* Thursday, October 18, 2018 10:27 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
> not match
>
>
>
> No, it's not a valid date. I would like if it an error happens, I would
> like to throw the flowfile to failure and continue.
>
>
>
> On Thu, 18 Oct 2018 at 12:19 Shawn Weeks 
> wrote:
>
> Any expression language syntax has to be correct or the processor won’t
> run. I’m not sure there is any way to work around that except to explicitly
> check that the value you are trying to evaluate is valid. Is the attribute
> “tradeDate” coming from the contents of a flow file or is it defined
> somewhere else. Can you ensure it is a valid date in that format before
> hand?
>
>
>
> Thanks
>
> Shawn Weeks
>
>
>
> *From:* Juan Pablo Gardella 
>
> *Sent:* Thursday, October 18, 2018 10:13 AM
>
>
> *To:* users@nifi.apache.org
> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
> not match
>
>
>
> Hi, the error is not in the processor itself. It's in the expression used
> against flowfile attributes. For example inside the text, I have:
>
>
>
>
> ${tradeDate:toDate('MM/dd/'):format("-MM-dd'T'00:00:00.000")}
>
> And that is the root issue. If it's unable to convert it, the flow cannot
> be consumed. How can I evaluate attributes in a non-blocker way?
>
>
>
> Juan
>
>
>
> On Thu, 18 Oct 2018 at 12:07 Shawn Weeks 
> wrote:
>
> Where is your expression? That’s not the entire configuration for that
> processor.
>
>
>
> Thanks
>
> Shawn Weeks
>
>
>
> *From:* Juan Pablo Gardella 
> *Sent:* Thursday, October 18, 2018 10:03 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
> not match
>
>
>
> Configuration:
>
> Replacement Strategy: Always replace
>
> EvaluationMode: Entire text
>
>
>
>
>
> On Thu, 18 Oct 2018 at 12:01 Juan Pablo Gardella <
> gardellajuanpa...@gmail.com> wrote:
>
> Hortonworks nifi based on 1.5.0:
>
>
>
> Configuration:
>
> Thanks
>
>
>
> On Thu, 18 Oct 2018 at 11:56 Peter Wicks (pwicks) 
> wrote:
>
> Hi Juan,
>
>
>
> What version of NiFi are you running on?
>
> What mode are you running ReplaceText in, all text or line by line?
>
> Other settings that might be important? What’s your RegEx look like (if
> your able to share).
>
>
>
> --Peter
>
>
>
>
>
> *From:* Juan Pablo Gardella [mailto:gardellajuanpa...@gmail.com]
> *Sent:* Thursday, October 18, 2018 8:53 AM
> *To:* users@nifi.apache.org
> *Subject:* [EXT] ReplaceText cannot consume messages if Regex does not
> match
>
>
>
> Hi all,
>
>
>
> I'm seeing that ReplaceText is not able to consume messages that does not
> match regex. It keeps all the messages in the input queue instead of
> sending them to failure relationship. Is this the intended behavior or I
> have to file a ticket in order to be fixed? In that way, the processor is
> not able to process bad messages and converts in the bottleneck of a flow
>
>
>
> Juan
>
>

Re: question about ConsumeKafka metrics for incoming data

[Jeff]

Dominique,

The "bytes in" value represents how many bytes have been received from an
incoming connection.  Since ConsumeKafka is directly consuming data from
Kafka (a "source" processor), and not from a connection from an upstream
processor, it won't show a value for "bytes in".  If you want to see how
much data ConsumeKafka has sent downstream, you can look at "bytes out".
The value should be close to how many bytes have been consumed from Kafka.
I'm not familiar with the Kafka processors, but I assume the messages are
encrypted/compressed.

On Thu, Oct 18, 2018 at 11:25 AM Dominique De Vito 
wrote:

> Hi,
>
> While running a ConsumeKafka processor, I noticed there are metrics for
> incoming data, but there was no value (like "bytes in") provided in the UI.
>
> And as far as I have digged into the ConsumeKafka processor, I didn't
> found any code related to feeding the metrics related to the incoming data.
>
> Is it true that ConsumeKafka processor provides no value for the metrics
> related to the incoming data ?
>
> Is it the same for all processors dealing with data coming from outside
> Nifi ?
>
> Thanks.
>
> Regards,
> Dominique
>
>

Re: question about ConsumeKafka metrics for incoming data

[Bryan Bende]

Hello,

Bytes in is the # of bytes read from incoming flow files. Since source
processors don't have incoming flow files they will never have bytes
in or flow files in.

-Bryan
On Thu, Oct 18, 2018 at 11:25 AM Dominique De Vito  wrote:
>
> Hi,
>
> While running a ConsumeKafka processor, I noticed there are metrics for 
> incoming data, but there was no value (like "bytes in") provided in the UI.
>
> And as far as I have digged into the ConsumeKafka processor, I didn't found 
> any code related to feeding the metrics related to the incoming data.
>
> Is it true that ConsumeKafka processor provides no value for the metrics 
> related to the incoming data ?
>
> Is it the same for all processors dealing with data coming from outside Nifi ?
>
> Thanks.
>
> Regards,
> Dominique
>

Who uses NiFi Cluster in Docker ?

[PICHARD, Guillaume]

Hi,

I'm looking for experiences and return on experience in running a Nifi Cluster 
in production using docker/kubernetes/mesos. Is it working well ? Is it stable 
? Does it handle well a high workload ?

Thanks for you feedbacks,
Guillaume.

RE: [EXT] ReplaceText cannot consume messages if Regex does not match

[Shawn Weeks]

What processor are you defining your expression in? I also may be 
misunderstanding the problem because I don’t see any regular expressions 
anywhere. Can you create a sample workflow showing your issue so I can take a 
look at it.

Thanks
Shawn Weeks

From: Juan Pablo Gardella 
Sent: Thursday, October 18, 2018 10:27 AM
To: users@nifi.apache.org
Subject: Re: [EXT] ReplaceText cannot consume messages if Regex does not match

No, it's not a valid date. I would like if it an error happens, I would like to 
throw the flowfile to failure and continue.

On Thu, 18 Oct 2018 at 12:19 Shawn Weeks 
mailto:swe...@weeksconsulting.us>> wrote:
Any expression language syntax has to be correct or the processor won’t run. 
I’m not sure there is any way to work around that except to explicitly check 
that the value you are trying to evaluate is valid. Is the attribute 
“tradeDate” coming from the contents of a flow file or is it defined somewhere 
else. Can you ensure it is a valid date in that format before hand?

Thanks
Shawn Weeks

From: Juan Pablo Gardella 
mailto:gardellajuanpa...@gmail.com>>
Sent: Thursday, October 18, 2018 10:13 AM

To: users@nifi.apache.org
Subject: Re: [EXT] ReplaceText cannot consume messages if Regex does not match

Hi, the error is not in the processor itself. It's in the expression used 
against flowfile attributes. For example inside the text, I have:

${tradeDate:toDate('MM/dd/'):format("-MM-dd'T'00:00:00.000")}
And that is the root issue. If it's unable to convert it, the flow cannot be 
consumed. How can I evaluate attributes in a non-blocker way?

Juan

On Thu, 18 Oct 2018 at 12:07 Shawn Weeks 
mailto:swe...@weeksconsulting.us>> wrote:
Where is your expression? That’s not the entire configuration for that 
processor.

Thanks
Shawn Weeks

From: Juan Pablo Gardella 
mailto:gardellajuanpa...@gmail.com>>
Sent: Thursday, October 18, 2018 10:03 AM
To: users@nifi.apache.org
Subject: Re: [EXT] ReplaceText cannot consume messages if Regex does not match

Configuration:
Replacement Strategy: Always replace
EvaluationMode: Entire text


On Thu, 18 Oct 2018 at 12:01 Juan Pablo Gardella 
mailto:gardellajuanpa...@gmail.com>> wrote:
Hortonworks nifi based on 1.5.0:

Configuration:
Thanks

On Thu, 18 Oct 2018 at 11:56 Peter Wicks (pwicks) 
mailto:pwi...@micron.com>> wrote:
Hi Juan,

What version of NiFi are you running on?
What mode are you running ReplaceText in, all text or line by line?
Other settings that might be important? What’s your RegEx look like (if your 
able to share).

--Peter


From: Juan Pablo Gardella 
[mailto:gardellajuanpa...@gmail.com]
Sent: Thursday, October 18, 2018 8:53 AM
To: users@nifi.apache.org
Subject: [EXT] ReplaceText cannot consume messages if Regex does not match

Hi all,

I'm seeing that ReplaceText is not able to consume messages that does not match 
regex. It keeps all the messages in the input queue instead of sending them to 
failure relationship. Is this the intended behavior or I have to file a ticket 
in order to be fixed? In that way, the processor is not able to process bad 
messages and converts in the bottleneck of a flow

Juan

Re: [EXT] ReplaceText cannot consume messages if Regex does not match

[Juan Pablo Gardella]

No, it's not a valid date. I would like if it an error happens, I would
like to throw the flowfile to failure and continue.

On Thu, 18 Oct 2018 at 12:19 Shawn Weeks  wrote:

> Any expression language syntax has to be correct or the processor won’t
> run. I’m not sure there is any way to work around that except to explicitly
> check that the value you are trying to evaluate is valid. Is the attribute
> “tradeDate” coming from the contents of a flow file or is it defined
> somewhere else. Can you ensure it is a valid date in that format before
> hand?
>
>
>
> Thanks
>
> Shawn Weeks
>
>
>
> *From:* Juan Pablo Gardella 
>
> *Sent:* Thursday, October 18, 2018 10:13 AM
>
>
> *To:* users@nifi.apache.org
> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
> not match
>
>
>
> Hi, the error is not in the processor itself. It's in the expression used
> against flowfile attributes. For example inside the text, I have:
>
>
>
>
> ${tradeDate:toDate('MM/dd/'):format("-MM-dd'T'00:00:00.000")}
>
> And that is the root issue. If it's unable to convert it, the flow cannot
> be consumed. How can I evaluate attributes in a non-blocker way?
>
>
>
> Juan
>
>
>
> On Thu, 18 Oct 2018 at 12:07 Shawn Weeks 
> wrote:
>
> Where is your expression? That’s not the entire configuration for that
> processor.
>
>
>
> Thanks
>
> Shawn Weeks
>
>
>
> *From:* Juan Pablo Gardella 
> *Sent:* Thursday, October 18, 2018 10:03 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
> not match
>
>
>
> Configuration:
>
> Replacement Strategy: Always replace
>
> EvaluationMode: Entire text
>
>
>
>
>
> On Thu, 18 Oct 2018 at 12:01 Juan Pablo Gardella <
> gardellajuanpa...@gmail.com> wrote:
>
> Hortonworks nifi based on 1.5.0:
>
> [image: image.png]
>
>
>
> Configuration:
>
> Thanks
>
>
>
> On Thu, 18 Oct 2018 at 11:56 Peter Wicks (pwicks) 
> wrote:
>
> Hi Juan,
>
>
>
> What version of NiFi are you running on?
>
> What mode are you running ReplaceText in, all text or line by line?
>
> Other settings that might be important? What’s your RegEx look like (if
> your able to share).
>
>
>
> --Peter
>
>
>
>
>
> *From:* Juan Pablo Gardella [mailto:gardellajuanpa...@gmail.com]
> *Sent:* Thursday, October 18, 2018 8:53 AM
> *To:* users@nifi.apache.org
> *Subject:* [EXT] ReplaceText cannot consume messages if Regex does not
> match
>
>
>
> Hi all,
>
>
>
> I'm seeing that ReplaceText is not able to consume messages that does not
> match regex. It keeps all the messages in the input queue instead of
> sending them to failure relationship. Is this the intended behavior or I
> have to file a ticket in order to be fixed? In that way, the processor is
> not able to process bad messages and converts in the bottleneck of a flow
>
>
>
> Juan
>
>

question about ConsumeKafka metrics for incoming data

[Dominique De Vito]

Hi,

While running a ConsumeKafka processor, I noticed there are metrics for
incoming data, but there was no value (like "bytes in") provided in the UI.

And as far as I have digged into the ConsumeKafka processor, I didn't found
any code related to feeding the metrics related to the incoming data.

Is it true that ConsumeKafka processor provides no value for the metrics
related to the incoming data ?

Is it the same for all processors dealing with data coming from outside
Nifi ?

Thanks.

Regards,
Dominique

RE: [EXT] ReplaceText cannot consume messages if Regex does not match

[Shawn Weeks]

Any expression language syntax has to be correct or the processor won’t run. 
I’m not sure there is any way to work around that except to explicitly check 
that the value you are trying to evaluate is valid. Is the attribute 
“tradeDate” coming from the contents of a flow file or is it defined somewhere 
else. Can you ensure it is a valid date in that format before hand?

Thanks
Shawn Weeks

From: Juan Pablo Gardella 
Sent: Thursday, October 18, 2018 10:13 AM
To: users@nifi.apache.org
Subject: Re: [EXT] ReplaceText cannot consume messages if Regex does not match

Hi, the error is not in the processor itself. It's in the expression used 
against flowfile attributes. For example inside the text, I have:

${tradeDate:toDate('MM/dd/'):format("-MM-dd'T'00:00:00.000")}
And that is the root issue. If it's unable to convert it, the flow cannot be 
consumed. How can I evaluate attributes in a non-blocker way?

Juan

On Thu, 18 Oct 2018 at 12:07 Shawn Weeks 
mailto:swe...@weeksconsulting.us>> wrote:
Where is your expression? That’s not the entire configuration for that 
processor.

Thanks
Shawn Weeks

From: Juan Pablo Gardella 
mailto:gardellajuanpa...@gmail.com>>
Sent: Thursday, October 18, 2018 10:03 AM
To: users@nifi.apache.org
Subject: Re: [EXT] ReplaceText cannot consume messages if Regex does not match

Configuration:
Replacement Strategy: Always replace
EvaluationMode: Entire text


On Thu, 18 Oct 2018 at 12:01 Juan Pablo Gardella 
mailto:gardellajuanpa...@gmail.com>> wrote:
Hortonworks nifi based on 1.5.0:
[image.png]

Configuration:
Thanks

On Thu, 18 Oct 2018 at 11:56 Peter Wicks (pwicks) 
mailto:pwi...@micron.com>> wrote:
Hi Juan,

What version of NiFi are you running on?
What mode are you running ReplaceText in, all text or line by line?
Other settings that might be important? What’s your RegEx look like (if your 
able to share).

--Peter


From: Juan Pablo Gardella 
[mailto:gardellajuanpa...@gmail.com]
Sent: Thursday, October 18, 2018 8:53 AM
To: users@nifi.apache.org
Subject: [EXT] ReplaceText cannot consume messages if Regex does not match

Hi all,

I'm seeing that ReplaceText is not able to consume messages that does not match 
regex. It keeps all the messages in the input queue instead of sending them to 
failure relationship. Is this the intended behavior or I have to file a ticket 
in order to be fixed? In that way, the processor is not able to process bad 
messages and converts in the bottleneck of a flow

Juan

Re: Whitelisting Proxy Host values in a Container Environment?

[Jon Logan]

Thanks Andy. Would a solution involving only checking the hostname and not
the port be considered? That's where our heartache is coming from. The
documentation hints that this is supported by referencing the
format host[:port] but the port is not currently optional (unless using
80/443 I suppose).

On Thu, Oct 18, 2018 at 4:58 AM Andy LoPresto  wrote:

> Jon,
>
> Sorry to hear that securing NiFi is proving difficult for you.
>
> The reason this feature was introduced is because prior to this check
> being enforced, a malicious Host header in requests could poison the server
> and return untrusted resources. Please see the Admin Guide [1] section on
> Proxy Configuration for more details. When not running behind a proxy, this
> feature is usually very simple to configure — the hostname of the NiFi
> service is automatically included in the whitelist, so requests are handled
> successfully. The original feature request (for proxy whitelisting) is in
> NIFI-4761 [2] (unit tests [3]) if you want more detail on why it was
> implemented, or see this article [4] for more details on host header
> attacks.
>
> If your dynamic hostnames fall in a list that is assigned on demand, you
> could provide the full list (comma-separated) as the whitelisted host value
> in all nodes, and this would allow each to be verified. If not, you could
> request a feature to allow regex-parsing of a pattern for that value as
> well, but I am opposed to this as it usually introduces more problems than
> it solves.
>
> I guess you could request an explicit disabling of this header check via
> nifi.properties, but I would certainly encourage you to find a solution
> that maintains the header check if possible.
>
> [1]
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#proxy_configuration
> [2] https://issues.apache.org/jira/browse/NIFI-4761
> [3] https://github.com/apache/nifi/pull/2279/files
> [4] https://dzone.com/articles/what-is-a-host-header-attack
>
>
>
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Oct 15, 2018, at 11:06 PM, Peter Wilcsinszky <
> peterwilcsins...@gmail.com> wrote:
>
> Jon,
>
> as for the port-forward you can bind NiFi to lo(required by the
> port-forward)+eth0 (I beleive eth0 is not provider dependent):
> nifi.web.https.network.interface.lo=lo
> nifi.web.https.network.interface.eth0=eth0
>
> Peter
>
> On Mon, Oct 15, 2018 at 3:46 PM Jon Logan  wrote:
>
>> Thanks Peter. I briefly was playing around with connecting to localhost
>> via the port-forwarding late last week but was having issues where Nifi was
>> internally trying to connect to 0.0.0.0 and failing...I'll toy with it some
>> more this morning (I had set the https listen property as 0.0.0.0 so that
>> it'd bind to localhost). Or I think the NodePort option will work -- but
>> will require mucking with the yaml file every deployment to avoid port
>> clashing between deployments. Mildly inconvenient, but certainly is doable.
>>
>> If this feature were disabled, certificates would still work -- in this
>> case, we know our entrypoint, but do not know the port. We could have certs
>> dynamically issued including the entrypoint hostname into the cert as an
>> alternate name. Our issue hasn't been knowing the hostname in advance, but
>> knowing the port, which isn't part of the certificate entry. We haven't
>> made it to that point yet, but it should work, and is on our roadmap.
>>
>> Again, thanks a lot -- securing this is turning out to be a lot more
>> complicated than originally anticipated.
>>
>>
>>
>>
>> On Mon, Oct 15, 2018 at 3:57 AM Peter Wilcsinszky <
>> peterwilcsins...@gmail.com> wrote:
>>
>>> Hey,
>>>
>>> I can't tell about the original intent and motivations, but this is the
>>> Jira that introduced this check [1].
>>>
>>> What I can tell is mutual TLS is not the only option to authenticate
>>> against NiFi. You can set up LDAP for example to authenticate the client
>>> and in that case MITM is possible I beleive.
>>>
>>> You will need a stable network identity that you can use to configure as
>>> your "proxy" in advance. For example in a testing scenario where you have
>>> access to the kubernetes cluster you can simply use "localhost" as the name
>>> of the proxy and use kubernetes port forward to tunnel requests from the
>>> localhost to your individual nodes (only one node at a time).
>>>
>>> Another option that could better work for non-local use cases is to use
>>> a LoadBalancer service in front of the nodes and configure DNS to point to
>>> your LoadBalancer IP. If you want to do this in advance it is possible to
>>> create floating IPs and preconfigure DNS for it at almost any cloud
>>> provider. Then add the configured DNS to nifi.web.proxy.host property when
>>> starting your cluster. If setting up DNS is not an option you can use the
>>> IP directly. If setting up the IP in advance is not an option you may use
>>> 

RE: [EXT] ReplaceText cannot consume messages if Regex does not match

[Peter Wicks (pwicks)]

Hi Juan,

What version of NiFi are you running on?
What mode are you running ReplaceText in, all text or line by line?
Other settings that might be important? What’s your RegEx look like (if your 
able to share).

--Peter


From: Juan Pablo Gardella [mailto:gardellajuanpa...@gmail.com]
Sent: Thursday, October 18, 2018 8:53 AM
To: users@nifi.apache.org
Subject: [EXT] ReplaceText cannot consume messages if Regex does not match

Hi all,

I'm seeing that ReplaceText is not able to consume messages that does not match 
regex. It keeps all the messages in the input queue instead of sending them to 
failure relationship. Is this the intended behavior or I have to file a ticket 
in order to be fixed? In that way, the processor is not able to process bad 
messages and converts in the bottleneck of a flow

Juan

ReplaceText cannot consume messages if Regex does not match

[Juan Pablo Gardella]

Hi all,

I'm seeing that ReplaceText is not able to consume messages that does not
match regex. It keeps all the messages in the input queue instead of
sending them to failure relationship. Is this the intended behavior or I
have to file a ticket in order to be fixed? In that way, the processor is
not able to process bad messages and converts in the bottleneck of a flow

Juan

Re: Whitelisting Proxy Host values in a Container Environment?

[Andy LoPresto]

Jon,

Sorry to hear that securing NiFi is proving difficult for you.

The reason this feature was introduced is because prior to this check being 
enforced, a malicious Host header in requests could poison the server and 
return untrusted resources. Please see the Admin Guide [1] section on Proxy 
Configuration for more details. When not running behind a proxy, this feature 
is usually very simple to configure — the hostname of the NiFi service is 
automatically included in the whitelist, so requests are handled successfully. 
The original feature request (for proxy whitelisting) is in NIFI-4761 [2] (unit 
tests [3]) if you want more detail on why it was implemented, or see this 
article [4] for more details on host header attacks.

If your dynamic hostnames fall in a list that is assigned on demand, you could 
provide the full list (comma-separated) as the whitelisted host value in all 
nodes, and this would allow each to be verified. If not, you could request a 
feature to allow regex-parsing of a pattern for that value as well, but I am 
opposed to this as it usually introduces more problems than it solves.

I guess you could request an explicit disabling of this header check via 
nifi.properties, but I would certainly encourage you to find a solution that 
maintains the header check if possible.

[1] 
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#proxy_configuration
 

[2] https://issues.apache.org/jira/browse/NIFI-4761 

[3] https://github.com/apache/nifi/pull/2279/files 

[4] https://dzone.com/articles/what-is-a-host-header-attack 





Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Oct 15, 2018, at 11:06 PM, Peter Wilcsinszky  
> wrote:
> 
> Jon,
> 
> as for the port-forward you can bind NiFi to lo(required by the 
> port-forward)+eth0 (I beleive eth0 is not provider dependent):
> nifi.web.https.network.interface.lo=lo
> nifi.web.https.network.interface.eth0=eth0
> 
> Peter
> 
> On Mon, Oct 15, 2018 at 3:46 PM Jon Logan  > wrote:
> Thanks Peter. I briefly was playing around with connecting to localhost via 
> the port-forwarding late last week but was having issues where Nifi was 
> internally trying to connect to 0.0.0.0 and failing...I'll toy with it some 
> more this morning (I had set the https listen property as 0.0.0.0 so that 
> it'd bind to localhost). Or I think the NodePort option will work -- but will 
> require mucking with the yaml file every deployment to avoid port clashing 
> between deployments. Mildly inconvenient, but certainly is doable.
> 
> If this feature were disabled, certificates would still work -- in this case, 
> we know our entrypoint, but do not know the port. We could have certs 
> dynamically issued including the entrypoint hostname into the cert as an 
> alternate name. Our issue hasn't been knowing the hostname in advance, but 
> knowing the port, which isn't part of the certificate entry. We haven't made 
> it to that point yet, but it should work, and is on our roadmap.
> 
> Again, thanks a lot -- securing this is turning out to be a lot more 
> complicated than originally anticipated.
> 
> 
> 
> 
> On Mon, Oct 15, 2018 at 3:57 AM Peter Wilcsinszky  > wrote:
> Hey,
> 
> I can't tell about the original intent and motivations, but this is the Jira 
> that introduced this check [1].
> 
> What I can tell is mutual TLS is not the only option to authenticate against 
> NiFi. You can set up LDAP for example to authenticate the client and in that 
> case MITM is possible I beleive.
> 
> You will need a stable network identity that you can use to configure as your 
> "proxy" in advance. For example in a testing scenario where you have access 
> to the kubernetes cluster you can simply use "localhost" as the name of the 
> proxy and use kubernetes port forward to tunnel requests from the localhost 
> to your individual nodes (only one node at a time).
> 
> Another option that could better work for non-local use cases is to use a 
> LoadBalancer service in front of the nodes and configure DNS to point to your 
> LoadBalancer IP. If you want to do this in advance it is possible to create 
> floating IPs and preconfigure DNS for it at almost any cloud provider. Then 
> add the configured DNS to nifi.web.proxy.host property when starting your 
> cluster. If setting up DNS is not an option you can use the IP directly. If 
> setting up the IP in advance is not an option you may use an arbitrary 
> hostname as the proxy host and add that hostname to your hosts file (or 
> dnsmasq or company dns) to point to the dynamically generated LoadBalancer IP 
> after NiFi