Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch

[Peter Moberg]

Hi Andy,

thanks for your suggestions. Here is what I have tried so far (still no luck).

Connecting with openssl and viewing the certs it presents

openssl s_client -connect quickstart-es-http.es-cluster -showcerts

If I then look inside the server cert I can find this

Server Cert:
Issuer: OU = quickstart, CN = quickstart-http

X509v3 Subject Alternative Name:
DNS:quickstart-es-http.es-cluster.es.local, DNS:quickstart-es-http, 
DNS:quickstart-es-http.es-cluster.svc, DNS:quickstart-es-http.es-cluster


If I look in to the self-signed root cert I find this:

Root Cert:
Subject: OU = quickstart, CN = quickstart-http


I now double check  my trust store to make sure the Root Cert is there.

Trust store content
Your keystore contains 1 entry

Alias name: ca_elastic
Creation date: Oct 16, 2019
Entry type: trustedCertEntry

Owner: CN=quickstart-http, OU=quickstart
Issuer: CN=quickstart-http, OU=quickstart
Serial number: 5aa50b6806d2394fff6f98d2b7d4c287
Valid from: Fri Oct 11 14:35:01 UTC 2019 until: Sat Oct 10 14:36:01 UTC 2020
Certificate fingerprints:
MD5: 1E:E3:33:13:EA:AC:B5:61:23:DE:2E:1A:D7:9C:AA:F0
SHA1: 62:EC:5B:EB:32:6A:38:3D:6A:6B:F7:10:5A:DE:E6:F1:F0:5B:07:99
SHA256: 
B4:B5:06:9C:50:5F:E8:A1:58:7C:C7:2C:37:52:2F:E0:CF:32:18:18:68:E4:C7:37:F8:82:B3:BC:61:EB:5B:CF
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
CA:true
PathLen:2147483647
]

#2: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
serverAuth
clientAuth
]

#3: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
DigitalSignature
Key_CertSign
]

So everything looks Ok. But when I run the ElasticSearchClientServicesImpl with 
a SSLContext pointing to my trust store I still get the following output in the 
log.

Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
at 
sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1633)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)
at 
org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)
... 9 common frames omitted
Caused by: sun.security.validator.ValidatorException: PKIX path building 
failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to 
find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
at sun.security.validator.Validator.validate(Validator.java:262)
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
at 
sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
at 
sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
at 
sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1620)
... 17 common frames omitted

Both the Nifi install and Elastic Search install is running in Kubernetes. The 
address I am using is a service address that is backed by 3 ES instances. 
However, I double checked all three of the ES nodes to make sure that they 
returned back the same SSL cert and they did.

The only thing I haven't been able to figure out is how to check if 
Kubernetes/ES reacts differently when you do a GET vs POST. Feels strange that 
it would return different SSL certs but stranger things have happened…



On Oct 17, 2019, 3:25 PM -0500, Andy LoPresto , wrote:
> Hi Peter,
>
> If you can use openssl’s s_client command (example below) to connect to the 
> endpoint and verify that the hostname matches the certificate and that the 
> certificate contains a SubjectAlternativeName entry with that hostname (see 
> RFC 6125 [1] for more details), this should help you debug the issue. The 
> cause of the PKIX error is that the truststore doesn’t contain a certificate 
> (or certificate chain) which matches the hostname presented by the remote 
> endpoint. I think you understand that based on your message. The underlying 
> reason for this is could be one of the following:
>
> * the server is behind an interface which responds differently to GET and 
> POST/PUT requests
> * there is a load-balancer which is directing the requests coincidentally to 
> different backend servers (one has 

Re: What UDFs are supported by QueryRecord?

[Mark Payne]

Eric,

I’m not sure that I understand. Can you provide an example illustrating what 
you’d like to do?

Thanks
-Mark

Sent from my iPhone

On Oct 17, 2019, at 4:49 PM, Eric Chaves  wrote:


Hi mark,

Thanks for pointing that out but from the docs I only got how to use RPATH to 
get a RecordPath value. How should I do for example if I wanted to apply a 
RecordPath function to in a Record Field?

Em qui, 17 de out de 2019 às 14:57, Mark Payne 
mailto:marka...@hotmail.com>> escreveu:
Eric,

You can use RecordPath with QueryRecord, via the RPATH, RPATH_STRING, 
RPATH_INT, etc. These are explained in the Processor's documentation. For 
example, see [1].

You can also use the Expression Language with QueryRecord. The Expression 
Language is evaluated before the SQL is parsed. So, for example, if you had an 
attribute named 'Field of Interest' you could actually use SQL like:

SELECT ${'Field of Interest'}
FROM FLOWFILE

Thanks
-Mark

[1] 
http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.2/org.apache.nifi.processors.standard.QueryRecord/index.html

On Oct 16, 2019, at 2:10 PM, Eric Chaves 
mailto:e...@uolet.com>> wrote:

Ok, figured out a way to do it. I noticed that QueryRecord uses Apache Calcite 
so I tried some of Calcite functions until I got into this statement that seems 
to work:

SELECT index FROM FLOWFILE WHERE CAST( SUBSTRING(index FROM 
CHAR_LENGTH(index)-9) AS DATE) <= CURRENT_DATE

Anyhow, I would still like to know if (and how) I could use either NiFi 
expression or RecordPath functions in a QueryRecord statement.

Thanks in advance,

Em qua, 16 de out de 2019 às 14:30, Eric Chaves 
mailto:e...@uolet.com>> escreveu:
Hi Folks,

I'd like to use a SQL statement in QueryRecords like this uses the extract the 
last 10 chars of  field and compare it to today, similar to the line below:

SELECT substring( '/index', 10, -1) as expired FROM FLOWFILE WHERE expired <= 
'${now():format("-MM-dd")}'

This statement is not work and I can't find a list of QueryRecord supported UDF 
or if (nd how) I can use RecordPath functions.

Is there any documentation where I can see the available UDFs?

Regards,

Re: What UDFs are supported by QueryRecord?

[Eric Chaves]

Hi mark,

Thanks for pointing that out but from the docs I only got how to use RPATH
to get a RecordPath value. How should I do for example if I wanted to apply
a RecordPath function to in a Record Field?

Em qui, 17 de out de 2019 às 14:57, Mark Payne 
escreveu:

> Eric,
>
> You can use RecordPath with QueryRecord, via the RPATH, RPATH_STRING,
> RPATH_INT, etc. These are explained in the Processor's documentation. For
> example, see [1].
>
> You can also use the Expression Language with QueryRecord. The Expression
> Language is evaluated before the SQL is parsed. So, for example, if you had
> an attribute named 'Field of Interest' you could actually use SQL like:
>
> SELECT ${'Field of Interest'}
> FROM FLOWFILE
>
> Thanks
> -Mark
>
> [1]
> http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.2/org.apache.nifi.processors.standard.QueryRecord/index.html
>
> On Oct 16, 2019, at 2:10 PM, Eric Chaves  wrote:
>
> Ok, figured out a way to do it. I noticed that QueryRecord uses Apache
> Calcite so I tried some of Calcite functions until I got into this
> statement that seems to work:
>
> SELECT index FROM FLOWFILE WHERE CAST( SUBSTRING(index FROM
> CHAR_LENGTH(index)-9) AS DATE) <= CURRENT_DATE
>
> Anyhow, I would still like to know if (and how) I could use either NiFi
> expression or RecordPath functions in a QueryRecord statement.
>
> Thanks in advance,
>
> Em qua, 16 de out de 2019 às 14:30, Eric Chaves  escreveu:
>
>> Hi Folks,
>>
>> I'd like to use a SQL statement in QueryRecords like this uses the
>> extract the last 10 chars of  field and compare it to today, similar to the
>> line below:
>>
>> SELECT substring( '/index', 10, -1) as expired FROM FLOWFILE WHERE
>> expired <= '${now():format("-MM-dd")}'
>>
>> This statement is not work and I can't find a list of QueryRecord
>> supported UDF or if (nd how) I can use RecordPath functions.
>>
>> Is there any documentation where I can see the available UDFs?
>>
>> Regards,
>>
>>
>

Re: ElasticSearchClientServiceImpl not working for secured ElasticSearch

[Andy LoPresto]

Hi Peter,

If you can use openssl’s s_client command (example below) to connect to the 
endpoint and verify that the hostname matches the certificate and that the 
certificate contains a SubjectAlternativeName entry with that hostname (see RFC 
6125 [1] for more details), this should help you debug the issue. The cause of 
the PKIX error is that the truststore doesn’t contain a certificate (or 
certificate chain) which matches the hostname presented by the remote endpoint. 
I think you understand that based on your message. The underlying reason for 
this is could be one of the following:

* the server is behind an interface which responds differently to GET and 
POST/PUT requests
* there is a load-balancer which is directing the requests coincidentally to 
different backend servers (one has the right cert; the other doesn’t)
* I recall something around the addition of (some) Elastic Search components 
which handled TLS in an ES client-specific manner; I remember advocating for 
standard NiFi TLS interaction here but I am not sure what was ultimately 
contributed. If it’s not one of the above issues, I can investigate further. 

Hopefully this helps. 

[1] https://tools.ietf.org/html/rfc6125#section-6.4.4 


s_client example: 

$ openssl s_client -connect  -debug -state -cert 
 -key  -CAfile 


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Oct 16, 2019, at 8:37 PM, Peter Moberg  wrote:
> 
> I have an Elastic Search cluster that is setup with SSL. It uses a 
> self-signed cert for this. I am working with Apache Nifi 1.9.2. I have a flow 
> that has the PutElasticSearchHttp component. I have setup a SSLContextService 
> for that component where I have specified a trust store that has the 
> self-signed cert from ES. I specify an https endpoint to access Elastic 
> Search and Im having no issues populating my Elastic Search instance using 
> this flow.
> 
> I have another flow where I want to do some lookups. So I have been using the 
> LookupRecord processor. That one I have associated with an 
> ElasticSearchClientServiceImpl which I have setup to  point to the same 
> SSLContextService as used above. I specified the same HTTPS Url (triple 
> checked this). However, when I run this second Flow I am not able to verify 
> the ES server's self-signed certificate.
> 
> I check the nifi-app.log and it says:
> Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable 
> to find valid certification path to requested target
> 
> I am a bit surprised that I am not able to verify the same server certificate 
> in the two different flows.
> 
> Completely stuck on this so if anyone have any pointers please let me know.
> 
> Thanks,
> 
> Peter

Re: What UDFs are supported by QueryRecord?

[Mark Payne]

Eric,

You can use RecordPath with QueryRecord, via the RPATH, RPATH_STRING, 
RPATH_INT, etc. These are explained in the Processor's documentation. For 
example, see [1].

You can also use the Expression Language with QueryRecord. The Expression 
Language is evaluated before the SQL is parsed. So, for example, if you had an 
attribute named 'Field of Interest' you could actually use SQL like:

SELECT ${'Field of Interest'}
FROM FLOWFILE

Thanks
-Mark

[1] 
http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.9.2/org.apache.nifi.processors.standard.QueryRecord/index.html

On Oct 16, 2019, at 2:10 PM, Eric Chaves 
mailto:e...@uolet.com>> wrote:

Ok, figured out a way to do it. I noticed that QueryRecord uses Apache Calcite 
so I tried some of Calcite functions until I got into this statement that seems 
to work:

SELECT index FROM FLOWFILE WHERE CAST( SUBSTRING(index FROM 
CHAR_LENGTH(index)-9) AS DATE) <= CURRENT_DATE

Anyhow, I would still like to know if (and how) I could use either NiFi 
expression or RecordPath functions in a QueryRecord statement.

Thanks in advance,

Em qua, 16 de out de 2019 às 14:30, Eric Chaves 
mailto:e...@uolet.com>> escreveu:
Hi Folks,

I'd like to use a SQL statement in QueryRecords like this uses the extract the 
last 10 chars of  field and compare it to today, similar to the line below:

SELECT substring( '/index', 10, -1) as expired FROM FLOWFILE WHERE expired <= 
'${now():format("-MM-dd")}'

This statement is not work and I can't find a list of QueryRecord supported UDF 
or if (nd how) I can use RecordPath functions.

Is there any documentation where I can see the available UDFs?

Regards,

Re: High CPU consumption

[Mark Payne]

Hey Evan,

Of note, it looks like the high CPU usage of the Port was resolved in NIFI-5464 
[1], which should be part of the next release.

Thanks
-Mark

https://issues.apache.org/jira/browse/NIFI-5464

On Oct 15, 2019, at 4:40 PM, Evan Reynolds 
mailto:e...@usermind.com>> wrote:

I have found two issues that can cause high CPU when idle (high being about 
200% CPU when idle.) I haven’t verified these with 1.9.2, but it doesn’t hurt 
to tell you.

If you are using ports, make sure each input port is connected. If you have a 
one that isn’t connected, that can bring your CPU to 200% and stay there.

If you have any processors that are set to run on the primary node of a 
cluster, that can also take your CPU to 200%. I know RouteOnAttribute will do 
that (again, haven’t tested 1.9.2, but it was a problem for me for a bit!) The 
fix – either don’t run it on the primary node, or else set the run schedule to 
5 seconds or something instead of 0.

To find out if this is the case – well, this is what I did. It worked, and 
wasn’t that hard, though isn’t exactly elegant.

Back up your flowfile (flow.xml.gz)
Stop all your processors and see what your CPU does
Start half of them and watch your CPU – basically do a binary search. If your 
CPU stays reasonable, then whatever group you started is fine. If not, then 
start stopping things until your CPU becomes reasonable.
Eventually you’ll find a processor that spikes your CPU when you start it and 
then you can figure out what to do about that processor. Record which processor 
it is and how you altered it to bring CPU down.
Repeat, as there may be more than one processor spiking CPU.
Stop NiFi and restore your flowfile by copying it back in place – since you 
were running around stopping things, this just makes sure everything is 
correctly back to where it should be

Then use the list of processors and fixes to make changes.

---

Evan Reynolds
e...@usermind.com


From: Jon Logan mailto:jmlo...@buffalo.edu>>
Reply-To: "users@nifi.apache.org" 
mailto:users@nifi.apache.org>>
Date: Sunday, October 13, 2019 at 6:12 PM
To: "users@nifi.apache.org" 
mailto:users@nifi.apache.org>>
Subject: Re: High CPU consumption

That isn't logback, that lists all jars on your classpath, the first of which 
happens to be logback.

If you send a SIGKILL3 (you can send it in HTOP) it will dump the thread stacks 
to stdout (probably the bootstrap log)...but that's just for one instant, and 
may or may not be helpful.

On Sun, Oct 13, 2019 at 8:58 PM Luis Carmona 
mailto:lcarm...@openpartner.cl>> wrote:
hi Aldrin,

thanks a  lot, by now I'm trying to learn how to make the profiling you 
mentioned.

One more question: Is it normal that the father java process has very low 
consumption while the child process related to logback jar is the one that is 
eating up all the CPU ?
Please take a look at the attached image.

Thanks,

LC


From: "Aldrin Piri" mailto:aldrinp...@gmail.com>>
To: "users" mailto:users@nifi.apache.org>>
Sent: Sunday, October 13, 2019 9:30:47 PM
Subject: Re: High CPU consumption

Luis, please feel free to give us some information on your flow so we can help 
you track down problematic areas as well.

On Sun, Oct 13, 2019 at 3:56 PM Jon Logan 
mailto:jmlo...@buffalo.edu>> wrote:
You should put a profiler on it to be sure.
Just because your processors aren't processing data doesn't mean they are idle 
though -- many have to poll for new data, especially sources -- ex. connecting 
to Kafka, etc, will itself consume some CPU.

But again, you should attach a profiler before participating in a wild goose 
chase of performance issues.

On Sun, Oct 13, 2019 at 12:20 PM Luis Carmona 
mailto:lcarm...@openpartner.cl>> wrote:
HI,

I've struggling to reduce my nifi installation CPU consumption. Even in idle 
state, all processors running but no data flowing, it is beyond 60% CPU 
consumption, with peaks of 200%.

What I've done so far
- Read and followed every instruction/post about tuning NIFI I've found 
googling.
- Verify scheduling is 1s for most consuming processors: http processors, 
wait/notify, jolt, etc.
- Scratch my head...

But nothing seems to have a major effect on the issue.

Can anyone give me some precise directions or tips about how to solve this 
please ?
Is this the regular situation, I mean this is the minimun NIFI consumption.

The server is configure with 4 CPU's, 8 GB RAM - 4 of them dedicated to heap at 
bootstrap.conf.

Thanks in advance.

LC

Unable to do unit testing Awscredentialprovidercontrolerservice

[sanjeet rath]

Hi Team,

I am new to this community, today i have joined and this is my first
query.(already from last 15 days i got stuck here)

I have build a nifi custom processor.to put a object in AWS s3 bucket.(Just
Modified Puts3Object processor in NIFI- AWS-PROCESSOR project)
In this processor  Awscredentialprovidercontrolerservice is the default one
to connect AWS.

But our project structure we have a custom controller service
,Awscredentialprovidercontrolerservice controller service  to connect AWS
with our defined attributes.(This is present in a separate project
NIFI-AWS-CUSTOM_PING_CONTROLER)
I want to use this custom controller service , instead of the default
Awscredentialprovidercontrolerservice one which is present in the NIFI-
AWS-PROCESSOR project

My question is I want to do the unit testing (Using JUnit & Mockito)  to
validate this custom Awscredentialprovidercontrolerservice is working or
not like if I get invalid bucket name then test case should failed.

My code structure:
@Test
public void testRetryLogin() {
final TestRunner runner = TestRunners.newTestRunner(new
puts3Object);
final Awscredentialprovidercontrolerservice  serviceimpl= new
Awscredentialprovidercontrolerservice()
Runner.setproperty(…)
Runner.enablecontrolerservice(serviceimpl)
runner.run();
   // will do assert  }

This gives me compiler error in error in
Awscredentialprovidercontrolerservice instance creation line as custom
controller service project is not available to this processor service
project. If we add dependency in pom file to make it available then it will
be circular dependency.
Because  already in custom controlerservice pom file , custom processor is
already added in dependency..

Thanks in advance :)

-- 
Sanjeet Kumar Rath,
mob- +91 8777577470

RE: Sudo service nifi command not working

[johannes.meixner]

Moin,

service(8) often drops whatever environment variables are in the environment,
so you'll want to set that in nifi-env.sh instead.

Best
Johannes

-Original Message-
From: Bryan Bende  
Sent: Thursday, October 17, 2019 4:24 PM
To: users@nifi.apache.org
Subject: Re: Sudo service nifi command not working

It is best to email only one list, so dropping dev...

If you are starting the service with sudo then its the root user starting it, 
so you need to check if echo "$JAVA_HOME" works as root.

On Thu, Oct 17, 2019 at 10:19 AM Praveen Anguralia 
 wrote:
>
> Hi team,
>
> I need your inputs on an issue that I am facing with respect to the command:
> sudo service nifi start
> Background: I have installed Nifi in Amazon EC2 linux machine. I have 
> installed Java as well. When I run command as:
> ./nifi.sh start
> I do not find issue. Nifi starts successfully.
> But when I run command as:
>   sudo service nifi start
> It says JAVA_HOME not set.
> Its not able to find out java.
> Howveer, echo "$JAVA_HOME" gives the installed path
>
> Please help me to sort out the issue.
>
> Thank you,
> Praveen

Re: Sudo service nifi command not working

[Bryan Bende]

It is best to email only one list, so dropping dev...

If you are starting the service with sudo then its the root user
starting it, so you need to check if echo "$JAVA_HOME" works as root.

On Thu, Oct 17, 2019 at 10:19 AM Praveen Anguralia
 wrote:
>
> Hi team,
>
> I need your inputs on an issue that I am facing with respect to the command:
> sudo service nifi start
> Background: I have installed Nifi in Amazon EC2 linux machine. I have
> installed Java as well. When I run command as:
> ./nifi.sh start
> I do not find issue. Nifi starts successfully.
> But when I run command as:
>   sudo service nifi start
> It says JAVA_HOME not set.
> Its not able to find out java.
> Howveer, echo "$JAVA_HOME" gives the installed path
>
> Please help me to sort out the issue.
>
> Thank you,
> Praveen