Re: NiFi V1.15.2 Conversion to Avro from JSON

[Mark Payne]

Thanks Nathan, the template is helpful. I put up a PR that I believe addresses 
the issues.

Thanks
-Mark


On Jan 25, 2022, at 7:21 AM, 
nathan.engl...@bt.com wrote:

Hey Mark,

Apologies if this is more of a dev mailing list issue. Happy to go across to it 
if needs be.

I've been testing the fix you've implemented for this and noticed a slight 
issue when the default value should be null. It took a little while to work out 
what was going on, but I think I've cracked it. I have attached a flow I've 
been using to test this.

So in AvroTypeUtil on line 630, it's creating the field as it doesn't exist. 
Instead of setting the default value to null, it is set to 
org.apache.avro.JsonProperties$Null, which throws an error when it tries to 
encode as Avro. For example, the Null String & Byte Type generator in the 
attached flow reproduces this.

In addition to this, I did a quick bit of testing with int and long types with 
default values in the Avro Schema set to 0. So, it looks like Long fields in 
the schema have the default values set as Integers? I have reproduced it with 
the attached flow's Int & Long Type generator.

Of course, this isn't an exhaustive list of the possible Avro types, but it 
does seem the changes made to the Avro Util class since v1.12.1 may be causing 
issues? Or at least it feels that way to me?

The only way I see forward is to restore the logic that was previously in place?

I have also created a bug NIFI-9629 [1].

Kind Regards,

Nathan

https://issues.apache.org/jira/browse/NIFI-9629
From: English,N,Nathan,VIR R
Sent: 20 January 2022 19:53
To: users@nifi.apache.org
Subject: Re: NiFi V1.15.2 Conversion to Avro from JSON

Hey Mark,

You beat me to the fix, Thanks for taking your time to look at it! I appreciate 
that one.

Cheers!

Nathan

From: Mark Payne mailto:marka...@hotmail.com>>
Sent: Wednesday, January 19, 2022 6:54:23 PM
To: users@nifi.apache.org 
mailto:users@nifi.apache.org>>
Subject: Re: NiFi V1.15.2 Conversion to Avro from JSON

Thanks Nathan. I created a Jira [1] for this. I was able to easily replicate 
with your template. Thanks for including that. Just put up a Pull Request for 
it, as well.

Thanks
-Mark



[1] 
https://issues.apache.org/jira/browse/NIFI-9594



On Jan 18, 2022, at 4:49 AM, 
nathan.engl...@bt.com wrote:

Hi There,

We've recently upgraded to NiFi V1.15.2 from V1.12.1 and have noticed a 
difference between the JSON Tree Reader Controller Service, which is a slight 
annoyance.

When we transform our records, we have some fields that will exist or won't 
after the transformation, depending on whether there is a value for the field. 
Previously with NiFi v1.12.1, when we would convert these records to Avro using 
the Avro Writer Controller and JSON Tree Reader Services (Both with Schemas 
Set), these fields would be created and set with the default value in the Avro 
Schema. But in NiFi v1.15.2, the conversion fails with "null of boolean in 
field field_name of Avro.Schema.Name", it doesn't matter which field, type or 
default value. It all seems to fail if the field doesn't exist.

Interestingly, there isn't an issue when I set the JSON Tree Reader to Infer 
Schema instead of 'use Schema Name' with a Schema Registry.

Annoyingly before we convert to Avro from JSON, we go through a Validate Schema 
Processor, which uses the same Schema, and JSON Tree Reader but instead uses a 
JSON Writer and successfully validates the record against Schema!

It seems similar to the NIFI-9335 issue on Jira [1], but maybe with the JSON 
Tree Reader Controller service instead?

I have attached a sample flow to reproduce the issue.

I didn't know if it was worth me raising a bug ticket?

Kind Regards,

Nathan

[1] 
https://issues.apache.org/jira/browse/NIFI-9335

Re: InvokeHTTP - SSL Cipher Selection Issue

[David Handermann]

Hi Shawn,

The InvokeHTTP processor relies on OkHttp to perform HTTP request and
response handling, so cipher suite negotiation is a combination of JVM
support and OkHttp configuration.  The InvokeHTTP does not implement any
direct configuration of the enabled cipher suites, and currently relies on
the default OkHttp behavior.  According to the following OkHttp
documentation on HTTPS, the default behavior is for OkHttp to use the
MODERN_TLS configuration option:

https://square.github.io/okhttp/https/

According to the list of APPROVED_CIPHER_SUITES associated with the
MODERN_TLS configuration, ECDHE-RSA-AES256-SHA384 is not one of supported
options:

https://github.com/square/okhttp/blob/parent-4.9.3/okhttp/src/main/kotlin/okhttp3/ConnectionSpec.kt#L290

Other NiFi components have different approaches to cipher suite support, so
OkHttp is a bit different in this scenario.  The OkHttp configuration
explains why you might see other TLS handshakes indicating support, but not
handshakes initiated through InvokeHTTP.

Regards,
David Handermann



On Tue, Jan 25, 2022 at 12:19 PM Shawn Weeks 
wrote:

> I need some help understanding how the InvokeHTTP processor negotiates SSL
> Ciphers with the remote host. I’ve got a remote system that only supports a
> weak cipher in this case ECDHE-RSA-AES256-SHA384. I’ve enabled Java SSL
> debugging with -Djavax.net.debug=ssl:handshake:verbose and I’ve noticed
> that the ClientHello logged does not list that cipher in the “cipher
> suites” value for that processor call however if I look at other
> ClientHello messages logged I see a much larger list of “cipher suites”
> listed including the one above. I’m trying to figure out what exactly is
> restricting the list of ciphers that are negotiated. I’ve tried using
> InvokeHTTP with a Restricted and Non Restricted SSL Controller. I’m not
> running Java in export mode so I have all the ciphers. I’ve also been doing
> some testing with the openssl s_client command and I’ve confirmed that only
> the cipher above is supported and any other cipher gives an error.
>
>
>
> Thanks
>
> Shawn
>

InvokeHTTP - SSL Cipher Selection Issue

[Shawn Weeks]

I need some help understanding how the InvokeHTTP processor negotiates SSL 
Ciphers with the remote host. I've got a remote system that only supports a 
weak cipher in this case ECDHE-RSA-AES256-SHA384. I've enabled Java SSL 
debugging with -Djavax.net.debug=ssl:handshake:verbose and I've noticed that 
the ClientHello logged does not list that cipher in the "cipher suites" value 
for that processor call however if I look at other ClientHello messages logged 
I see a much larger list of "cipher suites" listed including the one above. I'm 
trying to figure out what exactly is restricting the list of ciphers that are 
negotiated. I've tried using InvokeHTTP with a Restricted and Non Restricted 
SSL Controller. I'm not running Java in export mode so I have all the ciphers. 
I've also been doing some testing with the openssl s_client command and I've 
confirmed that only the cipher above is supported and any other cipher gives an 
error.

Thanks
Shawn

RE: NiFi V1.15.2 Conversion to Avro from JSON

[nathan.english]

Hey Mark,

Apologies if this is more of a dev mailing list issue. Happy to go across to it 
if needs be.

I've been testing the fix you've implemented for this and noticed a slight 
issue when the default value should be null. It took a little while to work out 
what was going on, but I think I've cracked it. I have attached a flow I've 
been using to test this.

So in AvroTypeUtil on line 630, it's creating the field as it doesn't exist. 
Instead of setting the default value to null, it is set to 
org.apache.avro.JsonProperties$Null, which throws an error when it tries to 
encode as Avro. For example, the Null String & Byte Type generator in the 
attached flow reproduces this.

In addition to this, I did a quick bit of testing with int and long types with 
default values in the Avro Schema set to 0. So, it looks like Long fields in 
the schema have the default values set as Integers? I have reproduced it with 
the attached flow's Int & Long Type generator.

Of course, this isn't an exhaustive list of the possible Avro types, but it 
does seem the changes made to the Avro Util class since v1.12.1 may be causing 
issues? Or at least it feels that way to me?

The only way I see forward is to restore the logic that was previously in place?

I have also created a bug NIFI-9629 [1].

Kind Regards,

Nathan

https://issues.apache.org/jira/browse/NIFI-9629
From: English,N,Nathan,VIR R
Sent: 20 January 2022 19:53
To: users@nifi.apache.org
Subject: Re: NiFi V1.15.2 Conversion to Avro from JSON

Hey Mark,

You beat me to the fix, Thanks for taking your time to look at it! I appreciate 
that one.

Cheers!

Nathan

From: Mark Payne mailto:marka...@hotmail.com>>
Sent: Wednesday, January 19, 2022 6:54:23 PM
To: users@nifi.apache.org 
mailto:users@nifi.apache.org>>
Subject: Re: NiFi V1.15.2 Conversion to Avro from JSON

Thanks Nathan. I created a Jira [1] for this. I was able to easily replicate 
with your template. Thanks for including that. Just put up a Pull Request for 
it, as well.

Thanks
-Mark



[1] 
https://issues.apache.org/jira/browse/NIFI-9594



On Jan 18, 2022, at 4:49 AM, 
nathan.engl...@bt.com wrote:

Hi There,

We've recently upgraded to NiFi V1.15.2 from V1.12.1 and have noticed a 
difference between the JSON Tree Reader Controller Service, which is a slight 
annoyance.

When we transform our records, we have some fields that will exist or won't 
after the transformation, depending on whether there is a value for the field. 
Previously with NiFi v1.12.1, when we would convert these records to Avro using 
the Avro Writer Controller and JSON Tree Reader Services (Both with Schemas 
Set), these fields would be created and set with the default value in the Avro 
Schema. But in NiFi v1.15.2, the conversion fails with "null of boolean in 
field field_name of Avro.Schema.Name", it doesn't matter which field, type or 
default value. It all seems to fail if the field doesn't exist.

Interestingly, there isn't an issue when I set the JSON Tree Reader to Infer 
Schema instead of 'use Schema Name' with a Schema Registry.

Annoyingly before we convert to Avro from JSON, we go through a Validate Schema 
Processor, which uses the same Schema, and JSON Tree Reader but instead uses a 
JSON Writer and successfully validates the record against Schema!

It seems similar to the NIFI-9335 issue on Jira [1], but maybe with the JSON 
Tree Reader Controller service instead?

I have attached a sample flow to reproduce the issue.

I didn't know if it was worth me raising a bug ticket?

Kind Regards,

Nathan

[1] 
https://issues.apache.org/jira/browse/NIFI-9335





Avro_Default_Value_Conversion_Issue_-_20220125.xml
Description: Avro_Default_Value_Conversion_Issue_-_20220125.xml