Re: is there a trick to running Nifi on AWS

[Matthew Hawkins]

Hi Mark,

NiFi supports many different forms of authentication. The "simple" way is
"http basic" and you can roll your own /etc/passwd style authenticator with
it. I actually recommend NOT doing this unless it's a test system as http
basic is trash and can be a right pest if you need to "unauth".

Single Sign-On refers to the literal SSO auth framework, where you'll need
an IdP and so forth per the SSO spec. You can allegedly get this to work
with AWS IAM but I've never tried. Note NiFi is designed first up to
integrate into an Apache/HDFS environment so there's often assumptions like
you have HDFS and you have a schema service and you have Ranger for auth.
Bear this in mind because it'll make flow configuration make more sense
once you've got the system up.

Note you can also mix different kinds; such as keep a break-glass admin
account with http basic but do more enterprise auth with LDAP which
integrates okish with RBAC in NiFi for daily users incl those with admin
privs via RBAC. This can be handy for those times when *cough* AD *cough*
dies again.

Authentication is one of the key configuration items to understand prior to
deployment because your choice will drive what steps are necessary. E.g.
with TLS auth you will need to add the CA chain to the truststore and make
sure the users.xml is entirely accurate including spaces with the DN from
the x509 data. If you have a cluster, you need to understand auth is not
synced, each cluster member is its own individual instance & so some steps
have to be repeated on each member. There's a fantastic ansible script from
cavemandaveman to manage deployments properly and I do recommend that path
over manual editing even with a single instance.

Kind regards,

On Thu, 4 Apr 2024, 08:59 Mark Woodcock,  wrote:

> What does the documentation mean by this paragraph:
>
> >For Single sign-on authentication, NiFi will redirect users to the
> Identity Provider before returning to NiFi. NiFi will then process
> responses and convert attributes to application token information.
>
> I've gotten local instances of Nifi to work before, but getting it to run
> on an AWS/EC2 instance (across a couple of proxies, to manage local
> firewalls) is not going well. I can see (from login-identity-providers.xml)
> that I've successfully set a simple username/password, but it always fails
> when I get to the UI.   But, is the single-sign-on stuff not part of NiFi?
> What sort of redirection is going on at this point?
>
> thx,
>
> mew
>
>
> On Thu, Mar 28, 2024 at 4:31 PM Mark Woodcock  wrote:
>
> > Howdy,
> >
> > Cranked up an EC2 instance.
> > Installed Java 11.
> > set up JAVA_HOME
> > Downloaded Nifi 1.25.0
> > unzipped Nifi
> > set a nifi.sensitive.properties.key
> > (https.port is default 8443)
> >
> > bin/nifi.sh start
> >
> > But, I can't even seem to access the most basic bit of the UI:
> >
> > curl -vvvk https://54.91.56.55:8443
> > *   Trying 54.91.56.55:8443...
> > * connect to 54.91.56.55 port 8443 failed: Connection refused
> > * Failed to connect to 54.91.56.55 port 8443 after 17 ms: Connection
> > refused
> > * Closing connection 0
> > curl: (7) Failed to connect to 54.91.56.55 port 8443 after 17 ms:
> > Connection refused
> >
> > I have no doubt, I'm doing something astonishingly dumb.  Would someone
> be
> > kind enough to point it out?
> >
> > thx,
> >
> > mew
> >
> >
>

Re: is there a trick to running Nifi on AWS

[Matthew Hawkins]

It's more than that,

You also need to setup TLS keys for the hostname, user accounts, etc. Also,
I would never expose a service directly on a public IP that way either. Use
a priv subnet and access via a VPN. Route 53 can manage DNS for a private
domain so you can have eg nifi-1.mycloud as the hostname and expose all the
necessary ports (not just 8443) to the necessary security groups. With no
other information on the deployment, it's not possible to specify
instructions.

It might be worthwhile adding an AWS CDK setup to the code so admins have
"cdk deploy" as a post-build option, even if it's just a basic ec2 host
like above and not getting into integration with logging, authentication,
etc.

On Fri, 29 Mar 2024, 07:59 Mark Bean,  wrote:

> Try adding your EC2 instance's private IP address mapping to public DNS
> name in /etc/hosts. And, in that case, you can use
> nifi.web.https.host=
>
> -Mark
>
>
> On Thu, Mar 28, 2024 at 4:40 PM Joe Witt  wrote:
>
> > Mark
> >
> > I believe you will need to tell NiFi you want it to listen on more than
> the
> > localhost/loopback address.
> >
> > nifi.web.https.host=localhost
> >
> > Is a default in nifi.properties for instance.
> >
> > Def take a look through the admin/install guide as well.
> >
> > Thanks
> >
> > On Thu, Mar 28, 2024 at 1:32 PM Mark Woodcock  >
> > wrote:
> >
> > > Howdy,
> > >
> > > Cranked up an EC2 instance.
> > > Installed Java 11.
> > > set up JAVA_HOME
> > > Downloaded Nifi 1.25.0
> > > unzipped Nifi
> > > set a nifi.sensitive.properties.key
> > > (https.port is default 8443)
> > >
> > > bin/nifi.sh start
> > >
> > > But, I can't even seem to access the most basic bit of the UI:
> > >
> > > curl -vvvk https://54.91.56.55:8443
> > > *   Trying 54.91.56.55:8443...
> > > * connect to 54.91.56.55 port 8443 failed: Connection refused
> > > * Failed to connect to 54.91.56.55 port 8443 after 17 ms: Connection
> > > refused
> > > * Closing connection 0
> > > curl: (7) Failed to connect to 54.91.56.55 port 8443 after 17 ms:
> > > Connection refused
> > >
> > > I have no doubt, I'm doing something astonishingly dumb.  Would someone
> > be
> > > kind enough to point it out?
> > >
> > > thx,
> > >
> > > mew
> > >
> >
>

Re: NiFi Manual Keystore - Generate cluster node certificate chain

[Matthew Hawkins]

A reminder I tossed up a gist over here with a java keytool version of a
local CA with a NiFi bent:

https://gist.github.com/hawko2600/922b727634784614465b83e52ec2be52

For clusters, you just need a key per host and share them around the common
truststore.jks

Consider it Apache License 2.0.

On Mon, 19 Feb 2024, 11:04 Paul Grey,  wrote:

> Matthew,
>
> Thanks much for identifying this issue with the documentation.  I see the
> same error message when running step 8.  I've created a JIRA and a pull
> request to correct the documentation.
>
> https://issues.apache.org/jira/browse/NIFI-12814
> https://github.com/apache/nifi/pull/8424
>
> On Fri, Feb 16, 2024 at 10:05 PM Matthew Wilson
>  wrote:
>
> > Hi Devs,
> > Recently found that in the steps to create Manual Keystore the step 6 for
> > "Generate cluster node certificate chain" currently shows as:
> > cat ca.cer nifi1.cer >nifi1.chain.cercat ca.cer nifi2.cer
> >nifi2.chain.cer
> >
> > However this creates the chain.cer files incorrectly which generates  the
> > error :error:05800074:x509 certificate
> routines:X509_check_private_key:key
> > values mismatch:crypto/x509/x509_cmp.c:408:
> > When running step 8 "Generate cluster node keystore"
> > Correct syntax for step 6 that works is below:
> > cat nifi1.cer ca.cer >nifi1.chain.cercat nifi2.cer ca.cer
> >nifi2.chain.cer
> > Reference URL Page:
> >
> https://nifi.apache.org/documentation/nifi-2.0.0-M2/html/walkthroughs.html#manual-keystore
> >
> > Please review and correct documentation as appropriate.  If you need
> more,
> > information let me know.
> >
> >
> > Kind Regards,
> > Matthew WilsonEmail: wilson...@yahoo.com.au
> >
> > This e-mail may contain confidential and privileged material for the sole
> > use of the intended recipient. Any review, use, distribution or
> disclosure
> > by others is strictly prohibited. If you are not the intended recipient
> (or
> > authorised to receive for the recipient), please contact the sender by
> > reply e-mail and delete all copies of this message.
>

Re: How to get files after a condition has been met

[Matthew Hawkins]

The wait/notify pattern might work here too.

https://pierrevillard.com/2018/06/27/nifi-workflow-monitoring-wait-notify-pattern-with-split-and-merge/

Wait the getfile and notify when the validatejson is stopped?


On Tue, 6 Feb 2024, 22:55 Mike Thomsen,  wrote:

> I think the easiest way to do the conditional fetch would be to use
> ConsumeKafka (or something equivalent for this use case like ListenHttp)
> and FetchFile.
>
> On Mon, Feb 5, 2024 at 4:23 PM Dan S  wrote:
>
> > I have a requirement to load Json Schemas into the
> > StandardJsonSchemaRegistry from the file system for use in validating
> JSON
> > with the ValidateJson processor. I am trying to use the NIFI Rest API to
> > update the StandardJsonSchemaRegistry. I am able to stop the ValidateJson
> > processor and disable the StandardJsonSchemaRegistry using the NIFI Rest
> > API. I do not know though how I can indicate either to GetFile or
> ListFile
> > (to allow for retrieval of the JSON schemas) that ValidateJson has been
> > stopped and StandardJsonSchemaRegistry has been disabled since both of
> > these processors do not allow for an incoming relationship. How can I do
> > this and if this is not possible how can I accomplish this?
> >
>

Re: Docker container for 2.0.0-M1

[Matthew Hawkins]

Hi Joe,

I have it running against Bellsoft's v21 alpine-musl base fine, with python
enabled. Happy to submit the Dockerfile if you want. In a nutshell,

apk add --no-cache --no-interactive --no-progress coreutils unzip curl jq
xmlstarlet procps-ng python3 py3-pip py3-virtualenv ripgrep

And other than obviously Alpine not Debian, Busybox wants adduser/addgroup
not the util-linux variants. I moved the apk call to the scripts section so
only one apk update is needed, and deleted the apt cache cleanup. I added
the ripgrep package to grok logs.

I initially missed the VOLUME line in the dockerfile. Unfortunately I can't
seem to get the -v flag to work to use local folders instead of anonymous
volumes. I appreciate it's probably done for performance but it would be
nice if they had sensible names. Host is Win11 with docker desktop and
wsl2. For brevity, I'm adding a -v
c:\users\me\nifi\conf:/opt/nifi/nifi-current/conf format line for each of
the exported volumes to the docker run invocation (wrote a small cmd to
launch as I'm not typing that monstrosity out daily by hand)
I know I can manually docker volume some nice named ones and alter the
VOLUME line accordingly. Should a docker compose be used instead? Because
this is all feeling hackish.

KR,

On Wed, 29 Nov 2023, 17:24 Joe Witt,  wrote:

> Matt
>
> Alpine is still desirable but it was about availability of Alpine w Java 21
> at that time.  I think Alpine also required some different command and such
> in the dockerfile.
>
> As far as the repo volumes they can be mapped as you wish now.  It is how I
> keep updating my own flow deployments I test with when I want to retain
> state across deployment.  Are you not able to map them?
>
> Thanks
>
>
>
> On Tue, Nov 28, 2023 at 9:41 PM Matthew Hawkins 
> wrote:
>
> > Two Q's regarding the docker container;
> >
> > 1. Why the Debian version instead of Alpine? Was it compatibility with
> > external stuff? I can confirm 2.0.0-M1 works fine with the alpine version
> > of liberica-jdk on at least basic flows, with python enabled. It'll save
> > disk space. I love Debian but Alpine is king of containers!
> >
> > 2. If we shift the repository folders out of NIFI_HOME per the regular
> best
> > practice doco, then a persistent  docker volume could easily be used to
> > house these, making upgrades painless. I guess the flow.json, keystores
> and
> > configs could move too.
> >
> > Kind regards,
> > Matt
> >
>

Docker container for 2.0.0-M1

[Matthew Hawkins]

Two Q's regarding the docker container;

1. Why the Debian version instead of Alpine? Was it compatibility with
external stuff? I can confirm 2.0.0-M1 works fine with the alpine version
of liberica-jdk on at least basic flows, with python enabled. It'll save
disk space. I love Debian but Alpine is king of containers!

2. If we shift the repository folders out of NIFI_HOME per the regular best
practice doco, then a persistent  docker volume could easily be used to
house these, making upgrades painless. I guess the flow.json, keystores and
configs could move too.

Kind regards,
Matt

Re: Removing JRuby?

[Matthew Hawkins]

Hi Team,

As a user I think re-implementing certain scripting languages in Java is a
bad idea compared to having a robust API that any language current or
future can interface with. I make a lot of use of ExecuteProcess and
anecdotally haven't experienced a situation where externally running the
script vs the internal ExecuteScript made a poomteenth of difference in
load or timing. It may have been the case in the 1980's that forking a
process was time consuming; we've moved on.

I think we have (almost) enough options now to expedite data transfer with
an external system so it wouldn't even be necessary to bloat out data sizes
4x with encoded JSON via the REST API, just load flowfiles in via Kafka or
whatever to and from the script. It is no longer your concern; NiFi exposed
the ability to in a generic way and you're done. Stop worrying about things
out of your control like the only available java version of ruby has a
billion and 1 CVE's and is unmaintained, or Jython has a working 3.x
version but it's been labelled "beta" for 15 years. You shouldn't have to
care what a user uses to interface with the NiFi API, and certainly don't
bloat the NiFi installer by including each and every option!

Let someone else write a native API library in scripting language of choice
to expose eg "from NiFi import FlowFileQueue" so an even more expansive
library is available than what was exposed via ExecuteScript. This is what
friends on GitHub are for 
Concentrate on making the NiFi API robust, complete and performant.

(I know the REST API exists, to me the missing piece is being able to
register a script in a generic fashion so that it integrates with the flow
and hence understands which IDs it requires to call esp FlowFile Queues via
the rest API without having to always trawl through eg /flow and
dynamically generate it that way, which would be expensive imo on each
schedule)

Kind regards,


On Thu, 16 Nov 2023, 08:07 Joe Witt,  wrote:

> Team,
>
> Coming back to this as I've been reviewing all the great output of OWASP
> that we're getting daily now (thanks David) as well as some findings from a
> nice vendor tool by JFrog.  And what is popping up left and right are
> issues in terms of vulnerabilities tied to Jython-standalone.  It is very
> clear this component needs to go and is not actively maintained.  It is
> also as noted tied to Python 2 which is eol.  I'll file a JIRA to deprecate
> in 1.x and remove in 2.x.
>
> The discussion about Javascript/Ruby, etc.. I think the best course going
> forward is a specific component appropriate to each desired mechanism is
> introduced in 2.x as interest/effort dictates.  The native Python support
> will eliminate the need for Jython obviously so not worried about that
> coming back.
>
> Thanks
>
> On Tue, Nov 7, 2023 at 5:17 AM Mike Thomsen 
> wrote:
>
> > Matt,
> >
> > I don't use JRuby, but the reason it raised my eyebrow a little was that
> > AFAIK JRuby is the complete opposite of Jython in terms of enterprise
> > readiness and parity with its C-based counterpart.
> >
> > Edward,
> >
> > The problem with Lua here is that the Java versions of Lua aren't being
> > maintained. I think the most recent release on any branch of them was 3
> > years ago. They're basically dead in the water as projects.
> >
> > On Mon, Nov 6, 2023 at 2:57 PM Edward Armes 
> > wrote:
> >
> > > While I do like the ExecuteScript processors as they are great at
> digging
> > > you out of a hole. The performance of them isn't that great.
> > >
> > > That being said I would ve careful about dropping Lua support as there
> > is a
> > > growing list of software that supports end user/administrator
> extensions
> > > via Lua for those that don't want to have to re-compile software
> > > themselves. On the other hand given that Jython doesn't yet have a
> > Python 3
> > > implementation it could be argued dropping Jython support is a must
> given
> > > that the Python 2.x line is basically end of life.
> > >
> > > Now I wonder if its worth re-factoring the ExecuteScript processors to
> be
> > > per language implementations inheriting from a common base like a few
> > > processors do already.
> > >
> > > Edward
> > >
> > > On Mon, 6 Nov 2023, 16:24 Matt Burgess,  wrote:
> > >
> > > > I believe it is because in both ExecuteScript and ExecuteGroovyScript
> > > > you can do "regular" groovy, but EGS has extra built-ins such as easy
> > > > access to controller services, Groovy SQL stuff, etc, and we could
> > > > keep building it out. But IMO we'd have to port the rest of the
> > > > scripted components (ScriptedReader/Writer, etc.) over to the Groovy
> > > > bundle and make sure there's a drop-in replacement in the Python
> stuff
> > > > before we'd want to deprecate the scripted bundle.
> > > >
> > > > On the JRuby front, is that something you use actively? This question
> > > > is for you and the entire community of course.
> > > >
> > > > Regards,
> > > > Matt
> > > >
> > > > On Mon, Nov 6, 2023 

Re: JOLTTransformRecord problem

[Matthew Hawkins]

I've had similar issues with the different processing done by
JoltTransformJSON and JoltTransformRecord and threw my hands in the air and
just ExcuteScript out to some Python that transforms the data correctly.

Try to minimise content transformations so the content repository doesn't
bloat with interim forms.

On Fri, 13 Oct 2023, 01:26 Mark Woodcock,  wrote:

> Chris,
>
> I guess I was so wrapped up in the ConvertRecord, it didn't occur to me to
> have the JOLT processor just emit a different format; not sure how I missed
> that.
> So, I re-rigged my flow to have  ReplaceText -> JOLTTransform ->
> UpdateAttribute -> etc.   Unfortunately, I got the attached error.
>
> This is running against the set of data that I provided earlier (that is
> the test case for the bug).  It's first 3 records all lack dimension and
> eta fields (which my ReplaceText process has removed all evidence of);
> followed by one record that has record values for both fields.  The error
> seems to be saying that those early records (at the time of writing to
> Avro) don't have the appropriate format, because they have "null" values
> (in the ETA and dimensions fields) which are not valid for the JSON schema
> which has been inferred.
>
> Why is it creating a single JSON schema for the whole file instead of
> managing each record independently?  (and not that it really matters, but
> hey, the short records are coming first, why precompute the schema before
> finishing the processing of any record?)
>
> And, is there something else I can try?
>
> thx,
>
> mew
>
>
>
>
> On Wed, Oct 11, 2023 at 4:35 PM Chris Sampson
>  wrote:
>
>> You could use an AvroWriter to output the results of the
>> JoltTransformRecord - it doesn’t need to be JSON (in or out), that’s one of
>> the great things of NiFi’s Record processors - if there’s a Reader and
>> Writer in the format you want, you can use that data and the Writer doesn’t
>> need to be the same format as the Reader.
>>
>> Good news: I’ve identified the problem in my NIFI-8135 PR [1] by adding a
>> cut-down version of your example as a unit test for the JoltTransformRecord
>> processor.
>>
>> However, I’m not so sure the output is quite what you were expecting -
>> see the
>> nifi-nar-bundles/nifi-jolt-record-bundle/nifi-jolt-record-processors/src/test/resources/TestJoltTransformRecord/flattenedOutput.json
>> file in the linked PR, the “Eta”’s “value” field appears as a Java Map
>> serialised as a String, I imagine you were wanting this to be a nested
>> Object?
>>
>> If the latter, I think we’re then running into NIFI-8134 [2], for which I
>> have a separate PR ready for review [3].
>>
>> [1]: https://github.com/apache/nifi/pull/7746/files
>>
>> [2]: https://issues.apache.org/jira/browse/NIFI-8134
>>
>> [3]: https://github.com/apache/nifi/pull/7745
>>
>>
>> Cheers,
>>
>> ---
>> Chris Sampson
>> IT Consultant
>> chris.samp...@naimuri.com
>>
>>
>> > On 11 Oct 2023, at 19:22, Mark Woodcock 
>> wrote:
>> >
>> > Chris,
>> >
>> > 1) well, reassuring to learn that I've found an actual bug; and
>> pleasing to
>> > know that I constructed an effective and illuminating test.  hurrah.
>> >
>> > 2) So, I can certainly use the ReplaceText (is there a better choice?)
>> > processor to ditch any field that looks like "whatever": "", (and I
>> > successfully implemented it), but unfortunately when I pass the
>> resulting
>> > json onto another processor (e.g. a ConvertRecord, so I can spit out
>> AVRO),
>> > the fact that the data now has different schemas causes an error.  Is
>> this
>> > just kicking the can down the road?
>> >
>> > thx,
>> >
>> > mew
>> >
>> >
>> > On Wed, Oct 11, 2023 at 6:00 AM Chris Sampson
>> >  wrote:
>> >
>> >> FYI - original thread in the archives for reference [1].
>> >>
>> >> Thanks for your more complete example, this does indeed fail with the
>> >> error you indicate. I think it’s related to NIFI-8135 [2], which
>> identified
>> >> a deficiency in the way Records are converted to Java Maps,
>> particularly
>> >> where CHOICE types are involved.
>> >>
>> >> The example data you’ve provided does indeed have a mix of String and
>> >> Record (JSON Object) values for the affected fields - this is a little
>> >> unusual, but certainly nothing that’s banned in the world of JSON, so
>> >> should probably be handled better by NiFi.
>> >>
>> >> I’ve had a go at providing a PR for NIFI-8135 (as yet unreviewed) [3].
>> I’d
>> >> been struggling to re-create the error for the ticket, but I think your
>> >> example does it nicely, so provides a good test for whether the
>> problem is
>> >> fixed - unfortunately, when I run this example data against my branch,
>> it
>> >> still fails albeit with a different error:
>> >>
>> >> java.lang.ClassCastException: class java.lang.String cannot be cast to
>> >> class org.apache.nifi.serialization.record.Record (java.lang.String is
>> in
>> >> module java.base of loader 'bootstrap';
>> >> org.apache.nifi.serialization.record.Record is in unnamed module 

NiFi Documentation on offline system

[Matthew Hawkins]

Hi devs,

I figured this one out from earlier in the month;
/nifi-docs/css/component-usage.css has a hard-coded import from
https://fonts.googleapis.com/ which of course cannot work on an offline
system. This causes 30+ seconds of waiting for DNS to not resolve the
Internet host before the page renders.

There might be other similar issues throughout the code, I haven't grepped
it yet.

Kind regards,
Matt

Re: [DISCUSS] Deprecate TLS Toolkit for Removal?

[Matthew Hawkins]

I've put up the base form here; happy for it to be considered under the
Apache-2.0 License like NiFi

https://gist.github.com/hawko2600/922b727634784614465b83e52ec2be52

It sets up a standard root and intermediate signing CA if that doesn't
already exist, then creates and signs a NiFi server cert. For ease of
scripting it has a default password which it then gets you to change on the
stores at the end for security.

I use env vars so you can create multiple certs in a loop eg if you have a
cluster.

My preference is PKCS12 for identity certs and JKS for the trust store (so
you can work with other web servers under multiple CAs) so that's
effectively enforced. Cipher attributes are secure enough for today in most
cases.

Out of scope for this demo are two important things;
* adequately protecting the rootCA store
* managing a CRL
Implementers should bear that in mind and take the actions they
deem appropriate.

Kr,
Matt


On Tue, 19 Sept 2023, 01:58 Paul Grey,  wrote:

> I spent a bit of time working on a draft of the needed openssl commands to
> generate equivalent keystores / truststores.  Was not able to finish this
> draft before I got side tracked.
>
>
> https://github.com/greyp9/nifi-tidbits/blob/main/cluster/tls-toolkit-via-openssl/cluster-certs.md
>
> If you're still offering, I would be interested to see the equivalent
> keytool commands, to compare and contrast.  Maybe both could be
> documented...
>
> If we can reduce or eliminate the needed maintenance efforts associated
> with TLS Toolkit, this seems worth some one-time effort.
>
>
> On Sun, Sep 17, 2023 at 1:46 AM Matthew Hawkins 
> wrote:
>
> > NiFi in certain configurations requires a specific set of x509_v3
> > extensions, e.g. when using InvokeHTTP. This is automatically added when
> > using TLS Toolkit but all other options relies upon the CA either
> accepting
> > and passing through extensions in the request, or specifically adding
> them
> > in.
> >
> > You cannot use a stock standard web cert signing authority as they
> usually
> > ignore passed extensions and just sign a server-only cert, which will
> > result in a dysfunctional NiFi. ADCS doesn't have a valid default
> template
> > for NiFi and many installations of that don't deviate from default
> > templates.
> >
> > If you don't use certain processors it's certainly easy to just sign a
> > server cert and not know anything is amiss. This shouldn't be the
> default.
> > It's 15 sequential Java keytool commands to build a proper CA chain and
> > sign a NiFi cert. If TLS Toolkit is being pulled, a script to invoke
> > keytool should be included at the same time.
> >
> > I'm happy to provide a basic solution to this if asked.
> >
> > I like keytool as it is available on all platforms and uses the right
> > formats NiFi requires for keystore and truststore. The open source
> Keytool
> > Explorer is also a nice way to build and manage the stores, not strictly
> > necessary but nice to have.
> >
> >
> >
> >
> > On Thu, 14 Sept 2023, 05:58 Kevin Doran,  wrote:
> >
> > >  I agree that tis toolkit can probably be removed in 2.0, and I would
> add
> > > that tinycert.org provides another option for teams that need to setup
> > > dev/test environments with trusted certificates.
> > >
> > > Thanks,
> > > Kevin
> > >
> > > On Sep 13, 2023 at 11:46:19, David Handermann <
> > exceptionfact...@apache.org
> > > >
> > > wrote:
> > >
> > > > Hi Isha,
> > > >
> > > > Thanks for the helpful reply. I agree that the TLS Toolkit is most
> > > > convenient for development and lab deployments, and that's where we
> > > > should be able to provide some documentation for alternatives. The
> > > > existing Secure Cluster Walkthrough is a helpful reference for TLS
> > > > Toolkit usage, so if we updated that to provide similar guidance
> using
> > > > other tools, that would be useful.
> > > >
> > > > Getting everything right with TLS can be challenging, but when it
> > > > comes to project maintenance, it seems better to focus on core
> > > > capabilities and not maintain the TLS Toolkit if the primary use case
> > > > is for non-production deployments.
> > > >
> > > > The encrypt-config command is a different question, but a very good
> > > > question. It includes functionality that is very specific to NiFi,
> and
> > > > it is also in need of refactoring. The threat model for containerized
> > > > deployments may be somewhat

Re: [DISCUSS] Deprecate TLS Toolkit for Removal?

[Matthew Hawkins]

NiFi in certain configurations requires a specific set of x509_v3
extensions, e.g. when using InvokeHTTP. This is automatically added when
using TLS Toolkit but all other options relies upon the CA either accepting
and passing through extensions in the request, or specifically adding them
in.

You cannot use a stock standard web cert signing authority as they usually
ignore passed extensions and just sign a server-only cert, which will
result in a dysfunctional NiFi. ADCS doesn't have a valid default template
for NiFi and many installations of that don't deviate from default
templates.

If you don't use certain processors it's certainly easy to just sign a
server cert and not know anything is amiss. This shouldn't be the default.
It's 15 sequential Java keytool commands to build a proper CA chain and
sign a NiFi cert. If TLS Toolkit is being pulled, a script to invoke
keytool should be included at the same time.

I'm happy to provide a basic solution to this if asked.

I like keytool as it is available on all platforms and uses the right
formats NiFi requires for keystore and truststore. The open source Keytool
Explorer is also a nice way to build and manage the stores, not strictly
necessary but nice to have.




On Thu, 14 Sept 2023, 05:58 Kevin Doran,  wrote:

>  I agree that tis toolkit can probably be removed in 2.0, and I would add
> that tinycert.org provides another option for teams that need to setup
> dev/test environments with trusted certificates.
>
> Thanks,
> Kevin
>
> On Sep 13, 2023 at 11:46:19, David Handermann  >
> wrote:
>
> > Hi Isha,
> >
> > Thanks for the helpful reply. I agree that the TLS Toolkit is most
> > convenient for development and lab deployments, and that's where we
> > should be able to provide some documentation for alternatives. The
> > existing Secure Cluster Walkthrough is a helpful reference for TLS
> > Toolkit usage, so if we updated that to provide similar guidance using
> > other tools, that would be useful.
> >
> > Getting everything right with TLS can be challenging, but when it
> > comes to project maintenance, it seems better to focus on core
> > capabilities and not maintain the TLS Toolkit if the primary use case
> > is for non-production deployments.
> >
> > The encrypt-config command is a different question, but a very good
> > question. It includes functionality that is very specific to NiFi, and
> > it is also in need of refactoring. The threat model for containerized
> > deployments may be somewhat different than running directly on
> > physical hardware. With the need to support various approaches,
> > however, some type of configuration encryption remains a relevant
> > concern.
> >
> > Regards,
> > David Handermann
> >
> > On Wed, Sep 13, 2023 at 10:19 AM Isha Lamboo
> >  wrote:
> >
> >
> > Hi David,
> >
> >
> > My primary use for the TLS toolkit is for lab deployments, mostly during
> > in-house trainings. I will miss the convenience of having a full set of
> > keystores and truststores ready to go with a single command, but then
> > again, a few commands in a script should replicate this well enough,
> > without the need for maintaining the toolkit.
> >
> >
> > I see no obstacles to adopting NiFi 2.0 if the TLS toolkit is phased out,
> > from the perspective of the deployments I manage.
> >
> >
> > On a side note: How relevant is the encrypt config part of the toolkit
> > still in a mostly containerized world?
> >
> >
> > Regards,
> >
> >
> > Isha
> >
> >
> > -Oorspronkelijk bericht-
> >
> > Van: David Handermann 
> >
> > Verzonden: woensdag 13 september 2023 15:16
> >
> > Aan: dev@nifi.apache.org
> >
> > Onderwerp: [DISCUSS] Deprecate TLS Toolkit for Removal?
> >
> >
> > Team,
> >
> >
> > The TLS Toolkit provides a number of useful features for securing NiFi
> > server communication, but it also presents several maintenance concerns.
> In
> > light of other available tools, I am raising the question of removing the
> > TLS Toolkit from the repository as part of NiFi 2.0 technical debt
> > reduction.
> >
> >
> > With the addition of automatic self-signed certificate generation in NiFi
> > 1.14.0, the TLS Toolkit is much less relevant to standalone or
> development
> > deployments. The validity period of the automatic certificate is limited,
> > but it provides a method of getting started without any need for the TLS
> > Toolkit.
> >
> >
> > On the other end of the spectrum, orchestrated deployments of Kubernetes
> > can take advantage of cert-manager [1] for declarative and configurable
> > certificate generation and distribution.
> >
> >
> > Cluster deployments on physical hardware or virtual machines may have
> > organization-specific Certificate Authorities, which require certificate
> > request processing external to NiFi itself. For this scenario,
> documenting
> > several standard OpenSSL commands may help to describe converting between
> > PEM and PKCS12 files for common use cases.
> >
> >
> > Back to standalone deployments, Let's 

Re: JoltTransformJSON EL when using file

[Matthew Hawkins]

Hi Matt,

Had this happen again.
Spec:
[{
"operation": "default",
"spec": {
"PartitionID": "${distribute.load.relationship}"
}}]

I'm abusing DistributeLoad in round robin mode to add a cycling id in a
range to the content by sending all relationships to the next processor.

Using JoltTransformRecord, running this spec as a file reference results in
the literal characters ${distribute.load.relationship} being added to the
content. Running the same spec inline results in the correct expectation of
the attribute value being inserted.

Removing the double quotes around the EL expression results in a Java
exception for the spec being invalid. Doing the same with JoltTransformJSON
results in a Java exception for the type conversion failure when
marshalling the object.

On Thu, 24 Aug 2023, 14:06 Matthew Hawkins,  wrote:

> Hi Matt,
>
> Many of the deployments I get involved with are not in a Hadoop ecosystem,
> so building in resilience becomes part of the challenge, especially when
> data input is out of your control and often has bad data. In these
> situations, one thing I like doing is a test flow using DuplicateFlowFile
> and DistributeLoad configured as a valve so the main queue isn't
> backlogged. You can then use live data to work on massaging bad input.
> Manually creating schemas for record based processors and JoltTransformJSON
> becomes a large part of the job.
>
> In one such experiment I was either trying to work around GenerateRecord
> not having capability for some data types (e.g. Boolean) and faking it with
> a jolt transform, or I was putting parent key/values back into a content
> after splitting on a sub-arrray. Either way, I ended up with a jolt
> transform inserting a timestamp.
> Current time is pretty useful for downstream systems to understand latency
> between an event and it getting to that system, I often ensure Nifi is
> dropping it into content.
>
> I'm not near my test system to get the exact spec file but will endeavour
> to at the next opportunity.
>
>
>
> On Fri, 18 Aug 2023, 07:13 Matt Burgess,  wrote:
>
>> Matthew,
>>
>> What was your second case trying to use ${now():toNumber()} ? My unit
>> test evaluated the expression into an integer and it worked (versus
>> putting the expression in quotes which would make it a string).
>>
>> Regards,
>> Matt
>>
>> On Tue, Aug 15, 2023 at 4:09 AM Matthew Hawkins 
>> wrote:
>> >
>> > Hi Matt,
>> >
>> > The tag will be ${firstname}, the spec is correct.
>> > (well, I also get lastname, but it's unimportant)
>> >
>> > Weird thing is this failed similarly with a file based input on the
>> Record
>> > based jolt processor as well on a completely different system.
>> >
>> > Both Linux (Ubuntu 22.04), both OpenJDK 17, both recompiled nifi from
>> > source with
>> include-graph,include-media,include-rules,include-sql-reporting
>> >
>> > In the second case I was trying to get ${now():toNumber()} into a json
>> > record using either default or modify-overwrite-beta (tried both). The
>> > following success processor failed on reading schema as the literal
>> > ${now...} was a string not a long and the JVM refused the type cast. If
>> I
>> > insert the transform directly then it still fails with a type cast
>> problem
>> > from generic Object :/ (that was midnight last night, and I tossed my
>> hands
>> > in the air and turned the server off)
>> >
>> > I'm pulling mqtt and opc-ua off a raspberry pi and dumping it into
>> > postgresql. Trying to be a good nifi citizen and use record based
>> > processing where possible. Since I don't run Hadoop at home, it's all
>> > manual schemas and using postgres as the perm data store. I'm using
>> json as
>> > mqtt does it intrinsically and I can jolt transform other data into json
>> > form and then store it easily in postgres. Well, that was the idea 
>> >
>> > Kr,
>> >
>> > On Sun, 13 Aug 2023, 10:44 Matt Burgess,  wrote:
>> >
>> > > Just to follow up, I added a unit test to put EL in a JOLT spec and it
>> > > worked. I noticed you referred to "attrname" in your post but your
>> > > spec refers to "firstname", is that a typo?
>> > >
>> > > Regards,
>> > > Matt
>> > >
>> > > On Thu, Aug 10, 2023 at 3:03 PM Matt Burgess 
>> wrote:
>> > > >
>> > > > I added file support to JoltTransformJSON under NIFI-4957 [1], a
>> first
>> > > &g

Re: Refreshing HTML displayed for View usage

[Matthew Hawkins]

Ive got a wks/offline server scenario where reverse DNS doesn't work and
things like the online Usage help and processor state don't update in a
similar way. If I was to guess it'd be something like a recent jetty
upgrade behaving differently; but just a guess. When the 30s timeout
occurs, things render as expected.

It's also a pain trying to drag connectors into a funnel, that processor
type in particular has difficulty turning green.

On Mon, 21 Aug 2023, 23:41 Russell Bateman,  wrote:

> Thanks, Matt. This is happening locally during development where I'm not
> using DNS (just localhost). I find this problem less acute on older
> versions of NiFi (1.1.3.2) than more recent ones (1.23.0, 1.19.1).
>
> Thanks,
> Russ
>
> On 8/20/23 19:03, Matthew Hawkins wrote:
> > Hi Russell,
> >
> > Something I've noticed myself, and it's easily reproducible on the Usage
> > pages for processors, is that Nifi is doing a reverse DNS lookup when
> > loading the page. If your DNS is broken, it can take 10-30 seconds for
> the
> > page content to appear.
> >
> > Kr,
> >
> > On Fri, 18 Aug 2023, 07:29 Russell Bateman,
> wrote:
> >
> >> Matt,
> >>
> >> I experimented with the Open in New Window button, but the new window,
> >> which probably isn't an iframe, nevertheless doesn't respond to attempts
> >> to get around the cache.
> >>
> >> You're probably right about the version not changing being at the root
> >> of the browser's inability to respond to a change. Sadly, the
> >> edit-display-debug-edit cycle doesn't excite me into changing versions
> >> just so I can see the result.
> >>
> >> I'm certainly not going to clamor for work to be done to get around
> >> this. It's annoying, but not crippling.
> >>
> >> Thanks,
> >>
> >> Russ
> >>
> >>
> >> On 8/17/23 13:54, Matt Gilman wrote:
> >>> Russell,
> >>>
> >>> Thanks for the response. The suggestion to open a new tab was for the
> >>> generated documentation, not the NiFi canvas itself. The generated
> >>> documentation is shown in an iframe which will offer you the menu item
> in
> >>> the context menu for opening in a new tab. IIRC correctly, the path to
> >> the
> >>> generated documentation contains the version of the component. For
> folks
> >>> simply using a released version of NiFi this works great since versions
> >>> will differ and any browser caching will only optimize page loading for
> >> the
> >>> user. If you're a developer, however, you're likely making changes to
> >> your
> >>> component and the version is not changing (x.y.z-SNAPSHOT). This is
> where
> >>> the browser cache could result in the behavior your seeing.
> >>>
> >>> Matt
> >>>
> >>> On Thu, Aug 17, 2023 at 3:21 PM Russell Bateman
> >>> wrote:
> >>>
> >>>> Thanks, Matt,
> >>>>
> >>>> (Since I use Chrome 99% of the time, I'm using Chrome here.)
> >>>>
> >>>> Clicking on the reload control next to the address bar, while holding
> >>>> down Shift, reloads the canvas, but the NiFi Documentation page
> >>>> disappears and I have to reload it using the View usage context menu
> of
> >>>> my processor. Once reloaded, nothing has changed.
> >>>>
> >>>> This is also the behavior of the Configure Processor dialog. As I
> reload
> >>>> View usage, ...
> >>>>
> >>>> In Chrome, there's no effective content-menu option to right-click in
> >>>> the page contents and open in a new tab. I can...
> >>>>
> >>>>   Back
> >>>>   Forward
> >>>>   Reload
> >>>>   ---
> >>>>   Save As...
> >>>>   Print
> >>>>   Cast
> >>>>   Search images with Google
> >>>>   -
> >>>>   Send to your devices
> >>>>   Create QR code for this page
> >>>>   
> >>>>   Translate
> >>>>   -
> >>>>   View page source
> >>>>   View frame source
> >>>>   Reload frame
> >>>>   Inspect
> >>>>
> >>>>
> >>>> If I right-click the current tab itself, I

Re: JoltTransformJSON EL when using file

[Matthew Hawkins]

Hi Matt,

Many of the deployments I get involved with are not in a Hadoop ecosystem,
so building in resilience becomes part of the challenge, especially when
data input is out of your control and often has bad data. In these
situations, one thing I like doing is a test flow using DuplicateFlowFile
and DistributeLoad configured as a valve so the main queue isn't
backlogged. You can then use live data to work on massaging bad input.
Manually creating schemas for record based processors and JoltTransformJSON
becomes a large part of the job.

In one such experiment I was either trying to work around GenerateRecord
not having capability for some data types (e.g. Boolean) and faking it with
a jolt transform, or I was putting parent key/values back into a content
after splitting on a sub-arrray. Either way, I ended up with a jolt
transform inserting a timestamp.
Current time is pretty useful for downstream systems to understand latency
between an event and it getting to that system, I often ensure Nifi is
dropping it into content.

I'm not near my test system to get the exact spec file but will endeavour
to at the next opportunity.



On Fri, 18 Aug 2023, 07:13 Matt Burgess,  wrote:

> Matthew,
>
> What was your second case trying to use ${now():toNumber()} ? My unit
> test evaluated the expression into an integer and it worked (versus
> putting the expression in quotes which would make it a string).
>
> Regards,
> Matt
>
> On Tue, Aug 15, 2023 at 4:09 AM Matthew Hawkins 
> wrote:
> >
> > Hi Matt,
> >
> > The tag will be ${firstname}, the spec is correct.
> > (well, I also get lastname, but it's unimportant)
> >
> > Weird thing is this failed similarly with a file based input on the
> Record
> > based jolt processor as well on a completely different system.
> >
> > Both Linux (Ubuntu 22.04), both OpenJDK 17, both recompiled nifi from
> > source with
> include-graph,include-media,include-rules,include-sql-reporting
> >
> > In the second case I was trying to get ${now():toNumber()} into a json
> > record using either default or modify-overwrite-beta (tried both). The
> > following success processor failed on reading schema as the literal
> > ${now...} was a string not a long and the JVM refused the type cast. If I
> > insert the transform directly then it still fails with a type cast
> problem
> > from generic Object :/ (that was midnight last night, and I tossed my
> hands
> > in the air and turned the server off)
> >
> > I'm pulling mqtt and opc-ua off a raspberry pi and dumping it into
> > postgresql. Trying to be a good nifi citizen and use record based
> > processing where possible. Since I don't run Hadoop at home, it's all
> > manual schemas and using postgres as the perm data store. I'm using json
> as
> > mqtt does it intrinsically and I can jolt transform other data into json
> > form and then store it easily in postgres. Well, that was the idea 
> >
> > Kr,
> >
> > On Sun, 13 Aug 2023, 10:44 Matt Burgess,  wrote:
> >
> > > Just to follow up, I added a unit test to put EL in a JOLT spec and it
> > > worked. I noticed you referred to "attrname" in your post but your
> > > spec refers to "firstname", is that a typo?
> > >
> > > Regards,
> > > Matt
> > >
> > > On Thu, Aug 10, 2023 at 3:03 PM Matt Burgess 
> wrote:
> > > >
> > > > I added file support to JoltTransformJSON under NIFI-4957 [1], a
> first
> > > > glance at the code seems like it should work, but I'll try to
> > > > reproduce it and follow up, thanks for bringing this to our
> attention!
> > > >
> > > > Regards,
> > > > Matt
> > > >
> > > > [1] https://issues.apache.org/jira/browse/NIFI-4957
> > > >
> > > > On Thu, Aug 10, 2023 at 6:53 AM Matthew Hawkins  >
> > > wrote:
> > > > >
> > > > > Hi devs,
> > > > >
> > > > > Using 1.23.0 I have a simple transform adding some flowfile
> attributes
> > > into
> > > > > the content.
> > > > > When putting the spec directly to the processor it works fine. If I
> > > have
> > > > > the spec in an external file however it seems to put in the literal
> > > string
> > > > > ${attrname} - ie the EL didn't process.
> > > > >
> > > > > Have I done something incorrectly or should I be now asking for an
> > > account
> > > > > to log a bug?
> > > > >
> > > > > Sample spec run using Chain DSL:
> > > > >
> > > > > [{
> > > > >   "operation": "shift",
> > > > >   "spec": {
> > > > > "@": "values"
> > > > >   },
> > > > >   {
> > > > > "operation": "default",
> > > > > "spec": {
> > > > >   "firstname": "${firstname}"
> > > > > }
> > > > >   }
> > > > > ]
> > > > >
> > > > > Kind regards,
> > > > > Matthew
> > >
>

Re: Refreshing HTML displayed for View usage

[Matthew Hawkins]

Hi Russell,

Something I've noticed myself, and it's easily reproducible on the Usage
pages for processors, is that Nifi is doing a reverse DNS lookup when
loading the page. If your DNS is broken, it can take 10-30 seconds for the
page content to appear.

Kr,

On Fri, 18 Aug 2023, 07:29 Russell Bateman,  wrote:

> Matt,
>
> I experimented with the Open in New Window button, but the new window,
> which probably isn't an iframe, nevertheless doesn't respond to attempts
> to get around the cache.
>
> You're probably right about the version not changing being at the root
> of the browser's inability to respond to a change. Sadly, the
> edit-display-debug-edit cycle doesn't excite me into changing versions
> just so I can see the result.
>
> I'm certainly not going to clamor for work to be done to get around
> this. It's annoying, but not crippling.
>
> Thanks,
>
> Russ
>
>
> On 8/17/23 13:54, Matt Gilman wrote:
> > Russell,
> >
> > Thanks for the response. The suggestion to open a new tab was for the
> > generated documentation, not the NiFi canvas itself. The generated
> > documentation is shown in an iframe which will offer you the menu item in
> > the context menu for opening in a new tab. IIRC correctly, the path to
> the
> > generated documentation contains the version of the component. For folks
> > simply using a released version of NiFi this works great since versions
> > will differ and any browser caching will only optimize page loading for
> the
> > user. If you're a developer, however, you're likely making changes to
> your
> > component and the version is not changing (x.y.z-SNAPSHOT). This is where
> > the browser cache could result in the behavior your seeing.
> >
> > Matt
> >
> > On Thu, Aug 17, 2023 at 3:21 PM Russell Bateman
> > wrote:
> >
> >> Thanks, Matt,
> >>
> >> (Since I use Chrome 99% of the time, I'm using Chrome here.)
> >>
> >> Clicking on the reload control next to the address bar, while holding
> >> down Shift, reloads the canvas, but the NiFi Documentation page
> >> disappears and I have to reload it using the View usage context menu of
> >> my processor. Once reloaded, nothing has changed.
> >>
> >> This is also the behavior of the Configure Processor dialog. As I reload
> >> View usage, ...
> >>
> >> In Chrome, there's no effective content-menu option to right-click in
> >> the page contents and open in a new tab. I can...
> >>
> >>  Back
> >>  Forward
> >>  Reload
> >>  ---
> >>  Save As...
> >>  Print
> >>  Cast
> >>  Search images with Google
> >>  -
> >>  Send to your devices
> >>  Create QR code for this page
> >>  
> >>  Translate
> >>  -
> >>  View page source
> >>  View frame source
> >>  Reload frame
> >>  Inspect
> >>
> >>
> >> If I right-click the current tab itself, I'm able to choose to
> >> Duplicate., but that doesn't seem to lead to anything more helpful.
> >>
> >> By the way, I'm doing this using NiFi 1.23.0. I have also verified the
> >> date of my NAR to ensure that I'm not repeatedly working using a NAR
> >> with old content. Also, the version displayed for the custom processor
> >> is that of the various /pom.xml/ files I'm building with.
> >>
> >> I usually develop on the latest NiFi version. However, my company has
> >> NARs that won't load beginning 1.14.0. I just brought up 1.13.2, what
> >> most of our customers run, and tried my latest NAR which *does* display
> >> my changes.Same with 1.1.2, which represents the oldest platform
> >> executing at any custom site. Exceptionally, I set up 1.19.1 where it
> >> *also works*.
> >>
> >> This is unexpected; maybe it does point, as unlikely as it seems, to
> >> something changed in 1.23.0 (instead of being caused by anything between
> >> the dossier of my chair and the space bar of my keyboard as I have been
> >> expecting to learn. ;-)  )
> >>
> >> It could still be me and I'm just not seeing the obvious yet.
> >>
> >>
> >> On 8/17/23 12:44, Matt Gilman wrote:
> >>> Russell,
> >>>
> >>> Assuming this is a browser cache issue, can you try right-clicking
> >> anywhere
> >>> in the page contents of the generated documentation and open it in a
> new
> >>> tab. Once that is open, can you try doing a hard refresh by holding
> Shift
> >>> while clicking Reload next to the address bar? This should clear the
> >> cache
> >>> and fetch the updated generated documentation. At this point, you
> should
> >> be
> >>> able to close that tab and retry from the NiFi UI.
> >>>
> >>> Let us know if this doesn't help and we can see if something isn't
> >> getting
> >>> generated and updated correctly.
> >>>
> >>> Matt
> >>>
> >>> On Thu, Aug 17, 2023 at 2:38 PM Russell Bateman
> >>> wrote:
> >>>
>  Seems like a really stupid user/browser question, but I cannot seem to
>  get changes I've made to properties, relationships, attributes read or
>  written, etc. *for custom 

Re: NiFi 1.19.1 TLS Cipher syntax

[Matthew Hawkins]

Just be aware that simple regex would also include undesirable outcomes
like null hashes and weak bit strengths, effecting a downgrade attack. You
should also explicitly exclude them using something like

 nifi.web.https.ciphersuites.exclude=.*128.*,.*NULL.*

Or whatever suits your needs. It should be and possibly is a default
setting but always best to enforce and note in documentation to stay on top
of.

I acknowledge it's a pest to set these manually, but that's what tooling
like ansible is for, you can do it once and apply to all your Nifi servers
easily. Also, even if you did it manually in windows notepad even, the hour
spent would cost something like $35-100 whereas the cyber incident could
cost hundreds of millions. Never shy putting your hand to the till with
mindless administrative tasks that are actually achieving something.

Kr,


On Fri, 18 Aug 2023, 00:23 Phillip Lord,  wrote:

> I think you just need to adjust your regex here...
>
> Have you tried something like this...   ^TLS_ECDHE.*
>
> Thanks,
> Phil
>
> On Thu, Aug 17, 2023 at 8:26 AM Martin Fong 
> wrote:
>
> > I would like to find out the syntax to set only ECDHE*.
> >
> > The following works:
> > nifi.web.https.ciphersuites.include=^.*GCM_SHA256$
> >
> > The following does not work:
> > nifi.web.https.ciphersuites.include=^.*TLS_ECDHE$
> >
> > This will work but I want the whole ECDHE* and nothing else but it will
> be
> > a very long line to set them up.
> > nifi.web.https.ciphersuites.include=
> >
> TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
> >
> > 2023-08-16 13:54:52,811 INFO [main] o.e.jetty.util.ssl.SslContextFactory
> > No Cipher Suite matching '^.*TLS_ECDHE$' is supported
> > 2023-08-16 13:54:52,812 WARN [main] o.e.jetty.util.ssl.SslContextFactory
> > No supported Cipher Suite from [TLS_AES_256_GCM_SHA384,
> > TLS_AES_128_GCM_SHA256, TLS_CHACHA20_POLY1305_SHA256,
> > TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
> > TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
> > TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,
> > TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
> > TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,
> > TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
> TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,
> > TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256,
> > TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,
> > TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,
> > TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,
> > TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,
> > TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,
> TLS_DHE_RSA_WITH_AES_256_CBC_SHA256,
> > TLS_DHE_RSA_WITH_AES_128_CBC_SHA256,
> TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,
> > TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,
> > TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA,
> > TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384,
> > TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA256,
> > TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA,
> > TLS_RSA_WITH_AES_128_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV,
> > TLS_RSA_WITH_NULL_SHA256, TLS_ECDHE_ECDSA_WITH_NULL_SHA,
> > TLS_ECDHE_RSA_WITH_NULL_SHA, SSL_RSA_WITH_NULL_SHA]
> >
> > Please advise if there is a correct syntax just only for ECDHE*.
> >
> > Thanks,
> > Martin Fong
> > Enterprise Technical Support Specialist, Infrastructure & Platform (IAG)
> > Technology Services Division, Technology Infrastructure Services
> > City of Toronto
> > 703 Don Mills Road, 2nd Floor
> > Toronto, ON
> > M3C 3N3
> > Tel:   416-397-7565
> > e-mail: martin.f...@toronto.ca
> >
> > This e-mail message is confidential and subject to copyright. Any
> > unauthorized use or disclosure is prohibited. If you have received this
> > email and are not the intended recipient, please advise and delete it.
> > Thank you.
> >
> >
>

Re: JoltTransformJSON EL when using file

[Matthew Hawkins]

Hi Matt,

The tag will be ${firstname}, the spec is correct.
(well, I also get lastname, but it's unimportant)

Weird thing is this failed similarly with a file based input on the Record
based jolt processor as well on a completely different system.

Both Linux (Ubuntu 22.04), both OpenJDK 17, both recompiled nifi from
source with include-graph,include-media,include-rules,include-sql-reporting

In the second case I was trying to get ${now():toNumber()} into a json
record using either default or modify-overwrite-beta (tried both). The
following success processor failed on reading schema as the literal
${now...} was a string not a long and the JVM refused the type cast. If I
insert the transform directly then it still fails with a type cast problem
from generic Object :/ (that was midnight last night, and I tossed my hands
in the air and turned the server off)

I'm pulling mqtt and opc-ua off a raspberry pi and dumping it into
postgresql. Trying to be a good nifi citizen and use record based
processing where possible. Since I don't run Hadoop at home, it's all
manual schemas and using postgres as the perm data store. I'm using json as
mqtt does it intrinsically and I can jolt transform other data into json
form and then store it easily in postgres. Well, that was the idea 

Kr,

On Sun, 13 Aug 2023, 10:44 Matt Burgess,  wrote:

> Just to follow up, I added a unit test to put EL in a JOLT spec and it
> worked. I noticed you referred to "attrname" in your post but your
> spec refers to "firstname", is that a typo?
>
> Regards,
> Matt
>
> On Thu, Aug 10, 2023 at 3:03 PM Matt Burgess  wrote:
> >
> > I added file support to JoltTransformJSON under NIFI-4957 [1], a first
> > glance at the code seems like it should work, but I'll try to
> > reproduce it and follow up, thanks for bringing this to our attention!
> >
> > Regards,
> > Matt
> >
> > [1] https://issues.apache.org/jira/browse/NIFI-4957
> >
> > On Thu, Aug 10, 2023 at 6:53 AM Matthew Hawkins 
> wrote:
> > >
> > > Hi devs,
> > >
> > > Using 1.23.0 I have a simple transform adding some flowfile attributes
> into
> > > the content.
> > > When putting the spec directly to the processor it works fine. If I
> have
> > > the spec in an external file however it seems to put in the literal
> string
> > > ${attrname} - ie the EL didn't process.
> > >
> > > Have I done something incorrectly or should I be now asking for an
> account
> > > to log a bug?
> > >
> > > Sample spec run using Chain DSL:
> > >
> > > [{
> > >   "operation": "shift",
> > >   "spec": {
> > > "@": "values"
> > >   },
> > >   {
> > > "operation": "default",
> > > "spec": {
> > >   "firstname": "${firstname}"
> > > }
> > >   }
> > > ]
> > >
> > > Kind regards,
> > > Matthew
>

JoltTransformJSON EL when using file

[Matthew Hawkins]

Hi devs,

Using 1.23.0 I have a simple transform adding some flowfile attributes into
the content.
When putting the spec directly to the processor it works fine. If I have
the spec in an external file however it seems to put in the literal string
${attrname} - ie the EL didn't process.

Have I done something incorrectly or should I be now asking for an account
to log a bug?

Sample spec run using Chain DSL:

[{
  "operation": "shift",
  "spec": {
"@": "values"
  },
  {
"operation": "default",
"spec": {
  "firstname": "${firstname}"
}
  }
]

Kind regards,
Matthew

Re: Re: Does Apache NiFi 1.19.1 works on RHEL 9.1?

[Matthew Hawkins]

Nifi is a Java based application and so the underlying OS is irrelevant as
long as you're;
a) using a supported JVM version (8-17 LTS should be fine, I believe
release is built using 11)
b) reading the release notes of your JVM on your OS to comprehend any local
issues, which you can take up with Redhat and/or your JVM provider if
you're using a 3rd party JVM

On Fri, May 26, 2023, 21:18 Martin Fong  wrote:

> Jim
> We are upgrading our OS from RH7 to RH9 so wanted to make sure it is
> compatible as I couldn't find the certified/supported OS page for NiFi.
> Thanks!
>
> On 2023/05/26 11:04:36 Jim Halfpenny wrote:
> > Hi Martin,
> > Apache NiFi is Java-based so there's no reason why it shouldn't work on
> RHEL 9.1. Did you have any specific concerns or was your question around
> which OS versions are supported?
>
> >
> > Kind regards,
> > Jim
> >
> > > On 26 May 2023, at 11:43, Martin Fong  wrote:
> > >
> > > Does Apache NiFi 1.19.1 works on RHEL 9.1?
> > >
> > > Please advise,
> > > Martin Fong
> > > Enterprise Technical Support Specialist, Infrastructure & Platform
> (IAG)
> > > Technology Services Division, Technology Infrastructure Services
> > > City of Toronto
> > > 703 Don Mills Road, 2nd Floor
> > > Toronto, ON
> > > M3C 3N3
> > > Tel:   416-397-7565
> > > e-mail: martin.f...@toronto.ca
> > >
> > > This e-mail message is confidential and subject to copyright. Any
> unauthorized use or disclosure is prohibited. If you have received this
> email and are not the intended recipient, please advise and delete it.
> Thank you.
>
> > >
> >
> >
>

Re: PostHTTP Deprecation Concerns

[Matthew Hawkins]

Hi Adam,

PostHTTP was marked deprecated 3 years ago (aka six technology lifetimes).
The successive technologies to replace it's functionality are well
documented and proven in production. The technical reason to remove it is
that it is superfluous code that has a cost to maintain and zero benefit.
Backwards compatibility is never guaranteed for components marked
deprecated for such a long length of time in any software product let alone
nifi specifically.

Your organisation is free to continue using the version of nifi it is on
today and not take any further action. It is unhelpful to suggest every
other organisation should be held back in progress because yours refuses to
take the necessary flow maintenance action. One of the impetus for a major
version upgrade is to specifically jettison deprecated components. It's
quite remarkable you're advocating against standard practice presumably for
your own convenience.

Site to site connectivity is conducted with either raw sockets or http
(which is https on secured nifi) so I'm highly skeptical there is any
performance deprecation in InvokeHTTP or S2S over PostHTTP, given the
former can take advantage of http/2 and the latter not. It's easy to
monitor nifi and prove through metrics in any case. Sadly in enterprise
environments this is sometimes necessary to defeat the political layer
around change management.

You can run records-based processing over either current method and it is
ridiculously fast. The bottleneck in my last engagement ended up being
network hardware limitations, not nifi. Having contributed in this domain,
I also recommend tossing CompressContent into the flow to minimise
bandwidth. On modern hardware the decompression is minimal time and you can
plug a *lot* more data through in less CPU and wall clock time. It's easy
to bench with DuplicateFlowfile on your production flow and metrics
analysis, just make sure your provenance db has sufficient space.

Kind regards,

On Thu, Jan 12, 2023, 10:09 Adam Taft  wrote:

> Just wanted to note a concern on the deprecation (and presumed removal) of
> the PostHTTP processor in the upcoming 2.0 release.
>
> While yes, for traditional client interactions with an external HTTP
> service, utilizing InvokeHTTP for your POST operation is probably sensible.
> The concern is that there are a number of NiFi-to-NiFi transfers that
> leverage the "special sauce" that exists between PostHTTP and ListenHTTP.
>
> What special sauce? Namely, the extra negotiation that enables an automated
> serialization of NiFi flowfiles from one system to another. InvokeHTTP is
> just a "raw" HTTP client and doesn't share any special concern or support
> for NiFi-to-NiFi data transfer.
>
> Of course, if you remember the history, before there was any site-to-site
> functionality built into processor groups, the primary means of flowfile
> transport between NiFi systems was the PostHTTP / ListenHTTP combo. It was
> an easy way to facilitate transfer between two nifi systems.
>
> And from what I can tell, this "legacy" approach to NiFi data transfer is
> still being used heavily in certain operational contexts. Why? Because
> often it's the case that the _only_ traffic allowed between network
> boundaries is done via HTTPS. The site-to-site protocol provides its own
> ports and protocol operations that don't necessarily comply with such a
> network policy. And I believe there's still some lingering and/or
> demonstrated concern over the performance characteristics of the
> site-to-site protocol by dataflow managers. They have often reverted to
> using PostHTTP / ListenHTTP instead.
>
> While many of the other deprecated components seem logical, getting rid of
> this one just seems like change-for-the-sake-of-change.
>
> Is there any actual technical reason to deprecate and remove PostHTTP from
> the standard nar? Is it causing a burden to the product itself? Or was the
> decision just more like, "hey it's dumb not to use InvokeHTTP for all HTTP
> client operations" and maybe not realize the alternative use case that
> PostHTTP enables?
>
> Thanks for any feedback.
>
> /Adam
>

Compression as Controller Service

[Matthew Hawkins]

Hi team,

Whilst giving a first go at adding brotli and zstd to CompressContent I
noticed a few things that I think we should address.

1. compression levels are hard-coded as 0-9 in the UI, this prevents
accessing every level possible under certain compressors, e.g. zstd has 22
levels, brotli has 15. I hacked it, it's ugly, we should do the "right"
thing instead.
  - I think this should be addressed by having a compressor register itself
along with metadata like how many levels it supports to a factory & the UI
utilise that property in a more dynamic fashion
  - we could also use metadata to let a compressor flag known high cpu /
high mem situations so the flow Admin can avoid silly mistakes in config or
the JVM can be hinted at.

2. Some other processors such as PublishKafka, InvokeHTTP,
JsonRecordSetWriter, and others implement some compression algorithms
directly.
  - This made me realise it would IMO be better offered as a Service and
each of these use cases can have the simple API of using the controller
service rather than duplicating compression algorithm specific code each
time the functionality is required.

Am I on the right track? Java isn't my forte and I'm new to contributing to
NiFi so keen to hear what experienced devs think.

-- 
Kind regards,
Matthew Hawkins