Re: [DISCUSS] NiFi 2.0 Release Goals

[Jon Logan]

I would be extremely careful about moving to Java 11 -- especially if NiFi
1.x is not actively maintained. I am sure it's not news to anyone, but a
lot (most?) of people are still on Java 8, for better or worse, and I do
not think moving without a strong, compelling reason is advisable. It's not
as simple to tell users to just update Java -- there is a lot involved with
that.

This is from last year, but probably still proves the point:
https://www.jetbrains.com/lp/devecosystem-2020/java/

On Fri, Jul 23, 2021 at 10:32 AM Joe Witt  wrote:

> David,
>
> I think this is a highly reasonable approach and such a focus will
> greatly help make a 2.0 release far more approachable to knock out.
> Not only that but tech debt reduction would help make work towards
> major features we'd think about in a 'major release' sense more
> approachable.
>
> We should remove all deprecated things (as well as verify we have the
> right list).  We should remove/consider removal of deprecated concepts
> like templates.  We should consider whether we can resolve the various
> ways we've handled what are now parameters down to one clean approach.
> We should remove options in the nifi.properties which turn out to
> never be used quite right (if there are).  There is quite a bit we can
> do purely in the name of tech debt reduction.
>
> Lots to consider here but I think this is the right discussion.
>
> Than ks
>
> On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende  wrote:
> >
> > I'm a +1 for this... Not sure if this falls under "Removing Deprecated
> > Components", but I think we should also look at anything that has been
> > marked as deprecated throughout the code base as a candidate for
> > removal. There are quite a few classes, methods, properties, etc that
> > have been waiting for a chance to be removed.
> >
> > On Fri, Jul 23, 2021 at 10:13 AM David Handermann
> >  wrote:
> > >
> > > Team,
> > >
> > > With all of the excellent work that many have contributed to NiFi over
> the
> > > years, the code base has also accumulated some amount of technical
> debt. A
> > > handful of components have been marked as deprecated, and some
> components
> > > remain in the code base to support integration with old versions of
> various
> > > products. Following the principles of semantic versioning, introducing
> a
> > > major release would provide the opportunity to remove these deprecated
> and
> > > unsupported components.
> > >
> > > Rather than focusing the next major release on new features, what do
> you
> > > think about focusing on technical debt removal? This approach would not
> > > make for the most interesting release, but it provides the opportunity
> to
> > > clean up elements that involve breaking changes.
> > >
> > > Focusing on technical debt, at least three primary goals come to mind
> for
> > > the next major release:
> > >
> > > 1. Removal of deprecated and unmaintained components
> > > 2. Require Java 11 as the minimum supported version
> > > 3. Transition internal date and time handling to JSR 310 java.time
> > > components
> > >
> > > *Removing Deprecated Components*
> > >
> > > Removing support for older and deprecated components provides a great
> > > opportunity to improve the overall security posture when it comes to
> > > maintaining dependencies. The OWASP dependency plugin report currently
> > > generates 50 MB of HTML for questionable dependencies, many of which
> are
> > > related to old versions of various libraries.
> > >
> > > As a starting point, here are a handful of components and extension
> modules
> > > that could be targeted for removal in a major version:
> > >
> > > - PostHTTP and GetHTTP
> > > - ListenLumberjack and the entire nifi-lumberjack-bundle
> > > - ListenBeats and the entire nifi-beats-bundle
> > > - Elasticsearch 5 components
> > > - Hive 1 and 2 components
> > >
> > > *Requiring Java 11*
> > >
> > > Java 8 is now over seven years old, and NiFi has supported general
> > > compatibility with Java 11 for several years. NiFi 1.14.0 incorporated
> > > internal improvements specifically related to TLS 1.3, which allowed
> > > closing out the long-running Java 11 compatibility epic NIFI-5174.
> Making
> > > Java 11 the minimum required version provides the opportunity to
> address
> > > any lingering edge cases and put NiFi in a better position to support
> > > current Java versions.
> > >
> > > *JSR 310 for Date and Time Handling*
> > >
> > > Without making the scope too broad, transitioning internal date and
> time
> > > handling to use DateTimeFormatter instead of SimpleDateFormat would
> provide
> > > a number of advantages. The Java Time components provide much better
> > > clarity when it comes to handling localized date and time
> representations,
> > > and also avoid the inherent confusion of java.sql.Date extending
> > > java.util.Date. Many internal components, specifically Record-oriented
> > > processors and services, rely on date parsing, leading to confusion and
> > > various work

Re: [DISCUSS] rename master branch, look through code for other related issues

[Jon Logan]

I think this blog post from the founder of Redis sums things up:
http://antirez.com/news/122

>From the post:

For example if I’m terminally ill, the “short living request”
terminology may be offensive, it reminds me that I’m going to die, or
that my father is going to die. Instead of banning every word out
there, we should make the mental effort to do better than the
political correctness movement that stops at the surface.


On Thu, Jun 18, 2020 at 11:05 AM Edward Armes 
wrote:

> I agree with this, and maybe that is the potential the step forward here
> is: issue a statement is issued saying something like this is a complex
> issue and instead of making changes that could cause further division
> within the community we are looking for those that are interested to help
> form a constructive working group that will help influence and resolve all
> of these issues in a positive way for all not only for project but also
> within the wider group of apache projects.
>
> Edward
>
>
>
> On Thu, 18 Jun 2020, 11:17 u...@moosheimer.com,  wrote:
>
> > Language is always changing and the meaning of words is changing,
> > sometimes positively and sometimes negatively.
> > I think that now is time for change again and we should discuss the use
> > of phrases and meanings.
> >
> > Of course we should change "Master Branch" to "Main Branch".
> > But I also think that we shouldn't just make quick changes because it's
> > opportune and hastily change a few words.
> >
> > An example: We could change Master/Slave to Leader/Follower. This may be
> > a perfect choice for most people in the world.
> > In German Leader is the English word for "Führer". And it is precisely
> > this word that we in Germany do not actually want to use for it.
> >
> > What I mean is that every country and every group (e.g. religion etc.)
> > has its own history and certain words or phrases are just not a perfect
> > choice.
> > We should try to go the ethically correct way worldwide.
> >
> > This concerns the adaptation of current words and phrases with a view to
> > all: in English, Indian, Chinese, German etc. but also for indigenous
> > peoples, different religions etc.
> > And cultural differences should also be taken into account.
> >
> > What I would wish for:
> > Apache.org should set up an "Ethics Board". A group of people of
> > different genders, all colors, religions and from different countries
> > and cultures all over our world.
> > This Ethics Board should find good and for no one discriminating words
> > or phrases for all the areas that stand out today as offensive.
> >
> > And it would be nice if not only computer scientists participated, but
> > also ethicists, philosophers, engineers, various religious people,
> > chemists, biologists, physicists, sociologists, etc.
> >
> > And this Council should set binding targets for all projects.
> >
> > Am 18.06.2020 um 09:36 schrieb Pierre Villard:
> > >> In my perspective this should be an issue for the entire community.
> > Being
> > >> able to identify an issue that directly affects another person but not
> > >> one’s self is the definition of privilege. If I can look at how the
> use
> > of
> > >> these words in someone’s daily life or career impacts them negatively,
> > > when
> > >> the change would not harm me at all, I see that as a failure on my
> > part. I
> > >> understand the desire to hear from the silent majority, but active
> > >> participation and discussion on the mailing list is the exact measure
> > >> described by the Apache process for participation in the community.
> > Those
> > >> who speak here are the ones who will have a voice.
> > > I could not agree more with the above.
> > >
> > > Le jeu. 18 juin 2020 à 04:29, Tony Kurc  a écrit :
> > >
> > >> I suppose I was a bit remiss in not unwinding and/or summarizing some
> of
> > >> what was in that yetus thread to prime the discussion, but a some of
> > what
> > >> Andy is mentioning is expanded on a bit in this ietf document [1],
> > which is
> > >> linked in one of the articles.
> > >>
> > >> 1. https://tools.ietf.org/id/draft-knodel-terminology-00.html
> > >>
> > >>
> > >> On Wed, Jun 17, 2020, 10:02 PM Andy LoPresto 
> > wrote:
> > >>
> > >>> Hi Edward, thanks for sharing your thoughts. I’ll reply inline.
> > >>>
> >  - Some of the terms proposed are not industry standard and may
> > >>> potentially
> >  cause significant issue for non-english speakers.
> > >>> I actually believe making these changes will _improve_ the clarity
> for
> > >>> non-english speakers. “Whitelist” and “blacklist” confer no inherent
> > >> reason
> > >>> to mean allow and deny other than connotative biases. “Allow” and
> > “deny”
> > >>> explicitly indicate the verb that is happening. Another example is
> > branch
> > >>> naming. “Masters” don’t have “branches”. “Trunks” do. These terms
> make
> > >>> _more_ sense for a non-English speaker than the current terms.
> > >>>
> >  - For each change that is made can we guarantee t

Re: kafka-clients version

[Jon Logan]

Is there a reason why this question is being posed? If the version used is
compatible with the latest server version, and there's no missing
functionality or bugs being reported, I would strongly discourage updating,
as you're likely mandating server upgrades for at least some users. Even if
you make new processors, just creating new processors for no benefit is not
beneficial.

On Fri, Jun 12, 2020 at 9:50 AM Joe Witt  wrote:

> Chris
>
> In my experience with the clients the answer is a definite no.  An older
> client will generally work (with important caveats at times) with a newer
> broker.  A newer client will generally not work with an older broker.  At
> the very least both major and minor version of the client matters so users
> can appropriately map to their broker.
>
> We just need to put out a 2.5 one now
>
> thanks
>
> On Fri, Jun 12, 2020 at 6:46 AM  wrote:
>
> > Hi
> >
> > Please see the discussion below.
> > Would it make sense to always pack the latest minor version of
> > kafka-clients-jar in nifi-kafka-bundle?
> >
> > The processors could be named “2.x” (or simply “2” really) and the
> > specific minor version can change in NiFi releases.
> >
> > Regards
> > Chris
> >
> > From: Bryan Bende 
> > Sent: Friday, 12. June 2020 15:33
> > To: us...@nifi.apache.org
> > Subject: Re: kafka-clients version
> >
> > Gotcha. Yes, we can obviously change the version of the client lib, but
> > it's misleading because they were named "2_0" because they use the "2_0"
> > client, and now that won't be true anymore, but if others agree with this
> > then maybe it's fine.
> >
> > Maybe more of a question for the dev list.
> >
> > On Fri, Jun 12, 2020 at 9:28 AM  > christophe.mon...@post.ch>> wrote:
> > Yes: https://issues.apache.org/jira/browse/KAFKA-9815
> >
> > > we'd have to be sure the latest 2.x client also works well with all
> > previous 2.x versions back to 2.0
> >
> > I think kafka-clients should make sure of that.
> >
> > The processors could still be named 2_0, it’s only the client jar that
> > should be the latest 2.x version.
> > Specifically
> >
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-kafka-bundle/pom.xml#L31
> >
> > Regards
> > Chris
> >
> > From: Bryan Bende mailto:bbe...@gmail.com>>
> > Sent: Friday, 12. June 2020 15:18
> > To: us...@nifi.apache.org
> > Subject: Re: kafka-clients version
> >
> > Usually we make a new version of the processors when something changes
> > that causes a previous version of the client to not work well with the
> > newer broker (i.e. 0.9 client with 0.10 broker, in the older versions).
> >
> > The latest processors are named with "2_0" so not sure if it makes sense
> > to upgrade the client lib and then the name is no longer accurate, plus
> > we'd have to be sure the latest 2.x client also works well with all
> > previous 2.x versions back to 2.0.
> >
> > The other option is to make a new copy of the processor like "2_5" or
> > whatever the latest version is, but I think we only want to do this if
> it's
> > really necessary. AFAIK the 2.0 client works with the latest 2.x brokers.
> >
> > Is there any specific issue you are looking to solve by using the latest
> > client?
> >
> >
> > On Fri, Jun 12, 2020 at 8:20 AM  > christophe.mon...@post.ch>> wrote:
> > Hi
> >
> > Any reason why nifi-kafka-bundle is still bundled with
> > kafka-clients-2.0.0.jar and not a newer version?
> >
> > Regards
> > Chris
> >
>

Re: Running Nifi on OpenShift

[Jon Logan]

I think this describes what you would need to do.

https://cookbook.openshift.org/users-and-role-based-access-control/how-can-i-enable-an-image-to-run-as-a-set-user-id.html


On Thu, Feb 13, 2020 at 11:38 AM Jon Logan  wrote:

> That's a OpenShift security feature so that your user IDs are more unique,
> and have less access between containers. I would suggest trying to alter
> your range of user IDs on your cluster if you don't want to modify the
> image.
>
> On Thu, Feb 13, 2020 at 11:09 AM Fill, Natalia 
> wrote:
>
>> Public
>>
>> Hi Shawn,
>> First I tried modifying securityContect first and the familiar error is
>> appeared. I remember trying to run as user 1000 a few days ago and had
>> error similar to below. OpenShift has restrictions on this value.
>>
>> Error creating: pods "nifi-4-" is forbidden: unable to validate
>> against any security context constraint: [fsGroup: Invalid value:
>> []int64{1000}: 1000 is not an allowed group
>> spec.containers[0].securityContext.securityContext.runAsUser: Invalid
>> value: 1000: must be in the ranges: [100047, 100047]]
>>
>> So if Nifi has to run as user 1000 and OpenShift only allows range
>> [100047, 100047] then the issue is not resolvable in the current
>> image.
>> Let me know if you have other views on it.
>>
>> Thanks
>>
>> Natalia Fill
>> Analyst Software Developer
>>
>> -Original Message-
>> From: Fill, Natalia [mailto:natalia.f...@lgim.com]
>> Sent: 13 February 2020 14:32
>> To: dev@nifi.apache.org; Endre Kovacs
>> Cc: Ali, Rizwan
>> Subject: RE: Running Nifi on OpenShift
>>
>> Public
>>
>> Hi Shawn,
>>
>> Thank you for your message. I will add your suggested configs and try it
>> out today. It certainly has new content not present in my yml so hopefully
>> it will resolve the issue.
>>
>> Thanks
>>
>> Natalia Fill
>> Analyst Software Developer
>>
>> -Original Message-
>> From: Shawn Weeks [mailto:swe...@weeksconsulting.us]
>> Sent: 13 February 2020 14:26
>> To: dev@nifi.apache.org; Endre Kovacs
>> Cc: Ali, Rizwan
>> Subject: Re: Running Nifi on OpenShift
>>
>> Your attachment didn't make it through but here are a couple of things to
>> note. First of all if you try and put the ./conf directory in a volume
>> you'll have to run a init container to copy the initial contents to the
>> volume. Kubernetes unlike Docker does not replicate from the container.
>>
>> Here is how I did that and I'm generally available on Slack if you want
>> quicker answers.
>>
>>   initContainers:
>> - name: init-nifi-conf
>>   image: apache/nifi:latest
>>   volumeMounts:
>> - mountPath: "/opt/nifi/nifi-current/new-conf"
>>   name: nifi-conf-claim
>>   command:
>> - sh
>> - '-c'
>> - '\cp /opt/nifi/nifi-current/conf/*
>> /opt/nifi/nifi-current/new-conf/'
>>
>> The other thing you'll want to include is this to set the user and group
>> id to 1000 which is what the apache image container expects since your not
>> running as root.
>>
>>   securityContext:
>> runAsUser: 1000
>> runAsGroup: 1000
>> fsGroup: 1000
>>
>> Here is my complete yaml.
>>
>> apiVersion: v1
>> kind: Service
>> metadata:
>>   name: nifi-service
>>   namespace: nifi
>> spec:
>>   clusterIP: None
>>   selector:
>> app: nifi
>>   ports:
>> - protocol: TCP
>>   port: 8080
>>   type: ClusterIP
>> ---
>> apiVersion: networking.k8s.io/v1beta1
>> kind: Ingress
>> metadata:
>>   name: nifi-ingress
>>   namespace: nifi
>> spec:
>>   rules:
>>   - host: nifi.dev.example.com
>> http:
>>   paths:
>>   - backend:
>>   serviceName: nifi-service
>>   servicePort: 8080
>>   tls:
>>   - hosts:
>> - nifi.dev.example.com
>> secretName: nifi-ssl-cert
>> ---
>> apiVersion: apps/v1
>> kind: StatefulSet
>> metadata:
>>   name: nifi-workload
>>   namespace: nifi
>> spec:
>>   replicas: 3
>>   podManagementPolicy: Parallel
>>   updateStrategy:
>> type: RollingUpdate
>>   serviceName: nifi-service
>>   selector:
>> matchLabels:
>>   app: nifi
>>  

Re: Running Nifi on OpenShift

[Jon Logan]

That's a OpenShift security feature so that your user IDs are more unique,
and have less access between containers. I would suggest trying to alter
your range of user IDs on your cluster if you don't want to modify the
image.

On Thu, Feb 13, 2020 at 11:09 AM Fill, Natalia 
wrote:

> Public
>
> Hi Shawn,
> First I tried modifying securityContect first and the familiar error is
> appeared. I remember trying to run as user 1000 a few days ago and had
> error similar to below. OpenShift has restrictions on this value.
>
> Error creating: pods "nifi-4-" is forbidden: unable to validate
> against any security context constraint: [fsGroup: Invalid value:
> []int64{1000}: 1000 is not an allowed group
> spec.containers[0].securityContext.securityContext.runAsUser: Invalid
> value: 1000: must be in the ranges: [100047, 100047]]
>
> So if Nifi has to run as user 1000 and OpenShift only allows range
> [100047, 100047] then the issue is not resolvable in the current
> image.
> Let me know if you have other views on it.
>
> Thanks
>
> Natalia Fill
> Analyst Software Developer
>
> -Original Message-
> From: Fill, Natalia [mailto:natalia.f...@lgim.com]
> Sent: 13 February 2020 14:32
> To: dev@nifi.apache.org; Endre Kovacs
> Cc: Ali, Rizwan
> Subject: RE: Running Nifi on OpenShift
>
> Public
>
> Hi Shawn,
>
> Thank you for your message. I will add your suggested configs and try it
> out today. It certainly has new content not present in my yml so hopefully
> it will resolve the issue.
>
> Thanks
>
> Natalia Fill
> Analyst Software Developer
>
> -Original Message-
> From: Shawn Weeks [mailto:swe...@weeksconsulting.us]
> Sent: 13 February 2020 14:26
> To: dev@nifi.apache.org; Endre Kovacs
> Cc: Ali, Rizwan
> Subject: Re: Running Nifi on OpenShift
>
> Your attachment didn't make it through but here are a couple of things to
> note. First of all if you try and put the ./conf directory in a volume
> you'll have to run a init container to copy the initial contents to the
> volume. Kubernetes unlike Docker does not replicate from the container.
>
> Here is how I did that and I'm generally available on Slack if you want
> quicker answers.
>
>   initContainers:
> - name: init-nifi-conf
>   image: apache/nifi:latest
>   volumeMounts:
> - mountPath: "/opt/nifi/nifi-current/new-conf"
>   name: nifi-conf-claim
>   command:
> - sh
> - '-c'
> - '\cp /opt/nifi/nifi-current/conf/*
> /opt/nifi/nifi-current/new-conf/'
>
> The other thing you'll want to include is this to set the user and group
> id to 1000 which is what the apache image container expects since your not
> running as root.
>
>   securityContext:
> runAsUser: 1000
> runAsGroup: 1000
> fsGroup: 1000
>
> Here is my complete yaml.
>
> apiVersion: v1
> kind: Service
> metadata:
>   name: nifi-service
>   namespace: nifi
> spec:
>   clusterIP: None
>   selector:
> app: nifi
>   ports:
> - protocol: TCP
>   port: 8080
>   type: ClusterIP
> ---
> apiVersion: networking.k8s.io/v1beta1
> kind: Ingress
> metadata:
>   name: nifi-ingress
>   namespace: nifi
> spec:
>   rules:
>   - host: nifi.dev.example.com
> http:
>   paths:
>   - backend:
>   serviceName: nifi-service
>   servicePort: 8080
>   tls:
>   - hosts:
> - nifi.dev.example.com
> secretName: nifi-ssl-cert
> ---
> apiVersion: apps/v1
> kind: StatefulSet
> metadata:
>   name: nifi-workload
>   namespace: nifi
> spec:
>   replicas: 3
>   podManagementPolicy: Parallel
>   updateStrategy:
> type: RollingUpdate
>   serviceName: nifi-service
>   selector:
> matchLabels:
>   app: nifi
>   template:
> metadata:
>   labels:
> app: nifi
> spec:
>   nodeSelector:
> node-role.nifi: "true"
>   securityContext:
> runAsUser: 1000
> runAsGroup: 1000
> fsGroup: 1000
>   initContainers:
> - name: init-nifi-conf
>   image: apache/nifi:latest
>   volumeMounts:
> - mountPath: "/opt/nifi/nifi-current/new-conf"
>   name: nifi-conf-claim
>   command:
> - sh
> - '-c'
> - '\cp /opt/nifi/nifi-current/conf/*
> /opt/nifi/nifi-current/new-conf/'
>   containers:
> - image: apache/nifi:latest
>   imagePullPolicy: Always
>   name: nifi
>   ports:
> - containerPort: 8080
> - containerPort: 1
>   volumeMounts:
> - mountPath: "/opt/nifi/nifi-current/conf"
>   name: nifi-conf-claim
> - mountPath: "/opt/nifi/nifi-current/database_repository"
>   name: nifi-db-claim
> - mountPath: "/opt/nifi/nifi-current/flowfile_repository"
>   name: nifi-flow-claim
> - mountPath: "/opt/nifi/nifi-current/content_repository"
>   name: nifi-conte

Re: Creating Smaller NiFi Images for Docker

[Jon Logan]

Apologies for my ignorance in this effort -- I haven't been following it
particularly closely -- but has there been any discussion in integrating
with existing solutions for retrieving artifacts, versus rolling out yet
another artifact repository? In particular, supporting Maven repository
spec would seem to make a lot of sense here.


On Fri, Sep 27, 2019 at 9:41 AM Joseph Thweatt 
wrote:

> Hi Erik, this tool currently works by looking at an existing flow and
> seeing which nars are being used by it. So I think if you are using
> something like a PRODUCTION nar in your flow, then this would try to pull
> that version in.
>
> On Fri, Sep 27, 2019 at 12:33 PM Erik Anderson  wrote:
>
> > This is important for our usages too. We dont want to redeploy NiFi
> > containers just because we added a new internal systems NAR file.
> >
> > Hopefully you will allow tags/versioning on the NAR, like LATEST or
> > PRODUCTION vs DEVELOPMENT, v1.0 or v2.1.43.
> >
> > This way we can test new(er) NAR's without using separate NiFi systems.
> > Even test 2 identical flows with different NAR versions.
> >
> > Erik Anderson
> > Bloomberg
> >
> > On Fri, Sep 27, 2019, at 12:07 PM, Joe Witt wrote:
> > > Joseph
> > >
> > > Cool - this is precisely a key part of the motivation behind the NiFi
> Reg
> > > effort and all the work to make extensions sourced from the registry
> > > on-demand.  As a result we'll no longer package most (perhaps even any)
> > > nars with a nifi distribution going forward.  We've been steadily
> making
> > > progress on this front.  It would be good for you to collaborate in
> those
> > > efforts if you're interested.  The driver here is not docker but rather
> > all
> > > forms of nifi consumption/deployment models but the docker world would
> > > certainly benefit.
> > >
> > > Thanks
> > >
> > > On Fri, Sep 27, 2019 at 12:01 PM Joseph Thweatt <
> josephthwe...@gmail.com
> > >
> > > wrote:
> > >
> > > > tl:dr I made this to make the NiFi docker image smaller, let me know
> if
> > > > it's useful: https://github.com/josephthweatt/skinifi
> > > >
> > > > Hello everyone,
> > > >
> > > > My team has been been using NiFi for a couple months now and we have
> > > > recently begun adding the docker image into a stack. I noticed that
> > NiFi's
> > > > docker image is pretty big (1.91 GB) and takes a while to download
> as a
> > > > result. Adding to that, the image itself is only a part of the
> > multistage
> > > > build. After adding custom processors we were actually looking at
> > something
> > > > closer to 2.8 GB, making itesting/building/downloading a nightmare.
> > > >
> > > > The main reason the image is so large is that the lib is full of nar
> > files,
> > > > the majority of which are never used. Those same nars are then
> > duplicated
> > > > to the work directory when nifi is running, making it even larger.
> > > >
> > > > My solution to this was the github project above. Basically if you
> > provide
> > > > it a Flow from a registry or a template it will find which nars are
> > needed
> > > > to run nifi and pack them into a smaller image. A bare-bones image
> > (NiFi
> > > > with only the processors needed to run without breaking) is about 680
> > MB
> > > > and in my team's case we got the file size reduced to 1.2 GB.
> > Integration
> > > > tests for our stack also halved as a result, from 35 mins down to 17.
> > > >
> > > > I'll probably continue to add some useful features that I feel we
> > need, but
> > > > I figured I should put this out there if anyone else wanted to try
> it.
> > > >
> > > > Suggestions are also appreciated!
> > > >
> > >
> >
>

Re: Weird spring issue if I run a snapshot w/out internet access

[Jon Logan]

I haven't looked at that specifically but we've ran into issues before with
Spring quietly pulling XSD's over the internet if they're not packaged
correctly in your artifact, and that breaking miserably if you don't have
the internet available. That's just a wild guess though.

On Tue, Jun 18, 2019 at 2:14 PM Mike Thomsen  wrote:

> Sometimes when I start a snapshot build when I don't have Internet access I
> get this. Any ideas?
>
> 2019-06-18 14:11:54,040 WARN [main] org.apache.nifi.web.server.JettyServer
> Failed to start web server... shutting down.
> org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line
> 19 in XML document from class path resource [nifi-context.xml] is invalid;
> nested exception is org.xml.sax.SAXParseException; lineNumber: 19;
> columnNumber: 139; cvc-elt.1: Cannot find the declaration of element
> 'beans'.
> at
>
> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:399)
> at
>
> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:336)
> at
>
> org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:304)
> at
>
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:181)
> at
>
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:217)
> at
>
> org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:188)
> at
>
> org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitionsFromImportedResources(ConfigurationClassBeanDefinitionReader.java:354)
> at
>
> org.springframework.context.annotation.ConfigurationClassBeanDefinitionReader.loadBeanDefinitionsForConfigurationClass(ConfigurationClassBeanDefinitionReader.java:143)
>

Re: New NiFi Metric

[Jon Logan]

It sounds interesting. The one potential issue I could see would be related
to permissions, particularly if it's an implicit global thing.

Would this integrate with the existing Metrics capabilities (ex.
MetricsReportingTask)? Maybe there could be a way to decouple it into a
metrics reporting task and have a UI Metrics Reporter Service that would
allow for a more general metrics status page? This would allow you to take
the same metric and publish it externally if desired (ex. to Cloudwatch),
and trigger notification alarms, etc. Just a thought...

On Fri, Mar 15, 2019 at 8:46 AM Joe Witt  wrote:

> Mark
>
> Certainly sounds interesting and adding such a metric makes good sense.
>
> On Fri, Mar 15, 2019 at 11:39 AM Owens, Mark  wrote:
>
> > Hi,
> >
> > I'm looking to implement a new metric in NiFi based upon some research
> > performed by an intern last summer. It would be a 'time-to-capacity'
> > estimate that would predict capacity overloads in dataflows. The goal
> would
> > be to predict estimated time to overload within data flows and provide
> > means to alert interested parties prior to that overload failure. The
> > initial method would involve sampling real-time information directly from
> > NiFi and calculating the rate of input/output for data connections using
> a
> > sliding window in time. A value would be calculated estimating the
> > remaining time until capacity overload by assuming a constant
> input/output
> > difference over the sampled timeframe. This value would be updated at
> > regular intervals using the latest input/output information. This method
> > could be refined and improved as needed in time.
> >
> >  I'm seeking comments on the level of difficulty you think this would
> > entail, i.e., does it sound feasible? I'm also seeking a sense on how
> > amenible the community would be to adding a new metric to NiFi and having
> > it added to the stats page. Suggestions as to good portions of code where
> > existing metric calculations are made and presented would be helpful as
> > well.
> >
> > Thanks,
> > Mark
> >
> >
>

NiFi and MiNiFi Compatability?

[Jon Logan]

Hi All,

I was wondering if there are compatibility concerns between NiFi and
MiNiFi? Specifically, it seems like the last release of MiNiFi is against
1.7 -- should we expect problems if we were to point this against the
latest version of NiFi?

If so, are there any plans of trying to coordinate releases between the
project and sub-project, even if its just to link against the same
binaries? I was also curious if MiNiFi compatibility is considered during
the NiFi release / testing process?


Thanks!
Jon

Re: Status of MetricsReportingTask?

[Jon Logan]

We were planning on having a reporter to pipe the metrics off to AWS
CloudWatch. The one issue being ran into -- and we've ran into it for other
components -- is the classloading. What is the recommended way of adding
alternate implementations of services (in this case MetricReporterService)?
We run into issues that I think stem from classloading isolation between
NARs, such that our custom implementations of services are not being
detected from the other NAR's context. We end up having to duplicate the
service into our custom NAR, but that hardly seems like a good idea.

Thanks!
Jon

On Thu, Nov 1, 2018 at 10:27 PM Koji Kawamura 
wrote:

> Hi Jon,
>
> About reporting counter values, there is an existing JIRA for the
> improvement idea to expose counters to Reporting task context. That
> requires NiFi framework level improvements. I'd suggest taking a look
> on it, and resuming discussion there if needed.
> https://issues.apache.org/jira/browse/NIFI-3293
>
> Although MetricsReportingTask currently only supports Graphite, the
> component is well designed for generic reporting use-cases. I don't
> think it is a legacy component. The underlying dropwizard metrics
> project seems active, too.
> https://github.com/dropwizard/metrics
>
> I'm interested in what service implementation you're going to write.
> Is it going to use one of already available reporters?
> https://metrics.dropwizard.io/3.1.0/manual/third-party/
>
> Thanks,
> Koji
> On Thu, Nov 1, 2018 at 3:17 AM Jon Logan  wrote:
> >
> > Hi All,
> >
> > I was looking at utilizing the MetricsReportingTask service, but I was
> > wondering what the status of it is -- it seems to be lacking some
> features
> > that I thought it'd have (ex. reporting counters), and I'm not sure
> there's
> > an ability to extend the metrics being produced. Is this something that
> is
> > still being worked on, or is a legacy component? We are going to have to
> > write our own Service implementation, as the only supported one seems to
> be
> > Graphite, and wanted to make sure we're not going down a legacy path.
> >
> > Thanks!
>

Status of MetricsReportingTask?

[Jon Logan]

Hi All,

I was looking at utilizing the MetricsReportingTask service, but I was
wondering what the status of it is -- it seems to be lacking some features
that I thought it'd have (ex. reporting counters), and I'm not sure there's
an ability to extend the metrics being produced. Is this something that is
still being worked on, or is a legacy component? We are going to have to
write our own Service implementation, as the only supported one seems to be
Graphite, and wanted to make sure we're not going down a legacy path.

Thanks!

Re: [External] Re: Expression language and NiFi registry

[Jon Logan]

Speaking of EL and Secrets -- we have been setting some secret values as EL
expressions to substitute in properties for passwords and such. If we
switch to NiFi registry, this would still be considered a sensitive value
and not be stored? Is there a better way to handle secret distribution?

On Fri, Oct 26, 2018 at 3:38 PM Michael Moser  wrote:

> For the specific case of "host name of the AMQP processors", EL support for
> that property was added to NiFi in NIFI-5489 [1] which will be released
> with NiFi 1.8.0 (available real soon!)
>
> [1] - https://issues.apache.org/jira/browse/NIFI-5489
>
> -- Mike
>
>
> On Wed, Oct 24, 2018 at 3:06 PM Pierre Villard <
> pierre.villard...@gmail.com>
> wrote:
>
> > Hi there,
> >
> > There is a JIRA [1] about this idea and, as Joe said, we need to
> carefully
> > think about the implications. Unless we can do something nice and
> backward
> > compatible, that could be something for NiFi 2.0.
> >
> > [1] https://issues.apache.org/jira/browse/NIFI-5367
> >
> > Pierre
> >
> > Le mer. 24 oct. 2018 à 20:59, Joe Witt  a écrit :
> >
> > > Jon,
> > >
> > > There is very little overhead really.  The answer more generally has
> > > to do with history.  Expression language was added later in the game
> > > and there can be different scopes to what a given EL statement has
> > > access to.  Some EL are evaluated against flowfiles and some arent.
> > > We didn't have a good way to show users this information though this
> > > has recently improved.  It also has to do with 'evaluation time'
> > > meaning processors might evaluate some properties frequently/while
> > > executing a task (as would be the case when EL statements have a given
> > > flowfile in scope) or during enabling/scheduling such as pulling a
> > > property value they intend to honor throughout an entire runtime.
> > >
> > > I suspect at this point that we can make most properties (not
> > > enumerations most likely) EL enabled.  But, each one requires
> > > review/consideration to ensure the processor is coded to use it
> > > correctly.
> > >
> > > Alternatively, we could explore making all text entry fields allow
> > > users to explicitly state 'this is a variable and here is the variable
> > > name' (but in a cool UI way) and then the actual value used would come
> > > from the variable registry for that process group.  This would be
> > > something we can do across the board as it wouldnt' change the
> > > processor's logic/behavior at all.  Variables get set at and retained
> > > throughout the life of a processor run.  If a used variable is changed
> > > we'd automatically stop/start each impacted component/processor so it
> > > wouldnt' require any additional review/changes in processor
> > > properties.
> > >
> > > ThanksOn Wed, Oct 24, 2018 at 2:46 PM Jonathan Meran
> > >  wrote:
> > > >
> > > > Hello there,
> > > > Regarding expression language properties, why aren't all or most
> > > properties enabled by default.
> > > >
> > > > Is there substantial processor overhead on these?
> > > >
> > > > Thanks,
> > > > Jon
> > > >
> > > > On 10/24/18, 1:49 PM, "Joe Witt"  wrote:
> > > >
> > > > Mark
> > > >
> > > > There are two scenarios here to discuss:
> > > > 1) What do to with sensitive properties
> > > > 2) What to do with properties that dont allow expression language
> > > statements
> > > >
> > > > For #1 we dont send those properties into the registry and they
> are
> > > > set (properly) in each environment where the secret belongs.  All
> > > good
> > > > as that doesn't change the flow definition.
> > > >
> > > > For #2 we should just look at which properties are causing
> trouble
> > > and
> > > > see about expression language enabling them.  So please share
> > > > precisely which ones you're hitting where that would help you out
> > and
> > > > lets see what can be done.
> > > >
> > > > Thanks
> > > > On Wed, Oct 24, 2018 at 1:39 PM Mark Littleton
> > > >  wrote:
> > > > >
> > > > > Hi Everyone,
> > > > >
> > > > > I'm currently doing a lot of work with Nifi and recently we
> have
> > > been trying to come up with a solution to a problem. We installed Nifi
> > > registry backed by our Git repository for versioning our flows. This
> has
> > > worked out great for us as we can now track the version of our flows
> > > correctly and make sure they are backed up in source control.
> > > > >
> > > > > However when we want to do deployment between our Development
> > Nifi
> > > cluster and our Qa Nifi cluster we have to ofcourse change some values.
> > > These could be amqp queues, directories on the file system etc.
> > > > >
> > > > > So ofcourse we use variables so that we can configure the
> values
> > > without it being detected as a change to the flow. A problem arises
> > however
> > > when we need to configure an option that does not support expression
> > > language. For example the host name of the amqp proce

Re: NIFI single node in cluster mode

[Jon Logan]

It waits for election for a specific period of time, which if I recall is
fairly high (I think 5 minutes?). If you lower this it'll still wait for an
election but will complete faster (we do 30 seconds, but you could do
lower). There's a property controlling this.

On Fri, Oct 12, 2018 at 5:41 PM Milan Das  wrote:

> Hello Nifi team,
>
> Is it possible to run a single NIFI node in cluster mode ? I have this
> requirement because we will add other nodes soon down line.
>
> I tried that by setting  “nifi.cluster.is.node” but and zookeeper setting.
> But seems it waits ever for election.
>
>
>
> Appreciate your thoughts.
>
>
>
> Thanks,
>
> Milan Das
>
>

Re: Variable / Nifi-EL for RPG Endpoint?

[Jon Logan]

Yeah, I meant to say NiFi Registry but substituted in Docker instead! Part
of the reason we've approached NiFi registry with caution is the lack of
connectivity between installations. Would the recommended approach be
shipping a registry containing the flows desired, and then shipping the
NiFi containers without a baked-in flow? Unfortunately that would require a
manual step step out of the box as the RPG and other potential secrets
would need to be specified when pulling from the registry into the NiFi
cluster (though it could probably be done via the API)?

On Thu, Sep 6, 2018 at 4:11 PM, Andy LoPresto  wrote:

> I think Bryan was referring to Apache NiFi Registry [1] which is the
> complementary standalone application which hosts version controlled flow
> snippets.
>
> [1] https://nifi.apache.org/registry.html
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Sep 6, 2018, at 1:09 PM, Jon Logan  wrote:
>
> Thanks Bryan -- Is that in terms of Docker Registry? We don't currently use
> Docker Registry, but our plan was to bundle a flow.gz into a Docker image
> directly. I'm not sure if maybe we're approaching this problem in a weird
> way? How do most people handle multiple instances in different environments
> that are disconnected from one another? We considered running a Docker
> Registry container as an option, or specifying the flow.gz directly as an
> option...both seem to have some issues and inconveniences.
>
> Thanks!
> Jon
>
> On Thu, Sep 6, 2018 at 4:01 PM, Bryan Bende  wrote:
>
> Jon,
>
> The RPG URL is treated similar to sensitive properties or parent
> controller services, meaning that after deploying your flow you can
> issue an update to the RPG to set the URL to given environment's value
> and that change will not be considered a change as far as version
> control and will be retained when upgrading the process group to newer
> versions.
>
> Thanks,
>
> Bryan
>
> On Thu, Sep 6, 2018 at 3:48 PM, Jon Logan  wrote:
>
> We are running into a slight issue when we wish to migrate between NiFi
> cluster instances...the RPG endpoint in our flow seems to disallow
>
> setting
>
> of a variable, which makes our flow have to be specific to the specific
> environment it's being deployed in. It seems odd that it won't let us use
> an expression there and set the value to a variable -- is this an
> oversight? Is there some design decision here that we're missing?
>
> Thanks!
>
>
>
>

Re: Variable / Nifi-EL for RPG Endpoint?

[Jon Logan]

Thanks Bryan -- Is that in terms of Docker Registry? We don't currently use
Docker Registry, but our plan was to bundle a flow.gz into a Docker image
directly. I'm not sure if maybe we're approaching this problem in a weird
way? How do most people handle multiple instances in different environments
that are disconnected from one another? We considered running a Docker
Registry container as an option, or specifying the flow.gz directly as an
option...both seem to have some issues and inconveniences.

Thanks!
Jon

On Thu, Sep 6, 2018 at 4:01 PM, Bryan Bende  wrote:

> Jon,
>
> The RPG URL is treated similar to sensitive properties or parent
> controller services, meaning that after deploying your flow you can
> issue an update to the RPG to set the URL to given environment's value
> and that change will not be considered a change as far as version
> control and will be retained when upgrading the process group to newer
> versions.
>
> Thanks,
>
> Bryan
>
> On Thu, Sep 6, 2018 at 3:48 PM, Jon Logan  wrote:
> > We are running into a slight issue when we wish to migrate between NiFi
> > cluster instances...the RPG endpoint in our flow seems to disallow
> setting
> > of a variable, which makes our flow have to be specific to the specific
> > environment it's being deployed in. It seems odd that it won't let us use
> > an expression there and set the value to a variable -- is this an
> > oversight? Is there some design decision here that we're missing?
> >
> > Thanks!
>

Variable / Nifi-EL for RPG Endpoint?

[Jon Logan]

We are running into a slight issue when we wish to migrate between NiFi
cluster instances...the RPG endpoint in our flow seems to disallow setting
of a variable, which makes our flow have to be specific to the specific
environment it's being deployed in. It seems odd that it won't let us use
an expression there and set the value to a variable -- is this an
oversight? Is there some design decision here that we're missing?

Thanks!

Version-Agnostic Paths in Docker?

[Jon Logan]

Hi,

I was looking at utilizing the Docker distribution but was wondering if any
thought was given to making the paths version-agnostic? Specifically, the
conf folder is in /opt/nifi/nifi-1.7.1/conf. This would mean any
modifications we would make in a container or RPM would be tied to 1.7.1.

I have seen other projects accomplish this with a symlink (ex.
/opt/nifi/latest -> opt/nifi/nifi-1.7.1), and think that approach might
work as well here?

Is there a better approach?


Thanks!
Jon

Re: SSLPeerUnverifiedException Hostname "xyz" not verified

[Jon Logan]

Is this a correct assumption? I would think the only way would be to have a
cert server and have servers generate certs on initialization? We have our
NiFi cluster autoscaling so we'd need this server stood up versus manually
provisioning ahead of time. Are there any viable alternatives? And
regarding site-to-site visibility, is there a way to change the default so
when nodes join they can see ports, etc?

On Fri, Jul 20, 2018 at 7:55 PM, Jon Logan  wrote:

> The other issue we have is that we would now have to run two additional
> separate services in any deployment we would run -- namely, a certificate
> server and LDAP. We try to reduce complexity of deployments, but it's stuff
> like this that quickly becomes a maintenance burden.
>
> On Fri, Jul 20, 2018 at 3:51 PM, Josefz 
> wrote:
>
>> @Andy LoPresto
>>
>> I fully understand what you wrote regarding certs in the admin guide,
>> however as you already mentioned, in my point of view this certificate
>> stuff
>> is really a pain. We have lost multiple days to get it running together
>> with
>> LDAP, just because of the complexity of the whole configuration. And after
>> the upgrade to 1.7.0 we had again issues because of certs and the bug...
>>
>> Let me explain why we use wildcard certs. We have to use our company CA
>> and
>> we have to manually insert the CSR on a website (with some additional
>> parameters) to get the certificate signed. If we have to do that for 20
>> nodes or even more, this would be a huge work. Additionally our network
>> where the NiFi Nodes are, is a subnet secured by a firewall, so it's not
>> possible to connect from outside through the cluster port. If an attacker
>> is
>> inside the subnet and is able to create a NiFi Node who can join the
>> cluster
>> (with the certificate and the password for the keystore), then we would
>> anyway have bigger problems. But yes of course, wildcard certs are less
>> secure.
>>
>> *Two questions for you:*
>>
>> 1. We used the wildcard certs already in NiFi 1.5.0 in our lab, however we
>> would like to go live with 1.7.1 now. If we haven't seen any issues on
>> NiFi
>> 1.5.0 with the wildcard certs, how likely would it be that we see some
>> issues on 1.7.1?
>>
>> 2. Somewhere I've read that in an optimal world (eg. with the NiFi TLS
>> Certkit) we should have a Cert with a unique DN and as well use the same
>> DN
>> for the SAN per node. Would it be ok to have the following:
>>
>> 3-Node Cluster Environment: nifi-node-1, nifi-node-2, nifi-node-3
>>
>> One Keystore Certificate for all NiFi nodes with the following attributes:
>> -> DN "CN=NiFi Apache";
>> -> SAN = nifi-node-1, nifi-node-2, nifi-node-3
>>
>> Background is the following, we are planning a loadbalancer in front of
>> NiFi
>> Webgui and I don't see any solution to get the whole thing work without
>> the
>> procedure above. Today we use wildcard, with that we are good to go. But
>> as
>> you already mentioned multiple times that wildcards are not supported we
>> are
>> looking for some alternatives.
>>
>> Thanks in advance
>> Josef
>>
>>
>>
>> --
>> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
>>
>
>

Re: SSLPeerUnverifiedException Hostname "xyz" not verified

[Jon Logan]

The other issue we have is that we would now have to run two additional
separate services in any deployment we would run -- namely, a certificate
server and LDAP. We try to reduce complexity of deployments, but it's stuff
like this that quickly becomes a maintenance burden.

On Fri, Jul 20, 2018 at 3:51 PM, Josefz  wrote:

> @Andy LoPresto
>
> I fully understand what you wrote regarding certs in the admin guide,
> however as you already mentioned, in my point of view this certificate
> stuff
> is really a pain. We have lost multiple days to get it running together
> with
> LDAP, just because of the complexity of the whole configuration. And after
> the upgrade to 1.7.0 we had again issues because of certs and the bug...
>
> Let me explain why we use wildcard certs. We have to use our company CA and
> we have to manually insert the CSR on a website (with some additional
> parameters) to get the certificate signed. If we have to do that for 20
> nodes or even more, this would be a huge work. Additionally our network
> where the NiFi Nodes are, is a subnet secured by a firewall, so it's not
> possible to connect from outside through the cluster port. If an attacker
> is
> inside the subnet and is able to create a NiFi Node who can join the
> cluster
> (with the certificate and the password for the keystore), then we would
> anyway have bigger problems. But yes of course, wildcard certs are less
> secure.
>
> *Two questions for you:*
>
> 1. We used the wildcard certs already in NiFi 1.5.0 in our lab, however we
> would like to go live with 1.7.1 now. If we haven't seen any issues on NiFi
> 1.5.0 with the wildcard certs, how likely would it be that we see some
> issues on 1.7.1?
>
> 2. Somewhere I've read that in an optimal world (eg. with the NiFi TLS
> Certkit) we should have a Cert with a unique DN and as well use the same DN
> for the SAN per node. Would it be ok to have the following:
>
> 3-Node Cluster Environment: nifi-node-1, nifi-node-2, nifi-node-3
>
> One Keystore Certificate for all NiFi nodes with the following attributes:
> -> DN "CN=NiFi Apache";
> -> SAN = nifi-node-1, nifi-node-2, nifi-node-3
>
> Background is the following, we are planning a loadbalancer in front of
> NiFi
> Webgui and I don't see any solution to get the whole thing work without the
> procedure above. Today we use wildcard, with that we are good to go. But as
> you already mentioned multiple times that wildcards are not supported we
> are
> looking for some alternatives.
>
> Thanks in advance
> Josef
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
>

Re: SSLPeerUnverifiedException Hostname "xyz" not verified

[Jon Logan]

Can you clarify that statement? If you have a completely empty
authorizers.xml file, would the node join the cluster and inherit the
clusters authorizers.xml? Also, what would happen if the whole cluster was
restarted? Are changes persisted? And if they are, if a node is restarted
at the same time a new node is added -- would that node not have an
outdated authorizers.xml and get rejected from rejoining?


Thanks again.

On Thu, Jul 19, 2018 at 4:55 AM, Peter Wilcsinszky <
peterwilcsins...@gmail.com> wrote:

> On Thu, Jul 19, 2018 at 12:38 AM Andy LoPresto 
> wrote:
>
> > Hi Jon,
> >
> > There are automatable, scalable, and non-restart-required ways to
> > horizontally scale a secure cluster without requiring wildcard
> > certificates. I should collect the various instructions / notes together
> > into an article and people can reference that, but until that time, the
> > Admin Guide [1] and the conversation Prashanth and I had on the ticket
> [2]
> > are the best documentation for this.
> >
> > Basically:
> >
> > * run the TLS toolkit as a server and have each node connect to it to
> > obtain a valid cert. All of these will be signed by the same CA and each
> > node will be able to communicate with all others. Add a new user with the
> > node CN and RW on /proxy resource via the UI/API of the existing nodes.
> >
>
>
> > You should not need to modify authorizers.xml directly.
> >
>
> I would highlight that in that case you should use the original
> authorizers.xml that do not contain any nodes, not even the initial admin
> identity. This way new nodes can figure to accept authorizers information
> from the cluster instead of having a conflict and failing to start.
> (This failure scenario happened to me when tried to spin up new nodes with
> the same authorizers.xml that I used for the inital cluster.)
>
>
> > * same permissions steps but use the toolkit in standalone mode. In this
> > case, you must run it from the same directory every time (so it uses the
> > same CA key to sign).
> > * same permissions steps but run toolkit in standalone on each node. In
> > this case, you must import the generated CA into existing node
> truststores
> > (requires restart).
> >
> > [1]
> > https://nifi.apache.org/docs/nifi-docs/html/administration-
> guide.html#tls-generation-toolkit
> > [2] https://issues.apache.org/jira/browse/NIFI-5370
> >
> >
> >
> >
> > Andy LoPresto
> > alopre...@apache.org
> > *alopresto.apa...@gmail.com *
> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >
> > On Jul 18, 2018, at 2:49 PM, Jon Logan  wrote:
> >
> > I saw this in the release notes...specifically that wildcard certs are
> not
> > supported. How do most people handle this in practice? We can run a cert
> > server or get them from other means (AWS cert manager, etc) but am not
> sure
> > how to overcome the authorizers.xml issue -- would we need to have a
> > provisioning script register each new server certificate with NiFi before
> > it can actually do anything useful? Will new servers then have issues
> > joining because their authorizers will not match the rest of the cluster?
> >
> > On Thu, Jul 5, 2018 at 8:04 AM, Pierre Villard <
> > pierre.villard...@gmail.com>
> > wrote:
> >
> > Hi Josef,
> >
> > I don't have a solution for you but it seems it has already been reported
> > and a JIRA has been opened:
> > https://issues.apache.org/jira/browse/NIFI-5370
> >
> > Andy might be able to give more insights about it.
> >
> > Pierre
> >
> > 2018-07-05 13:19 GMT+02:00 Josefz :
> >
> > Hi expert
> >
> > I've just done an upgrade from NiFi 1.5.0 to 1.7.0 in a SSL secured
> >
> > cluster
> >
> > with LDAP authentication. Now I'm not anymore able to login into the
> > webgui.
> > After I have entered the login/password I'm getting the following
> >
> > message:
> >
> >
> >
> >
> > And nifi-app.log reports the following error messages:
> >
> >
> >
> > I'm having a wildcard SSL certificate and I'm using the same
> > keystore/truststore combination for three usecases:
> > - for cluster connectivity (in nifi.properties)
> > - in "authorizer.xml"
> > - in "login-identity-providers.xml".
> >
> > The keystore.jks (private/public) keypair has been signed by our internal
> > root CA and the root CA cert has been imported into the truststore.jks.
> >
> > As
> >
> > the ldap login works with certificates I'm more or less sure that the
> >
> > certs
> >
> > in general are fine. Has anybody an idea if wildcard CN and SAN names
> > should
> > work in a cluster or where the problem could be? I've tried the same
> >
> > certs
> >
> > as well in standalone mode, no issue at all.
> >
> > The following parameters in nifi.properties are enabled:
> > nifi.security.needClientAuth=true
> > nifi.cluster.protocol.is.secure=true
> >
> > Thanks in advance
> >
> >
> >
> >
> > --
> > Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
> >
> >
> >
> >
>

Re: SSLPeerUnverifiedException Hostname "xyz" not verified

[Jon Logan]

I saw this in the release notes...specifically that wildcard certs are not
supported. How do most people handle this in practice? We can run a cert
server or get them from other means (AWS cert manager, etc) but am not sure
how to overcome the authorizers.xml issue -- would we need to have a
provisioning script register each new server certificate with NiFi before
it can actually do anything useful? Will new servers then have issues
joining because their authorizers will not match the rest of the cluster?

On Thu, Jul 5, 2018 at 8:04 AM, Pierre Villard 
wrote:

> Hi Josef,
>
> I don't have a solution for you but it seems it has already been reported
> and a JIRA has been opened:
> https://issues.apache.org/jira/browse/NIFI-5370
>
> Andy might be able to give more insights about it.
>
> Pierre
>
> 2018-07-05 13:19 GMT+02:00 Josefz :
>
> > Hi expert
> >
> > I've just done an upgrade from NiFi 1.5.0 to 1.7.0 in a SSL secured
> cluster
> > with LDAP authentication. Now I'm not anymore able to login into the
> > webgui.
> > After I have entered the login/password I'm getting the following
> message:
> >
> >
> >
> > And nifi-app.log reports the following error messages:
> >
> >
> >
> > I'm having a wildcard SSL certificate and I'm using the same
> > keystore/truststore combination for three usecases:
> > - for cluster connectivity (in nifi.properties)
> > - in "authorizer.xml"
> > - in "login-identity-providers.xml".
> >
> > The keystore.jks (private/public) keypair has been signed by our internal
> > root CA and the root CA cert has been imported into the truststore.jks.
> As
> > the ldap login works with certificates I'm more or less sure that the
> certs
> > in general are fine. Has anybody an idea if wildcard CN and SAN names
> > should
> > work in a cluster or where the problem could be? I've tried the same
> certs
> > as well in standalone mode, no issue at all.
> >
> > The following parameters in nifi.properties are enabled:
> > nifi.security.needClientAuth=true
> > nifi.cluster.protocol.is.secure=true
> >
> > Thanks in advance
> >
> >
> >
> >
> > --
> > Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
> >
>

Re: Java 10

[Jon Logan]

Will any PRs with Java 9 features be rejected until then? (ie. preserving
Java-8 compatibility) Is there a plan for supporting multiple Java versions?

On Thu, Jun 7, 2018 at 10:18 AM, Mike Thomsen 
wrote:

> Once that is done, it should run on Java 10. However, under the new Oracle
> version rules Java 10 is a short term iteration. Java 11 will probably be
> the next version officially supported by NiFi because it's supposed to be
> Java's next LTS release.
>
> Java 10 features like the var key will also be rejected in PRs until that
> point.
>
> On Thu, Jun 7, 2018 at 9:48 AM Otto Fowler 
> wrote:
>
> > You can track progress here:
> > https://issues.apache.org/jira/browse/NIFI-5174
> > There has been some progress very recently.
> >
> >
> >
> > On June 7, 2018 at 09:40:50, Sivaprasanna (sivaprasanna...@gmail.com)
> > wrote:
> >
> > Nope. Not yet. AFAIK, there is a Jira to support Java 9.
> >
> > On Thu, Jun 7, 2018 at 7:09 PM, kirilzilla  wrote:
> >
> > > does nifi support java 10 ? or even java 9 ?
> > >
> > >
> > >
> > > --
> > > Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
> > >
> >
>

Re: Signing AWS Elasticsearch Requests?

[Jon Logan]

Thanks, a few notes -- as far as not using AWS ES, we have been having
issues with standard-ES, mainly the lack of encryption and authentication
without the X-Pack (yes, I could proxy but that seems like a band-aid)..
note that AWS VPCs are not encrypted internally, and any machine that you
want to have updates must have a public-facing IP address (no VPC-only).

This could be done using the AWS ES SDK, but it seems like that's more
trouble than it's worth...the AWS ES endpoints are identical, yet support
the additional signing attribute. It would seem unnecessary to have two
disjoint implementations, rather than just adding the signatures to the
request...similar to this (I found this on Google, so no guarantees!)
https://gist.github.com/missingfaktor/664b01b7989e91033657e2e703f14ed6 . I
think the only piece of the AWS SDK this would require would be the things
to get the credentials, and the signing part itself.

The new ES client service looks much easier to use than the previous one.
Is this going to eventually be used by the existing processors (ex.
FetchEsHttp, PutEsHttp, etc)? I suppose this could be even have a
TransportClient implementation and merge the FetchEs FetchEsHttp and
FetchEs5 (and put, etc) processors?

On Mon, Apr 9, 2018 at 9:47 PM, Mike Thomsen  wrote:

> Related, interesting take on AWS ES:
>
> https://code972.com/blog/2017/12/111-why-you-shouldnt-use-
> aws-elasticsearch-service
>
> I saw some other critical posts that went into detail on other issues.
> Considering how easy it is to stand up ES yourself, I think it would be
> worthwhile to strongly consider going that route.
>
> On Mon, Apr 9, 2018 at 9:02 PM, Otto Fowler 
> wrote:
>
> > Sorry, just like the gateway api class, it is for managing ES not
> calling.
> >
> >
> > On April 9, 2018 at 20:29:33, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
> >
> > The aws java sdk has a purpose built ElasticSearchClient class.
> > The way to do this and be consistent would be to add a new nifi-aws
> > processor for the ES client,
> > as there is for s3 and dynamoDB etc.
> >
> > https://github.com/apache/nifi/pull/2588 is my outstanding PR for
> > HttpInvoke for AWS Gateway Web APIs.
> > A similar approach being having version of the ElasticSearchHttp but in
> the
> > aws nar.
> >
> >
> >
> > On April 9, 2018 at 19:11:45, Jon Logan (jmlo...@buffalo.edu) wrote:
> >
> > Hi All,
> >
> > We are looking to use the built-in Nifi Elasticsearch Http processors
> with
> > signed AWS ES requests. I tried to extend them to do so, but ran into
> > issues with things being in-extensible (private-static-final in some
> > cases), and was wondering if this would be something that would be
> > considered to be merged into the baseline?
> >
> > There's two main ways I saw doing it -- either modifying the existing ES
> > code to allow for it, or making new AWS-specific extended adapters to do
> > it. In the former, it might require dependencies between the ES code and
> > the AWS code for optional CredentialProviders, etc., and am not sure how
> > isolated you all try to keep things.
> >
> > In either case, AWS essentially adds an HTTP-header signature to
> > authenticate the request against your ES instance. The easiest path to do
> > this seems to be writing a bridge between the ES OkHttpClient requests
> and
> > the AWS requests to generate the signature correctly.
> >
> >
> > Just wanted to get some feedback to be sure this wouldn't be a waste of
> > time.
> >
> >
> > Thanks!
> > Jon
> >
>

Signing AWS Elasticsearch Requests?

[Jon Logan]

Hi All,

We are looking to use the built-in Nifi Elasticsearch Http processors with
signed AWS ES requests. I tried to extend them to do so, but ran into
issues with things being in-extensible (private-static-final in some
cases), and was wondering if this would be something that would be
considered to be merged into the baseline?

There's two main ways I saw doing it -- either modifying the existing ES
code to allow for it, or making new AWS-specific extended adapters to do
it. In the former, it might require dependencies between the ES code and
the AWS code for optional CredentialProviders, etc., and am not sure how
isolated you all try to keep things.

In either case, AWS essentially adds an HTTP-header signature to
authenticate the request against your ES instance. The easiest path to do
this seems to be writing a bridge between the ES OkHttpClient requests and
the AWS requests to generate the signature correctly.


Just wanted to get some feedback to be sure this wouldn't be a waste of
time.


Thanks!
Jon