Re: Extending StandardOidcIdentityProvider

[Vijay Jammi]

 Thank You Bryan.

It seems like using assertions to authenticate a client with the token
endpoint is more common within enterprises. Here is a link to the RFC
 and a sample code

from Microsoft.

Vijay Jammi


On Tue, Jan 19, 2021 at 9:44 AM Bryan Bende  wrote:

> Hello,
>
> 1) I don't think NiFi currently supports the client id + x509
> scenario, is this part of the OIDC standard? If so then maybe it can
> be an improvement that is implemented.
>
> 2) The OIDC code in NiFi is not part of an extension point, so you
> can't just plug in your own version. You would have to modify the code
> in NiFi and rebuild the nifi-framework-nar with your changes.
>
> -Bryan
>
> On Fri, Jan 15, 2021 at 1:25 PM Vijay Jammi 
> wrote:
> >
> > Hello there,
> >
> > I am trying to enable OIDC [OpenIDConnect/OAuth2.0] for our on prem Nifi
> > with our on prem Identity Provider [Microsoft ADFS].
> >
> > Now, it looks like Nifi's authorization code flow requires a client id [
> > nifi.security.user.oidc.client.id] and client secret
> > [nifi.security.user.oidc.client.secret] to be able to exchange
> > Authorization Code for an Access and Id Token. However, our Authorization
> > Server only supports client id and x509 client certificate based
> > authentication [Client Assertion] for the exchange. So my question here
> is
> >
> >  1. Is there way to configure Nifi for client id and x509 client
> > certificate for the exchange?
> >  2. If not, how can we extend Nifi for our need?
> >
> > I am new to Nifi so please excuse me if this is trivial within the Nifi
> > development. I see a StandardOidcIdentityProvider under
> nifi-web-security.
> > Can I override the default functionality by making a custom bundle to
> > override or will I need to rebuild the bundle associated to
> > nifi-web-security and drop it into the Nifi lib?  Any guidance will be
> much
> > appreciated.
> >
> > Thank you in advance.
> >
> > Vijay Jammi
>

Re: Setting thread counts based on worker node size

[Mark Payne]

Hi Chris,

There’s a YouTube video [1] that I created a few months ago that discusses a 
lot of the considerations to take into account when determining the number of 
threads to allocate to the overall thread pool and how many to allocate to 
individual processors. Unfortunately, it’s pretty non-trivial. One thing to 
note, though, is that the number of nodes shouldn’t really determine the size 
of the thread pool, as the thread pool is per-node.

Thanks
-Mark

[1] https://www.youtube.com/watch?v=pZq0EbfDBy4


On Jan 25, 2021, at 12:05 PM, Chris Nava 
mailto:chrisnav...@gmail.com>> wrote:

Good morning from Denver!

I'm having a hard time trying to find the answer within the documentation
online. I am fairly new to NiFi and was trying to research the possibility
of adjusting the maximum thread count of a process based on the
environment and the node size.  We have the following code and I FEEL like
doing some kind of request to see how many nodes are on the given process
and then replacing the thread_count with some equation based on the return
is the way to go. Am I on the right track or way off-base here?:

def update_thread():
   """ Updates the overall thread count for NiFi """
   if (os.getenv('NIFI_THREAD_OVERRIDE') == 'true'):
   thread_count = int(os.getenv('NIFI_THREAD_COUNT'))
   else:
   thread_count = int(round(multiprocessing.cpu_count()*0.8*4))
   api = nipyapi.nifi.apis.controller_api.ControllerApi()
   controller = api.get_controller_config()
   controller.component =
nipyapi.nifi.models.controller_configuration_dto.ControllerConfigurationDTO(thread_count)
   api.update_controller_config(controller)

Setting thread counts based on worker node size

[Chris Nava]

Good morning from Denver!

I'm having a hard time trying to find the answer within the documentation
online. I am fairly new to NiFi and was trying to research the possibility
of adjusting the maximum thread count of a process based on the
environment and the node size.  We have the following code and I FEEL like
doing some kind of request to see how many nodes are on the given process
and then replacing the thread_count with some equation based on the return
is the way to go. Am I on the right track or way off-base here?:

def update_thread():
""" Updates the overall thread count for NiFi """
if (os.getenv('NIFI_THREAD_OVERRIDE') == 'true'):
thread_count = int(os.getenv('NIFI_THREAD_COUNT'))
else:
thread_count = int(round(multiprocessing.cpu_count()*0.8*4))
api = nipyapi.nifi.apis.controller_api.ControllerApi()
controller = api.get_controller_config()
controller.component =
nipyapi.nifi.models.controller_configuration_dto.ControllerConfigurationDTO(thread_count)
api.update_controller_config(controller)

Re: Require access to MiNiFi confluence page

[Joe Witt]

Gabor. - I got you all set.  Thanks

On Mon, Jan 25, 2021 at 7:30 AM Gábor Gyimesi  wrote:

> Hi,
>
> I am a contributor to the Apache MiNiFi project currently working on an
> additional pause-resume feature on the C2 protocol. I would like to update
> the C2 protocol documentation with this feature on the page
> https://cwiki.apache.org/confluence/display/MINIFI/C2+Design for which I
> would like to ask access to with the username lordgamez.
>
> BR,
> Gabor
>

Require access to MiNiFi confluence page

[Gábor Gyimesi]

Hi,

I am a contributor to the Apache MiNiFi project currently working on an
additional pause-resume feature on the C2 protocol. I would like to update
the C2 protocol documentation with this feature on the page
https://cwiki.apache.org/confluence/display/MINIFI/C2+Design for which I
would like to ask access to with the username lordgamez.

BR,
Gabor

Re: [DISCUSS] Release of Apache NiFi 1.13.0

[Joe Witt]

Team,

There is a TZ handling issue https://issues.apache.org/jira/browse/NIFI-8023
that needs a resolution and there is also a new issue that has emerged on
Github Actions/CI https://github.com/apache/nifi/actions which is resulting
in broken builds.  This emerged in the latest Java 11 Azul release and in
Github CI.  I've tried to replicate with no success on a local linux
machine with the exact same JVM/version.

Once these are wrapped I'll initiate the RC.  Great work to all involved in
helping get the release candidate this close.

Thanks
Joe

On Thu, Jan 21, 2021 at 8:30 AM Joe Witt  wrote:

> Team,
>
> A couple 1.13 tagged items still lingering.  Going to either kick them out
> or merge them and get going.  Hopefully an RC today or tomorrow.
>
> Thanks
>
> On Wed, Jan 20, 2021 at 7:21 AM Mike Thomsen 
> wrote:
>
>> Bug confirmed. Working on a PR now.
>>
>> On Wed, Jan 20, 2021 at 8:45 AM Mike Thomsen 
>> wrote:
>> >
>> > This morning, I think I found a serious bug in PutCassandraRecord WRT
>> > writing binary data to Cassandra. I'm debugging now. If it's really a
>> > bug, I'd appreciate it if we could squeak a fix into 1.13.
>> >
>> > On Tue, Jan 19, 2021 at 9:54 AM Joe Witt  wrote:
>> > >
>> > > Team,
>> > >
>> > > I'm going to initiate RM processes for 1.13. I'll be watching closely
>> for
>> > > currently tagged 1.13 items to include.  If anything is dragging I'll
>> just
>> > > remove it from 1.13 and we'll pick it up later.  Other items that get
>> > > dev/reviewer traction before I cut the RC we can include.  I will
>> > > *probably/hopefully* start that process first thing tomorrow morning.
>> > >
>> > > https://issues.apache.org/jira/projects/NIFI/versions/12348700 is
>> what i'm
>> > > working from.
>> > >
>> > > Thanks
>> > >
>> > > On Fri, Jan 15, 2021 at 6:12 PM Waleed Aibany 
>> wrote:
>> > >
>> > > > Hi,
>> > > >
>> > > > I did rebase to this old PR
>> https://github.com/apache/nifi/pull/4167
>> > > > Is it possible to include it in 1.13.0
>> > > >
>> > > > Thank you
>> > > >
>> > > > On 2021/01/05 15:03:32, Pierre Villard > >
>> > > > wrote:
>> > > > > Hi all,
>> > > > >
>> > > > > Happy new year to everyone, hope you all had a great time. I'm
>> bumping
>> > > > this
>> > > > > thread up as I think it'd make a lot of sense to release NiFi
>> 1.13.0. We
>> > > > > have a ton of bug fixes and improvements that would be very
>> useful for
>> > > > our
>> > > > > community.
>> > > > >
>> > > > > Let's try to see if there are any blockers for a release and
>> start the
>> > > > > process:
>> > > > >
>> > > >
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20NIFI%20AND%20resolution%20%3D%20Unresolved%20AND%20fixVersion%20%3D%201.13.0%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
>> > > > >
>> > > > > Thanks!
>> > > > >
>> > > > >
>> > > > > Le mar. 1 déc. 2020 à 14:49, Bryan Bende  a
>> écrit :
>> > > > >
>> > > > > > Chris,
>> > > > > >
>> > > > > > The illegal reflective access warnings are expected when
>> running on
>> > > > Java
>> > > > > > 11.
>> > > > > >
>> > > > > > The second warning about the provenance repo seems like a
>> possible
>> > > > > > issue due to some recent refactorings made for stateless, we
>> should
>> > > > > > take a look at that.
>> > > > > >
>> > > > > > Thanks,
>> > > > > >
>> > > > > > Bryan
>> > > > > >
>> > > > > > On Tue, Dec 1, 2020 at 3:58 AM Chris Sampson
>> > > > > >  wrote:
>> > > > > > >
>> > > > > > > Just spotted the following during startup of NiFi
>> 1.13.0-SNAPSHOT on
>> > > > my
>> > > > > > > local machine, which might be worth looking at (although I've
>> not
>> > > > deleted
>> > > > > > > and re-created the binaries in a little while, so I could have
>> > > > something
>> > > > > > > wrong in my /lib folder or such):
>> > > > > > >
>> > > > > > > 2020-12-01 08:52:48,382 ERROR [NiFi logging handler]
>> > > > > > org.apache.nifi.StdErr
>> > > > > > > WARNING: An illegal reflective access operation has occurred
>> > > > > > > 2020-12-01 08:52:48,383 ERROR [NiFi logging handler]
>> > > > > > org.apache.nifi.StdErr
>> > > > > > > WARNING: Illegal reflective access by
>> > > > > > > com.sun.xml.bind.v2.runtime.reflect.opt.Injector
>> > > > > > >
>> (file:.../nifi-1.13.0-SNAPSHOT/lib/java11/jaxb-impl-2.3.0.jar) to
>> > > > method
>> > > > > > >
>> java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int)
>> > > > > > > 2020-12-01 08:52:48,383 ERROR [NiFi logging handler]
>> > > > > > org.apache.nifi.StdErr
>> > > > > > > WARNING: Please consider reporting this to the maintainers of
>> > > > > > > com.sun.xml.bind.v2.runtime.reflect.opt.Injector
>> > > > > > > 2020-12-01 08:52:48,383 ERROR [NiFi logging handler]
>> > > > > > org.apache.nifi.StdErr
>> > > > > > > WARNING: Use --illegal-access=warn to enable warnings of
>> further
>> > > > illegal
>> > > > > > > reflective access operations
>> > > > > > > 2020-12-01 08:52:48,383 ERROR [NiFi logging handler]
>> > > > > > org.apache.nifi.StdErr
>> > > > > > > WARNING: All i

Submitting a new NiFi processor

[Andrea Castello]

Hello,

   I've developed a NiFi processor that enriches data from an HTTP request
(User-Agent and other HTTP request headers) with data from WURFL device
detection API (detect device make and model and provide attributes and
capabilities of the browsing device).

The device detection microservice (docker images, AWS ami or Azure/GCP vms)
is accessed via a client API - the WM Java client
 - which is
released under the Apache License but requires a commercial WURFL
Microservice to work. So, before forking the NIfi repo and submitting any
PR, I'd like to understand if connecting to a service with a commercial
license may cause my contribution to be refused.

Thank you. Best,

Andrea Castello

Disclosure: I am one of the developers of WURFL Microservice, whose client
library has been used to develop this processor.