Re: Most memory-efficient way in NiFi to fetch an entire RDBMS table?

[Mike Thomsen]

Thanks, Matt.

On Wed, Jun 22, 2022 at 10:46 AM Matt Burgess  wrote:
>
> Mike,
>
> I recommend QueryDatabaseTableRecord with judicious choices for the
> following properties:
>
> Fetch Size: This should be tuned to return the most number of rows
> without causing network issues such as timeouts. Can be set to the
> same value as Max Rows Per Flow File ensuring one fetch per outgoing
> FlowFile
> Max Rows Per Flow File: This should be set to a reasonable number of
> rows per FlowFile, maybe 100K or even 1M if that doesn't cause issues
> (see above)
> Output Batch Size: This is the key to doing full selects on huge
> tables, as it allows FlowFiles to be committed to the session and
> passed downstream while the rest of the fetch is being processed. In
> your case if you set Max Rows to 100K then this could be 10, or if you
> set it to 1M it could be 1. Note that with this property set, the
> maxvalue.* and fragment.count attributes will not be set on these
> FlowFiles, so you can't merge them.  I believe the maxvalue state will
> still be updated even if this property is used, so it should turn into
> an incremental fetch after the first full fetch is complete.
>
> Regards,
> Matt
>
> On Wed, Jun 22, 2022 at 10:00 AM Mike Thomsen  wrote:
> >
> > We have a table with 68M records that will blow up to over 250M soon,
> > and need to do a full table fetch on it. What's the best practice for
> > efficiently doing a partial or full select on it?
> >
> > Thanks,
> >
> > Mike

Re: Most memory-efficient way in NiFi to fetch an entire RDBMS table?

[Matt Burgess]

Mike,

I recommend QueryDatabaseTableRecord with judicious choices for the
following properties:

Fetch Size: This should be tuned to return the most number of rows
without causing network issues such as timeouts. Can be set to the
same value as Max Rows Per Flow File ensuring one fetch per outgoing
FlowFile
Max Rows Per Flow File: This should be set to a reasonable number of
rows per FlowFile, maybe 100K or even 1M if that doesn't cause issues
(see above)
Output Batch Size: This is the key to doing full selects on huge
tables, as it allows FlowFiles to be committed to the session and
passed downstream while the rest of the fetch is being processed. In
your case if you set Max Rows to 100K then this could be 10, or if you
set it to 1M it could be 1. Note that with this property set, the
maxvalue.* and fragment.count attributes will not be set on these
FlowFiles, so you can't merge them.  I believe the maxvalue state will
still be updated even if this property is used, so it should turn into
an incremental fetch after the first full fetch is complete.

Regards,
Matt

On Wed, Jun 22, 2022 at 10:00 AM Mike Thomsen  wrote:
>
> We have a table with 68M records that will blow up to over 250M soon,
> and need to do a full table fetch on it. What's the best practice for
> efficiently doing a partial or full select on it?
>
> Thanks,
>
> Mike

Most memory-efficient way in NiFi to fetch an entire RDBMS table?

[Mike Thomsen]

We have a table with 68M records that will blow up to over 250M soon,
and need to do a full table fetch on it. What's the best practice for
efficiently doing a partial or full select on it?

Thanks,

Mike

Re: Nifi Adding more users other than Default

[Ben .T.George]

Hello,

i was trying to configure AD integration by following link
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldap_login_identity_provider

file: nifi.properties - changed like below
nifi.security.user.login.identity.provider=ldap-provider

file: login-identity-providers.xml


ldap-provider
org.apache.nifi.ldap.LdapProvider
SIMPLE

OU=Application
Servers,DC=example,DC=com
Password











FOLLOW
10 secs
10 secs

ldap://corp.mtc.com.kw:389
DC=example,DC=com
sAMAccountName

USE_DN
12 hours



Error logged:

ERROR [NiFi logging handler] org.apache.nifi.StdErr Failed to start web
server: Error creating bean with name 'niFiWebApiConfiguration':
BeanPostProcessor before instantiation of bean failed; nested exception is
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'metaDataSourceAdvisor': Cannot resolve reference to bean
'methodSecurityMetadataSource' while setting constructor argument; nested
exception is
org.springframework.beans.factory.UnsatisfiedDependencyException: Error
creating bean with name
'org.springframework.security.config.annotation.method.configuration.GlobalMethodSecurityConfiguration':
Unsatisfied dependency expressed through method 'setObjectPostProcessor'
parameter 0; nested exception is
org.springframework.beans.factory.UnsatisfiedDependencyException: Error
creating bean with name
'org.apache.nifi.web.security.configuration.AuthenticationSecurityConfiguration':
Unsatisfied dependency expressed through constructor parameter 2; nested
exception is org.springframework.beans.factory.BeanCreationException: Error
creating bean with name 'authorizer': FactoryBean threw exception on object
creation; nested exception is
org.apache.nifi.authorization.exception.AuthorizerCreationException:
SingleUserAuthorizer requires
org.apache.nifi.authentication.single.user.SingleUserLoginIdentityProvider
to be configured
2022-06-22 15:00:21,286 ERROR [NiFi logging handler] org.apache.nifi.StdErr
Shutting down...

On Wed, Jun 22, 2022 at 12:30 AM Ben .T.George 
wrote:

> Thanks, I'll check it out.
>
> Regards
> Ben
>
> On Wed, 22 Jun 2022, 12:26 am Josh Friberg,  wrote:
>
>> Yes. This is done with LDAP.
>> The documentation may be a bit intimidating but I know someone did a
>> really nice article about how to do it that I followed but that domain is
>> no longer active.  A quick google search for "nifi active directory
>> authentication" will give you lots of information.
>>
>> On Tue, Jun 21, 2022 at 12:41 PM Ben .T.George 
>> wrote:
>>
>>> Hello,
>>>
>>> Thanks for the explanation.
>>>
>>> Is it possible to configure active directory based authentication.?
>>>
>>> regards,
>>> ben
>>>
>>> On Tue, Jun 21, 2022 at 8:31 PM Josh Friberg 
>>> wrote:
>>>
 The default used to be NO authentication.  It was then changed to
 single user authentication for localhost only to add a bit more security
 out of the box.

 If you want multiple users for the system then you need to select one
 of the authentication methods provided in the documentation.


 On Tue, Jun 21, 2022 at 12:23 PM Ben .T.George 
 wrote:

> Hello,
>
> Thanks for the response.
>
> one question, is it possible to add more than one single user?
>
> or as the name/provider says only one single user?
>
> Regards,
> ben
>
> On Tue, Jun 21, 2022 at 8:18 PM Josh Friberg 
> wrote:
>
>>
>> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#user_authentication
>>
>> On Tue, Jun 21, 2022 at 12:06 PM Ben .T.George 
>> wrote:
>>
>>> Hello,
>>>
>>> I am very new to Nifi and somehow maneged to install it, Currently i
>>> using the username which i taken from log(created while installing)
>>>
>>> How can i add more users for accessing platform?
>>>
>>> regards,
>>> Ben
>>>
>>
>
> --
> Yours Sincerely
> Ben.T.George
> Phone : +965 - 50629829 / 94100799
>
> *" Live like you will die tomorrow, learn like you will live forever "*
>

>>>
>>> --
>>> Yours Sincerely
>>> Ben.T.George
>>> Phone : +965 - 50629829 / 94100799
>>>
>>> *" Live like you will die tomorrow, learn like you will live forever "*
>>>
>>

-- 
Yours Sincerely
Ben.T.George
Phone : +965 - 50629829 / 94100799

*" Live like you will die tomorrow, learn like you will live forever "*