Re: Queue Prioritization

[Endre Kovács]

Please upload it to somewhere and link it here. ASF mailing lists are striping 
the attachments/images.

> On 2 Mar 2020, at 13:35, Midhun Mohan  wrote:
> 
> Hi Eduardo,
> Can you please send the Image  again, I am not able to open it
> 
> On Mon, 2 Mar 2020 at 18:02, Eduardo Fontes  >
> wrote:
> 
>> Hi Midhun,
>> 
>> I think this flow will do what you want.
>> 
>> [image: image.png]
>> 
>> Be advised that use of Wait/Notify will slow down the average throughput
>> of your flow because the aditional steps with the Cache.
>> 
>> 
>> 
>> 
>> On Sun, Mar 1, 2020 at 12:57 PM Midhun Mohan 
>> wrote:
>> 
>>> So I would like to know your suggestions on how to implement wait and
>>> notify.
>>> 
>>> In my case messages will flow one after another
>>> 
>>> If a failure occurs I need to retry the failure before the orginal ones,
>>> and notify users if it fails again.
>>> 
>>> 
>>> On Sun, 1 Mar 2020, 6:31 pm Emanuel Oliveira,  wrote:
>>> 
 Hi,
 
 Invokehttp have 4 exits
 - success
 - error retry -> loop/count retries (updateattribite) -> if retries >
>>> max
 then logmessage(error) flow stops here ff discarded  Otherwise loop
>>> back to
 invokehttp.
 - error type 2 (bad password etc, )
 - error type 3 (connectivity problems) you may want also to loop this
>>> one
 with some delay , count retries and exit /log/stop after max.
 
 I just highly advise prior to invokehttp to have an updateattribute
 deleting all attributes that don't start with some prefix.. to be sure
>>> your
 ff looping doesnt keep adfing different errors, attributes wriiten by
 invoke http.
 
 As good practice we prefix name all control attributes with
>>> flow_xxx so
 our "cleanup," Updateattribute before Invokehttp have special regex
>>> sayng
 "delete all attributes that don't start with flow_
 (If you interested tomorrow when im back in office i csn share the
>>> regex a
 bit tricky).
 
 Hope this helps, i just would love all processors to be as good as
 invokehttp providing hard multipke exit points for specifics + all the
>>> nice
 write attributes invokehttp adds that ptovide high details on errors or
 success (atttibutes created/added by invokehttp to ff are prefixed with
 invoke.x
 
 Cheers,
 Emanuel O.
 
 
 
 
 
 
 On Sun, 1 Mar 2020, 12:42 Midhun Mohan,  wrote:
 
> Hi my use case is
> 
> When a invokehttp processor fails I need to stop the incoming message
>>> and
> retry the failed message 2 times and if it is success start the
>>> execution
> normally or if it fails again need to notify admins and stop the main
> invoke http processor.
> 
> 
> 
> Hope you can get the scenario from this
> 
> On Sun, 1 Mar 2020, 5:30 pm Eduardo Fontes, >>> 
> wrote:
> 
>> Hi Midhun,
>> 
>> You can use DistributedMapCacheServer which starts a cache service
 within
>> NiFi's process, but I not recommend in production. Instead you could
 use
>> RedisConnectionPoolService.
>> 
>> To use solution Wait/Notify the NiFi must know the number of
>>> flowfiles
 to
>> process in one queue before start another queue. If it is a batch/on
> demand
>> flow I think it's easier. If it is a continuos/streamming flow you
>>> must
>> define a window, by number of flowfiles or by time, to switch
>>> between
>> queues.
>> 
>> Please let me know what is your case.
>> 
>> On Fri, Feb 28, 2020 at 2:39 PM Midhun Mohan <
>>> midhun.mo...@esginc.us>
>> wrote:
>> 
>>> I think wait and notify can solve my problem. I checked the
 documents ,
>>> which is bit unclear to me. Can anyone help me?
>>> 
>>> Just to know few things like whether we need to configure a cache
> server,
>>> if so which one is best
>>> And also getting confused on how to configure wait and notify
>>> signals
>>> 
>>> On Fri, 28 Feb 2020, 10:59 pm Eduardo Fontes, <
> eduardo.fon...@gmail.com>
>>> wrote:
>>> 
 Hi! Maybe you should try another approach, like Wait/Notify or
>> configure
 prioritizer just before the funnel, depending on your demand.
 
 On Fri, Feb 28, 2020 at 11:40 AM Midhun Mohan <
> midhun.mo...@esginc.us>
 wrote:
 
> Hi all,
> How to prioritize one queue before a funnel?
> 
> I need to make sure one queue is emptied before another
>>> queue is
>> taken
> over for executing, I tried with a priority attribute but it
 fails
>>> when a
> queue with less priority has messages before the queue with
>>> high
 priority.
> 
> --
> 
> 
> Regards,
> Midhun Mohan
> 
 
>>> 
>> 
> 
 
>>> 
>> 
> 
> -- 
> 
> 
> Regards,
> Midhun Mohan

Re: Importing Maven Projects Error

[Endre Kovács]

Hi Richie,

Images do not make it through this apache mailing list.
Could you please upload the picture somewhere and provide a link to it? 
Or Provide the error message as text?

Otherwise what,  JDK do you use (and selected to be used in eclipse)?

Best regards,
Endre

> On 15 Jan 2020, at 17:53, Richie Howell  
> wrote:
> 
> Hi Everyone,
>  
> When I import the NiFi project into Eclipse as an existing maven project I 
> get this error, I’ve done some research but haven’t found a solution.
>  
> 
>  
> After importing, I was actually able to build the project in Eclipse 
> successfully, without any errors. However, there are still many project 
> problems and when I try to Update Maven Project I receive that same JAX-RS 
> error. Has anyone else encountered this or know a fix?
>  
> Thanks,
> Richie

Re: NIFI PutDatabaseRecord question

[Endre Kovács]

Hi Imre,

The images usually do not pass through the apache mailing lists.
Please upload the images and provide a link to them, or describe their content 
textually.

Best regards,
Endre Zoltan Kovacs


> On 14 Jan 2020, at 15:10,   wrote:
> 
> Hi,
>  
> I’m struggling with a problem in building a simple flow, that is Nifi should 
> read records form a postgreSQL(EnterpiseDB) table
> and sholud put data into a Oracle table:
> 
> The table and field names are all the same in the source and the target DB .
> The PutDatabaseRecord processor looks like this:
> 
> I get the following error message:
> none of the fields in the records map to the columns defined by the 
> target_table_ora
>  
> Would you please suggest a solution.
>  
> Best regards
>  
> Imre Fábián
> 
>  
>  
>  
>  
>  
>  
> Fontos!  
> Ezt az elektronikus levelet - beleértve az esetleges mellékleteit is - a 
> levél feladója a címzettként megjelölt személynek szánta. Ha Ön nem címzettje 
> a levélnek kérjük, haladéktalanul értesítse a feladót és a levelet törölje! 
> Felhívjuk szíves figyelmét, hogy az üzenet tartalmának nyilvánosságra 
> hozatala, másolása, terjesztése az Aegon Magyarország engedélye nélkül 
> szigorúan tilos.  
> ATTENTION:  
> This email message (including any attachments) may be confidential and is 
> intended solely for the use of the individual or entity to which it is 
> addressed. If you have received it by mistake please notify the sender by 
> return e-mail and delete this message (including any attachments) from your 
> system. You are hereby notified that any disclosure, reproduction, 
> distribution or use of this message without the authority of AEGON N.V. is 
> strictly prohibited and that no rights can be derived from such distribution.

Re: Cluster question - interval vs external routing

[Endre Kovács]

Hi,

Have you considered putting a proxy in front, for UI / external access
and then let the nodes bind to their internal addresses?

https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#proxy_configuration
 


Best regards,
Endre


> On 7 Nov 2019, at 03:49, Phil H  wrote:
> 
> Hi there,
> 
> We have two addresses on each of our NiFi cluster hosts. One is externally
> facing, which we use to access the UI, and connect to receiving processors
> from the wider network. We are trying to configure NiFi to use the internal
> network for its cluster traffic. However if I use the internal address IPs
> in the nifi.cluster.node.address and nifi.remote.input.host properties, I
> end up getting two entries for each host in the cluster (both showing the
> external address) and only one of them will be online and connected. The
> other shows as disconnected.
> 
> What am I missing here?
> 
> Regards,
> Phil

Re: Restore nifi after /database_repository/ folder being cleared

[Endre Kovács]

Hi,

I think the problem is that the path:


/disk_r5_01/nifi-1.7.1/database_repository/nifi-user-keys.lock.db


is not writable by the user running NiFi.

this database holds the keys which can verify the bearer / jwt token,
passed by each, authenticated client request.

Best regards,
Endre


On Tue, Aug 20, 2019, 12:37 PM Кочетов Николай Александрович <
na.koche...@severstal.com> wrote:

> Hello, guys and girls!
>
> Could you kindly help me to restore my nifi instance and get it up and
> running again after the content of the nifi/database_repository/ folder was
> being removed?
> Now, while I'm trying to start it I'm getting the error
> (nifi-bootstrap.log):
>
> 2019-08-19 18:55:32,970 ERROR [NiFi logging handler]
> org.apache.nifi.StdErr Failed to start web server: Error creating bean with
> name
> 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration':
> Unsatisfied dependency expressed through method
> 'setFilterChainProxySecurityConfigurer' parameter 1; nested exception is
> org.springframework.beans.factory.BeanExpressionException: Expression
> parsing failed; nested exception is
> org.springframework.beans.factory.UnsatisfiedDependencyException: Error
> creating bean with name
> 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied
> dependency expressed through method 'setJwtAuthenticationProvider'
> parameter 0; nested exception is
> org.springframework.beans.factory.BeanCreationException: Error creating
> bean with name 'jwtAuthenticationProvider' defined in class path resource
> [nifi-web-security-context.xml]: Cannot resolve reference to bean
> 'jwtService' while setting constructor argument; nested exception is
> org.springframework.beans.factory.BeanCreationException: Error creating
> bean with name 'jwtService' defined in class path resource
> [nifi-web-security-context.xml]: Cannot resolve reference to bean
> 'keyService' while setting constructor argument; nested exception is
> org.springframework.beans.factory.BeanCreationException: Error creating
> bean with name 'keyService' defined in class path resource
> [nifi-administration-context.xml]: Cannot resolve reference to bean
> 'keyTransactionBuilder' while setting bean property 'transactionBuilder';
> nested exception is
> org.springframework.beans.factory.BeanCreationException: Error creating
> bean with name 'keyTransactionBuilder' defined in class path resource
> [nifi-administration-context.xml]: Cannot resolve reference to bean
> 'keyDataSource' while setting bean property 'dataSource'; nested exception
> is org.springframework.beans.factory.BeanCreationException: Error creating
> bean with name 'keyDataSource': FactoryBean threw exception on object
> creation; nested exception is org.h2.jdbc.JdbcSQLException: Error opening
> database: "Could not save properties
> /disk_r5_01/nifi-1.7.1/database_repository/nifi-user-keys.lock.db"
> [8000-176]
>
> And quite a long stacktrace in nifi.app.log:
>
> 2019-08-19 18:55:32,971 WARN [main] org.apache.nifi.web.server.JettyServer
> Failed to start web server... shutting down.
> org.springframework.beans.factory.UnsatisfiedDependencyException: Error
> creating bean with name
> 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration':
> Unsatisfied dependency expressed through method
> 'setFilterChainProxySecurityConfigurer' parameter 1; nested exception is
> org.springframework.beans.factory.BeanExpressionException: Expression
> parsing failed; nested exception is
> org.springframework.beans.factory.UnsatisfiedDependencyException: Error
> creating bean with name
> 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied
> dependency expressed through method 'setJwtAuthenticationProvider'
> parameter 0; nested exception is
> org.springframework.beans.factory.BeanCreationException: Error creating
> bean with name 'jwtAuthenticationProvider' defined in class path resource
> [nifi-web-security-context.xml]: Cannot resolve reference to bean
> 'jwtService' while setting constructor argument; nested exception is
> org.springframework.beans.factory.BeanCreationException: Error creating
> bean with name 'jwtService' defined in class path resource
> [nifi-web-security-context.xml]: Cannot resolve reference to bean
> 'keyService' while setting constructor argument; nested exception is
> org.springframework.beans.factory.BeanCreationException: Error creating
> bean with name 'keyService' defined in class path resource
> [nifi-administration-context.xml]: Cannot resolve reference to bean
> 'keyTransactionBuilder' while setting bean property 'transactionBuilder';
> nested exception is
> org.springframework.beans.factory.BeanCreationException: Error creating
> bean with name 'keyTransactionBuilder' defined in class path resource
> [nifi-administration-context.xml]: Cannot resolve reference to bean
> 'keyDataSource' while setting bean property 'dataSource'; nested exception
> is 

Re: Need some help in creating Regex Expression

[Endre Kovács]

Hi,

Also when in doubt, you can always experiment, check and learn regex in online 
places like:

https://regex101.com/#pcre 
or
https://regexr.com/ 

where you can also specify a test string you want to match for.
Best regards,
Endre

> On 31 May 2019, at 18:47, ravva  wrote:
> 
> Hello - I have input files as below 
> 
> CanadaCST_prices.csv
> CanadaEST_prices.csv
> CanadaMST_prices.csv
> CanadaPST_prices.csv
> cst_prices.csv
> est_prices.csv
> mst_prices.csv
> pst_prices.csv
> 
> I have to write 2  regex expressions where in one of them I need to get only
> the file names that are starting with cst,est,mst,pst . In the second regex
> I need to pull the files that are starting with
> CanadaCST,CanadaEST,CanadaPST,CanadaMST. 
> 
> Can someone suggest on how to create a regex for this scenario.
> 
> Thanks,
> 
> 
> 
> 
> 
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: Execute SQL support for nvarchar(max) datatype

[Endre Kovács]

Hi Venu,
I belive it would only be possible if the PDF would be base64 encoded
before DB insertion, and then in NiFi it would need to be base64 decoded.
Seems possible, but adds a bit of computing overhead to the flow.
Best regards,
Endre

On Thu, Apr 18, 2019, 5:09 PM Venugopal Iyengar 
wrote:

> Thanks Peter.
> We are trying to avoid using nvarchar going forward. However, for some
> schema it is implemented that way by some consultants. If you can give me
> the steps to debug that would be helpful. Is it possible to get this
> working by using some converters as part of the flow? I appreciate your
> help on this.
>
> Venu
>
> On Tue, Apr 16, 2019 at 5:23 AM Peter Turcsanyi
>  wrote:
>
> > Hi Venu,
> >
> > If I'm not wrong, you're using MS SQL Server and nvarchar / varbinary are
> > the database column types.
> > nvarchar can be used for storing character data (strings) but it is not
> > suitable for binary data (like pdf).
> >
> > I think NiFi tries to handle the bytes of the pdf file as Unicode
> > characters. It could be debugged but my short answer is that nvarchar
> > should not be used for binary data.
> >
> > Regards,
> > Peter
> >
> > On Mon, Apr 15, 2019 at 10:05 PM Venugopal Iyengar <
> > iyengar.g.v...@gmail.com>
> > wrote:
> >
> > > Hello there,
> > > I am using Execute SQL, Split AVro and put file processors to read some
> > > pdf documents stored in SQL database and store that in the file system.
> > >
> > > When I use varbinary(max)  I was able to pull and view the pdf without
> > any
> > > issues.
> > >
> > > When the datatype is nvarchar(max) I am unable to open the pdf file.
> > >
> > > I appreciate if some body can shed some light on this. How can I debug
> > > this?
> > >
> > > Thanks
> > > Venu
> > >
> > >
> > > [image: image.png]
> > >
> >
>