Re: Multiplying a record filed value by a constant

are? > > Kind Regards, > > Nathan > > Get Outlook for iOS <https://aka.ms/o0ukef> > -- > *From:* Vibhath Ileperuma > *Sent:* Sunday, July 31, 2022 8:49:04 AM > *To:* users@nifi.apache.org > *Subject:* Multiplying a record filed value by a constant > > Hi

Multiplying a record filed value by a constant

1, container=default, section=1], offset=0, length=23202895],offset=0,name=abc.csv,size=23202895]; will route to failure: For input string: ""* Could you please let me know whether there is a way to mutiply the record only if it is not null. Thank You. Best Regards, Vibhath Ileperuma.

Nifi custom processor to consume two flow files at once.

Hi All, We are planning to develop a custom Nifi processor which consumes two files at once. We have a set of files which contains two types of data; say 'A' type data and 'B' type data. This processor receives these two type files from two upstream connections. Processor needs to get one file

Handling floating point numbers

Hi All, I have created a Nifi flow to query from Postgresql database and write data into csv files. However, I noticed that the floating point values (Double values) can be changed slightly when writing to csv files. For an example, value 4313681553.3 was written as 4313681553.292. Since some

Re: FetchS3Object fails to fetch small files from S3

Hi Joey, It happens only for 0B files. Sorry for the confusion caused. Vibhath.

Re: FetchS3Object fails to fetch small files from S3

hen you say small, exactly how small are the files? >> >> -joey >> >> On Aug 16, 2021, at 10:53 PM, Vibhath Ileperuma < >> vibhatharunapr...@gmail.com> wrote: >> >>  >> Hi All, >> >> I'm using Nifi 1.14.0 to fetch a set of files fro

FetchS3Object fails to fetch small files from S3

Hi All, I'm using Nifi 1.14.0 to fetch a set of files from a S3 bucket. The 'FetchS3Object' processor throws the following error only for small files. I tried to fetch the same set of files using Nifi 1.13.2. There was no issue with 1.13.2. Can you please let me know a way to overcome this issue.

Re: Execute Script Processor is not working some random times

Hi Mark, Thank You for the information. It was really helpful to resolve the issue. I was able to increase the disk size and this issue didn't occur. Just for my knowledge, when does nifi start to clean the content repository? Does it always wait till the repository usage percentage limit or

Re: Execute Script Processor is not working some random times

.* On Mon, Jul 26, 2021 at 7:11 PM Mark Payne wrote: > Vibhath, > > This was a known issue in 1.13. It has been fixed in 1.14. Please update > to the latest, and you should be good to go! > > Thanks > -Mark > > On Jul 26, 2021, at 8:01 AM, Vibhath Ileperuma < > v

Re: Execute Script Processor is not working some random times

Thank You Mark for the information. On Mon, Jul 26, 2021 at 7:11 PM Mark Payne wrote: > Vibhath, > > This was a known issue in 1.13. It has been fixed in 1.14. Please update > to the latest, and you should be good to go! > > Thanks > -Mark > > On Jul 26, 2021, at

Execute Script Processor is not working some random times

Hi All, I'm using an ExecuteScript processor to write a set of flow file attributes to a csv file located in the local file system. Note that I'm writing attribute values of all the flow files in a single csv file. Even Though this works fine most of the times, in some times, the processor does

Re: Getting a notification once a request is finished

king the status using db query for these tables and Mark as complete > based on db query status > > Thanks, > Asmath > > > On May 19, 2021, at 6:32 AM, Vibhath Ileperuma < > vibhatharunapr...@gmail.com> wrote: > > > >  > > Hi all, > > > > I'm de

Getting a notification once a request is finished

Hi all, I'm developing a NIFI flow to extract data in a database and write into files.This flow starts with a ListenHttp processor. A python script sends a list of table names required to be extracted to the port ListenHttp processor is listening to. NIFI flow has been developed to iterate

Re: Issue when extracting a large dataset into CSV files

the GC.log file and it also shows the large heap growing. I have attached the GC log with this email. Could you please assist me to resolve this issue. Thanks & Regards, Vibhath On Wed, May 12, 2021 at 10:30 AM Vibhath Ileperuma < vibhatharunapr...@gmail.com> wrote: > Hi Matt, &

Re: Issue when extracting a large dataset into CSV files

own the performance too much. > > Regards, > Matt > > [1] https://jdbc.postgresql.org/documentation/head/query.html > > On Mon, May 10, 2021 at 3:41 AM Vibhath Ileperuma > wrote: > > > > Hi Vijay, > > > > Eventhough that property is set to a postive

Re: Issue when extracting a large dataset into CSV files

d in > a single FlowFile. > Which seems to be what is happening in your case. > > > > On May 9, 2021, at 12:11 PM, Vibhath Ileperuma < > vibhatharunapr...@gmail.com> wrote: > > executeSQLRecord > > >

Re: Warnings printed when NIFI is starting

o|__> ///, ,\\ >> \ ` / /// || \\ >> ): :( / \) || (/ >> :o_o:/ \ / >> "-"/ `w - w' >> >> >> On Wed, May 5, 2021 at 10:13 AM Vibhath Ileperuma

Warnings printed when NIFI is starting

Hi All, I tried to start NIFI binary downloaded from apache NIFI website, in a linux environment in which java openjdk is installed. When I start the NIFI, the following warnings are printed. These warnings are written in nifi-boostrap.log file too. WARNING: An illegal reflective access

Accessing NIFI processors through rest api

a template. Is there a way to define the UUIDs which are not changing? Or is it possible to access a processor using the name or a custom defined id ? Thanks & Regards *Vibhath Ileperuma*

Re: Clarification of controller services

ngle thread > at a time will perform the action “doSomethingExpensive” and any other > threads will block, executing only one at a time. > > Thanks > -Mark > > > > On Apr 20, 2021, at 7:59 AM, Vibhath Ileperuma < > vibhatharunapr...@gmail.com> wrote: > > H

Clarification of controller services

. Is it like accessing the same java object from multiple threads? Is there any negative or positive impact on performance when using multiple objects of the same service instead of a single service object? Thanks & Regards *Vibhath Ileperuma* Graduate | ENTC University of Moratuwa E vibhatharu

Re: Nifi throws an error when reading a large csv file

postgre generated CSV from NIFI.? Any workaround for this issue. postgreSQL COPY command documentation : https://www.postgresql.org/docs/13/sql-copy.html Thanks & Regards *Vibhath Ileperuma* On Thu, Apr 15, 2021 at 4:59 AM Mike Thomsen wrote: > I could be totally barking up the wro

Re: Nifi throws an error when reading a large csv file

=c9a981db-0178-1000-363d-c767653a6f34] due to uncaught Exception: java.lang.OutOfMemoryError: Requested array size exceeds VM limit* *java.lang.OutOfMemoryError: Requested array size exceeds VM limit* Thanks & Regards *Vibhath Ileperuma* On Wed, Apr 14, 2021 at 7:47 PM Otto Fowler wrote: &g

Nifi throws an error when reading a large csv file

essor Administratively Yielded for 1 sec: java.lang.OutOfMemoryError: Requested array size exceeds VM limit" I would be grateful if someone can suggest a way to overcome this error. Thanks & Regards *Vibhath Ileperuma*

Understanding few points regarding a NIFI flow

Hi All. I'm developing a NIFI flow to fetch csv files from a S3 bucket and write those data to Apache kudu. I need to understand a few things regarding this flow. 1. Does ListS3 processor list down S3 objects in alphabetical order? Or is there any other specific order it uses to list down

Re: Writing custom logs

to different log files based on the table name. Thanks & Regards *Vibhath Ileperuma* On Thu, Mar 25, 2021 at 4:18 PM Etienne Jouvin wrote: > Hello. > > What you can do it to prefix your log message, with something like > [MY_TABLE] > and then in the logback configuration use

Writing custom logs

. But it's better if I can maintain log files separately for different tables. Is there a way to do this in NIFI. Thanks & Regards *Vibhath Ileperuma*

Issue when using parquet record set writer

, Is it possible for a row group to exceed that memory limit.? Could you please explain how the parquertRecorSetWriter behaves. Thanks & Regards *Vibhath Ileperuma*

Re: Writing parquet files to S3

, it always merges 19 files and generates files of size 415 MB - 417 MB. I'm using NIFI 1.13.1. Could you please let me know how to resolve this issue. Thanks & Regards *Vibhath Ileperuma* On Fri, Mar 19, 2021 at 8:45 PM Bryan Bende wrote: > Hello, > > What would the reason be

Writing parquet files to S3

required for a 256MB row group, multiple parquet files should be generated. further,* It's better if we can merge multiple csvs when a csv file contains a very small amount of data*). I would be grateful if you could suggest a way to do this. Thanks & Regards *Vibhath Ileperuma*

Writing parquet files to S3

required for a 256MB row group, multiple parquet files should be generated). I would be grateful if you could suggest a way to do this. Thanks & Regards *Vibhath Ileperuma*

Re: Data duplication When NIFI is restarted

where it stopped. Thanks & Regards *Vibhath Ileperuma* On Wed, Mar 17, 2021 at 5:51 PM Pierre Villard wrote: > Hi Vibhath, > > How is NiFi terminated / restarted ? > > Thanks, > Pierre > > Le mer. 17 mars 2021 à 15:04, Vibhath Ileperuma < > vibhatharunapr..

Data duplication When NIFI is restarted

& Regards *Vibhath Ileperuma*

Re: Adding a new column which contains a column value of the previous row.

wfile is very non-scalable, as the data must be processed sequentially. > If your data volumes are high at all, you’ll certainly want to spend some > time thinking about this approach to see if there’s a better way to attack > your problem. > > Hope this is helpful. > > Thanks &g

Adding a new column which contains a column value of the previous row.

to implement this logic. If it is required to create a custom processor, could you please suggest the best practices for state management between two flow files. Thanks & Regards *Vibhath Ileperuma*

Re: NiFI fronend disconnects after some time

Hi, I have configured heap size to 25GB On Mon, 1 Mar 2021, 12:18 pm Jorge Machado, wrote: > How much memory do you have on your instance? > I would say the fetch size should be bigger then the Max Rows… > > > On 1. Mar 2021, at 06:37, Vibhath Ileperuma > wrote: >

NiFI fronend disconnects after some time

, but there were no errors. I would be grateful if you could suggest to me the reason for this issue and a way to overcome this. Thanks & Regards *Vibhath Ileperuma*