NiFi 1.9.0 : Unable to change the state of a processor from DISABLED to ENABLED from rest api

to go into the NiFi UI and enable that processor manually. This is not feasible in the production environment. Any help would be appreciated. Regards, Mohit

NiFi 1.9.0: ListFile throws NoSuchFileException

folder to the processed folder. 3. Go ahead with the flow. It's a 3 Node NiFi 1.9.0 cluster, and the flow is reading from the NFS mount. ListFile is scheduled to run on the primary node. Any help would be appreciated. Thanks, Mohit

Re: Nifi takes too long to start(~ 30 minutes)

and NiFi is up after 2-3 minutes max. There are 5 custom processors being used in the flow. Thanks, Mohit On Wed, Aug 19, 2020 at 5:07 AM Andy LoPresto wrote: > Mohit, > > I’m a little confused by some of the details you report. Initially you > said there were ~100 flows with 2

Re: Nifi takes too long to start(~ 30 minutes)

. On a similar note, NiFi Rest api response is also quite slow when there are a lot of components. Would it improve by disabling the components which are not in use? Or is there something else that could have been causing the issue? Thanks, Mohit On Thu, Aug 13, 2020 at 9:49 PM Pierre Villard wrote: >

Re: Nifi takes too long to start(~ 30 minutes)

Hi Brandon, There are no flow files in the queue. Thanks, Mohit On Thu, Aug 13, 2020 at 9:56 PM Brandon DeVries wrote: > Mohit, > > How many flowfiles are currently on the instance? Sometimes a very large > number of flowfiles can result in slower start times.

Nifi takes too long to start(~ 30 minutes)

started in a minute. Is this an expected behaviour that startup time is directly proportional to the number of processors in the Canvas? Or is there a way to reduce the NiFi startup time? Any leads would be appreciated. Thanks, Mohit

Re: Updating the Concurent Tasks of a NiFi processor using Rest API

uly 31, 2020 at 06:47:06, Mohit Jain (mo...@open-insights.com) wrote: > > Hi Team, > > Is there a Rest API to change the number of concurrent tasks? I wasn't > able to find any in the doc. > > Any leads would be appreciated. > > Thanks, > Mohit > >

Updating the Concurent Tasks of a NiFi processor using Rest API

Hi Team, Is there a Rest API to change the number of concurrent tasks? I wasn't able to find any in the doc. Any leads would be appreciated. Thanks, Mohit

NiFi 1.9.0 MoveHDFS error: Failed to rename on HDFS

not move file '/path/to/source' to its final filename. Any help would be appreciated. Regards, Mohit

Re: Urgent: HDFS processors throwing OOM - Compressed class space exception

essors throwing OOM - Compressed class space exception How big are the files that you are trying to store? How much memory did you configure nifi ? > On 22. Jul 2020, at 06:13, Mohit Jain wrote: > > Hi team, > > I’ve been facing the issue while using any HDFS processor, e.g. - PutHDF

Re: Urgent: HDFS processors throwing OOM - Compressed class space exception

Apologies. I intended to send it on the user list. I mistakenly sent it on the dev list.. that’s why sent it here again. Thanks, Mohit Get Outlook for iOS<https://aka.ms/o0ukef> From: Joe Witt Sent: Wednesday, July 22, 2020 10:05:16 AM To: users@nifi.apac

Re: Urgent: Facing issue while trying to open a NiFi processor

Hi Team, It seems like the nifi-flow-audit.h2.db is corrupted. I'm not able to execute the same query on the database directly. Anything I can do to fix this? And what are the scenarios when this might happen again? Regards, Mohit On Sat, Jul 11, 2020 at 2:48 PM Mohit Jain wrote: > Hi T

Urgent: Facing issue while trying to open a NiFi processor

.java:4537) at org.apache.nifi.web.StandardNiFiServiceFacade$$FastClassBySpringCGLIB$$358780e0.invoke() Complete nifi-user.log is attached in the log. Any input would be appreciated. Regards, Mohit 2020-07-11 09:06:53,063 ERROR [NiFi Web Server-28918] o.a.n.w.a.c.AdministrationExcep

HiveConnectionPool doesn't get enabled with the Nifi-Api in NiFi 1.9.2

, AvroWriter, etc) gets enabled. I'm not able to figure out the reason behind it. One thing I witnessed is that the PutHiveQL is the last processor in the flow. Seems like the last controller service doesn't get enabled Kindly Help. Regards, Mohit

Unable to login to Nifi UI with ranger

Hi, I've integrated Nifi-1.9.2 with ranger. When I login to the UI, following error is shown - No applicable policies could be found. Contact the system administrator. I'm not able to figure out what I'm missing. Kindly help. Regards, Mohit

Re: GenerateTable fetch throws error with MS SQL 2012+

Hi Matt, Does it means that all the other DBs(Oracle, MySQL, etc) might fetch duplicates rows with GenerateTableFetch? Thanks, Mohit On Thu, Aug 8, 2019 at 8:16 PM Matt Burgess wrote: > Mohit, > > MSSQL seems to have the only parser (of the more popular DBs) that > complains ab

GenerateTable fetch throws error with MS SQL 2012+

paging; Processor Administratively Yielded for 1 sec: java.lang.IllegalArgumentException: Order by clause cannot be null or empty when using row paging* It seems like a bug to me, is it resolved in the latest versions of Nifi? Thanks, Mohit

Re: Unable to login to secured nifi cluster via UI

Thanks Pierre. On Wed, Aug 7, 2019 at 5:51 PM Pierre Villard wrote: > Hi Mohit, > > The initial admin you configured is the user you should use for the first > connection in order to grant authorizations to additional users/groups. The > initial admin should have been auto

Unable to login to secured nifi cluster via UI

Hi team, I'm not able to open the nifi canvas in the secured NiFi. It shows the following error message once I provides the credentials - *No applicable policies could be found. Contact the system administrator.* [image: image.png] Kindly help. Thanks, Mohit

RE: Unable to see Nifi data lineage in Atlas

) at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:7 9) at org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:27 7) ... 20 common frames omitted Thanks, Mohit From: Mohit Sent: 25 July 2018 17:46 To: users@nifi.apache.org Subject: Unable to see Nifi data lineage in Atlas

Unable to see Nifi data lineage in Atlas

can't figure out anything wrong in the configuration. Is there something in the Ranger configuration that I'm missing? Regards, Mohit

GenerateTableFetch -> RPG -> ExecuteSQL fetching duplicate records from Netezza but the count is same.

to table size. For table with ~300 million rows, size is 30 million and likewise. For Example - Table : abc Netezza count - 3265421 Hive Count - 3265421 Duplicate rows in Hive - 97070 Is this the expected behaviour while fetching from Netezza? Regards, Mohit

RE: SelectHiveQl gets stuck when query table containning 12 Billion rows

Thanks Shawn, I followed a similar approach. Regards, Mohit From: Shawn Weeks Sent: 27 June 2018 19:22 To: users@nifi.apache.org Subject: Re: SelectHiveQl gets stuck when query table containning 12 Billion rows Well to get the partitions you can execute a 'show partitions

RE: SelectHiveQl gets stuck when query table containning 12 Billion rows

Hi, Yes I tried to fetch around 40 million rows which took time but it was executed. I'll try with the Avro thing. How to break the select into multiple part? Can you explain in brief the partition flow to start with? Thanks, Mohit From: Shawn Weeks Sent: 27 June 2018 18:51

SelectHiveQl gets stuck when query table containning 12 Billion rows

a 4 node NiFi cluster with 150GB RAM memory each. Is there any configuration which is to be manipulated to make this work? Regards, Mohit

RE: Unable to read the Parquet file written by NiFi through Spark when Logical Data Type is set to true.

Hi, Spark 2.3 has avro-1.7.7.jar. But hive 1.2 also uses avro-1.7.5.jar and I’m able to read the parquet file from hive. Not sure if this is the reason. Thanks, Mohit From: Mike Thomsen Sent: 31 May 2018 14:15 To: users@nifi.apache.org Subject: Re: Unable to read the Parquet file

RE: Unable to read the Parquet file written by NiFi through Spark when Logical Data Type is set to true.

(ParquetSchemaConverter.scala:141) Thanks, Mohit From: Mike Thomsen Sent: 30 May 2018 17:28 To: users@nifi.apache.org Subject: Re: Unable to read the Parquet file written by NiFi through Spark when Logical Data Type is set to true. What's the error from Spark? Logical data types are just

Unable to read the Parquet file written by NiFi through Spark when Logical Data Type is set to true.

spark as well? Regards, Mohit

RE: PutDatabaseRecord Error while ingesting to Netezza

with both CSVReader and AvroReader. For Netezza it is not able to write. Thanks, Mohit From: Pierre Villard <pierre.villard...@gmail.com> Sent: 24 May 2018 13:49 To: users@nifi.apache.org Subject: Re: PutDatabaseRecord Error while ingesting to Netezza Hi, Can we hav

PutDatabaseRecord Error while ingesting to Netezza

to the flowfile in the csv format. Please let me know if I'm doing anything wrong. Thanks, Mohit

RE: Nifi Remote Process Group FlowFile Distribution among nodes

Thanks Koji, It was helpful. Mohit. -Original Message- From: Koji Kawamura <ijokaruma...@gmail.com> Sent: 09 May 2018 05:53 To: users@nifi.apache.org Subject: Re: Nifi Remote Process Group FlowFile Distribution among nodes Hi, You can see the description for default be

RE: Nifi Remote Process Group FlowFile Distribution among nodes

Hi, What are the configurations in default batch settings? Thanks, Mohit -Original Message- From: Koji Kawamura <ijokaruma...@gmail.com> Sent: 08 May 2018 13:19 To: users@nifi.apache.org Subject: Re: Nifi Remote Process Group FlowFile Distribution among nodes Hi Mohit, Ni

RE: Validate Record issue

Mark, Thanks for the input. I’ll implement the same. Regards, Mohit From: Mark Payne <marka...@hotmail.com> Sent: 02 May 2018 23:13 To: users@nifi.apache.org Subject: Re: Validate Record issue Mohit, Correct - I was saying that it *should* allow that but due to a bug

RE: Validate Record issue

Hi Mark, I set the Strict type checking to false, still it doesn’t allowed. Thanks, Mohit From: Mark Payne <marka...@hotmail.com> Sent: 02 May 2018 23:00 To: users@nifi.apache.org Subject: Re: Validate Record issue Mohit, If you look at the Provenance events that are e

Validate Record issue

ot;type":["int","null"],"default":0},{"name":"hs_kbps","t ype":["double","null"],"default":0.0},{"name":"eul_kbps","type":["int","null "],"default":0}, {"name":"tech","type":["null","string"],"default":null},{"name":"site_id","t ype":["double","null"],"default":0.0},{"name":"longitude","type":["double"," null"],"default":0.0},{"name":"latitude","type":["double","null"],"default": 0.0}]} Sample record - May-16,KA4371D,95,100,0,151,,2G,4371,-1.606926,6.67223 Is there something I'm doing wrong? Regards, Mohit

RE: Nifi 1.6.0 ValidateRecord Processor- AvroRecordSetWriter issue

Matt, I have used all default compression settings as of now, i.e., it is set to none for AvroRecordSetWriter, PutHDFS, ConverAvroToORC. Regards, Mohit -Original Message- From: Matt Burgess <mattyb...@apache.org> Sent: 24 April 2018 20:58 To: users@nifi.apache.org Subject: Re

RE: Nifi 1.6.0 ValidateRecord Processor- AvroRecordSetWriter issue

ormat. ". Is this format different than what ConvertCsvToAvro writes? Because same flow is working with ValidateRecord(write it to csv) +ConvertCsvToAvro processor. Thanks, Mohit -Original Message- From: Matt Burgess <mattyb...@apache.org> Sent: 24 April 2018 18:43 To: users@nifi.apache.org S

Nifi 1.6.0 ValidateRecord Processor- AvroRecordSetWriter issue

FS I manually created hive table with avro format. When I use ValidateRecord + ConvertCSVToAvro, it is working fine. Flow - ValidateRecord -> ConvertCSVToAvro -> ConvertAvroToORC -> PutHDFS -> CreateHiveTable Is there anything I'm doing wrong? Thanks, Mohit

RE: Generate a surrogate key/sequence for each avro record

I'm able to achieve it using QueryRecord processor. Thanks, Mohit From: Mohit <mohit.j...@open-insights.co.in> Sent: 23 April 2018 15:59 To: users@nifi.apache.org Subject: Generate a surrogate key/sequence for each avro record Hi, I'm trying to added some extra inf

Generate a surrogate key/sequence for each avro record

Hi, I'm trying to added some extra info like a surrogate key/sequence number to the data fetch from rdbms. Is there a way to generate surrogate key for each avro record in nifi? Regards, Mohit

Nifi Custom processor code coverage

code coverage is affected by it. It would be helpful if there is any document available for nifi-mock. Regards, Mohit

RE: NiFi 1.6

Joe, What is the expected date? From: Joe Witt Sent: 06 April 2018 02:20 To: users@nifi.apache.org Subject: Re: NiFi 1.6 dan It is presently working through the release candidate vote process. As it stands now it could be out tomorrow. Please do help by

RE: ConvertCSVToAvro taking a lot of time when passing single record as an input.

Hi Mike, I intentionally did this, just to check how the processor handles invalid record. Thanks, Mohit From: Mike Thomsen <mikerthom...@gmail.com> Sent: 03 April 2018 18:17 To: users@nifi.apache.org Subject: Re: ConvertCSVToAvro taking a lot of time when passing single

RE: ConvertCSVToAvro taking a lot of time when passing single record as an input.

Pierre, Thanks for the information. This would be really helpful if Nifi 1.6.0 releases this week. There is a lot of pending tasks dependent on it.  Mohit From: Pierre Villard <pierre.villard...@gmail.com> Sent: 03 April 2018 18:03 To: users@nifi.apache.org Subje

RE: ConvertCSVToAvro taking a lot of time when passing single record as an input.

,0139LA_PALM,LABADI,ATEBSC2,0139,Greater Accra Region,Operational Is there any way to maintain the order of the record? Thanks, Mohit From: Mark Payne <marka...@hotmail.com> Sent: 02 April 2018 20:23 To: users@nifi.apache.org Subject: Re: ConvertCSVToAvro taking a lot of tim

RE: ConvertCSVToAvro taking a lot of time when passing single record as an input.

Mark, Error:- ValidateRecord[id=5a9c3616-ab7c-17c1--e6c2fc5d] ValidateRecord[id=5a9c3616-ab7c-17c1--e6c2fc5d] failed to process due to org.apache.nifi.serialization.record.util.IllegalTypeConversionException: Cannot convert value mohit of type class java.lang.String

RE: ConvertCSVToAvro taking a lot of time when passing single record as an input.

Hi Mark, I tried the ValidateRecord processor, it is converting the flowfile if it is valid. But If the records are not valid, it is passing to the invalid relationship. Instead it keeps on throwing bulletins keeping the flowfile in the queue. Any suggestion? Mohit From: Mark

RE: ConvertCSVToAvro taking a lot of time when passing single record as an input.

Mark, Agree with you. I’ll try the ValidateRecord Processor, if it serve the purpose. Thanks, Mohit From: Mark Payne <marka...@hotmail.com> Sent: 02 April 2018 19:02 To: users@nifi.apache.org Subject: Re: ConvertCSVToAvro taking a lot of time when passing single record as an

RE: ConvertCSVToAvro taking a lot of time when passing single record as an input.

ConvertCSVToAvro is able to convert 6-7k per second, which though not optimum but quite better than 45-50 records per seconds. Is there any other improvement I can do? Mohit From: Mark Payne <marka...@hotmail.com> Sent: 02 April 2018 18:30 To: users@nifi.apache.org Subje

ConvertCSVToAvro taking a lot of time when passing single record as an input.

I'm doing wrong in the flow. I'm using Nifi 1.5.0 . Any quick input would be appreciated. Thanks, Mohit