e has to go
into the NiFi UI and enable that processor manually. This is not feasible
in the production environment.
Any help would be appreciated.
Regards,
Mohit
ource folder
to the processed folder.
3. Go ahead with the flow.
It's a 3 Node NiFi 1.9.0 cluster, and the flow is reading from the NFS
mount. ListFile is scheduled to run on the primary node.
Any help would be appreciated.
Thanks,
Mohit
and NiFi is up after 2-3 minutes max.
There are 5 custom processors being used in the flow.
Thanks,
Mohit
On Wed, Aug 19, 2020 at 5:07 AM Andy LoPresto wrote:
> Mohit,
>
> I’m a little confused by some of the details you report. Initially you
> said there were ~100 flows with 2
similar note, NiFi Rest api response is also quite slow when there are
a lot of components. Would it improve by disabling the components which are
not in use? Or is there something else that could have been causing the
issue?
Thanks,
Mohit
On Thu, Aug 13, 2020 at 9:49 PM Pierre Villard
wrote:
>
Hi Brandon,
There are no flow files in the queue.
Thanks,
Mohit
On Thu, Aug 13, 2020 at 9:56 PM Brandon DeVries
wrote:
> Mohit,
>
> How many flowfiles are currently on the instance? Sometimes a very large
> number of flowfiles can result in slower start times.
started
in a minute.
Is this an expected behaviour that startup time is directly proportional to
the number of processors in the Canvas? Or is there a way to reduce the
NiFi startup time?
Any leads would be appreciated.
Thanks,
Mohit
uly 31, 2020 at 06:47:06, Mohit Jain (mo...@open-insights.com) wrote:
>
> Hi Team,
>
> Is there a Rest API to change the number of concurrent tasks? I wasn't
> able to find any in the doc.
>
> Any leads would be appreciated.
>
> Thanks,
> Mohit
>
>
Hi Team,
Is there a Rest API to change the number of concurrent tasks? I wasn't able
to find any in the doc.
Any leads would be appreciated.
Thanks,
Mohit
d not move file '/path/to/source' to
its final filename.
Any help would be appreciated.
Regards,
Mohit
essors throwing OOM - Compressed class space
exception
How big are the files that you are trying to store? How much memory did you
configure nifi ?
> On 22. Jul 2020, at 06:13, Mohit Jain wrote:
>
> Hi team,
>
> I’ve been facing the issue while using any HDFS processor, e.g. - PutHDF
Apologies.
I intended to send it on the user list. I mistakenly sent it on the dev list..
that’s why sent it here again.
Thanks,
Mohit
Get Outlook for iOS<https://aka.ms/o0ukef>
From: Joe Witt
Sent: Wednesday, July 22, 2020 10:05:16 AM
To: users@nifi.apac
Hi Team,
It seems like the nifi-flow-audit.h2.db is corrupted. I'm not able to
execute the same query on the database directly.
Anything I can do to fix this?
And what are the scenarios when this might happen again?
Regards,
Mohit
On Sat, Jul 11, 2020 at 2:48 PM Mohit Jain wrote:
>
537)
at
org.apache.nifi.web.StandardNiFiServiceFacade$$FastClassBySpringCGLIB$$358780e0.invoke()
Complete nifi-user.log is attached in the log.
Any input would be appreciated.
Regards,
Mohit
2020-07-11 09:06:53,063 ERROR [NiFi Web Server-28918]
o.a.n.w.a.c.Administration
,
AvroWriter, etc) gets enabled.
I'm not able to figure out the reason behind it. One thing I witnessed is
that the PutHiveQL is the last processor in the flow. Seems like the last
controller service doesn't get enabled
Kindly Help.
Regards,
Mohit
Hi,
I've integrated Nifi-1.9.2 with ranger. When I login to the UI, following
error is shown -
No applicable policies could be found. Contact the system administrator.
I'm not able to figure out what I'm missing. Kindly help.
Regards,
Mohit
Hi Matt,
Does it means that all the other DBs(Oracle, MySQL, etc) might fetch
duplicates rows with GenerateTableFetch?
Thanks,
Mohit
On Thu, Aug 8, 2019 at 8:16 PM Matt Burgess wrote:
> Mohit,
>
> MSSQL seems to have the only parser (of the more popular DBs) that
> complains ab
g
row paging; Processor Administratively Yielded for 1 sec:
java.lang.IllegalArgumentException: Order by clause cannot be null or empty
when using row paging*
It seems like a bug to me, is it resolved in the latest versions of Nifi?
Thanks,
Mohit
Thanks Pierre.
On Wed, Aug 7, 2019 at 5:51 PM Pierre Villard
wrote:
> Hi Mohit,
>
> The initial admin you configured is the user you should use for the first
> connection in order to grant authorizations to additional users/groups. The
> initial admin should have been automatical
Hi team,
I'm not able to open the nifi canvas in the secured NiFi. It shows the
following error message once I provides the credentials -
*No applicable policies could be found. Contact the system administrator.*
[image: image.png]
Kindly help.
Thanks,
Mohit
org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:7
9)
at
org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:27
7)
... 20 common frames omitted
Thanks,
Mohit
From: Mohit
Sent: 25 July 2018 17:46
To: users@nifi.apache.org
Subject: Unable to see Nifi data lineage in Atlas
can't figure out anything wrong in the configuration.
Is there something in the Ranger configuration that I'm missing?
Regards,
Mohit
can't figure out anything wrong in the configuration.
Is there something in the Ranger configuration that I'm missing?
Regards,
Mohit
to table
size. For table with ~300 million rows, size is 30 million and likewise.
For Example -
Table : abc
Netezza count - 3265421
Hive Count - 3265421
Duplicate rows in Hive - 97070
Is this the expected behaviour while fetching from Netezza?
Regards,
Mohit
Thanks Shawn,
I followed a similar approach.
Regards,
Mohit
From: Shawn Weeks
Sent: 27 June 2018 19:22
To: users@nifi.apache.org
Subject: Re: SelectHiveQl gets stuck when query table containning 12 Billion
rows
Well to get the partitions you can execute a 'show parti
Hi,
Yes I tried to fetch around 40 million rows which took time but it was
executed. I'll try with the Avro thing.
How to break the select into multiple part? Can you explain in brief the
partition flow to start with?
Thanks,
Mohit
From: Shawn Weeks
Sent: 27 June 2018 18:
have a 4 node NiFi cluster with 150GB RAM memory each.
Is there any configuration which is to be manipulated to make this work?
Regards,
Mohit
Hi,
Spark 2.3 has avro-1.7.7.jar. But hive 1.2 also uses avro-1.7.5.jar and I’m
able to read the parquet file from hive.
Not sure if this is the reason.
Thanks,
Mohit
From: Mike Thomsen
Sent: 31 May 2018 14:15
To: users@nifi.apache.org
Subject: Re: Unable to read the Parquet file
(ParquetSchemaConverter.scala:141)
Thanks,
Mohit
From: Mike Thomsen
Sent: 30 May 2018 17:28
To: users@nifi.apache.org
Subject: Re: Unable to read the Parquet file written by NiFi through Spark when
Logical Data Type is set to true.
What's the error from Spark? Logical data types are
t readable from spark as well?
Regards,
Mohit
with both CSVReader and
AvroReader. For Netezza it is not able to write.
Thanks,
Mohit
From: Pierre Villard
Sent: 24 May 2018 13:49
To: users@nifi.apache.org
Subject: Re: PutDatabaseRecord Error while ingesting to Netezza
Hi,
Can we have more details: describe of the Hive
nd writing to the flowfile in the csv
format.
Please let me know if I'm doing anything wrong.
Thanks,
Mohit
Thanks Koji,
It was helpful.
Mohit.
-Original Message-
From: Koji Kawamura
Sent: 09 May 2018 05:53
To: users@nifi.apache.org
Subject: Re: Nifi Remote Process Group FlowFile Distribution among nodes
Hi,
You can see the description for default behavior by hovering your mouse on the
Hi,
What are the configurations in default batch settings?
Thanks,
Mohit
-Original Message-
From: Koji Kawamura
Sent: 08 May 2018 13:19
To: users@nifi.apache.org
Subject: Re: Nifi Remote Process Group FlowFile Distribution among nodes
Hi Mohit,
NiFi RPG batches multiple FlowFiles
t the
flowfiles are evenly distributed across the nodes even if there are less
number of flowfiles.
Regards,
Mohit
Mark,
Thanks for the input. I’ll implement the same.
Regards,
Mohit
From: Mark Payne
Sent: 02 May 2018 23:13
To: users@nifi.apache.org
Subject: Re: Validate Record issue
Mohit,
Correct - I was saying that it *should* allow that but due to a bug (NIFI-5141)
it currently
Hi Mark,
I set the Strict type checking to false, still it doesn’t allowed.
Thanks,
Mohit
From: Mark Payne
Sent: 02 May 2018 23:00
To: users@nifi.apache.org
Subject: Re: Validate Record issue
Mohit,
If you look at the Provenance events that are emitted by the processor, they
;,"type":["int","null"],"default":0},{"name":"hs_kbps","t
ype":["double","null"],"default":0.0},{"name":"eul_kbps","type":["int","null
"],"default":0},
{"name":"tech","type":["null","string"],"default":null},{"name":"site_id","t
ype":["double","null"],"default":0.0},{"name":"longitude","type":["double","
null"],"default":0.0},{"name":"latitude","type":["double","null"],"default":
0.0}]}
Sample record -
May-16,KA4371D,95,100,0,151,,2G,4371,-1.606926,6.67223
Is there something I'm doing wrong?
Regards,
Mohit
Matt,
I have used all default compression settings as of now, i.e., it is set to none
for AvroRecordSetWriter, PutHDFS, ConverAvroToORC.
Regards,
Mohit
-Original Message-
From: Matt Burgess
Sent: 24 April 2018 20:58
To: users@nifi.apache.org
Subject: Re: Nifi 1.6.0 ValidateRecord
". Is this format different than what ConvertCsvToAvro
writes? Because same flow is working with ValidateRecord(write it to csv)
+ConvertCsvToAvro processor.
Thanks,
Mohit
-Original Message-
From: Matt Burgess
Sent: 24 April 2018 18:43
To: users@nifi.apache.org
Subject: Re: Nifi 1.6.0
PutHDFS I manually created
hive table with avro format.
When I use ValidateRecord + ConvertCSVToAvro, it is working fine.
Flow - ValidateRecord -> ConvertCSVToAvro -> ConvertAvroToORC -> PutHDFS
-> CreateHiveTable
Is there anything I'm doing wrong?
Thanks,
Mohit
I'm able to achieve it using QueryRecord processor.
Thanks,
Mohit
From: Mohit
Sent: 23 April 2018 15:59
To: users@nifi.apache.org
Subject: Generate a surrogate key/sequence for each avro record
Hi,
I'm trying to added some extra info like a surrogate key/sequence num
Hi,
I'm trying to added some extra info like a surrogate key/sequence number to
the data fetch from rdbms.
Is there a way to generate surrogate key for each avro record in nifi?
Regards,
Mohit
o test that part? My code coverage is affected by it.
It would be helpful if there is any document available for nifi-mock.
Regards,
Mohit
Joe,
What is the expected date?
From: Joe Witt
Sent: 06 April 2018 02:20
To: users@nifi.apache.org
Subject: Re: NiFi 1.6
dan
It is presently working through the release candidate vote process. As it
stands now it could be out tomorrow.
Please do help by reviewing the rc if you
Hi Mike,
I intentionally did this, just to check how the processor handles invalid
record.
Thanks,
Mohit
From: Mike Thomsen
Sent: 03 April 2018 18:17
To: users@nifi.apache.org
Subject: Re: ConvertCSVToAvro taking a lot of time when passing single record
as an input.
Mohit
Pierre,
Thanks for the information. This would be really helpful if Nifi 1.6.0 releases
this week. There is a lot of pending tasks dependent on it. 😊
Mohit
From: Pierre Villard
Sent: 03 April 2018 18:03
To: users@nifi.apache.org
Subject: Re: ConvertCSVToAvro taking a lot of time when
,0139LA_PALM,LABADI,ATEBSC2,0139,Greater Accra Region,Operational
Is there any way to maintain the order of the record?
Thanks,
Mohit
From: Mark Payne
Sent: 02 April 2018 20:23
To: users@nifi.apache.org
Subject: Re: ConvertCSVToAvro taking a lot of time when passing single record
as an
Mark,
Error:-
ValidateRecord[id=5a9c3616-ab7c-17c1--e6c2fc5d]
ValidateRecord[id=5a9c3616-ab7c-17c1--e6c2fc5d] failed to process due
to org.apache.nifi.serialization.record.util.IllegalTypeConversionException:
Cannot convert value mohit of type class java.lang.String
Hi Mark,
I tried the ValidateRecord processor, it is converting the flowfile if it is
valid. But If the records are not valid, it is passing to the invalid
relationship. Instead it keeps on throwing bulletins keeping the flowfile in
the queue.
Any suggestion?
Mohit
From: Mark
Mark,
Agree with you. I’ll try the ValidateRecord Processor, if it serve the purpose.
Thanks,
Mohit
From: Mark Payne
Sent: 02 April 2018 19:02
To: users@nifi.apache.org
Subject: Re: ConvertCSVToAvro taking a lot of time when passing single record
as an input.
Mohit,
You can
ConvertCSVToAvro is able to convert 6-7k per second, which
though not optimum but quite better than 45-50 records per seconds.
Is there any other improvement I can do?
Mohit
From: Mark Payne
Sent: 02 April 2018 18:30
To: users@nifi.apache.org
Subject: Re: ConvertCSVToAvro taking a lot
nclude where I'm doing wrong in the flow.
I'm using Nifi 1.5.0 .
Any quick input would be appreciated.
Thanks,
Mohit
52 matches
Mail list logo