Processor for Firestore GCP

Hello all, I'm looking for the connection and fetch data from FIRESTORE GCP. Have we something in NIFI to do this thing ? Thanks   Minh     

Re: java.lang.OutOfMemoryError: Java heap space : NIFI 1.23.2

Hello Joe,   Thanks for the response. But I found the solution.    Like the table size is around 7,6GB (avro) for 161000 rows  And I setup Fetch size : 20 so the processor ExecuteSQL try to fetch all datas.   Like the heap size is setup to 8GB I have reduced the "Fetch Size" to 1

java.lang.OutOfMemoryError: Java heap space : NIFI 1.23.2

Hello    I got the executeSQL processor does the sql command "select * from public.table1" It is a postgresql database.   Here the end of properties of processor.   Max Wait Time 0 seconds Normalize Table/Column Names false Use Avro Logical Types false Compression Format NONE

Re: Track dataflow with counter / updatecounter / statefull

Hello Lehel,   Yes, I have already try to use UpdateAttribute with "Store State Locally". The issue I saw it is when the same dataflow run multiple time the value "token" is increase all the times I need to have the "counter" reset to Zero when the dataflow is consider to done.   Also, I

Track dataflow with counter / updatecounter / statefull

  Hello all,   Somebody has already use the Counter or UpdateCounter ?   My use case is sending a message (success) to topic pubsub if my nifi dataflow has consider to finish.   The input of dataflow is for example 3 flowfiles (do so treatements) and the ouput will be push to the GCS

Error MQSerie

  Hi all,  I don't how explain this error.   We use the MQV9.2 as producer and consume by NIFI 1.19.1 with processor ConsumeJMS and drivers :    ls -rtl mqm_ibm total 11384 -rw-r--r-- 1 nifi nifi 2111220 Jun 26  2018 ojdbc-11.2.0.3-jdk16.jar -rw-r--r-- 1 nifi nifi   88374 Jun 26  2018

Content repository error

  Hello all,   Somebody has already seen this error and how to correct it ?   Unable to write flowfile content to content repository container repo0 due to archive file size constraints; waiting for archive cleanup. Total number of files currently archived = 1   my filesystem is up

Re: Hardware requirement for NIFI instance

Hello Mark,   I shared the screenshot about ExecuteSQL properties.   This 3 properties together work very well for PostGRESQL in the NIFI with 2CPU/8Gb.   Without the properties "Set Auto Commit" to FALSE, the NIFI stuck and crash.   Thanks a lot for help   Minh        Envoyé: 

Re: Hardware requirement for NIFI instance

So after some tests, here the result perhaps could help someone.    With nifi (2CPU / 8Go Ram) I have tested with these couples properties :   > 1 executeSQL with "select * from table" Output Batch Size : 1 Fetch Size : 10    > 2 executeSQL with "select * from table" Output Batch

Re: Hardware requirement for NIFI instance

And the important thing for us it has only one avro file by table.    So it is possible to merge avro files to one avro file ?   Regards      Envoyé: jeudi 4 janvier 2024 à 19:01 De: e-soci...@gmx.fr À: users@nifi.apache.org Cc: users@nifi.apache.org Objet: Re: Hardware requirement for NIFI

Re: Hardware requirement for NIFI instance

  Hello all,   Thanks a lot for the reply.   So for more details.   All the properties for the ExecuteSQL are set by default, except "Set Auto Commit:  false".   The sql command could not be more simple than "select * from ${db.table.fullname}"   The nifi version is 1.16.3 and 1.23.2  

Hardware requirement for NIFI instance

  Hello all,   Who could help me to determine the cpu/memory need for nifi instance to fetch the data from Postgresql hosted in google ?   We got this error : ==> Error : executesql.error.message Ran out of memory retrieving query results.   The procesor ExecuteSQL has this config : Set

Listenhttp 1.23.2 / JDK jdk-11.0.17

Hello all,   It was working with NIFI 1.18.0 and JDK 1.8.0_311   Have you got some issues with ListenHTTP with the length of file receive by NIFI ? Error : "Form is larger than max length 20"   Do you know how correct it ?   Regards   Minh    java.lang.IllegalStateException: Form

Re: RE: invokeHTTP SSL error NIFI : 1.23.2

Hi Steve, I found the error :) It seems block by the security team... "This category is not allowed by the company IS security policies."   Regards   Minh    Envoyé: lundi 27 novembre 2023 à 16:47 De: e-soci...@gmx.fr À: users@nifi.apache.org, stephen.hindma...@bt.com Cc: 

Re: RE: invokeHTTP SSL error NIFI : 1.23.2

Thanks, for the informations, I will recheck from my side   Minh     Envoyé: lundi 27 novembre 2023 à 16:38 De: "stephen.hindmarch.bt.com via users" À: users@nifi.apache.org Objet: RE: invokeHTTP SSL error NIFI : 1.23.2 Minh,   I would be reluctant to simply trust an external web site’s

Re: invokeHTTP SSL error NIFI : 1.23.2

Hello,    It is not working    I have used :    # true | openssl s_client -showcerts -connect api-eu.reputation.com:443   I saw it is manage by Let Encrypt   And the let Encrypt CA is already setup in the file 11.0.17/lib/security/cacerts => alias name: letsencryptisrgx1 [jdk]   so I

Re: invokeHTTP SSL error NIFI : 1.23.2

Hello   Thank for reply, the weird thing it is until now, I don't use SSL context and it is working.   Good anyway, I will get the server certificate and add it in the truststore and configure invokeHTTP to user SSL context also   Thanks   Minh     Envoyé: lundi 27 novembre 2023 à 10:48

invokeHTTP SSL error NIFI : 1.23.2

  Hello all,   Since I've upgraded the nifi version from 1.18 to 1.23.2 - Java Version 11.0.17 I got the error concerning the invokeHTTP (GET https://api-eu.reputation.com/v3/ ..) even if I setup SSL Context or not   Do you have informations about what has changed between the 2 nifi version

Re: ListSFTP Processor CRON doesn't start

Hello Quentin,   In the normal way, if the processor is started correctly, you have this logs   My schedule ==> 0 0/2 * * * ?   ==> 2023-11-16 10:28:00,002 DEBUG [Timer-Driven Process Thread-8] o.a.nifi.processors.standard.ListSFTP ListSFTP[id=7edea667-2e42-39c5-7252-bda580ba683a] Returning

Re: ListSFTP Processor CRON doesn't start

Hello Quentin,   It seems strange to have log at 06h00 PM, it is not related to your CRON strategy.   What is you have you do this CRON "«0 15 1/1 * * ? *»   Anyway, logs seems saying it is running once time. Have you got "clear state" for ListSFTP to be sure ?     Regards   Minh 

Re: ListSFTP Processor CRON doesn't start

Hello   Thanks Mark   What could be a good way to debug "google processor"       (not working)       Thanks     Envoyé: mardi 14 novembre 2023 à 16:11 De: "Mark Payne" À: "users@nifi.apache.org" Objet: Re: ListSFTP Processor CRON doesn't start Hi Minh,   No - you can configure

Re: ListSFTP Processor CRON doesn't start

  Hello Mark,   Have we got the documentation about exhaustive list about logger we got have in NIFI ?   Regards    Minh  Envoyé: mardi 14 novembre 2023 à 15:25 De: "Mark Payne" À: "users" Objet: Re: ListSFTP Processor CRON doesn't start Hi Quentin, What is the CRON schedule that

Re: Bug : PutGCSObject : Version NIFI 1.23.2

Hello,   Seems not working, I don't have any logs concerning google/putgcsObjects in the nifi-app.log    Missing something ?   Minh   Envoyé: vendredi 10 novembre 2023 à 15:36 De: "Pierre Villard" À: users@nifi.apache.org Objet: Re: Bug : PutGCSObject : Version NIFI 1.23.2 You'd just

Re: Bug : PutGCSObject : Version NIFI 1.23.2

Hello Pierre,   I'm not developper, so it is a bit confuse for me.   I'm trying to temporarily push this config in the logback.xml but it is not working.     . . .   . .   It is missing something ?   Thanks a lot for your help   Regards    Minh Envoyé: 

PutGCSObject version 1.23.2

  Hi all,    It is strange for the putGCSObject with properties "GZIP Compression Enabled"   The description said the most time is better to setup at "false" but by default is "true"   "Signals to the GCS Blob Writer whether GZIP compression during transfer is desired. False means do not

Re: Bug : PutGCSObject : Version NIFI 1.23.2

Hi Pierre,   Thanks for this information.   Minh     Envoyé: mercredi 8 novembre 2023 à 19:57 De: "Pierre Villard" À: users@nifi.apache.org Objet: Re: Bug : PutGCSObject : Version NIFI 1.23.2 So I believe the client library, when using the private endpoint, is making multiple calls, the

Re: Bug : PutGCSObject : Version NIFI 1.23.2

Hi Pierre,   I confirm, it works with Proxy.    But how we are sure that the data are transfert from the "private connection" and not through the proxy...   Regards     Envoyé: mercredi 8 novembre 2023 à 16:36 De: e-soci...@gmx.fr À: users@nifi.apache.org Objet: Re: Bug : PutGCSObject :

Re: Bug : PutGCSObject : Version NIFI 1.23.2

Hi Pierre,   But it seems weird because when I put the wrong "storage api url : https://storage-euwest1.p.googleapis.com/storage/v1/"   We got the wrong url but has the correct hostname   Not Found POST

Re: Bug : PutGCSObject : Version NIFI 1.23.2

Hi    I have checked the DNS entry for "storage-euwest1.p.googleapis.com" it seems OK From the VM nifi, in the terminal, I could reach the port 443 it works fine.   I check with the devops google team also.   Thanks   Envoyé: mercredi 8 novembre 2023 à 15:40 De: "Joe Witt" À: 

Re: Bug : PutGCSObject : Version NIFI 1.23.2

Hello Joe   Until now we leave the "storage api url" empty by default, but we use "proxy configuration service" because we are behind the corporate network.   Since some month ago we have configured "private connection" with google. So I have setup "storage api url" with the private url and

Bug : PutGCSObject : Version NIFI 1.23.2

Hello,   I'm trying to user the overwriting storage url parameter in the nifi version 1.23.2.   Do you know what is the correct parameter to set for the "Storage API URL" in the processor PutGCSObject ?   If I setup "https://storage-euwest1.p.googleapis.com/storage/v1/" It got this error :

Re: Analyze : "Diagnostics on Stop" logs

  Hello Joe,   It is a VM with Rocky 8 as OS. We use the systemd to start/stop nifi process   The nifi process is die. I could see with the journalctl -u nifi   Oct 27 14:53:11 server01 systemd[1]: Started Apache NiFi. Oct 28 03:31:04 server01 systemd[1]: nifi.service: Supervising

Analyze : "Diagnostics on Stop" logs

Hello all,   I have activated the "diagnosticts on stop" in the nifi.properties. https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#nifi_diagnostics My node has stop twice time now but I don't find anything.   Have we got some documentations about the log of diagnosticts

Re: RE: How find the root cause "Node disconnect to the cluster"

Hello   Thanks for your reply, I think I'm doing something like you to find the roort cause but it is often I need to check a lot of thing.   For exemple, when I saw the "Java heap space" in the nifi-app.log  It is not easy to see what processor has cause this issue.   Anyway thanks  

How find the root cause "Node disconnect to the cluster"

  Hello all,   Since I'm working with NIFI, I got a lot of difficulty to find the root cause why the node is disconnected to the cluster. I check nifi-app.log nifi-bootstrap.log but it is a bit complicated to find the correct information.   Somebody has some informations for me ?   Regards 

Timezone Summer Winter Time

  Hello all,   Somebody has already done some configuration for schedule the processor with the particular (CET/France) timezone even if the NIFI is running in the UTC ?   Here the definition of summer/winter time     Metropolitan France uses Central European Time (heure d'Europe centrale,

Re: NIFI-REGISTRY Migration

Hello Pierre,   Thanks for new option.   So it means, the dataflow_1 register in the registry_A, will be register in the same way in registry_B after migration ? Or needs we also use the URL aliasing ?   Regards    Envoyé: vendredi 29 septembre 2023 à 12:41 De: "Pierre Villard" À: 

NIFI-REGISTRY Migration

  Hello all,   Someone has already migrate their nifi-registry from hostA to hostB (nifi-registry) and keep track the versionning for the nifi client points to the hostB ?   Have we got some documentations about migration nifi-registry ?   regards    Minh 

Re: GCS processors : endpoint Override URL: " storage-xyz.p.googleapis.com"

  Hello,    Sorry, I have not correctly checked in the release note.  Great new, thanks a lot   regards   Minh    Envoyé: mardi 19 septembre 2023 à 11:38 De: "Pierre Villard" À: users@nifi.apache.org Objet: Re: GCS processors : endpoint Override URL: " storage-xyz.p.googleapis.com"

GCS processors : endpoint Override URL: " storage-xyz.p.googleapis.com"

  Hello all,   Is it possible to add an option in the processor GCS so we can write the custom url for storage.googleapis.com something like "storage-xyz.p.googleapis.com"?   I know it is possible for the "S3 Amazon processor" : Endpoint Override URL Endpoint Override URL  

Re: NiFi hanging during large sql query

  Hello Mike,   Could you please give me the details about the resolution ? Have you change something in the processor or just changing the sql command ?   Regards   Envoyé: samedi 2 septembre 2023 à 00:00 De: "Mike Thomsen" À: users@nifi.apache.org Objet: NiFi hanging during large sql

Re: Help : LoadBalancer

  Hello Jens,   Yes it is a bit too long to read but very interesting. Thanks a lot for this information. I'm not sure can do this thing in my work. I do simple for testing and use only one Haproxy.   Thanks   Minh     Envoyé: jeudi 7 septembre 2023 à 18:58 De: "Jens M. Kofoed" À: 

Re: [NIFI] ExecuteScript

Hello Quentin,   I also used the python with ExecuteScript but it is not native and not really supported.   The better thing it is use executeGroovy processor.   Minh     Envoyé: vendredi 8 septembre 2023 à 22:55 De: "Rafael Fracasso" À: users@nifi.apache.org Objet: Re: [NIFI]

Re: Help : LoadBalancer

  Hello Jens   Thanks a lot for haproxy conf.   Could you give more details about this point :   - I have 2 proxy nodes, which is running in a HA setup with keepalived and with a vip.-  - I have a dns record nifi-cluster01.foo.bar pointing to the vip address to keepalived   Thanks   

Re: Re: Node/processor/queue status in grafana

Hello YolandaWhere is the document about accepted parameter to set after “includedRegistries”Thanks for helpEnvoyé à partir de l'app mail mobile sur 14/08/2023 le 17:38, Yolanda Davis écrivit:

PutGCSObject more option

Hello all,   We're facing a lot of issue concerning the timeout for "PutGCSObject"   PutGCSObject[id=d5a6d876-486f-3f6d-acba-8c661b3527af] Failed to put FlowFile[filename=xxx] to Google Cloud Storage due to Read timed out: com.google.cloud.storage.StorageException: Read timed out - Caused by: