RE: MiNiFi agent cannot update flow configuration

? If not the VolatileProvenanceRepository, can you try that as a workaround? Also are you using the 1.14.0 version of the C2 server? Regards, Matt On Tue, Sep 21, 2021 at 3:45 PM Tomislav Novosel wrote: > > Hi to all, > > > I'm using MiNiFi 1.14.0 with configured change ingestor to pull from &

MiNiFi agent cannot update flow configuration

Hi to all, I'm using MiNiFi 1.14.0 with configured change ingestor to pull from HTTP C2 server whenever there is a change in configuration (change in NiFi flow that suppose to be running on MiNiFi). MiNiFi agent is running on Raspberry Pi 3 with enough disk space. When I make a change and save

MiNiFi C2 server 0.5.0 bug or not?

Hi to all, I am using MiNiFi C2 server version 0.5.0 and MiNiFi version 1.14.0. C2 server is configured to pull templates from localhost NiFi installation and MiNiFi is configured with configuration change ingestor to pull config from C2 server. After I created MiNiFi flow on NiFi canvas

RE: Minifi 1.14.0 exception - sensitive props key

/administration-guide.html#updating-the-sensitive-properties-key ./bin/nifi.sh set-sensitive-properties-key Jeremy On Mon, Sep 20, 2021 at 9:14 PM Tomislav Novosel mailto:tomislav.novo...@clearpeaks.com>> wrote: According to migration guidance for NiFi property 'Sensitive Properties Key'

RE: Minifi 1.14.0 exception - sensitive props key

According to migration guidance for NiFi property 'Sensitive Properties Key' should be generated at startup, I believe this is the same behaviour in case of MiNiFi since they merged the codebase. Why is this happening? BR, Tom From: Tomislav Novosel Sent: 20 September 2021 11:05 To: users

Minifi 1.14.0 exception - sensitive props key

Hi to all, I was using MiNiFi 0.5.0 running on ubuntu, installed as a service. I switched now to MiNiFi 1.14.0 - I disabled minifi service, deleted installation folder and unpacked MiNiFi 1.14.0 folder at the same place where was 0.5.0 installed. I started new MiNiFi 1.14.0 and the service don't

PutDistributedMapCache

Hi to all, I have these error bulletin message showing now and there on PutDistributedMapCache processor. The error disappears after a few minutes, but now and there it shows up again, and it is self-resolving (I have failure relationship routed again to the processor). I increased connection

RE: NiFi configuration files changes

system. If you need a change log for config files, I'd look into CM api to fetch those instead. On Tue, Jun 29, 2021, 8:30 AM Tomislav Novosel mailto:tomislav.novo...@clearpeaks.com>> wrote: Hi to all, Is there a good way how to capture NiFi configuration files changes (nifi.prop

NiFi configuration files changes

Hi to all, Is there a good way how to capture NiFi configuration files changes (nifi.properties, authorizers.xml,...etc) and to forward that changes (or just to notify) some other system or app? Can I do it with NiFi itself? The question is in the context of Cloudera platform - CFM. Thanks,

RE: Nifi Registry git presistence

like prod where the registry is read-only, they periodically stop registry, delete the H2 database, git pull, and start again, and now everything is up to date. On Wed, Jun 23, 2021 at 10:21 AM Tomislav Novosel wrote: > > Hi Chris, Bryan, > > Thanks for the answers. > >

RE: Nifi Registry git presistence

order ot pick up the latest changes. > > On Wed, Jun 23, 2021 at 5:25 AM Tomislav Novosel > wrote: >> >> Hi to all, >> >> >> >> i sit possible to have one central Git repo for multiple NiFi >> registries >> >> as persistance layer? >

Nifi Registry git presistence

Hi to all, i sit possible to have one central Git repo for multiple NiFi registries as persistance layer? Ii it possible to configure, for example, three NiFi registries (through providers.xml file) to communicate with one git repo? Thanks, Tom

Setting Param Context to PG

Hi to all, How can I set parameter context to all processor groups at once? I imported root PG to another NiFi environment using template, but now I need to go manually on every nested PG and set param context. How to set it on all PGs at once? Can I use toolkit or NiFi REST API? In toolkit

Delete FlowFile content

Hi to all, what is the best way to delete flowfiles content so that flowfile keeps attributes, but to not take space in next relationship between two processors? I found this: https://stackoverflow.com/questions/53312069/what-is-a-fastest-way-to-remove-nifi-flowfile-content But then flowfile

Nifi registry - NiFi toolkit

Hi to all, Is it possible to use NiFi registry with github persistence provider instead of local file system provider for flows versioning but to be able to use nifi-toolkit and command nifi pg-import to import flows from registry to another nifi env? I configured github persistence provider

RE: Some retry flowfile questions

Hi Harald, Mark, I asked about RetryFlowfile the other day and its potential danger, but no answer yet. My question was not referred to penalty and yield really, but just to make consideration about it. @Harald, if on this Retry in your schema you are using RetryFlowfile processor, there can

Stopping processor after MAX number of retries

Hi guys, I want to stop the processor after exceeding maximum number of retries. For that I'm using RetryFlowFile processor, after 5 times of retry, it routes flowfile to retries_exceeded. When that kicks in, I want to stop the processor which was retried 5 times. What is the best approach? I

RE: Groovy script

Subject: Re: Groovy script If file_path is pointing to a folder as you said, it's going to check for the folder's existence. The fact that it's failing to return true there suggests that something is wrong with the path in the file_path attribute. On Wed, Feb 24, 2021 at 11:47 AM Tomislav Novosel

Groovy script

Hi guys, I want to check if file exists with this groovy script: flowfile = session.get() if(!flowfile) return file_path = flowfile.getAttribute('file_path') File file = new File(file_path) if(file.exists()){ session.transfer(flowfile, REL_FAILURE) } else{ session.transfer(flowfile, REL_SUCCESS)

DMC and ListFile

Hi team, is Redis good alternative to use it like DMC with ListFile processor in mode 'Tracking entities'? In general, can Listfile use external cache service located on separated server? Something similar is dscribed here by Bryan:

Monitoring big directory tree

Hi guys, I have following situation: There is SMB mounted folder on one Nifi worker and it has many subfolders with subfolders (the depth of nesting is not known in advance). If new files in that directory tree appears or file is moved or old file is copied/moved, modification timestamp

NiFi user and access rights

Hi guys, I'm having this situation inside my company projects. We are using NiFi as DataFlow platform and there are multiple projects. Every project has files on shared disk/folder from which one Nifi instance(standalone instance) is reading data. NiFi instance service is running under one

Merge and transform JSON

Hi Nifi Community, I have situation where I need to merge JSON content from multiple flowfiles into one JSON content (single flowfile). JSON in every flowfile looks like this: { "analyses": "prep_array", "args": "prep_array", "scriptId": "142", "libIds": "141", "job_name":

Re: Nifi errors - FetchFile and UnpackContent

Any more suggestions to this situation? Thanks, Tom On Thu, 3 Oct 2019 at 19:54, Tomislav Novosel wrote: > Hi Jeff, > > None of this is applied in pipeline and FetchFile processor. > It is not on cluster, it runs only on one standalone Nifi instance. > Completion strategy

Re: Nifi errors - FetchFile and UnpackContent

load of fetching the files across the cluster. > > On Thu, Oct 3, 2019 at 2:32 AM Tomislav Novosel > wrote: > >> Hi all, >> >> I'm getting errors from FetchFile and UnpackContent processors. >> I have pipeline where I fetch zip files as they come continu

Nifi errors - FetchFile and UnpackContent

Hi all, I'm getting errors from FetchFile and UnpackContent processors. I have pipeline where I fetch zip files as they come continuously on shared network drive with Minimum file age set to 30 sec to avoid fetching file before it is written to disk completely. Sometimes I get this error from

Re: Reading only latest file

Yes, I've got the same idea. Thanks. Tom On Tue, 27 Aug 2019, 10:49 Arpad Boda, wrote: > Quite a special case, I would go for executescript proc and do the logic > you need in Python. > > On Tue, Aug 27, 2019 at 8:45 AM Tomislav Novosel > wrote: > >> Hi Arpad, >>

Re: Reading only latest file

aximum file age lower than the frequency of the file creation you have? > > Regards, > Arpad > > On Mon, Aug 26, 2019 at 10:19 AM Tomislav Novosel > wrote: > >> Any ideas? >> Thanks, >> Tom >> >> On Thu, 22 Aug 2019 at 16:06, Tomislav Novosel >

Re: Reading only latest file

aximum file age lower than the frequency of the file creation you have? > > Regards, > Arpad > > On Mon, Aug 26, 2019 at 10:19 AM Tomislav Novosel > wrote: > >> Any ideas? >> Thanks, >> Tom >> >> On Thu, 22 Aug 2019 at 16:06, Tomislav Novosel >

Re: Reading only latest file

Any ideas? Thanks, Tom On Thu, 22 Aug 2019 at 16:06, Tomislav Novosel wrote: > Hi all, > > I have scenario where I need to read only the latest(the youngest) file > according to creation date. The files are: > > Load_Dump_1.001.2019-07-22_17-22-45994.ifl - creation date >

Reading only latest file

Hi all, I have scenario where I need to read only the latest(the youngest) file according to creation date. The files are: Load_Dump_1.001.2019-07-22_17-22-45994.ifl - creation date 2019-07-22T17:24:44+0200 Load_Dump_1.001.2019-07-22_17-25-09132.ifl - creation date 2019-07-22T17:26:14+0200 So I

JSON argument

Hi all, I have a case where I'm triggering python script with ExecuteStreamCommand processor and one of the parameters needs to be JSON string e.g. '{"foo":"bar"}', so when the script receives it, it need to convert it into python dictionary. The error is due to incorrect handover of parameter

Re: InvokeHTTP with SSL

ng an SSL Context Service in the processor which > points to a truststore that trusts the certificate of the service you > are calling. > > Alternatively, if the CA certs system truststore trusts the service > cert then it should also work. > > Thanks, > > Bryan > > On Mon

InvokeHTTP with SSL

Hi all, I have a case where I need to send POST request on one enpoint which is located on K8S cluster and behind reverse proxy. Only HTTPS can be used. If I put value of endpoint using https:// I get error 'Unable to find valid certification path to requested target'. I spoke to my admin/devops

Re: MergeContent processor

quot; to > say 30 seconds or 60 seconds. That > way, as soon as MergeContent sees a single file, it will wait 30 seconds > or 60 seconds or whatever you have set, and > then merge together all of the files that it has queued up. > > Does that help? > > Thanks > -Mark >

Re: MergeContent processor

, Tomislav Novosel wrote: > Hi all, > > I need to create one flowfile from multiple flowfiles which are of files > in one directory. So if I have e.g. 8 files in directory, I want to merge > them and the output I want is 1 flowfile for further processing ( I want to > extract fo

MergeContent processor

Hi all, I need to create one flowfile from multiple flowfiles which are of files in one directory. So if I have e.g. 8 files in directory, I want to merge them and the output I want is 1 flowfile for further processing ( I want to extract folder name and path of that 8 files). I tried with

Re: When do I download the older releases of Apache NiFi

Hi Vijay, here you can find older releases: https://archive.apache.org/dist/nifi/ BR, Tom On Wed, 27 Feb 2019, 18:52 Vijay Chhipa, wrote: > Hi, > > Noticed that with the release of NiFi 1.9.0 there is not a link to > download version 1.7.1 and older, > > Is there a link to the archived

Re: How to integrate Secured Ragistry with Secured Nifi

Any ideas? Tom On Tue, 19 Feb 2019 at 11:08, Mike Thomsen wrote: > Copy pasta. > > On Tue, Feb 19, 2019 at 12:51 AM Tomislav Novosel > wrote: > >> Hi Mike, Kevin >> >> Thank you for your answers, I appreciate it. >> >> @Mike, why are you set

Re: How to integrate Secured Ragistry with Secured Nifi

is preety much the same regarding, hosts and ports. BR, Tom On Tue, 19 Feb 2019 at 06:51, Tomislav Novosel wrote: > Hi Mike, Kevin > > Thank you for your answers, I appreciate it. > > @Mike, why are you setting WEB_HTTP_HOST and WEB_HTTP_PORT when you are > using secu

Re: How to integrate Secured Ragistry with Secured Nifi

repo in which I have an example of connecting NiFi and >> Registry and docker conatiners using docker-compose: >> >> https://github.com/kevdoran/flowops >> >> Hope this helps, >> Kevin >> >> >> On February 18, 2019 at 10:08:54, Tomislav Novosel

How to integrate Secured Ragistry with Secured Nifi

Hi all, I generated standalone certificate with nifi-toolkit for my two Nifi instances and for Nifi registry instance. All are on the same domain so I used one certificate and its credentials for properties file (trustore path, keystore path, keystore passw, trustore passw). Auth is configured

Can't open summary-systems diagnostics

Hi all, I installed Nifi in cluster mode with two nodes and both secured with Kerberos for user Auth. When I want to see summary->system diagnostics I've got error: An unexpected error has occurred in UI and in the log: ERROR [NiFi Web Server-80406] o.a.nifi.web.api.config.ThrowableMapper An

Re: Nifi registry Kerberos Auth with Docker

ting you set > policies in the UI without having to manually create users that match the > LDAP directory users. > > Hope this helps, > Kevin > > > On February 13, 2019 at 03:56:52, Tomislav Novosel (to.novo...@gmail.com) > wrote: > > Also, FYI. > &g

Re: Nifi registry Kerberos Auth with Docker

then you get sent to the login > page. > > Maybe try turning on debug logging by editing logback.xml name="org.apache.nifi.registry" level="INFO"/> and changing to DEBUG. > > On Fri, Feb 8, 2019 at 9:51 AM Tomislav Novosel > wrote: > > > >

Re: Nifi registry Kerberos Auth with Docker

the following properties is not > >> populated: > >> > >> nifi.registry.kerberos.spnego.principal= > >> nifi.registry.kerberos.spnego.keytab.location= > >> > >> On Fri, Feb 8, 2019 at 8:20 AM Tomislav Novosel > wrote: > >> > > &

Re: Nifi registry Kerberos Auth with Docker

int first and if it returns the > conflict response (as the log says) then you get sent to the login > page. > > Maybe try turning on debug logging by editing logback.xml name="org.apache.nifi.registry" level="INFO"/> and changing to DEBUG. > > On Fri, Feb 8, 2019 at 9

Re: Nifi registry Kerberos Auth with Docker

nifi.registry" level="INFO"/> and changing to DEBUG. > > On Fri, Feb 8, 2019 at 9:51 AM Tomislav Novosel > wrote: > > > > Hi Bryan, > > > > I don't have this properties populated in Nifi registry instance > > outside Docker (as a

Re: Nifi registry Kerberos Auth with Docker

et login not supported by this > NiFi Registry" means that one of the following properties is not > populated: > > nifi.registry.kerberos.spnego.principal= > nifi.registry.kerberos.spnego.keytab.location= > > On Fri, Feb 8, 2019 at 8:20 AM Tomislav Novosel > wrote: > &g

Re: Nifi registry Kerberos Auth with Docker

t you switch to the official apache image that Kevin mentioned > and try again? It is an up to date version and recommended by the community. > > On Thu, Feb 7, 2019 at 5:54 PM Tomislav Novosel > wrote: > >> Hi Kevin, >> >> I'm using image from Docker hub on this link: >>

Re: Nifi registry Kerberos Auth with Docker

he apache/nifi-registry image or a custom image for this? > > Have you configured TLS? > Can you share your complete conf dir (removing sensitive values such as > password or domains)? > > Thanks, > Kevin > > > On February 7, 2019 at 05:57:37, Tomislav Novosel (to

Nifi registry Kerberos Auth with Docker

Hi all, I'm trying to configure Nifi registry authentication with Kerberos while Nifi registry runs inside Docker container. I configured all security properties in nifi-registry.properties, login identity provider and authorizers.xml. Everything the same as for Nifi registry running as a

[no subject]

Hi all, I'm trying to configure Nifi registry authentication with Kerberos while Nifi registry runs inside Docker container. I configured all security properties in nifi-registry.properties, login identity provider and authorizers.xml. Everything the same as for Nifi registry running as a

Filter duplicates by modification date

Hi all, I have specific case to filter duplicate files by modification date. I want to fetch only files with maximum modification date(the youngest files) if they are duplicated by filename. How can I achieve this? Thanks in advance, I appreciate it a lot. BR, Tom

Re: Minimum file age

; > > > *From: *Arpad Boda > *Reply-To: *"users@nifi.apache.org" > *Date: *Monday, 28 January 2019 at 10:17 > *To: *"users@nifi.apache.org" > *Subject: *Re: Minimum file age > > > > Hi, > > > > It’s on the way: https://issues.apac

Re: Modify Flowfile attributes

gt; *week_att = **date(year=int(date_final[0:1]), > month=int(date_final[2:3]), day=int(date_final[4:5])).isocalendar()[1]* > > > > Yeah, it’s not a solution, but I wonder if skipping strptime fixes the > problem or not. > > > > *From: *Tomislav Novosel > *Reply

Re: Modify Flowfile attributes

: > This sounds very strange. > > > > What happens if you do this: > > > > date_obj = datetime.strptime(‘20’ + date_final,'%Y%m%d') > > > > ? > > > > > > *From: *Tomislav Novosel > *Reply-To: *"users@nifi.apache.org" > *Date: *Wednes

Re: Modify Flowfile attributes

quot;.")[0]* > > *date_obj = datetime.strptime(date_final,'%y%m%d')* > > My testing assumed “date_final” to be “181231”, which I guess doesn’t > apply for your case. > > > > Could you *modify* your python code to add the two variables (marked red) > as attributes to y

Re: Modify Flowfile attributes

> expected (the result is 1st week of 2019). > > > > *From: *Tomislav Novosel > *Reply-To: *"users@nifi.apache.org" > *Date: *Wednesday, 30 January 2019 at 11:10 > *To: *"users@nifi.apache.org" > *Subject: *Re: Modify Flowfile attributes > > &g

Re: Modify Flowfile attributes

that is exactly what I tried and date_final or date_file are applied to > the attribute of outgoing flowfile, it works.” > > > > It works as they are strings, so not working would be a surprise. The > question is: what are their values?  > > > > Regards, > &g

Re: Modify Flowfile attributes

> My gut feeling says there is some parsing magic going wrong here. > > > > Regards, > > Arpad > > > > *From: *Tomislav Novosel > *Reply-To: *"users@nifi.apache.org" > *Date: *Tuesday, 29 January 2019 at 20:13 > *To: *"users@nifi.apac

Re: Modify Flowfile attributes

bute(flowFile, "week_extracted", str_week) session.transfer(flowFile, REL_SUCCESS) session.commit() On Tue, 29 Jan 2019 at 16:59, Tomislav Novosel wrote: > Thank you all for answers. The reason why I want this to do with python > script is wrong calculation of w

Re: Modify Flowfile attributes

t in > general this is the approach I take when I need to do date munging. > > On Tue, Jan 29, 2019 at 10:06 AM Tomislav Novosel > wrote: > >> Hi Matt, thanks for suggestions. But performance is not crucial here. >> This is code i tried. but I get error: "AttributeErro

Re: Modify Flowfile attributes

; of attribute name/value pairs, and use putAllAttributes(flowFile, > attributes) instead, as it is more performant. > > Regards, > Matt > > On Tue, Jan 29, 2019 at 9:25 AM Tomislav Novosel > wrote: > > > > Thanks for the answer. > > > > Yes I know

Re: Modify Flowfile attributes

getDelimitedField() I listed > and I received 6.2. Then another UpdateAttribute could parse the 6.2 for > the second substring, or you might be able to chain them in the existing > UpdateProcessor. > > > ---- > On Tue, 1/29/19, Tomisla

Modify Flowfile attributes

Hi all, I'm trying to calculate week number and date from filename using ExecuteScript processor and Jython. Here is python script. How can I add calculated attributes week and year to flowfile? Please help, thank you. Tom P.S. Maybe I completely missed with this script. Feel free to correct

Re: Minimum file age

10:17 > *To: *"users@nifi.apache.org" > *Subject: *Re: Minimum file age > > > > Hi, > > > > It’s on the way: https://issues.apache.org/jira/browse/NIFI-5977 :) > > > > Regards, > > Arpad > > > > *From: *Tomislav Novosel > *

Minimum file age

Hi all, I'm having issue with SFTPList processor in Nifi. When reading files from folder where another process writes files, it lists the same file multiple time and ingests file multiple times because modification date of file changes rapidly as another process writes to it. It appears that

Nifi fetching files

Hi Nifi team, my usecase is to list files and fetch files using ListFile and FetchFile processors from intermediate folder which is also destination of my external .exe script. Fetching from that folder has completion strategy to delete files. How can I wait for let's say 10 minutes before

Re: Putting data to Solr index using Nifi

nsider > your experience with Python to be fungible with the Java APIs because they > use ZooKeeper for SolrCloud. > > Mike > > On Thu, Sep 27, 2018 at 9:24 AM Tomislav Novosel > wrote: > >> Hi Mike, thanks for the answer. >> >> How should I configure PutSolrCo

Re: Putting data to Solr index using Nifi

ng > ZooKeeper and going straight to Solr since the interface is the same > between cloud and non-cloud Solr for everything that matters here. > > On Thu, Sep 27, 2018 at 7:56 AM Tomislav Novosel > wrote: > >> Hi all, >> >> >> >> I'm trying to put data i

Putting data to Solr index using Nifi

Hi all, I'm trying to put data in Solr Index using Nifi. Solr is v7.5.0 and Nifi is v1.6.0. I'm using PutSolrContentStream processor and Solr in Solrcloud mode with embedded zookeeper Inside docker container. I exposed Solr admin port and zookeeper port to be accessible through browser. I