Re: Is there a configuration to limit the size of nifi's flowfile repository

...@gmail.com> > > > wrote: > > > > > > > Hi Ben, > > > > > > > > Since the flow file repository contains the information of the flow > > files > > > > currently being processed by NiFi, you don't want to limit that > &

Is there a configuration to limit the size of nifi's flowfile repository

Hi guys, I checked NIFI's system administrator guide trying to find a configuration item so that the size of the flowfile repository could be limited similar to the other repositories(e.g. content repository), but I didn't find such configuration items, is there currently any configuration to

Re: get access token inside custom processor

i framework and have no idea which user started/stopped > them. > > Thanks, > > Bryan > > On Tue, Feb 27, 2018 at 1:27 AM, 尹文才 <batman...@gmail.com> wrote: > > Hi guys, I'm trying to invoke some nifi rest apis inside my custom > > processor, the nifi I'm using i

get access token inside custom processor

Hi guys, I'm trying to invoke some nifi rest apis inside my custom processor, the nifi I'm using is nifi 1.4.0 and it's a 3 node secured cluster., the username and password are kept inside a ldap server. I know that in a secured nifi cluster, in order to make any request I need the access token,

Unable to start NiFi

Hi guys, one of my colleagues was working on a remote deploy tool trying to start NiFi remotely on a windows system(NiFi package is on the windows system). I don't know how exactly the deployment tool works, but the basic flow is to launch Ansible locally which uses winrm to remotely launch NiFi

Re: clear all flowfiles in all queues upon NiFi restart

epository > subdirectories under ${NIFI_ROOT}, it will result in what you seem to be > asking for. > > Hope this helps. > > > On 01/11/2018 09:48 PM, 尹文才 wrote: > >> Hi Andrew, sorry I didn't follow your idea, could you please elaborate >> with >> more detai

Re: clear all flowfiles in all queues upon NiFi restart

you could delete the repository directories when you need to > restart with no data? > > On Thu, Jan 11, 2018, 9:16 PM 尹文才 <batman...@gmail.com> wrote: > > > Hi Mark, forgot to ask about VolatileFlowFileRepository you mentioned, > if I > > switch to use Volatile

Re: clear all flowfiles in all queues upon NiFi restart

Hi Mark, forgot to ask about VolatileFlowFileRepository you mentioned, if I switch to use VolatileFlowFileRepository, will NiFi swap out all the other FlowFiles to disk if a queue is already full? Is it just simply keeping all FlowFiles in memory? Regards, Ben 2018-01-12 12:07 GMT+08:00 尹文才

Re: clear all flowfiles in all queues upon NiFi restart

FlowFile associated with it. > > I’m on my phone right now so can’t easily tell you the exact name of the > property to change but you’ll probably find it pretty quickly. The Admin > Guide may well explain the different repositories as well. > > Thanks > -Mark > > Sent

clear all flowfiles in all queues upon NiFi restart

Hi guys, I'm trying to clear all FlowFIles in all queues when NiFi is restarted, but I don't know the correct way to do this. I checked all NiFi's guide documentation, it seems there're 2 possible solutions: 1. write a custom notification service: a notification service could be notified when NiFi

Re: DBCP connection pool problem - connection already closed

wever; I’ve always used this driver for ms sql > server and never encountered issues. > > [1]: http://jtds.sourceforge.net/faq.html > > > On 11 Jan 2018, at 18:56, 尹文才 <batman...@gmail.com> wrote: > > > > Hi guys, I'm using the PutDatabaseRecord processor to w

Re: NiFi data HA in cluster mode

e.w...@gmail.com>: > I'd avoid setting any processor to primary node only unless it is a > source processor (something that brings data into the system). > > But, yes, I believe your description is accurate as of now. > > Thanks > > On Mon, Jan 8, 2018 at 11:21 PM, 尹

Re: NiFi data HA in cluster mode

d balancing or fail-over as-is > present in the Apache NiFi Site to Site protocol. > > Thanks > Joe > > On Mon, Jan 8, 2018 at 11:05 PM, 尹文才 <batman...@gmail.com> wrote: > > Hi guys, I have a question about data HA when NiFi is run in clustered > > mode, if one

NiFi data HA in cluster mode

Hi guys, I have a question about data HA when NiFi is run in clustered mode, if one node goes down, will the flowfiles owned by this node taken over and processed by another node? Or will the flowfiles be kept locally to that node and will only be processed when that node is back online? Thanks.

Re: proper way in nifi to sync status between custom processors

a processor instance you are interested in, then > select 'View data provenance' context menu. This way, NiFi displays > provenance events for the selected processor. > > Koji > > On Wed, Dec 27, 2017 at 6:17 PM, 尹文才 <batman...@gmail.com> wrote: > > Hi Koji, sorry abou

Re: proper way in nifi to sync status between custom processors

Hi Koji, sorry about the provenance exception, it was because there's no space left on the machine(filled up with logs) Regards, Ben 2017-12-27 17:11 GMT+08:00 尹文才 <batman...@gmail.com>: > Hi Koji, thanks, the names of the temp tables are created with format > "M

Re: proper way in nifi to sync status between custom processors

/html/user-guide. > html#viewing-flowfile-lineage > > I didn't mention about it earlier because you were having Provenance > repository performance issue, but I hope you can use it now with the > WriteAheadProvenanceRepository. > > Thanks, > Koji > > On

Re: proper way in nifi to sync status between custom processors

throw ex; > > } finally { > > logger.debug("重新设置autoCommit为true"); > > con.setAutoCommit(true); > > } > > } catch (Exception ex) { > > // HERE, the except

Re: proper way in nifi to sync status between custom processors

te cause of getting FlowFile repository > failed should be investigated. Is it possible to share WaitBatch code? > The reason why ask this is all 'FlowFile Repository failed to update' > is related to WaitBatch processor in the log that you shared earlier. > > Thanks, > Koji >

Re: proper way in nifi to sync status between custom processors

itory, then the same > FlowFile can be processed again after restarting NiFi. > > Thanks, > Koji > > > > On Wed, Dec 27, 2017 at 12:21 PM, 尹文才 <batman...@gmail.com> wrote: > > Thanks Koji, I will look into this article about the record model. > > > &

Re: proper way in nifi to sync status between custom processors

and DistributedMapCacheServer > pair is not supported at the moment. If you need HighAvailability, > RedisDistributedMapCacheClientService with Redis replication will > provide that, I haven't tried that myself though. > https://redis.io/topics/replication > > Thanks, > Koji >

Re: proper way in nifi to sync status between custom processors

e all FlowFiles in a single FlowFile, instead of batches > of FlowFiles. If you need logical boundaries, you can use NiFi Record > data model to embed multiple records within a FlowFile, Record should > perform better. > > Hope this helps. > > Thanks, > Koji > > > On Tue,

proper way in nifi to sync status between custom processors

Hi guys, I'm currently trying to find a proper way in nifi which could sync status between my custom processors. our requirement is like this, we're doing some ETL work using nifi and I'm extracting the data from DB into batches of FlowFiles(each batch of FlowFile has a flag FlowFile indicating

Re: The rate of the dataflow is exceeding the provenance recording rate. Slowing down flow to accommodate

> For example, adding following entry will reveal how NiFi repositories run: > > > > > > > Thanks, > Koji > > On Mon, Dec 25, 2017 at 4:30 PM, 尹文才 <batman...@gmail.com> wrote: > > Hi Koji, I also didn't find anything related to the

Re: The rate of the dataflow is exceeding the provenance recording rate. Slowing down flow to accommodate

o any further investigation > with only these logs. > > Thanks, > Koji > > On Mon, Dec 25, 2017 at 3:46 PM, 尹文才 <batman...@gmail.com> wrote: > > Hi Koji, one more thing, do you have any idea why my first issue leads to > > the unexpected shutdown of NIFI? accord

Re: The rate of the dataflow is exceeding the provenance recording rate. Slowing down flow to accommodate

Hi Koji, one more thing, do you have any idea why my first issue leads to the unexpected shutdown of NIFI? according to the words, it will just slow down the flow. thanks. Regards, Ben 2017-12-25 14:31 GMT+08:00 尹文才 <batman...@gmail.com>: > Hi Koji, thanks for your help, for the first

Re: The rate of the dataflow is exceeding the provenance recording rate. Slowing down flow to accommodate

> https://nifi.apache.org/docs/nifi-docs/html/administration- > guide.html#provenance-repository > > > > Thanks, > > Koji > > > > On Mon, Dec 25, 2017 at 12:56 PM, 尹文才 <batman...@gmail.com> wrote: > >> Hi guys, I'm using nifi 1.4.0 to do some ETL wo

The rate of the dataflow is exceeding the provenance recording rate. Slowing down flow to accommodate

Hi guys, I'm using nifi 1.4.0 to do some ETL work in my team and I have encountered 2 problems during my testing. The first problem is I found the nifi bulletin board was showing the following warning to me: 2017-12-25 01:31:00,460 WARN [Provenance Maintenance Thread-1]

Re: Unable to checkpoint FlowFile Repository (No space left on device)

itory.archive.enabled=false in > your nifi.properties. > > -- Mike > > > On Thu, Dec 14, 2017 at 2:22 AM, 尹文才 <batman...@gmail.com> wrote: > > > One Strange thing one of our testers found was that we're using the > default > > 12hours archive and 50% disk spac

Re: Unable to checkpoint FlowFile Repository (No space left on device)

of the archived files inside using the rm command, the inodes's count did go down. /Ben 2017-12-14 9:49 GMT+08:00 尹文才 <batman...@gmail.com>: > Hi Michael, the no space left on device occurred again and I checked the > inodes at the time and found it was indeed full, why would the inodes &

Re: Unable to checkpoint FlowFile Repository (No space left on device)

Hi Michael, the no space left on device occurred again and I checked the inodes at the time and found it was indeed full, why would the inodes become full and are there any solutions to get around this problem? Thanks. /Ben 2017-12-13 13:36 GMT+08:00 尹文才 <batman...@gmail.com>: > Hi M

Re: Unable to checkpoint FlowFile Repository (No space left on device)

u can check this with "df -i". > > -- Mike > > > On Tue, Dec 12, 2017 at 1:36 AM, 尹文才 <batman...@gmail.com> wrote: > > > sorry that I forgot to mention the environment that caused this problem, > > I'm using the latest nifi 1.4.0 release and installed i

Re: Unable to checkpoint FlowFile Repository (No space left on device)

sorry that I forgot to mention the environment that caused this problem, I'm using the latest nifi 1.4.0 release and installed it on centos 7. 2017-12-12 14:35 GMT+08:00 尹文才 <batman...@gmail.com>: > Hi guys, I'm running into a very weird problem, I wrote a processor > specificall

Unable to checkpoint FlowFile Repository (No space left on device)

Hi guys, I'm running into a very weird problem, I wrote a processor specifically to extract some data and I found starting from yesterday it kept showing errors in the log, as below: 2017-12-12 14:01:04,661 INFO [pool-10-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of

Re: save variables to template when creating a template

le registry in dev/test and a different one for > production, for example. If you want to keep the variable you would > likely want to transfer the variable registry along with the template > to the other NiFi instance. > > Regards, > Matt > > On Tue, Oct 10, 2017 at 11:45 P

save variables to template when creating a template

Hi guys, I've been using the new variable registry feature in 1.4.0 and I like it very much. But when I define a variable in a processor group and try to create a template out out it, I noticed that the variable I defined is not included in the template after I import the template to another NIFI

Re: route flow based on variable

something like ${env:equals(“prod”)} which would only > enable this path in prod. > > Variables on their own are not associated with flow files, so if you > wanted to do something more specific per-flow file, then the only way would > be what you described with using UpdateAttribute. >

route flow based on variable

Hi guys, I've played around with the latest NIFI 1.4.0 release for a while and I think the new variable registry feature is great, however I have 2 questions about this feature: 1. It seems that I could only add variables to a processor group, could I add a global variable in the NIFI root

Re: NIFI templates in template folder in sync with templates inside NIFI UI

t; Matt > > [1] https://nifi.apache.org/docs/nifi-docs/rest-api/index.html > > On Thu, Sep 14, 2017 at 2:40 AM, 尹文才 <batman...@gmail.com> wrote: > > > Hi guys, I put all my flow template xml files inside the nifi template > > directory specified inside the nifi.prope

NIFI templates in template folder in sync with templates inside NIFI UI

Hi guys, I put all my flow template xml files inside the nifi template directory specified inside the nifi.properties config file so that nifi could read in all my templates. But sometimes I need to clear all the templates currently inside nifi and then place in a new template, I would remove all

Re: how to execute code when processor is stopping

ache/nifi/processors/kafka/pubsub/ > ConsumeKafka_0_10.java#L316 > > Thanks, > Koji > > On Mon, Aug 14, 2017 at 11:12 AM, 尹文才 <batman...@gmail.com> wrote: > > Hi guys, about my case, I have another question, if I implement the retry > > logic inside the ontrigger method and I ne

Re: how to execute code when processor is stopping

ontrigger will keep executing the retry logic and the processor couldn't be stopped even if user tries to stop it, is there any way to solve this problem? Thanks. Regards, Ben 2017-08-12 6:30 GMT+08:00 尹文才 <batman...@gmail.com>: > Hi Bryan and Matt, thanks for all your suggestions, I w

Re: how to execute code when processor is stopping

be to start looping again right? > > > > An alternative that might avoid some of this would be to lazily > > initialize the connection in the onTrigger method of the processor. > > > > -Bryan > > > > > > On Fri, Aug 11, 2017 at 9:16 AM, 尹文才 <batman...@gmail.

Re: Assigning JIRAs to myself

Hi Aldrin, I would also like to be added to the list of NIFI contributors to start working on some easy JIRAS, thanks. Regards, Ben 2017-08-11 20:03 GMT+08:00 Aldrin Piri : > Hi Arun, > > I've added you to the list of contributors and you should be able to self > assign

Re: how to execute code when processor is stopping

one? Thanks. Regards, Ben 2017-08-11 15:51 GMT+08:00 Pierre Villard <pierre.villard...@gmail.com>: > Hi Ben, > > You might want to have a look here: > https://nifi.apache.org/docs/nifi-docs/html/developer- > guide.html#component-lifecycle > > Pierre > > 2017

how to execute code when processor is stopping

Hi guys, I'm trying to execute some code in my processor when the processor is asked to stop in the NIFI UI by the user, I checked the developer guide and only find OnUnscheduled will be called when the processor is no long scheduled to run. I've tested this OnUnscheduled, it will also be called

Re: get controller service's configuration

rties because DBCPConnectionPool.DB_DRIVERNAME is > specific to the DBCPConnectionPool implementation... another > implementation may not have that property, or may call it something > different. The interface would have to provide getDriverName() so that > each implementation could pro

Re: get controller service's configuration

> main/java/org/apache/nifi/processors/standard/GetHTTP.java#L295 > > Andy LoPresto > alopre...@apache.org > *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>* > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > > On Aug 9, 2017, at 6:57

Re: get controller service's configuration

n, this discussion might be helpful: > https://github.com/apache/nifi/pull/1417 > > Although I would not recommend, if you really need to obtain all > configurations, you can do so by calling NiFi REST API from your > processor. > > Thanks, > Koji > > On Thu, Aug 10, 2017 at

get controller service's configuration

Hi guys, I have a customized processor with a DBCP controller service as a property. I could get the DBCP controller service in my code, but does anyone know how to obtain all the configurations of the DBCP controller service in java code(e.g. Database Connection URL, Database Driver Location,

Re: how to submit bug found in NIFI code

; > Regards, > Janosch > > > > On 25.07.17, 11:30, "尹文才" <batman...@gmail.com> wrote: > > >Hi guys, I wonder if any of you knows the correct way to submit bugs found > >in current NIFI's code, I checked the JIRA page of NIFI and it seems that > I > >need to login to be able to submit an issue. Thanks. > > > >Regards, > >Ben >

how to submit bug found in NIFI code

Hi guys, I wonder if any of you knows the correct way to submit bugs found in current NIFI's code, I checked the JIRA page of NIFI and it seems that I need to login to be able to submit an issue. Thanks. Regards, Ben

Re: Trigger timer-driven processor to run from outside NIFI

I have an > example here. Please refer updateProcessorState function. > https://github.com/ijokarumawak/nifi-api-client-js/blob/master/nifi-api- > client.js > > Trying to reply from mobile at best effort. Excuse me for typos if any.. > > Thanks, > Koji > > > On Jul 1

Re: Trigger timer-driven processor to run from outside NIFI

led (next > 5min in your case). > > Thanks, > Koji > > On Wed, Jul 12, 2017 at 5:04 PM, 尹文才 <batman...@gmail.com> wrote: > > Hi guys, is it possible for a Java application outside the NIFI > environment > > to trigger a timer-driven processor to do its work(

Trigger timer-driven processor to run from outside NIFI

Hi guys, is it possible for a Java application outside the NIFI environment to trigger a timer-driven processor to do its work(I mean its ontrigger method will be called) when the processor is not yet due to be triggered? The reason why I'm asking about this is because I have a Java applicatiion

Re: FlowFile position when transferred to Relationship.SELF

i-docs/html/user-guide.html#prioritization > > I'm just curious about the functionality you added. Wait processor has > 'Releasable FlowFile Count' and it could be used to make a batch of > FlowFiles wait and go. Or Notify's 'Signal Counter Delta' could be > useful, too. > > R

FlowFile position when transferred to Relationship.SELF

Hi guys, I have written a customized processor whose functionality is similar to the NIFI's Wait processor, the difference is my processor needs to wait a batch of data and when the batch end flag is found, it will transfer the batch of data to destinations. I checked the source code of Wait

Re: get ControllerService inside Advanced custom UI

aster/nifi-nar- > bundles/nifi-standard-bundle/nifi-standard-processors/src/ > main/java/org/apache/nifi/processors/standard/InvokeHTTP.java#L205 > > On Sun, Jul 2, 2017 at 12:15 AM, 尹文才 <batman...@gmail.com> wrote: > > > Thanks Matt, how should I get the service identi

Re: get ControllerService inside Advanced custom UI

ill be the identifier of Processor. You'll need to invoke > > ComponentDetails getComponentDetails(NiFiWebRequestContext requestContext) > > in order to get the service identifier from the configured properties. > > Thanks > > Matt > > On Sat, Jul 1, 2017 at 4:17 AM, 尹

get ControllerService inside Advanced custom UI

Hi guys, I' m currently creating a custom UI for my processor and I need to get the DBCP Connection ControllerService in the backend code for the UI, the ControllerService is a property defined in the processor properties. I saw there's one method in class NiFiWebConfigurationContext as below: