How do you guys handle schedule (one source every 4 hrs, other source every 24 hrs,...) as a parameter?
Thanks Milind On Aug 13, 2017 11:32 AM, "dan young" <[email protected]> wrote: > We've done something similar as Carlos outlines here. Works really great > > Dano > > On Sun, Aug 13, 2017, 9:39 AM Carlos Manuel Fernandes (DSI) < > [email protected]> wrote: > >> Hi Vikram, >> >> >> >> I had the same requirements like you , and my solution is a service: >> >> >> >> HandleHttpRequest (prepared to handle http://host:8085/ods? >> sourceTable=tableName&targetTable=tableName&truncate=Y) -> >> >> ExecuteScript (Script reads the http parameters and/or read extra >> parameters from a database table and make the synchronization) -> >> >> HandleHttpResponse (return 200 if ok ) >> >> >> >> After this service are ready, you can create a scheduled invoker , like >> this: >> >> >> >> ExecuteScript (get all the tables names you need to synchronize, in my >> case based on a query : select source_table,target_Table,,truncate from >> ods_tables ) -> >> >> invokeHttp (http://host:8085/ods?sourceTable=${source_table}&; >> targetTable=${target_table}&truncate=${truncate}) -> >> >> putEmail (sending the result of the synchronization). >> >> >> >> >> >> With this you don’t need to repeat flows, I hope this help. >> >> >> >> Carlos Fernandes >> >> >> >> >> >> >> >> >> >> *From:* Andy LoPresto [mailto:[email protected]] >> *Sent:* sábado, 12 de agosto de 2017 01:36 >> *To:* [email protected] >> *Subject:* Re: Parameterizing the nifi flow >> >> >> >> The variable registry is a great tool for parameterizing values that >> differ between environments/deployments. In this case it sounds like >> setting up a flow that read from a flowfile attribute/content to determine >> the source table’s name is a better fit. You can create a master list of >> all source tables and store it as a plaintext file, then on your schedule, >> read the contents of that file, split the content by line, and send each >> 100+ resulting flowfiles into the part of the flow that reads from the >> database. Each would provide the name of the source table to read at that >> time. >> >> >> >> Andy LoPresto >> >> [email protected] >> >> *[email protected] <[email protected]>* >> >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 >> >> >> >> On Aug 11, 2017, at 3:26 PM, Andrew Grande <[email protected]> wrote: >> >> >> >> Hi, >> >> Read up on the variable registry in the docs, that sounds like a good >> fit. I don't remember if it were available in 1.1 though. >> >> Andrew >> >> >> >> On Fri, Aug 11, 2017, 5:12 PM More, Vikram (CONT) < >> [email protected]> wrote: >> >> Hi, >> >> >> >> I have a nifi flow which pulls/extracts from source database table and >> loads into target database table. This flow will run several times in a day >> to get delta records from source table (more like batch process running >> every 3-4 hrs). Now I need to replicate this same process for 100+ >> different source tables. So rather than creating 100+ nifi flows for each >> separate table, can I create main flow (let's say template) and pass >> parameter like source extract sql, target load sql to main flow. And repeat >> these steps for each source table . Has anyone tried parameterizing the >> nifi flows, can you please advice . We are using NiFi 1.1.0 >> >> >> >> Appreciate any thoughts here. >> >> >> >> >> >> Thanks & Regards, >> >> *Vikram* >> >> >> >> >> ------------------------------ >> >> The information contained in this e-mail is confidential and/or >> proprietary to Capital One and/or its affiliates and may only be used >> solely in performance of work or services for Capital One. The information >> transmitted herewith is intended only for use by the individual or entity >> to which it is addressed. If the reader of this message is not the intended >> recipient, you are hereby notified that any review, retransmission, >> dissemination, distribution, copying or other use of, or taking of any >> action in reliance upon this information is strictly prohibited. If you >> have received this communication in error, please contact the sender and >> delete the material from your computer. >> >> >> >
