I should add you can use NiFi to update the reference dataset in a database/backing store in one flow. And have another flow that handles the live stream/lookup,etc. MarkPayne/Others: I think there are blogs that describe this pattern. Anyone have links?
On Fri, Feb 22, 2019 at 12:27 PM Joe Witt <joe.w...@gmail.com> wrote: > Boris, > > Great. So have a process to load the periodic dataset into a lookup > service. COuld be backed by a simple file, a database, Hive, whatever. > Then have the live flow run against that. > > This reminds me - we should make a Kudu based lookup service i think. > I'll chat with some of our new Kudu friends on this. > > Thanks > > On Fri, Feb 22, 2019 at 12:25 PM Boris Tyukin <bo...@boristyukin.com> > wrote: > >> Thanks Joe and Bryan. In this case I don't need to do it in real-time, >> probably once a day only. >> >> I am thinking to trigger both pulls by generateflow processor, then merge >> datasets somehow since flowfile id will be the same for both sets. And then >> need to join somehow. >> >> Would like to use nifi still :) >> >
