Hi JB. I'm not sure, I could create two PCollections, the question is how do I make the PCollection from Redshift reflect the changes in the table? To refrease my initial question, each element in my PCollection has a foreing_key_id, I need to check if the row associated with the foreing_key_id in redshift is valid? issue is that my PCollection is unbound (new elements with different foreing_key_id can show) and redshift table is also changing.
Regards On Mon, 30 Jul 2018 at 08:16, Jean-Baptiste Onofré <[email protected]> wrote: > Hi Jose, > > so basically, you create two PCollections with the same keys and then > you join/filter/flatten ? > > Regards > JB > > On 30/07/2018 15:09, Jose Bermeo wrote: > > Hi, question guys. > > > > I have to filter an unbounded collection based on data from a redshift > > DB. I cannot use a side input as redshift data could change. One way to > > do it would be to group common elements, make a query to filter each > > group, finally flatten the pipe again.Do you know if this is the best > > way to do it? and what would be the way to run the query agains > redshift?. > > > > Thaks. > > -- > Jean-Baptiste Onofré > [email protected] > http://blog.nanthrax.net > Talend - http://www.talend.com >
