Hi Chamikara, I've looking the stable reads from Apache Beam, and I'm not sure if that is what I want, even thought I make redshift input stable, it is a bounded PCollection, Eventually new items are added to the redshift table and those changes are not reflected in the Pcollection. On Mon, 30 Jul 2018 at 12:03, Jose Bermeo <[email protected]> wrote:
> Hi JB. > > I'm not sure, I could create two PCollections, the question is how do I > make the PCollection from Redshift reflect the changes in the table? To > refrease my initial question, each element in my PCollection has a > foreing_key_id, I need to check if the row associated with the foreing_key_id > in redshift is valid? issue is that my PCollection is unbound (new elements > with different foreing_key_id can show) and redshift table is also > changing. > > Regards > > On Mon, 30 Jul 2018 at 08:16, Jean-Baptiste Onofré <[email protected]> > wrote: > >> Hi Jose, >> >> so basically, you create two PCollections with the same keys and then >> you join/filter/flatten ? >> >> Regards >> JB >> >> On 30/07/2018 15:09, Jose Bermeo wrote: >> > Hi, question guys. >> > >> > I have to filter an unbounded collection based on data from a redshift >> > DB. I cannot use a side input as redshift data could change. One way to >> > do it would be to group common elements, make a query to filter each >> > group, finally flatten the pipe again.Do you know if this is the best >> > way to do it? and what would be the way to run the query agains >> redshift?. >> > >> > Thaks. >> >> -- >> Jean-Baptiste Onofré >> [email protected] >> http://blog.nanthrax.net >> Talend - http://www.talend.com >> >
