I have been working on this, and have a question. The flow is: message -> by message get data -> continue flow.
I seem to need some of the information from the original message in the next parts of my flow. Is there a simple way to do this using side input and windows? do i need to add support for KV when getting the data to pass on the orig object? ideas? thanks chaim On Thu, Jul 18, 2019 at 12:45 PM Ismaël Mejía <ieme...@gmail.com> wrote: > > Just discovered that RedisIO exposes ReadAll too so you can take a > look at that one too. > > On Thu, Jul 18, 2019 at 11:39 AM Ismaël Mejía <ieme...@gmail.com> wrote: > > > > Yes this is done in multiple IOs now, you can see how it is done in > > JdbcIO or a simpler but without explicit ReadAll transform in SolrIO. > > Notice that this change requires a refactor of the IO to avoid code > > repetition. > > I filled https://issues.apache.org/jira/browse/BEAM-7769 and assigned > > it to you, feel free to unassign if you don' t plan to work on it. > > In the meantime I am going to try to expose ReadAll on SolrIO so we > > can have it as a reference. > > > > On Thu, Jul 18, 2019 at 11:08 AM Chaim Turkel <ch...@behalf.com> wrote: > > > > > > is there another source that does this so i can have a look and add it > > > to the MongoDBIO? > > > > > > On Wed, Jul 17, 2019 at 9:48 PM Eugene Kirpichov <kirpic...@google.com> > > > wrote: > > > > > > > > I think full-blown SDF is not needed for this - someone just needs to > > > > implement a MongoDbIO.readAll() variant, using a composite transform. > > > > The regular pattern for this sort of thing will do (ParDo split, > > > > reshuffle, ParDo read). > > > > Whether it's worth replacing MongoDbIO.read() with a redirect to > > > > readAll() is another matter - size estimation, available only in > > > > BoundedSource for now, may or may not be important. > > > > > > > > On Wed, Jul 17, 2019 at 2:39 AM Ryan Skraba <r...@skraba.com> wrote: > > > >> > > > >> Hello! To clarify, you want to do something like this? > > > >> > > > >> PubSubIO.read() -> extract mongodb collection and range -> > > > >> MongoDbIO.read(collection, range) -> ... > > > >> > > > >> If I'm not mistaken, it isn't possible with the implementation of > > > >> MongoDbIO (based on BoundedSource interface, requiring the collection > > > >> to be specified once at pipeline construction time). > > > >> > > > >> BUT -- this is a good candidate for an improvement in composability, > > > >> and the ongoing work to prefer the SDF for these types of use cases. > > > >> Maybe raise a JIRA for an improvement? > > > >> > > > >> All my best, Ryan > > > >> > > > >> > > > >> On Wed, Jul 17, 2019 at 9:35 AM Chaim Turkel <ch...@behalf.com> wrote: > > > >>> > > > >>> any ideas? > > > >>> > > > >>> On Mon, Jul 15, 2019 at 11:04 PM Rui Wang <ruw...@google.com> wrote: > > > >>> > > > > >>> > +u...@beam.apache.org > > > >>> > > > > >>> > > > > >>> > -Rui > > > >>> > > > > >>> > On Mon, Jul 15, 2019 at 6:55 AM Chaim Turkel <ch...@behalf.com> > > > >>> > wrote: > > > >>> >> > > > >>> >> Hi, > > > >>> >> I am looking to write a pipeline that read from a mongo > > > >>> >> collection. > > > >>> >> I would like to listen to a pubsub that will have a object that > > > >>> >> will > > > >>> >> tell me which collection and which time frame. > > > >>> >> Is there a way to do this? > > > >>> >> > > > >>> >> Chaim > > > >>> >> > > > >>> >> -- > > > >>> >> > > > >>> >> > > > >>> >> Loans are funded by > > > >>> >> FinWise Bank, a Utah-chartered bank located in Sandy, > > > >>> >> Utah, member FDIC, Equal > > > >>> >> Opportunity Lender. Merchant Cash Advances are > > > >>> >> made by Behalf. For more > > > >>> >> information on ECOA, click here > > > >>> >> <https://www.behalf.com/legal/ecoa/>. For important information > > > >>> >> about > > > >>> >> opening a new > > > >>> >> account, review Patriot Act procedures here > > > >>> >> <https://www.behalf.com/legal/patriot/>. > > > >>> >> Visit Legal > > > >>> >> <https://www.behalf.com/legal/> to > > > >>> >> review our comprehensive program terms, > > > >>> >> conditions, and disclosures. > > > >>> > > > >>> -- > > > >>> > > > >>> > > > >>> Loans are funded by > > > >>> FinWise Bank, a Utah-chartered bank located in Sandy, > > > >>> Utah, member FDIC, Equal > > > >>> Opportunity Lender. Merchant Cash Advances are > > > >>> made by Behalf. For more > > > >>> information on ECOA, click here > > > >>> <https://www.behalf.com/legal/ecoa/>. For important information about > > > >>> opening a new > > > >>> account, review Patriot Act procedures here > > > >>> <https://www.behalf.com/legal/patriot/>. > > > >>> Visit Legal > > > >>> <https://www.behalf.com/legal/> to > > > >>> review our comprehensive program terms, > > > >>> conditions, and disclosures. > > > > > > -- > > > > > > > > > Loans are funded by > > > FinWise Bank, a Utah-chartered bank located in Sandy, > > > Utah, member FDIC, Equal > > > Opportunity Lender. Merchant Cash Advances are > > > made by Behalf. For more > > > information on ECOA, click here > > > <https://www.behalf.com/legal/ecoa/>. For important information about > > > opening a new > > > account, review Patriot Act procedures here > > > <https://www.behalf.com/legal/patriot/>. > > > Visit Legal > > > <https://www.behalf.com/legal/> to > > > review our comprehensive program terms, > > > conditions, and disclosures. -- Loans are funded by FinWise Bank, a Utah-chartered bank located in Sandy, Utah, member FDIC, Equal Opportunity Lender. Merchant Cash Advances are made by Behalf. For more information on ECOA, click here <https://www.behalf.com/legal/ecoa/>. For important information about opening a new account, review Patriot Act procedures here <https://www.behalf.com/legal/patriot/>. Visit Legal <https://www.behalf.com/legal/> to review our comprehensive program terms, conditions, and disclosures.
