Yes this is done in multiple IOs now, you can see how it is done in JdbcIO or a simpler but without explicit ReadAll transform in SolrIO. Notice that this change requires a refactor of the IO to avoid code repetition. I filled https://issues.apache.org/jira/browse/BEAM-7769 and assigned it to you, feel free to unassign if you don' t plan to work on it. In the meantime I am going to try to expose ReadAll on SolrIO so we can have it as a reference.
On Thu, Jul 18, 2019 at 11:08 AM Chaim Turkel <ch...@behalf.com> wrote: > > is there another source that does this so i can have a look and add it > to the MongoDBIO? > > On Wed, Jul 17, 2019 at 9:48 PM Eugene Kirpichov <kirpic...@google.com> wrote: > > > > I think full-blown SDF is not needed for this - someone just needs to > > implement a MongoDbIO.readAll() variant, using a composite transform. The > > regular pattern for this sort of thing will do (ParDo split, reshuffle, > > ParDo read). > > Whether it's worth replacing MongoDbIO.read() with a redirect to readAll() > > is another matter - size estimation, available only in BoundedSource for > > now, may or may not be important. > > > > On Wed, Jul 17, 2019 at 2:39 AM Ryan Skraba <r...@skraba.com> wrote: > >> > >> Hello! To clarify, you want to do something like this? > >> > >> PubSubIO.read() -> extract mongodb collection and range -> > >> MongoDbIO.read(collection, range) -> ... > >> > >> If I'm not mistaken, it isn't possible with the implementation of > >> MongoDbIO (based on BoundedSource interface, requiring the collection to > >> be specified once at pipeline construction time). > >> > >> BUT -- this is a good candidate for an improvement in composability, and > >> the ongoing work to prefer the SDF for these types of use cases. Maybe > >> raise a JIRA for an improvement? > >> > >> All my best, Ryan > >> > >> > >> On Wed, Jul 17, 2019 at 9:35 AM Chaim Turkel <ch...@behalf.com> wrote: > >>> > >>> any ideas? > >>> > >>> On Mon, Jul 15, 2019 at 11:04 PM Rui Wang <ruw...@google.com> wrote: > >>> > > >>> > +u...@beam.apache.org > >>> > > >>> > > >>> > -Rui > >>> > > >>> > On Mon, Jul 15, 2019 at 6:55 AM Chaim Turkel <ch...@behalf.com> wrote: > >>> >> > >>> >> Hi, > >>> >> I am looking to write a pipeline that read from a mongo collection. > >>> >> I would like to listen to a pubsub that will have a object that will > >>> >> tell me which collection and which time frame. > >>> >> Is there a way to do this? > >>> >> > >>> >> Chaim > >>> >> > >>> >> -- > >>> >> > >>> >> > >>> >> Loans are funded by > >>> >> FinWise Bank, a Utah-chartered bank located in Sandy, > >>> >> Utah, member FDIC, Equal > >>> >> Opportunity Lender. Merchant Cash Advances are > >>> >> made by Behalf. For more > >>> >> information on ECOA, click here > >>> >> <https://www.behalf.com/legal/ecoa/>. For important information about > >>> >> opening a new > >>> >> account, review Patriot Act procedures here > >>> >> <https://www.behalf.com/legal/patriot/>. > >>> >> Visit Legal > >>> >> <https://www.behalf.com/legal/> to > >>> >> review our comprehensive program terms, > >>> >> conditions, and disclosures. > >>> > >>> -- > >>> > >>> > >>> Loans are funded by > >>> FinWise Bank, a Utah-chartered bank located in Sandy, > >>> Utah, member FDIC, Equal > >>> Opportunity Lender. Merchant Cash Advances are > >>> made by Behalf. For more > >>> information on ECOA, click here > >>> <https://www.behalf.com/legal/ecoa/>. For important information about > >>> opening a new > >>> account, review Patriot Act procedures here > >>> <https://www.behalf.com/legal/patriot/>. > >>> Visit Legal > >>> <https://www.behalf.com/legal/> to > >>> review our comprehensive program terms, > >>> conditions, and disclosures. > > -- > > > Loans are funded by > FinWise Bank, a Utah-chartered bank located in Sandy, > Utah, member FDIC, Equal > Opportunity Lender. Merchant Cash Advances are > made by Behalf. For more > information on ECOA, click here > <https://www.behalf.com/legal/ecoa/>. For important information about > opening a new > account, review Patriot Act procedures here > <https://www.behalf.com/legal/patriot/>. > Visit Legal > <https://www.behalf.com/legal/> to > review our comprehensive program terms, > conditions, and disclosures.
