is there another source that does this so i can have a look and add it to the MongoDBIO?
On Wed, Jul 17, 2019 at 9:48 PM Eugene Kirpichov <kirpic...@google.com> wrote: > > I think full-blown SDF is not needed for this - someone just needs to > implement a MongoDbIO.readAll() variant, using a composite transform. The > regular pattern for this sort of thing will do (ParDo split, reshuffle, ParDo > read). > Whether it's worth replacing MongoDbIO.read() with a redirect to readAll() is > another matter - size estimation, available only in BoundedSource for now, > may or may not be important. > > On Wed, Jul 17, 2019 at 2:39 AM Ryan Skraba <r...@skraba.com> wrote: >> >> Hello! To clarify, you want to do something like this? >> >> PubSubIO.read() -> extract mongodb collection and range -> >> MongoDbIO.read(collection, range) -> ... >> >> If I'm not mistaken, it isn't possible with the implementation of MongoDbIO >> (based on BoundedSource interface, requiring the collection to be specified >> once at pipeline construction time). >> >> BUT -- this is a good candidate for an improvement in composability, and the >> ongoing work to prefer the SDF for these types of use cases. Maybe raise a >> JIRA for an improvement? >> >> All my best, Ryan >> >> >> On Wed, Jul 17, 2019 at 9:35 AM Chaim Turkel <ch...@behalf.com> wrote: >>> >>> any ideas? >>> >>> On Mon, Jul 15, 2019 at 11:04 PM Rui Wang <ruw...@google.com> wrote: >>> > >>> > +u...@beam.apache.org >>> > >>> > >>> > -Rui >>> > >>> > On Mon, Jul 15, 2019 at 6:55 AM Chaim Turkel <ch...@behalf.com> wrote: >>> >> >>> >> Hi, >>> >> I am looking to write a pipeline that read from a mongo collection. >>> >> I would like to listen to a pubsub that will have a object that will >>> >> tell me which collection and which time frame. >>> >> Is there a way to do this? >>> >> >>> >> Chaim >>> >> >>> >> -- >>> >> >>> >> >>> >> Loans are funded by >>> >> FinWise Bank, a Utah-chartered bank located in Sandy, >>> >> Utah, member FDIC, Equal >>> >> Opportunity Lender. Merchant Cash Advances are >>> >> made by Behalf. For more >>> >> information on ECOA, click here >>> >> <https://www.behalf.com/legal/ecoa/>. For important information about >>> >> opening a new >>> >> account, review Patriot Act procedures here >>> >> <https://www.behalf.com/legal/patriot/>. >>> >> Visit Legal >>> >> <https://www.behalf.com/legal/> to >>> >> review our comprehensive program terms, >>> >> conditions, and disclosures. >>> >>> -- >>> >>> >>> Loans are funded by >>> FinWise Bank, a Utah-chartered bank located in Sandy, >>> Utah, member FDIC, Equal >>> Opportunity Lender. Merchant Cash Advances are >>> made by Behalf. For more >>> information on ECOA, click here >>> <https://www.behalf.com/legal/ecoa/>. For important information about >>> opening a new >>> account, review Patriot Act procedures here >>> <https://www.behalf.com/legal/patriot/>. >>> Visit Legal >>> <https://www.behalf.com/legal/> to >>> review our comprehensive program terms, >>> conditions, and disclosures. -- Loans are funded by FinWise Bank, a Utah-chartered bank located in Sandy, Utah, member FDIC, Equal Opportunity Lender. Merchant Cash Advances are made by Behalf. For more information on ECOA, click here <https://www.behalf.com/legal/ecoa/>. For important information about opening a new account, review Patriot Act procedures here <https://www.behalf.com/legal/patriot/>. Visit Legal <https://www.behalf.com/legal/> to review our comprehensive program terms, conditions, and disclosures.
