Just discovered that RedisIO exposes ReadAll too so you can take a look at that one too.
On Thu, Jul 18, 2019 at 11:39 AM Ismaël Mejía <ieme...@gmail.com> wrote: > > Yes this is done in multiple IOs now, you can see how it is done in > JdbcIO or a simpler but without explicit ReadAll transform in SolrIO. > Notice that this change requires a refactor of the IO to avoid code > repetition. > I filled https://issues.apache.org/jira/browse/BEAM-7769 and assigned > it to you, feel free to unassign if you don' t plan to work on it. > In the meantime I am going to try to expose ReadAll on SolrIO so we > can have it as a reference. > > On Thu, Jul 18, 2019 at 11:08 AM Chaim Turkel <ch...@behalf.com> wrote: > > > > is there another source that does this so i can have a look and add it > > to the MongoDBIO? > > > > On Wed, Jul 17, 2019 at 9:48 PM Eugene Kirpichov <kirpic...@google.com> > > wrote: > > > > > > I think full-blown SDF is not needed for this - someone just needs to > > > implement a MongoDbIO.readAll() variant, using a composite transform. The > > > regular pattern for this sort of thing will do (ParDo split, reshuffle, > > > ParDo read). > > > Whether it's worth replacing MongoDbIO.read() with a redirect to > > > readAll() is another matter - size estimation, available only in > > > BoundedSource for now, may or may not be important. > > > > > > On Wed, Jul 17, 2019 at 2:39 AM Ryan Skraba <r...@skraba.com> wrote: > > >> > > >> Hello! To clarify, you want to do something like this? > > >> > > >> PubSubIO.read() -> extract mongodb collection and range -> > > >> MongoDbIO.read(collection, range) -> ... > > >> > > >> If I'm not mistaken, it isn't possible with the implementation of > > >> MongoDbIO (based on BoundedSource interface, requiring the collection to > > >> be specified once at pipeline construction time). > > >> > > >> BUT -- this is a good candidate for an improvement in composability, and > > >> the ongoing work to prefer the SDF for these types of use cases. Maybe > > >> raise a JIRA for an improvement? > > >> > > >> All my best, Ryan > > >> > > >> > > >> On Wed, Jul 17, 2019 at 9:35 AM Chaim Turkel <ch...@behalf.com> wrote: > > >>> > > >>> any ideas? > > >>> > > >>> On Mon, Jul 15, 2019 at 11:04 PM Rui Wang <ruw...@google.com> wrote: > > >>> > > > >>> > +u...@beam.apache.org > > >>> > > > >>> > > > >>> > -Rui > > >>> > > > >>> > On Mon, Jul 15, 2019 at 6:55 AM Chaim Turkel <ch...@behalf.com> wrote: > > >>> >> > > >>> >> Hi, > > >>> >> I am looking to write a pipeline that read from a mongo collection. > > >>> >> I would like to listen to a pubsub that will have a object that > > >>> >> will > > >>> >> tell me which collection and which time frame. > > >>> >> Is there a way to do this? > > >>> >> > > >>> >> Chaim > > >>> >> > > >>> >> -- > > >>> >> > > >>> >> > > >>> >> Loans are funded by > > >>> >> FinWise Bank, a Utah-chartered bank located in Sandy, > > >>> >> Utah, member FDIC, Equal > > >>> >> Opportunity Lender. Merchant Cash Advances are > > >>> >> made by Behalf. For more > > >>> >> information on ECOA, click here > > >>> >> <https://www.behalf.com/legal/ecoa/>. For important information about > > >>> >> opening a new > > >>> >> account, review Patriot Act procedures here > > >>> >> <https://www.behalf.com/legal/patriot/>. > > >>> >> Visit Legal > > >>> >> <https://www.behalf.com/legal/> to > > >>> >> review our comprehensive program terms, > > >>> >> conditions, and disclosures. > > >>> > > >>> -- > > >>> > > >>> > > >>> Loans are funded by > > >>> FinWise Bank, a Utah-chartered bank located in Sandy, > > >>> Utah, member FDIC, Equal > > >>> Opportunity Lender. Merchant Cash Advances are > > >>> made by Behalf. For more > > >>> information on ECOA, click here > > >>> <https://www.behalf.com/legal/ecoa/>. For important information about > > >>> opening a new > > >>> account, review Patriot Act procedures here > > >>> <https://www.behalf.com/legal/patriot/>. > > >>> Visit Legal > > >>> <https://www.behalf.com/legal/> to > > >>> review our comprehensive program terms, > > >>> conditions, and disclosures. > > > > -- > > > > > > Loans are funded by > > FinWise Bank, a Utah-chartered bank located in Sandy, > > Utah, member FDIC, Equal > > Opportunity Lender. Merchant Cash Advances are > > made by Behalf. For more > > information on ECOA, click here > > <https://www.behalf.com/legal/ecoa/>. For important information about > > opening a new > > account, review Patriot Act procedures here > > <https://www.behalf.com/legal/patriot/>. > > Visit Legal > > <https://www.behalf.com/legal/> to > > review our comprehensive program terms, > > conditions, and disclosures.
