Have you taken a look at the PCollectionView? It allows you to use various views of a PCollection from within a DoFn. This <https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ViewTest.java#L461> is a short example where a multimap view is used to join two PCollections. In your pipeline you would have the bounded PCollection used as a map or multimap view. You would then use a DoFn that had a main input with an unbounded PCollection and a side input of the view.
On Mon, Nov 21, 2016 at 3:28 AM, Jean-Baptiste Onofré <[email protected]> wrote: > Sure, it's on a private repo, let me push on the public one. > > I will let you know as soon as it's done. > > Thanks ! > Regards > JB > > On 11/21/2016 10:25 AM, 陈竞 wrote: > >> ok, thank you very much. Could you show me your branch address? >> >> 2016-11-21 17:20 GMT+08:00 Jean-Baptiste Onofré <[email protected] >> <mailto:[email protected]>>: >> >> I have an example, but with the RedisIO. >> >> So, if you are interested, I can share my branch. >> >> Regards >> JB >> >> On 11/21/2016 10:18 AM, 陈竞 wrote: >> >> could you show the example code of redis query with PCollection? >> >> 2016-11-21 16:41 GMT+08:00 Jean-Baptiste Onofré <[email protected] >> <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>>>: >> >> >> Hi, >> >> you can convert your PCollection<KV<?,?>> to a >> PCollection<POJO> and >> then create a DoFn to do the query. >> >> By the way, I have a RedisIO mostly ready. >> >> Regards >> JB >> >> >> On 11/21/2016 09:14 AM, 陈竞 wrote: >> >> my dataflow case is like that: >> stream: >> a stream want to query some data from redis with a key, >> >> batch: >> a table left join another table in with a key >> >> i want to unify the two sence above by a transform like >> MapJoin, >> so i >> need to use >> PCollection to represent the data in redis, but the >> question is that >> PCollection has no interface to make PCollection >> queryable, so >> is there >> any solution for my case? >> >> >> -- >> Jean-Baptiste Onofré >> [email protected] <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>> >> http://blog.nanthrax.net >> Talend - http://www.talend.com >> >> >> >> >> -- >> 陈竞，中科院计算技术研究所，高性能计算机中心 >> Jing Chen HPCC.ICT.AC <http://HPCC.ICT.AC> <http://HPCC.ICT.AC> >> China >> >> >> -- >> Jean-Baptiste Onofré >> [email protected] <mailto:[email protected]> >> http://blog.nanthrax.net >> Talend - http://www.talend.com >> >> >> >> >> -- >> 陈竞，中科院计算技术研究所，高性能计算机中心 >> Jing Chen HPCC.ICT.AC <http://HPCC.ICT.AC> China >> > > -- > Jean-Baptiste Onofré > [email protected] > http://blog.nanthrax.net > Talend - http://www.talend.com >
