Hi Jean-Baptiste, Morning.I have quite similiar with Jing's case. I wanna join some relatively static data from HBase (which were bulk loaded everyday) in an unbounded pipeline. I'd like take a look at your code for a reference. I checked your github but couldn't found anything close to RedisIO you mentioned. Did I overlook anything? or could you send me a link to your RedisIO. thanks a lot.Ya-Feng ------------------------------------------------------------------发件人：Jean-Baptiste Onofré <[email protected]>发送时间：2016年11月22日(星期二) 03:29收件人：user <[email protected]>主 题：Re: how to use key-value storage like redis with PCollection? Hi Amir,
I'm working on MqttIO right now, I will push the RedisIO on my github just after. I will let you know. Regards JB On 11/21/2016 08:13 PM, amir bahmanyari wrote: > Am very curious about the RedisIO() example you mentioned JB... > Thanks ! > > > ------------------------------------------------------------------------ > *From:* Lukasz Cwik <[email protected]> > *To:* [email protected] > *Sent:* Monday, November 21, 2016 5:42 AM > *Subject:* Re: how to use key-value storage like redis with PCollection? > > Have you taken a look at the PCollectionView? > > It allows you to use various views of a PCollection from within a DoFn. > This > ><https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ViewTest.java#L461> > is > a short example where a multimap view is used to join two PCollections. > In your pipeline you would have the bounded PCollection used as a map or > multimap view. You would then use a DoFn that had a main input with an > unbounded PCollection and a side input of the view. > > On Mon, Nov 21, 2016 at 3:28 AM, Jean-Baptiste Onofré <[email protected] > <mailto:[email protected]>> wrote: > > Sure, it's on a private repo, let me push on the public one. > > I will let you know as soon as it's done. > > Thanks ! > Regards > JB > > On 11/21/2016 10:25 AM, 陈竞 wrote: > > ok, thank you very much. Could you show me your branch address? > > 2016-11-21 17:20 GMT+08:00 Jean-Baptiste Onofré <[email protected] > <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>>>: > > I have an example, but with the RedisIO. > > So, if you are interested, I can share my branch. > > Regards > JB > > On 11/21/2016 10:18 AM, 陈竞 wrote: > > could you show the example code of redis query with > PCollection? > > 2016-11-21 16:41 GMT+08:00 Jean-Baptiste Onofré > <[email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>> > <mailto:[email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>>>>: > > > Hi, > > you can convert your PCollection<KV<?,?>> to a > PCollection<POJO> and > then create a DoFn to do the query. > > By the way, I have a RedisIO mostly ready. > > Regards > JB > > > On 11/21/2016 09:14 AM, 陈竞 wrote: > > my dataflow case is like that: > stream: > a stream want to query some data from redis with > a key, > > batch: > a table left join another table in with a key > > i want to unify the two sence above by a > transform like > MapJoin, > so i > need to use > PCollection to represent the data in redis, but the > question is that > PCollection has no interface to make PCollection > queryable, so > is there > any solution for my case? > > > -- > Jean-Baptiste Onofré > [email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>> > <mailto:[email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>>> > http://blog.nanthrax.net <http://blog.nanthrax.net/> > Talend - http://www.talend.com <http://www.talend.com/> > > > > > -- > 陈竞，中科院计算技术研究所，高性能计算机中心 > Jing Chen HPCC.ICT.AC <http://hpcc.ict.ac/> > <http://HPCC.ICT.AC <http://hpcc.ict.ac/>> <http://HPCC.ICT.AC > <http://hpcc.ict.ac/>> > China > > > -- > Jean-Baptiste Onofré > [email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>> > http://blog.nanthrax.net <http://blog.nanthrax.net/> > Talend - http://www.talend.com <http://www.talend.com/> > > > > > -- > 陈竞，中科院计算技术研究所，高性能计算机中心 > Jing Chen HPCC.ICT.AC <http://hpcc.ict.ac/> <http://HPCC.ICT.AC > <http://hpcc.ict.ac/>> China > > > -- > Jean-Baptiste Onofré > [email protected] <mailto:[email protected]> > http://blog.nanthrax.net <http://blog.nanthrax.net/> > Talend - http://www.talend.com <http://www.talend.com/> > > > > -- Jean-Baptiste Onofré [email protected] http://blog.nanthrax.net Talend - http://www.talend.com
