Am very curious about the RedisIO() example you mentioned JB...Thanks !
From: Lukasz Cwik <[email protected]>
To: [email protected]
Sent: Monday, November 21, 2016 5:42 AM
Subject: Re: how to use key-value storage like redis with PCollection?
Have you taken a look at the PCollectionView?
It allows you to use various views of a PCollection from within a DoFn. This is
a short example where a multimap view is used to join two PCollections. In your
pipeline you would have the bounded PCollection used as a map or multimap view.
You would then use a DoFn that had a main input with an unbounded PCollection
and a side input of the view.
On Mon, Nov 21, 2016 at 3:28 AM, Jean-Baptiste Onofré <[email protected]> wrote:
Sure, it's on a private repo, let me push on the public one.
I will let you know as soon as it's done.
Thanks !
Regards
JB
On 11/21/2016 10:25 AM, 陈竞 wrote:
ok, thank you very much. Could you show me your branch address?
2016-11-21 17:20 GMT+08:00 Jean-Baptiste Onofré <[email protected]
<mailto:[email protected]>>:
I have an example, but with the RedisIO.
So, if you are interested, I can share my branch.
Regards
JB
On 11/21/2016 10:18 AM, 陈竞 wrote:
could you show the example code of redis query with PCollection?
2016-11-21 16:41 GMT+08:00 Jean-Baptiste Onofré <[email protected]
<mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>:
Hi,
you can convert your PCollection<KV<?,?>> to a
PCollection<POJO> and
then create a DoFn to do the query.
By the way, I have a RedisIO mostly ready.
Regards
JB
On 11/21/2016 09:14 AM, 陈竞 wrote:
my dataflow case is like that:
stream:
a stream want to query some data from redis with a key,
batch:
a table left join another table in with a key
i want to unify the two sence above by a transform like
MapJoin,
so i
need to use
PCollection to represent the data in redis, but the
question is that
PCollection has no interface to make PCollection
queryable, so
is there
any solution for my case?
--
Jean-Baptiste Onofré
[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
http://blog.nanthrax.net
Talend - http://www.talend.com
--
陈竞，中科院计算技术研究所，高性能计算机中心
Jing Chen HPCC.ICT.AC <http://HPCC.ICT.AC> <http://HPCC.ICT.AC>
China
--
Jean-Baptiste Onofré
[email protected] <mailto:[email protected]>
http://blog.nanthrax.net
Talend - http://www.talend.com
--
陈竞，中科院计算技术研究所，高性能计算机中心
Jing Chen HPCC.ICT.AC <http://HPCC.ICT.AC> China
--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com
