Hello, Probably it is not worth the effort to write a new RedisIO from zero considering there is an ongoing Pull Request for this.
https://github.com/apache/beam/pull/1687 Maybe you can take a look if the current WIP is enough for your needs, and eventually give a hand there to improve it if it is not the case. Regards, Ismaël On Friday, March 17, 2017, sowmya balasubramanian <[email protected]> wrote: > > Thanks a ton Raghu and Eugene! The Setup and Teardown is what I was looking > for. I will try it and see how it goes. > > Regards, > Sowmya > > On Thu, Mar 16, 2017 at 6:02 PM, Eugene Kirpichov <[email protected]> > wrote: >> >> Yes, please use a ParDo. The Custom Sink API is not intended for use with >> unbounded collections (i.e. in pretty much any streaming pipelines) and it's >> generally due for a redesign. ParDo is currently almost always a better >> choice when you want to implement a connector writing data to a third-party >> system, unless you're just implementing export to a particular file format >> (in which case FileBasedSink is appropriate). >> >> Concur with what Raghu said about @Setup/@Teardown. >> >> On Thu, Mar 16, 2017 at 3:02 PM Raghu Angadi <[email protected]> wrote: >>> >>> ParDo is ok. >>> >>> Do you open a connection in each processElement() invocation? If you can >>> reuse the connection, you can open once in @Setup method and close it in >>> @Teardown. >>> >>> Raghu. >>> >>> On Thu, Mar 16, 2017 at 2:19 PM, sowmya balasubramanian >>> <[email protected]> wrote: >>>> >>>> Hi All, >>>> >>>> I am newbie who has recently entered the world of GCP and pipelines. >>>> >>>> I have a streaming pipeline in which I write to a Redis sink at the end. >>>> The pipeline writes about 60,000 events per 15 minute window it processes. >>>> I implemented the writing to Redis using a ParDo. >>>> >>>> The prototype worked well for small set of streaming events. However, when >>>> I tested with my full dataset, every now and then I noticed the Redis >>>> client (Jedis) threw a SocketException. (The client opens connection every >>>> time it has to write to Redis, then closes the connection) >>>> >>>> Couple of questions I have: >>>> >>>> Is there a preferred Redis client for the pipeline? >>>> Does it make sense to write a Custom Redis sink instead of a ParDo? >>>> >>>> Thanks, >>>> Sowmya >>>> >>>> >>>> >>>> >>>> >>> >
