Hi All, I am newbie who has recently entered the world of GCP and pipelines.
I have a streaming pipeline in which I write to a Redis sink at the end. The pipeline writes about 60,000 events per 15 minute window it processes. I implemented the writing to Redis using a ParDo. The prototype worked well for small set of streaming events. However, when I tested with my full dataset, every now and then I noticed the Redis client (Jedis) threw a SocketException. (The client opens connection every time it has to write to Redis, then closes the connection) Couple of questions I have: 1. Is there a preferred Redis client for the pipeline? 2. Does it make sense to write a Custom Redis sink instead of a ParDo? Thanks, Sowmya
