Welcome Ufuk! I’m using in GCP dataflow and so far so good. However do you have any > problems that you foresee I may bump into, my data is safely stored so in > case of hard failures I can recover easily. >
Beam community as a whole, including all of its runners, always focuses on data consistency and semantic guarantees as a top priority. We have an extensive testing infrastructure to try to catch bugs as early as possible. Bugs may occasionally happen in any system -- but we'll certainly strive to avoid it and/or mitigate any impact as best as we can. - I didn’t wait for Dataflow 2.0 beta release and jumped into beam 0.4, do > you think I should go and use 2.0 now instead of 0.4 as I will be only > using Google’s managed dataflow service. > You are welcome to use either. The Beam community endorses Beam releases, and 0.4.0 is the newest and the recommended release. With my Google hat on -- Any vendor's distribution comes with additional vetting and support for that specific runner/scenario, and Dataflow is not an exception. If you want to use Dataflow service, using the Dataflow distribution makes sense. And, of course, you can always easily change your mind, or mix-and-match with Beam releases, without any modification to your code. - How can I try ‘TemplatingDataflowPipelineRunner’ on 0.4 or 2.0? I think > this will be renamed to ‘TemplatingPipelineRunner’? Do you have any > guidance? > Please see --templateLocation pipeline option and BEAM-551 [1] in our JIRA issue tracker. These are my questions so far. I will also add some feat requests for > Google Dataflow part, this may not be correct place to post those, if so > ignore those please: > These are great ideas, but they pertain to the Dataflow Service, so it would be best addressed by Dataflow support [2]. Once again, welcome! It is great to have you join the Beam user community. Davor [1] https://issues.apache.org/jira/browse/BEAM-551 [2] https://cloud.google.com/dataflow/support
