One of the patterns I've been using in some of my pipelines is "getOrCreate" where I define the DataSet and if it's not present I create and if it is I just load it instead, it's pretty good for pipelines which might fail transiently. Any thoughts or objections?
-- Twitter: https://twitter.com/holdenkarau Fight Health Insurance: https://www.fighthealthinsurance.com/ <https://www.fighthealthinsurance.com/?q=hk_email> Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> YouTube Live Streams: https://www.youtube.com/user/holdenkarau Pronouns: she/her
