On 2026-10-12 11:59 UTC Holden Karau wrote:
> One of the patterns I've been using in some of my pipelines is
> "getOrCreate" where I define the DataSet and if it's not present I create
> and if it is I just load it instead, it's pretty good for pipelines which
> might fail transiently. Any thoughts or objections?
>
> --
> Twitter: https://twitter.com/holdenkarau
> Fight Health Insurance: https://www.fighthealthinsurance.com/
> <https://www.fighthealthinsurance.com/?q=hk_email>
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
> Pronouns: she/her
>
+1, I would use this.
I am a maintainer on Apache Airflow. Holden and I talked about this at
Community Over Code last week, so here is the Airflow side of the story.
This hurts people who run their Spark jobs from Airflow. A long job dies
in a late stage. Airflow retries it. The retry redoes every stage that
already finished, because Spark has no idea they finished.
It appears that people build this by hand today. Save each stage to a folder,
check if the folder is there, skip the stage if the folder exists. Everyone
writes their own version, and some of them might be subtly wrong.
My one question is who names the folder. Please let the caller pass it
in. If Spark picks the name from the application code, then Monday's run and
Tuesday's run get the same name, and Tuesday loads Monday's data. Airflow
knows which run it is, and both attempts of a retry share that, so if I
can pass the name that should be great.
Another pointer: If the job crashes while saving, the folder exists, but the
data is half written, so the check probably needs a success marker.
---------------------------------------------------------------------
To unsubscribe e-mail: [email protected]
