Amogh Desai created SPARK-60775:
-----------------------------------
Summary: getOrCreate for Dataset / DataFrame to allow to resuming
after a failed stage
Key: SPARK-60775
URL: https://issues.apache.org/jira/browse/SPARK-60775
Project: Spark
Issue Type: Improvement
Components: PySpark, SQL
Affects Versions: 4.0.5
Reporter: Amogh Desai
Add a getOrCreate for DataFrames: given a path, load the dataset if it is
already there, otherwise compute it, save it, and return it.
{code:python}
df = spark.getOrCreate(path, lambda: expensive_transform(src))
{code}
h3. Why
A long batch job that dies in a late stage has to rerun from the start. Spark
has no record of which stages finished, so there is nothing to resume from.
For anyone running Spark under an orchestrator, this is routine. In Airflow,
the task retries and resubmits the whole application, so users pay for the same
stages twice. The cause is often a reclaimed spot instance or a preempted node,
where the same code would have worked on a rerun.
People build this by hand today: save each stage to a known path, check the
path on startup, skip the stage if it is there. Everyone writes their own
version, and some are quietly wrong.
h3. Design points
# Let the caller pass the path. A scheduled pipeline runs the same code every
day on different data. If Spark derives the path from the code, Monday and
Tuesday get the same path and Tuesday loads Monday's data while still reporting
success. Airflow has a run id that is stable across retries and distinct
between runs, so passing the path lets that flow through.
# Check for a success marker, not just that the path exists. A crash during the
write leaves a path holding partial data.
# Decide what happens when an input changed but the saved copy did not.
Returning the stale copy gives a wrong answer and the job still passes. There
should be a way to force recomputation.
# Decide who cleans up saved intermediates.
h3. Context
Dev list thread: [Thoughts around getOrCreate for Dataset /
DataFrame|https://lists.apache.org/thread.html/7mj5rwnylksnrvjocywc6sfxnrhl7c59]
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
