[
https://issues.apache.org/jira/browse/SPARK-60775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18135786#comment-18135786
]
Amogh Desai commented on SPARK-60775:
-------------------------------------
I'll try to POC this one out soon. [~holden] Can you assign this to me if you
don't mind?
> getOrCreate for Dataset / DataFrame to allow to resuming after a failed stage
> -----------------------------------------------------------------------------
>
> Key: SPARK-60775
> URL: https://issues.apache.org/jira/browse/SPARK-60775
> Project: Spark
> Issue Type: Improvement
> Components: PySpark, SQL
> Affects Versions: 4.0.5
> Reporter: Amogh Desai
> Priority: Major
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Add a getOrCreate for DataFrames: given a path, load the dataset if it is
> already there, otherwise compute it, save it, and return it.
> {code:python}
> df = spark.getOrCreate(path, lambda: expensive_transform(src))
> {code}
> h3. Why
> A long batch job that dies in a late stage has to rerun from the start. Spark
> has no record of which stages finished, so there is nothing to resume from.
> For anyone running Spark under an orchestrator, this is routine. In Airflow,
> the task retries and resubmits the whole application, so users pay for the
> same stages twice. The cause is often a reclaimed spot instance or a
> preempted node, where the same code would have worked on a rerun.
> People build this by hand today: save each stage to a known path, check the
> path on startup, skip the stage if it is there. Everyone writes their own
> version, and some are quietly wrong.
> h3. Design points
> # Let the caller pass the path. A scheduled pipeline runs the same code every
> day on different data. If Spark derives the path from the code, Monday and
> Tuesday get the same path and Tuesday loads Monday's data while still
> reporting success. Airflow has a run id that is stable across retries and
> distinct between runs, so passing the path lets that flow through.
> # Check for a success marker, not just that the path exists. A crash during
> the write leaves a path holding partial data.
> # Decide what happens when an input changed but the saved copy did not.
> Returning the stale copy gives a wrong answer and the job still passes. There
> should be a way to force recomputation.
> # Decide who cleans up saved intermediates.
> h3. Context
> Dev list thread: [Thoughts around getOrCreate for Dataset /
> DataFrame|https://lists.apache.org/thread.html/7mj5rwnylksnrvjocywc6sfxnrhl7c59]
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
