Sandy Ryza created SPARK-51727:
----------------------------------
Summary: SPIP: Declarative Pipelines
Key: SPARK-51727
URL: https://issues.apache.org/jira/browse/SPARK-51727
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 4.1.0
Reporter: Sandy Ryza
The SPIP proposes a new abstraction that combines multiple transformations into
a single declarative dataflow graph, to simplify the development and management
of data pipelines.
The approach extends Spark's lazy, declarative execution model beyond single
queries, to pipelines that keep multiple datasets up to date. This reduces
cognitive overhead and manual orchestration of dependencies.
Declarative pipelines can include both batch and streaming computations,
leveraging Spark Streaming for stream processing and new materialized view
syntax for batch processing.
SPIP doc:
https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4/edit?tab=t.0
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
