Hi Khalid – the CLI in the current proposal will need to be built on top of internal APIs for constructing and launching pipeline executions. We'll have the option to expose these in the future.
It would be worthwhile to understand the use cases in more depth before exposing these, because APIs are one-way doors and can be costly to maintain. On Sat, Apr 5, 2025 at 11:59 PM Khalid Mammadov <khalidmammad...@gmail.com> wrote: > Looks great! > QQ: will user able to run this pipeline from normal code? I.e. can I > trigger a pipeline from *driver* code based on some condition etc. or it > must be executed via separate shell command ? > As a background Databricks imposes similar limitation where as you cannot > run normal Spark code and DLT on the same cluster for some reason and > forces to use two clusters increasing the cost and latency. > > On Sat, 5 Apr 2025 at 23:03, Sandy Ryza <sa...@apache.org> wrote: > >> Hi all – starting a discussion thread for a SPIP that I've been working >> on with Chao Sun, Kent Yao, Yuming Wang, and Jie Yang: [JIRA >> <https://issues.apache.org/jira/browse/SPARK-51727>] [Doc >> <https://docs.google.com/document/d/1PsSTngFuRVEOvUGzp_25CQL1yfzFHFr02XdMfQ7jOM4/edit?tab=t.0> >> ]. >> >> The SPIP proposes extending Spark's lazy, declarative execution model >> beyond single queries, to pipelines that keep multiple datasets up to date. >> It introduces the ability to compose multiple transformations into a single >> declarative dataflow graph. >> >> Declarative pipelines aim to simplify the development and management of >> data pipelines, by removing the need for manual orchestration of >> dependencies and making it possible to catch many errors before any >> execution steps are launched. >> >> Declarative pipelines can include both batch and streaming computations, >> leveraging Structured Streaming for stream processing and new materialized >> view syntax for batch processing. Tight integration with Spark SQL's >> analyzer enables deeper analysis and earlier error detection than is >> achievable with more generic frameworks. >> >> Let us know what you think! >> >>
