This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new a771b02  [SPARK-61084][CONNECT][PYTHON] Register progress handlers, 
and call them with the progress of each execution, in the Python client as 
PySpark does
a771b02 is described below

commit a771b02250afc723c6b956efd7b6b48a570c303f
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 15:28:42 2026 +0900

    [SPARK-61084][CONNECT][PYTHON] Register progress handlers, and call them 
with the progress of each execution, in the Python client as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61084](https://issues.apache.org/jira/browse/SPARK-61084): progress 
handlers are now registered, and called, as PySpark Connect's are.
    
    As PySpark Connect's:
    - `spark.registerProgressHandler(handler)` registers a callable once and 
returns `None`.
    - `removeProgressHandler(handler)` removes it, and raises a `ValueError` if 
it is not registered. `clearProgressHandlers()` removes them all.
    - Each handler is called with the keyword arguments of `ProgressHandler`: 
`stages` (the `StageInfo` of each stage), `inflight_tasks`, `operation_id` and 
`done`. It is called for each progress of an execution that has tasks. It is 
called once more, with `done=True`, when the execution ends, also when it fails.
    - `SPARK_CONNECT_PROGRESS_BAR_ENABLED=1` prints the progress bar of each 
execution.
    
    `pyspark.sql.connect.shell.progress`, with `ProgressHandler`, `StageInfo` 
and `Progress`, is vendored from Apache Spark v4.3.0, and the drift check 
covers it. It imports `ExecutePlanResponse` from `pyspark.sql.connect.proto`, 
which now has a message class with the fields of an `ExecutionProgress` that 
`from_proto` reads.
    
    The core tracks the progress of each execution as PySpark's `Progress` 
does, from the operation ID that the client assigns before it sends the 
request. Every execution goes through it, including collects, local iterators, 
commands, catalog requests, streaming listeners and pipeline runs. The handlers 
of `SparkSession::register_progress_handler` now take a `ProgressUpdate` with 
the stages, the running tasks, the operation and whether it is done, rather 
than the raw `ExecutionProgress`  [...]
    
    One difference remains: the Rust transport calls the handlers from inside 
its execution. So an exception raised by a handler goes to 
`sys.unraisablehook`, rather than failing the execution as it does in PySpark.
    
    ### Why are the changes needed?
    
    The client:
    - returned an id from `registerProgressHandler`, and removed a handler by 
that id
    - called a handler with one dict of the progress, positionally, so a 
handler written for PySpark failed, silently
    - never called a handler when an execution ended
    - lacked `pyspark.sql.connect.shell.progress`
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Progress handlers, and the progress bar, work as in PySpark 4.3.0.
    
    ### How was this patch tested?
    
    - A new live test registers a handler with PySpark's keyword-only 
arguments, runs a query, and checks its calls: the `StageInfo`s, the running 
tasks, one operation ID, and one call with `done=True`, the last. It passes 
with official PySpark 4.3.0 and fails without this change.
    - A new offline test runs executions that receive a progress from a mock 
Connect server. It checks the calls of a handler, its registration and removal, 
the `ValueError`, an exception of a handler, and the progress bar.
    - A new lib test checks the progress of an execution in the core: 
progresses without tasks, the operation of the first response, and the last 
call, of `done`.
    - On a local server, the calls of a handler for fast and slow queries, a 
SQL command and a failed query have the same arguments as official PySpark 
4.3.0's. Only the number of progress messages differs, which depends on the 
timing of the server.
    - The lib tests of both crates (593 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2040 passed, 12 skipped), and the 
live drop-in and Arrow suites (145 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #289 from HyukjinKwon/SPARK-61084-progress-handlers.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/lib.rs                 |   1 +
 crates/pyspark-rs/src/progress.rs            | 156 ++++++++++++++++++++
 crates/pyspark-rs/src/session.rs             |  46 +++---
 crates/spark-connect/src/catalog.rs          |   2 +-
 crates/spark-connect/src/dataframe.rs        |  32 ++--
 crates/spark-connect/src/lib.rs              |   2 +
 crates/spark-connect/src/pipelines.rs        |   7 +-
 crates/spark-connect/src/progress.rs         | 189 ++++++++++++++++++++++++
 crates/spark-connect/src/session.rs          |  55 ++++---
 crates/spark-connect/src/streaming.rs        |   5 +-
 python/pyspark/sql/connect/proto.py          |  35 ++++-
 python/pyspark/sql/connect/shell/progress.py | 213 +++++++++++++++++++++++++++
 python/tests/test_dropin_e2e.py              |  24 +++
 python/tests/test_dropin_offline.py          |  63 ++++++++
 scripts/check_vendored_upstream.sh           |   4 +
 15 files changed, 763 insertions(+), 71 deletions(-)

diff --git a/crates/pyspark-rs/src/lib.rs b/crates/pyspark-rs/src/lib.rs
index 1041722..a2adc3b 100644
--- a/crates/pyspark-rs/src/lib.rs
+++ b/crates/pyspark-rs/src/lib.rs
@@ -40,6 +40,7 @@ mod origin;
 mod pandas_input;
 mod pipelines;
 mod profiler;
+mod progress;
 mod pyarrow_export;
 mod pyarrow_import;
 mod readwriter;
diff --git a/crates/pyspark-rs/src/progress.rs 
b/crates/pyspark-rs/src/progress.rs
new file mode 100644
index 0000000..10d54ee
--- /dev/null
+++ b/crates/pyspark-rs/src/progress.rs
@@ -0,0 +1,156 @@
+//! The progress handlers of the Python client, as PySpark Connect's client 
keeps them: a list
+//! of each session, whose handlers are called with the keyword arguments of 
`ProgressHandler`
+//! and the `StageInfo` of `pyspark.sql.connect.shell.progress`, and the 
progress bars of
+//! `SPARK_CONNECT_PROGRESS_BAR_ENABLED`, of its `Progress`.
+
+use std::collections::HashMap;
+use std::sync::atomic::{AtomicUsize, Ordering};
+use std::sync::{Arc, Mutex, OnceLock};
+
+use pyo3::prelude::*;
+use pyo3::types::{PyDict, PyList};
+use spark_connect::progress::ProgressUpdate;
+use spark_connect::SparkSession;
+
+/// The progress handlers of a session, and the progress bars of its 
executions.
+pub(crate) struct SessionProgress {
+    /// The handlers, as the `_progress_handlers` list of PySpark's client.
+    handlers: Py<PyList>,
+    /// The `Progress` of each execution of a progress bar, by its operation 
ID.
+    bars: Py<PyDict>,
+    /// The number of handlers, so that an execution takes the GIL only for a 
handler or a bar.
+    count: AtomicUsize,
+}
+
+/// The progress of each session of the client, by its session ID: every 
Python object of a
+/// session has its handlers.
+fn registry() -> &'static Mutex<HashMap<String, Arc<SessionProgress>>> {
+    static REGISTRY: OnceLock<Mutex<HashMap<String, Arc<SessionProgress>>>> = 
OnceLock::new();
+    REGISTRY.get_or_init(|| Mutex::new(HashMap::new()))
+}
+
+/// The progress of `session`: the first time, it registers the handler of the 
core session
+/// that calls the Python handlers and drives the progress bars.
+pub(crate) fn session_progress(py: Python<'_>, session: &SparkSession) -> 
Arc<SessionProgress> {
+    let mut registry = registry().lock().unwrap_or_else(|e| e.into_inner());
+    let id = session.session_id().to_string();
+    if let Some(progress) = registry.get(&id) {
+        return Arc::clone(progress);
+    }
+    let progress = Arc::new(SessionProgress {
+        handlers: PyList::empty(py).unbind(),
+        bars: PyDict::new(py).unbind(),
+        count: AtomicUsize::new(0),
+    });
+    let shared = Arc::clone(&progress);
+    session.register_progress_handler(move |update| shared.notify(update));
+    registry.insert(id, Arc::clone(&progress));
+    progress
+}
+
+/// Forget the progress of a stopped session.
+pub(crate) fn forget_session(session: &SparkSession) {
+    registry()
+        .lock()
+        .unwrap_or_else(|e| e.into_inner())
+        .remove(session.session_id());
+}
+
+/// Whether the progress bar is enabled, as 
`pyspark.sql.connect.shell.progress_bar_enabled`.
+fn progress_bar_enabled() -> bool {
+    std::env::var("SPARK_CONNECT_PROGRESS_BAR_ENABLED").as_deref() == Ok("1")
+}
+
+impl SessionProgress {
+    /// As PySpark's `register_progress_handler`: a handler is registered once.
+    pub(crate) fn register(&self, py: Python<'_>, handler: &Bound<'_, PyAny>) 
-> PyResult<()> {
+        let handlers = self.handlers.bind(py);
+        if !handlers.contains(handler)? {
+            handlers.append(handler)?;
+        }
+        self.count.store(handlers.len(), Ordering::SeqCst);
+        Ok(())
+    }
+
+    /// As PySpark's `remove_progress_handler`: a `ValueError` for a handler 
that is not
+    /// registered, of `list.remove`.
+    pub(crate) fn remove(&self, py: Python<'_>, handler: &Bound<'_, PyAny>) -> 
PyResult<()> {
+        let handlers = self.handlers.bind(py);
+        handlers.call_method1("remove", (handler,))?;
+        self.count.store(handlers.len(), Ordering::SeqCst);
+        Ok(())
+    }
+
+    /// As PySpark's `clear_progress_handlers`.
+    pub(crate) fn clear(&self, py: Python<'_>) -> PyResult<()> {
+        self.handlers.bind(py).call_method0("clear")?;
+        self.count.store(0, Ordering::SeqCst);
+        Ok(())
+    }
+
+    fn notify(&self, update: &ProgressUpdate) {
+        let bar = progress_bar_enabled();
+        if self.count.load(Ordering::SeqCst) == 0 && !bar {
+            return;
+        }
+        Python::attach(|py| {
+            if let Err(error) = self.notify_python(py, update, bar) {
+                // The Rust transport calls the handlers from its execution, 
which their
+                // exceptions do not fail: as an exception of a callback, it 
goes to
+                // `sys.unraisablehook`.
+                error.write_unraisable(py, None);
+            }
+        });
+    }
+
+    fn notify_python(&self, py: Python<'_>, update: &ProgressUpdate, bar: 
bool) -> PyResult<()> {
+        let module = py.import("pyspark.sql.connect.shell.progress")?;
+        let stage_info = module.getattr("StageInfo")?;
+        let stages = PyList::empty(py);
+        for stage in &update.stages {
+            let kwargs = PyDict::new(py);
+            kwargs.set_item("stage_id", stage.stage_id)?;
+            kwargs.set_item("num_tasks", stage.num_tasks)?;
+            kwargs.set_item("num_completed_tasks", stage.num_completed_tasks)?;
+            kwargs.set_item("num_bytes_read", stage.num_bytes_read)?;
+            kwargs.set_item("done", stage.done)?;
+            stages.append(stage_info.call((), Some(&kwargs))?)?;
+        }
+        if bar {
+            // The `Progress` of the execution, which prints its bar, as 
PySpark's client makes
+            // one of each request.
+            let bars = self.bars.bind(py);
+            let key = update.operation_id.as_deref();
+            let progress = match bars.get_item(key)? {
+                Some(progress) => progress,
+                None => {
+                    let kwargs = PyDict::new(py);
+                    kwargs.set_item("operation_id", key)?;
+                    let progress = module.getattr("Progress")?.call((), 
Some(&kwargs))?;
+                    bars.set_item(key, &progress)?;
+                    progress
+                }
+            };
+            if update.done {
+                bars.del_item(key)?;
+                progress.call_method0("finish")?;
+            } else {
+                progress.call_method1("update_ticks", (&stages, 
update.inflight_tasks, key))?;
+            }
+        }
+        // A copy of the handlers, as PySpark's `Progress` takes the list of 
the client.
+        let handlers: Vec<Bound<'_, PyAny>> = 
self.handlers.bind(py).iter().collect();
+        if handlers.is_empty() {
+            return Ok(());
+        }
+        let kwargs = PyDict::new(py);
+        kwargs.set_item("stages", &stages)?;
+        kwargs.set_item("inflight_tasks", update.inflight_tasks)?;
+        kwargs.set_item("operation_id", update.operation_id.as_deref())?;
+        kwargs.set_item("done", update.done)?;
+        for handler in handlers {
+            handler.call((), Some(&kwargs))?;
+        }
+        Ok(())
+    }
+}
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 6d9ceb0..a9ef96d 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -482,6 +482,8 @@ pub struct PySparkSession {
 
 impl PySparkSession {
     pub fn new(session: SparkSession) -> Self {
+        // The progress of the session, for the progress bars of its 
executions.
+        Python::attach(|py| crate::progress::session_progress(py, &session));
         PySparkSession { session }
     }
 
@@ -1468,6 +1470,7 @@ impl PySparkSession {
     /// to release the session on the server is logged, not raised. Also 
clears the
     /// active session and the cached `SQLContext` if they belong to this 
session.
     fn stop(&self, py: Python<'_>) -> PyResult<()> {
+        crate::progress::forget_session(&self.session);
         if let Err(e) = py.detach(|| self.session.stop()) {
             let logger = py
                 .import("logging")?
@@ -1724,41 +1727,30 @@ impl PySparkSession {
             })
     }
 
-    /// Register a progress handler invoked on each execution-progress message.
-    /// Returns an id for `removeProgressHandler`. Mirrors 
`registerProgressHandler`.
+    /// Register a handler of the progress of the executions of the session, 
as PySpark
+    /// Connect's: once, called with the keyword arguments of 
`ProgressHandler` (`stages`,
+    /// `inflight_tasks`, `operation_id` and `done`) for each progress of an 
execution, and
+    /// once more, as done, when it ends.
     #[pyo3(name = "registerProgressHandler")]
-    fn register_progress_handler(&self, handler: Py<PyAny>) -> u64 {
-        self.session.register_progress_handler(move |progress| {
-            Python::attach(|py| {
-                let dict = pyo3::types::PyDict::new(py);
-                let _ = dict.set_item("num_inflight_tasks", 
progress.num_inflight_tasks);
-                let stages = PyList::empty(py);
-                for s in &progress.stages {
-                    let sd = pyo3::types::PyDict::new(py);
-                    let _ = sd.set_item("stage_id", s.stage_id);
-                    let _ = sd.set_item("num_tasks", s.num_tasks);
-                    let _ = sd.set_item("num_completed_tasks", 
s.num_completed_tasks);
-                    let _ = sd.set_item("input_bytes_read", 
s.input_bytes_read);
-                    let _ = sd.set_item("done", s.done);
-                    let _ = stages.append(sd);
-                }
-                let _ = dict.set_item("stages", stages);
-                // Best-effort: ignore handler errors so progress never breaks 
execution.
-                let _ = handler.call1(py, (dict,));
-            });
-        })
+    fn register_progress_handler(
+        &self,
+        py: Python<'_>,
+        handler: &Bound<'_, PyAny>,
+    ) -> PyResult<()> {
+        crate::progress::session_progress(py, &self.session).register(py, 
handler)
     }
 
-    /// Remove a progress handler by id. Mirrors `removeProgressHandler`.
+    /// Remove a progress handler, as PySpark Connect's: a `ValueError` for 
one that is not
+    /// registered.
     #[pyo3(name = "removeProgressHandler")]
-    fn remove_progress_handler(&self, handler: u64) {
-        self.session.remove_progress_handler(handler);
+    fn remove_progress_handler(&self, py: Python<'_>, handler: &Bound<'_, 
PyAny>) -> PyResult<()> {
+        crate::progress::session_progress(py, &self.session).remove(py, 
handler)
     }
 
     /// Remove all progress handlers. Mirrors `clearProgressHandlers`.
     #[pyo3(name = "clearProgressHandlers")]
-    fn clear_progress_handlers(&self) {
-        self.session.clear_progress_handlers();
+    fn clear_progress_handlers(&self, py: Python<'_>) -> PyResult<()> {
+        crate::progress::session_progress(py, &self.session).clear(py)
     }
 
     /// Get the profiler collector for this session.
diff --git a/crates/spark-connect/src/catalog.rs 
b/crates/spark-connect/src/catalog.rs
index c211558..b5a1320 100644
--- a/crates/spark-connect/src/catalog.rs
+++ b/crates/spark-connect/src/catalog.rs
@@ -1056,7 +1056,7 @@ impl Catalog {
         let request = self.build_execute_catalog_request(catalog)?;
         // On the reattachable path, as PySpark's client runs a catalog 
operation, so that an
         // error carries the operation id of the request.
-        let mut stream = 
block_on(self.session.client().execute_plan_reattachable(request))?;
+        let mut stream = self.session.execute_plan(request)?;
         let mut rows = vec![];
         while let Some(resp) = block_on(stream.message())? {
             if let 
Some(proto::execute_plan_response::ResponseType::ArrowBatch(batch)) =
diff --git a/crates/spark-connect/src/dataframe.rs 
b/crates/spark-connect/src/dataframe.rs
index 0265375..ea078c0 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -5,7 +5,6 @@
 use std::collections::HashMap;
 use std::sync::{Arc, Mutex};
 
-use spark_connect_core::client::ReattachableResponseStream;
 use spark_connect_core::error::{Result, SparkError};
 use spark_connect_core::runtime::{block_on, get_runtime};
 use spark_connect_proto as proto;
@@ -13,6 +12,7 @@ use spark_connect_proto as proto;
 use crate::column::Column;
 use crate::expression::{Expression, ExpressionNode};
 use crate::plan::{AggregateGroupType, JoinType, LogicalPlan, SetOpType};
+use crate::progress::ExecutionStream;
 use crate::row::{Row, StructValue, Value};
 use crate::session::{ExecutionInfo, SparkSession};
 use crate::types::DataType;
@@ -74,7 +74,9 @@ enum RowSource {
     /// one is exhausted (`prefetchPartitions=False`).
     OnDemand {
         session: SparkSession,
-        stream: ReattachableResponseStream,
+        // Boxed: the stream, with the progress of its execution, is much 
larger than a
+        // receiver.
+        stream: Box<ExecutionStream>,
         execution_info: ExecutionInfo,
         execution_recorded: bool,
     },
@@ -90,7 +92,7 @@ impl LocalRowIterator {
     /// Create a new `LocalRowIterator` over an already-issued ExecutePlan 
stream.
     pub(crate) fn new(
         session: SparkSession,
-        stream: ReattachableResponseStream,
+        stream: ExecutionStream,
         prefetch_partitions: bool,
     ) -> Self {
         let schema = SchemaSlot::default();
@@ -101,7 +103,7 @@ impl LocalRowIterator {
         } else {
             RowSource::OnDemand {
                 session,
-                stream,
+                stream: Box::new(stream),
                 execution_info: ExecutionInfo::default(),
                 execution_recorded: false,
             }
@@ -202,7 +204,7 @@ impl Iterator for LocalRowIterator {
 /// stream ends. Backs `to_local_iterator(prefetch_partitions = true)`.
 fn spawn_prefetch(
     session: SparkSession,
-    mut stream: ReattachableResponseStream,
+    mut stream: ExecutionStream,
     schema: SchemaSlot,
 ) -> tokio::sync::mpsc::Receiver<Result<Vec<Row>>> {
     let (tx, rx) = tokio::sync::mpsc::channel::<Result<Vec<Row>>>(1);
@@ -798,7 +800,7 @@ impl DataFrame {
     /// Unlike the Arrow schema of the rows, it has the user-defined types of 
the columns.
     pub fn collect_with_schema(&self) -> Result<(Vec<Row>, Option<DataType>)> {
         let request = self.build_execute_request()?;
-        let mut stream = 
block_on(self.session.client().execute_plan_reattachable(request))?;
+        let mut stream = self.session.execute_plan(request)?;
 
         let mut rows = vec![];
         let mut schema = None;
@@ -837,7 +839,7 @@ impl DataFrame {
     ///   fetched on demand only once the previous batch is exhausted.
     pub fn to_local_iterator(&self, prefetch_partitions: bool) -> 
Result<LocalRowIterator> {
         let request = self.build_execute_request()?;
-        let stream = 
block_on(self.session.client().execute_plan_reattachable(request))?;
+        let stream = self.session.execute_plan(request)?;
         Ok(LocalRowIterator::new(
             self.session.clone(),
             stream,
@@ -874,7 +876,7 @@ impl DataFrame {
         &self,
     ) -> Result<(Vec<arrow::record_batch::RecordBatch>, Option<DataType>)> {
         let request = self.build_execute_request()?;
-        let mut stream = 
block_on(self.session.client().execute_plan_reattachable(request))?;
+        let mut stream = self.session.execute_plan(request)?;
 
         let mut batches = vec![];
         let mut schema = None;
@@ -2225,7 +2227,7 @@ pub(crate) fn execute_command_with_info(
     request.tags = session.tags();
     request.plan = Some(plan);
 
-    let mut stream = 
block_on(session.client().execute_plan_reattachable(request))?;
+    let mut stream = session.execute_plan(request)?;
     // Drain the response stream so the command runs to completion server-side,
     // capturing any metrics/progress emitted along the way.
     let mut info = ExecutionInfo::default();
@@ -2238,9 +2240,10 @@ pub(crate) fn execute_command_with_info(
     Ok((responses, info))
 }
 
-/// Pull execution metrics/observed-metrics off a response and fire progress
-/// handlers. `metrics`/`observed_metrics` are top-level fields (not part of 
the
-/// `response_type` oneof), so this must run before `response_type` is 
consumed.
+/// Pull execution metrics/observed-metrics off a response; the 
`ExecutionStream` of the
+/// response updates the progress of the execution. 
`metrics`/`observed_metrics` are top-level
+/// fields (not part of the `response_type` oneof), so this must run before 
`response_type` is
+/// consumed.
 fn capture_execution(
     resp: &mut proto::ExecutePlanResponse,
     info: &mut ExecutionInfo,
@@ -2259,11 +2262,6 @@ fn capture_execution(
         session.profiler().accumulate_observed_metrics(&metrics);
         info.observed_metrics.extend(metrics);
     }
-    if let 
Some(proto::execute_plan_response::ResponseType::ExecutionProgress(progress)) =
-        &resp.response_type
-    {
-        session.notify_progress(progress);
-    }
 }
 
 /// Set the metrics that the server observed in the Observations of their 
relations, as
diff --git a/crates/spark-connect/src/lib.rs b/crates/spark-connect/src/lib.rs
index 38fb6a9..ad07a2d 100644
--- a/crates/spark-connect/src/lib.rs
+++ b/crates/spark-connect/src/lib.rs
@@ -21,6 +21,7 @@ pub mod origin;
 pub mod pipelines;
 pub mod plan;
 pub mod profiler;
+pub mod progress;
 pub mod readwriter;
 pub mod resource;
 pub mod row;
@@ -50,6 +51,7 @@ pub use group::{CoGroupedData, GroupedData};
 pub use merge::MergeIntoWriter;
 pub use observation::Observation;
 pub use profiler::ProfilerCollector;
+pub use progress::{ProgressUpdate, StageInfo};
 pub use readwriter::{
     DataFrameReader, DataFrameWriter, DataFrameWriterV2, ReadType, SaveMode, 
TableSaveMethod,
 };
diff --git a/crates/spark-connect/src/pipelines.rs 
b/crates/spark-connect/src/pipelines.rs
index a30d8e7..3aad22b 100644
--- a/crates/spark-connect/src/pipelines.rs
+++ b/crates/spark-connect/src/pipelines.rs
@@ -8,7 +8,6 @@
 
 use std::collections::HashMap;
 
-use spark_connect_core::client::ReattachableResponseStream;
 use spark_connect_core::error::{Result, SparkError};
 use spark_connect_core::runtime::block_on;
 use spark_connect_proto as proto;
@@ -229,7 +228,7 @@ pub fn define_sql_graph_elements(
 pub struct PipelineRunStream {
     session: SparkSession,
     request: Option<proto::ExecutePlanRequest>,
-    stream: Option<ReattachableResponseStream>,
+    stream: Option<crate::progress::ExecutionStream>,
 }
 
 impl PipelineRunStream {
@@ -241,9 +240,7 @@ impl PipelineRunStream {
                 .request
                 .take()
                 .expect("pipeline run stream already exhausted its request");
-            self.stream = Some(block_on(
-                self.session.client().execute_plan_reattachable(request),
-            )?);
+            self.stream = Some(self.session.execute_plan(request)?);
         }
         block_on(self.stream.as_mut().unwrap().message())
     }
diff --git a/crates/spark-connect/src/progress.rs 
b/crates/spark-connect/src/progress.rs
new file mode 100644
index 0000000..5677f7c
--- /dev/null
+++ b/crates/spark-connect/src/progress.rs
@@ -0,0 +1,189 @@
+//! The progress of the executions of a session, as PySpark Connect's
+//! `pyspark.sql.connect.shell.progress`: the progress handlers of the session 
are called with
+//! each progress of the server that has tasks, and once more, as done, when 
the execution ends.
+
+use spark_connect_core::client::ReattachableResponseStream;
+use spark_connect_core::error::Result;
+use spark_connect_proto as proto;
+
+use crate::session::SparkSession;
+
+/// The progress of a stage of an execution, as PySpark's `StageInfo`.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct StageInfo {
+    pub stage_id: i64,
+    pub num_tasks: i64,
+    pub num_completed_tasks: i64,
+    pub num_bytes_read: i64,
+    pub done: bool,
+}
+
+impl StageInfo {
+    /// The stage of a progress of the server, as PySpark's `from_proto`.
+    fn from_proto(stage: 
&proto::execute_plan_response::execution_progress::StageInfo) -> Self {
+        StageInfo {
+            stage_id: stage.stage_id,
+            num_tasks: stage.num_tasks,
+            num_completed_tasks: stage.num_completed_tasks,
+            num_bytes_read: stage.input_bytes_read,
+            done: stage.done,
+        }
+    }
+}
+
+/// A progress of an execution, of which PySpark calls each `ProgressHandler` 
with the keyword
+/// arguments: the stages and the running tasks of the last progress of the 
server, the
+/// operation, and whether the execution ended.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct ProgressUpdate {
+    pub stages: Vec<StageInfo>,
+    pub inflight_tasks: i64,
+    pub operation_id: Option<String>,
+    pub done: bool,
+}
+
+/// The progress of one execution, as PySpark's `Progress` of the execution of 
a request.
+pub(crate) struct ProgressTracker {
+    session: SparkSession,
+    operation_id: Option<String>,
+    stages: Vec<StageInfo>,
+    inflight_tasks: i64,
+}
+
+impl ProgressTracker {
+    pub(crate) fn new(session: &SparkSession, operation_id: Option<String>) -> 
Self {
+        ProgressTracker {
+            session: session.clone(),
+            operation_id,
+            stages: Vec::new(),
+            inflight_tasks: 0,
+        }
+    }
+
+    /// A progress of the server, as PySpark's `Progress.update_ticks`: the 
handlers are called
+    /// only for one that has tasks, whose stages and running tasks the last 
call, of `done`,
+    /// has too.
+    pub(crate) fn update(
+        &mut self,
+        progress: &proto::execute_plan_response::ExecutionProgress,
+        operation_id: &str,
+    ) {
+        if self.operation_id.as_deref().is_none_or(str::is_empty) {
+            self.operation_id = Some(operation_id.to_string());
+        }
+        let stages: Vec<StageInfo> = 
progress.stages.iter().map(StageInfo::from_proto).collect();
+        if stages.iter().map(|stage| stage.num_tasks).sum::<i64>() > 0 {
+            self.stages = stages;
+            self.inflight_tasks = progress.num_inflight_tasks;
+            self.notify(false);
+        }
+    }
+
+    fn notify(&self, done: bool) {
+        self.session.notify_progress(|| ProgressUpdate {
+            stages: self.stages.clone(),
+            inflight_tasks: self.inflight_tasks,
+            operation_id: self.operation_id.clone(),
+            done,
+        });
+    }
+}
+
+impl Drop for ProgressTracker {
+    /// As PySpark's `Progress.finish` when its execution ends, also of an 
error.
+    fn drop(&mut self) {
+        self.notify(true);
+    }
+}
+
+/// The responses of an execution, with its progress for the handlers of the 
session: each
+/// `ExecutionProgress` updates it, and it ends when the stream is dropped.
+pub(crate) struct ExecutionStream {
+    stream: ReattachableResponseStream,
+    progress: ProgressTracker,
+}
+
+impl ExecutionStream {
+    pub(crate) fn new(stream: ReattachableResponseStream, progress: 
ProgressTracker) -> Self {
+        ExecutionStream { stream, progress }
+    }
+
+    /// The next response of the execution, as 
`ReattachableResponseStream::message`.
+    pub(crate) async fn message(&mut self) -> 
Result<Option<proto::ExecutePlanResponse>> {
+        let response = self.stream.message().await?;
+        if let Some(response) = &response {
+            if let 
Some(proto::execute_plan_response::ResponseType::ExecutionProgress(progress)) =
+                &response.response_type
+            {
+                self.progress.update(progress, &response.operation_id);
+            }
+        }
+        Ok(response)
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use std::sync::{Arc, Mutex};
+
+    use super::*;
+
+    fn progress(
+        stages: &[(i64, i64)],
+        inflight: i64,
+    ) -> proto::execute_plan_response::ExecutionProgress {
+        proto::execute_plan_response::ExecutionProgress {
+            stages: stages
+                .iter()
+                .map(|&(stage_id, num_tasks)| {
+                    
proto::execute_plan_response::execution_progress::StageInfo {
+                        stage_id,
+                        num_tasks,
+                        num_completed_tasks: 1,
+                        input_bytes_read: 8,
+                        done: false,
+                    }
+                })
+                .collect(),
+            num_inflight_tasks: inflight,
+        }
+    }
+
+    /// As PySpark's `Progress`: the handlers are called with each progress 
that has tasks, of the
+    /// operation of the first response, and once more, as done, with the last 
one, when the
+    /// execution ends.
+    #[test]
+    fn handlers_are_called_as_the_progress_of_pyspark_calls_them() {
+        let session = SparkSession::builder()
+            .remote("sc://localhost:15002")
+            .get_or_create()
+            .expect("session");
+        let updates = Arc::new(Mutex::new(Vec::new()));
+        let seen = Arc::clone(&updates);
+        let id = session.register_progress_handler(move |update| {
+            seen.lock().unwrap().push(update.clone());
+        });
+        {
+            let mut tracker = ProgressTracker::new(&session, None);
+            tracker.update(&progress(&[(1, 0)], 0), "op");
+            tracker.update(&progress(&[(1, 4), (2, 2)], 3), "other");
+        }
+        let updates = updates.lock().unwrap().clone();
+        assert_eq!(updates.len(), 2);
+        assert!(!updates[0].done && updates[1].done);
+        assert_eq!(updates[0].operation_id.as_deref(), Some("op"));
+        assert_eq!(updates[1].stages, updates[0].stages);
+        assert_eq!(updates[1].inflight_tasks, 3);
+        assert_eq!(
+            updates[0].stages[1],
+            StageInfo {
+                stage_id: 2,
+                num_tasks: 2,
+                num_completed_tasks: 1,
+                num_bytes_read: 8,
+                done: false,
+            }
+        );
+        session.remove_progress_handler(id);
+    }
+}
diff --git a/crates/spark-connect/src/session.rs 
b/crates/spark-connect/src/session.rs
index 7824d8f..a938f37 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -14,13 +14,14 @@ use spark_connect_proto as proto;
 use crate::dataframe::DataFrame;
 use crate::plan::{LogicalPlan, PlanIds};
 use crate::profiler::ProfilerCollector;
+use crate::progress::{ExecutionStream, ProgressTracker, ProgressUpdate};
 use crate::row::{Row, Value};
 use crate::types::DataType;
 
-/// A progress handler invoked with each `ExecutionProgress` message received 
on
-/// the response stream. Mirrors `SparkSession.registerProgressHandler`.
-pub type ProgressHandler =
-    Arc<dyn Fn(&proto::execute_plan_response::ExecutionProgress) + Send + 
Sync>;
+/// A progress handler of the executions of a session, as PySpark's 
`ProgressHandler`: it is
+/// called with each progress of the server that has tasks, and once more, as 
done, when the
+/// execution ends. Mirrors `SparkSession.registerProgressHandler`.
+pub type ProgressHandler = Arc<dyn Fn(&ProgressUpdate) + Send + Sync>;
 
 /// Metrics captured from the last executed query. Mirrors the data behind
 /// `pyspark.sql.DataFrame.executionInfo` / `client.core.ExecutionInfo`.
@@ -83,8 +84,8 @@ pub struct SparkSession {
     /// Metrics/observed-metrics captured from the most recent execution on 
this
     /// session. Backs `DataFrame::execution_info` and the profiler accessor.
     last_execution: Arc<Mutex<Option<ExecutionInfo>>>,
-    /// Registered progress handlers, keyed by id (for removal). Invoked on 
each
-    /// `ExecutionProgress` message. Mirrors `registerProgressHandler`.
+    /// Registered progress handlers, keyed by id (for removal). Invoked with 
each progress of
+    /// an execution, and when it ends. Mirrors `registerProgressHandler`.
     progress_handlers: Arc<Mutex<Vec<(u64, ProgressHandler)>>>,
     /// Monotonic id source for `register_progress_handler`.
     progress_handler_id: Arc<AtomicU64>,
@@ -162,12 +163,13 @@ impl SparkSession {
         self.tags.update(Vec::clear);
     }
 
-    /// Register a handler invoked for every `ExecutionProgress` message 
received
-    /// during query execution. Returns an id usable with
-    /// [`SparkSession::remove_progress_handler`]. Mirrors 
`registerProgressHandler`.
+    /// Register a handler of the progress of the executions of this session: 
it is called with
+    /// each progress of the server that has tasks, and once more, as done, 
when the execution
+    /// ends, also of an error, as PySpark's `Progress` calls its handlers. 
Returns an id usable
+    /// with [`SparkSession::remove_progress_handler`]. Mirrors 
`registerProgressHandler`.
     pub fn register_progress_handler(
         &self,
-        handler: impl Fn(&proto::execute_plan_response::ExecutionProgress) + 
Send + Sync + 'static,
+        handler: impl Fn(&ProgressUpdate) + Send + Sync + 'static,
     ) -> u64 {
         let id = self.progress_handler_id.fetch_add(1, Ordering::SeqCst);
         self.progress_handlers
@@ -191,12 +193,10 @@ impl SparkSession {
         self.progress_handlers.lock().unwrap().clear();
     }
 
-    /// Invoke all registered progress handlers with a progress message. 
Called by
-    /// the execute loop; not part of the public API.
-    pub(crate) fn notify_progress(
-        &self,
-        progress: &proto::execute_plan_response::ExecutionProgress,
-    ) {
+    /// Invoke all registered progress handlers with the progress of an 
execution, which
+    /// `update` builds only when there is a handler. Called by the progress 
of executions; not
+    /// part of the public API.
+    pub(crate) fn notify_progress(&self, update: impl FnOnce() -> 
ProgressUpdate) {
         // Clone the handler `Arc`s out of the lock so a handler can't 
deadlock by
         // touching the session's handler list.
         let handlers: Vec<ProgressHandler> = {
@@ -207,11 +207,32 @@ impl SparkSession {
                 .map(|(_, h)| Arc::clone(h))
                 .collect()
         };
+        if handlers.is_empty() {
+            return;
+        }
+        let update = update();
         for h in handlers {
-            h(progress);
+            h(&update);
         }
     }
 
+    /// Execute `request` on the server, with the progress of the execution 
for the progress
+    /// handlers. Its operation ID is that of the request, or a new one, which 
the client
+    /// assigns before it sends the request, as PySpark's `Progress` of the 
request has it.
+    pub(crate) fn execute_plan(
+        &self,
+        mut request: proto::ExecutePlanRequest,
+    ) -> Result<ExecutionStream> {
+        let operation_id = request
+            .operation_id
+            .get_or_insert_with(|| uuid::Uuid::new_v4().to_string())
+            .clone();
+        // Before the request: an execution that fails to start ends, as done, 
too.
+        let progress = ProgressTracker::new(self, Some(operation_id));
+        let stream = 
block_on(self.client().execute_plan_reattachable(request))?;
+        Ok(ExecutionStream::new(stream, progress))
+    }
+
     /// Store the metrics captured from the most recent execution. Called by 
the
     /// execute loop; not part of the public API.
     pub(crate) fn record_execution(&self, info: ExecutionInfo) {
diff --git a/crates/spark-connect/src/streaming.rs 
b/crates/spark-connect/src/streaming.rs
index 80f17ac..bd864c0 100644
--- a/crates/spark-connect/src/streaming.rs
+++ b/crates/spark-connect/src/streaming.rs
@@ -7,7 +7,6 @@ use std::collections::HashMap;
 use std::sync::{Arc, Mutex};
 use uuid;
 
-use spark_connect_core::client::ReattachableResponseStream;
 use spark_connect_core::error::{Result, SparkError};
 use spark_connect_core::runtime::block_on;
 use spark_connect_proto as proto;
@@ -733,7 +732,7 @@ pub struct StreamingQueryException {
 /// An iterator over streaming query listener events from the server.
 /// Yields events incrementally as they arrive, without buffering the entire 
stream.
 pub struct ListenerEventStream {
-    stream: ReattachableResponseStream,
+    stream: crate::progress::ExecutionStream,
     buffered_events: std::vec::IntoIter<(i32, String)>,
     done: bool,
 }
@@ -1106,7 +1105,7 @@ impl StreamingQueryManager {
             ..Default::default()
         };
 
-        let response_stream = 
block_on(self.session.client().execute_plan_reattachable(request))?;
+        let response_stream = self.session.execute_plan(request)?;
 
         Ok(ListenerEventStream {
             stream: response_stream,
diff --git a/python/pyspark/sql/connect/proto.py 
b/python/pyspark/sql/connect/proto.py
index 20fdb02..ec887d0 100644
--- a/python/pyspark/sql/connect/proto.py
+++ b/python/pyspark/sql/connect/proto.py
@@ -45,4 +45,37 @@ except ImportError:
         """Placeholder for protobuf Command message."""
         pass
 
-__all__ = ["Plan", "Relation", "Command"]
+
+
+class ExecutePlanResponse:
+    """The ``ExecutePlanResponse`` of ``base.proto``, of the progress of an 
execution: the Rust
+    transport reads the responses of the server, of which 
``ExecutionProgress`` holds the
+    fields that ``pyspark.sql.connect.shell.progress.from_proto`` reads."""
+
+    class ExecutionProgress:
+        """``ExecutePlanResponse.ExecutionProgress``: the stages of an 
execution, and its
+        running tasks."""
+
+        class StageInfo:
+            """``ExecutionProgress.StageInfo``: the tasks and the bytes read 
of a stage."""
+
+            def __init__(
+                self,
+                stage_id=0,
+                num_tasks=0,
+                num_completed_tasks=0,
+                input_bytes_read=0,
+                done=False,
+            ):
+                self.stage_id = stage_id
+                self.num_tasks = num_tasks
+                self.num_completed_tasks = num_completed_tasks
+                self.input_bytes_read = input_bytes_read
+                self.done = done
+
+        def __init__(self, stages=(), num_inflight_tasks=0):
+            self.stages = list(stages)
+            self.num_inflight_tasks = num_inflight_tasks
+
+
+__all__ = ["Plan", "Relation", "Command", "ExecutePlanResponse"]
diff --git a/python/pyspark/sql/connect/shell/progress.py 
b/python/pyspark/sql/connect/shell/progress.py
new file mode 100644
index 0000000..ada3b4d
--- /dev/null
+++ b/python/pyspark/sql/connect/shell/progress.py
@@ -0,0 +1,213 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Implementation of a progress bar that is displayed while a query is 
running."""
+
+import abc
+from dataclasses import dataclass
+import time
+import sys
+import typing
+from types import TracebackType
+from typing import Iterable, Any
+
+from pyspark.sql.connect.proto import ExecutePlanResponse
+
+try:
+    from IPython.utils.terminal import get_terminal_size
+except ImportError:
+
+    def get_terminal_size(defaultx: Any = None, defaulty: Any = None) -> Any:  
# type: ignore[misc]
+        return (80, 25)
+
+
+from pyspark.sql.connect.shell import progress_bar_enabled
+
+
+@dataclass
+class StageInfo:
+    stage_id: int
+    num_tasks: int
+    num_completed_tasks: int
+    num_bytes_read: int
+    done: bool
+
+
+class ProgressHandler(abc.ABC):
+    @abc.abstractmethod
+    def __call__(
+        self,
+        stages: typing.Optional[Iterable[StageInfo]],
+        inflight_tasks: int,
+        operation_id: typing.Optional[str],
+        done: bool,
+    ) -> None:
+        pass
+
+
+def from_proto(
+    proto: ExecutePlanResponse.ExecutionProgress,
+) -> typing.Tuple[Iterable[StageInfo], int]:
+    result = []
+    for stage in proto.stages:
+        result.append(
+            StageInfo(
+                stage_id=stage.stage_id,
+                num_tasks=stage.num_tasks,
+                num_completed_tasks=stage.num_completed_tasks,
+                num_bytes_read=stage.input_bytes_read,
+                done=stage.done,
+            )
+        )
+    return (result, proto.num_inflight_tasks)
+
+
+class Progress:
+    """This is a small helper class to visualize a textual progress bar.
+    he interface is very simple and assumes that nothing else prints to the
+    standard output."""
+
+    SI_BYTE_SIZES = (1 << 60, 1 << 50, 1 << 40, 1 << 30, 1 << 20, 1 << 10, 1)
+    SI_BYTE_SUFFIXES = ("EiB", "PiB", "TiB", "GiB", "MiB", "KiB", "B")
+
+    def __init__(
+        self,
+        char: str = "*",
+        min_width: int = 80,
+        output: typing.IO = sys.stdout,
+        enabled: bool = False,
+        handlers: Iterable[ProgressHandler] = [],
+        operation_id: typing.Optional[str] = None,
+    ) -> None:
+        """
+        Constructs a new Progress bar. The progress bar is typically used in
+        the blocking query execution path to process the execution progress
+        methods from the server.
+
+        Parameters
+        ----------
+        char : str
+          The Default character to be used for printing the bar.
+        min_width : numeric
+          The minimum width of the progress bar
+        output : file
+          The output device to write the progress bar to.
+        enabled : bool
+          Whether the progress bar printing should be enabled or not.
+        handlers : list of ProgressHandler
+          A list of handlers that will be called when the progress bar is 
updated.
+        """
+        self._ticks: typing.Optional[int] = None
+        self._tick: typing.Optional[int] = None
+        x, y = get_terminal_size()
+        self._min_width = min_width
+        self._char = char
+        self._width = max(min(min_width, x), self._min_width)
+        self._max_printed = 0
+        self._started = time.time()
+        self._enabled = enabled or progress_bar_enabled()
+        self._bytes_read = 0
+        self._out = output
+        self._running = 0
+        self._handlers = handlers
+        self._stages: Iterable[StageInfo] = []
+        self._operation_id = operation_id
+
+    def _notify(self, done: bool = False) -> None:
+        for handler in self._handlers:
+            handler(
+                stages=self._stages,
+                inflight_tasks=self._running,
+                operation_id=self._operation_id,
+                done=done,
+            )
+
+    def __enter__(self) -> "Progress":
+        return self
+
+    def __exit__(
+        self,
+        exc_type: typing.Optional[typing.Type[BaseException]],
+        exception: typing.Optional[BaseException],
+        exc_tb: typing.Optional[TracebackType],
+    ) -> typing.Any:
+        self.finish()
+        return False
+
+    def update_ticks(
+        self,
+        stages: Iterable[StageInfo],
+        inflight_tasks: int,
+        operation_id: typing.Optional[str] = None,
+    ) -> None:
+        """This method is called from the execution to update the progress bar 
with a new total
+        tick counter and the current position. This is necessary in case new 
stages get added with
+        new tasks and so the total task number will be updated as well.
+
+        Parameters
+        ----------
+        stages : list
+          A list of StageInfo objects reporting progress in each stage.
+        inflight_tasks : int
+          The number of tasks that are currently running.
+        """
+        if self._operation_id is None or len(self._operation_id) == 0:
+            self._operation_id = operation_id
+
+        total_tasks = sum(map(lambda x: x.num_tasks, stages))
+        completed_tasks = sum(map(lambda x: x.num_completed_tasks, stages))
+        if total_tasks > 0:
+            self._ticks = total_tasks
+            self._tick = completed_tasks
+            self._bytes_read = sum(map(lambda x: x.num_bytes_read, stages))
+            if self._tick is not None and self._tick >= 0:
+                self.output()
+            self._running = inflight_tasks
+            self._stages = stages
+            self._notify(False)
+
+    def finish(self) -> None:
+        """Clear the last line. Called when the processing is done."""
+        self._notify(True)
+        if self._enabled:
+            print("\r" + " " * self._max_printed, end="", flush=True, 
file=self._out)
+            print("\r", end="", flush=True, file=self._out)
+
+    def output(self) -> None:
+        """Writes the progress bar out."""
+        if self._enabled and self._tick is not None and self._ticks is not 
None:
+            val = int((self._tick / float(self._ticks)) * self._width)
+            bar = self._char * val + "-" * (self._width - val)
+            percent_complete = (self._tick / self._ticks) * 100
+            elapsed = int(time.time() - self._started)
+            scanned = self._bytes_to_string(self._bytes_read)
+            running = self._running
+            buffer = (
+                f"\r[{bar}] {percent_complete:.2f}% Complete "
+                f"({running} Tasks running, {elapsed}s, Scanned {scanned})"
+            )
+            self._max_printed = max(len(buffer), self._max_printed)
+            print(buffer, end="", flush=True, file=self._out)
+
+    @staticmethod
+    def _bytes_to_string(size: int) -> str:
+        """Helper method to convert a numeric bytes value into a 
human-readable representation"""
+        i = 0
+        while i < len(Progress.SI_BYTE_SIZES) - 1 and size < 2 * 
Progress.SI_BYTE_SIZES[i]:
+            i += 1
+        result = float(size) / Progress.SI_BYTE_SIZES[i]
+        return f"{result:.1f} {Progress.SI_BYTE_SUFFIXES[i]}"
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index ceaca2d..befa3fd 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1449,6 +1449,30 @@ def 
test_server_errors_carry_their_condition_and_details(session):
     assert "\n\nJVM stacktrace:\norg.apache.spark.SparkArithmeticException" in 
str(e)
 
 
+def 
test_progress_handlers_receive_the_progress_of_executions_as_pyspark(session):
+    # As PySpark Connect's: a handler is called with the keyword arguments of 
ProgressHandler,
+    # of each execution: its stages, its running tasks, its operation, and, 
last, done.
+    from pyspark.sql.connect.shell.progress import StageInfo
+
+    calls = []
+
+    def handler(*, stages, inflight_tasks, operation_id, done):
+        calls.append((list(stages), inflight_tasks, operation_id, done))
+
+    session.registerProgressHandler(handler)
+    try:
+        session.range(0, 8, 1, 4).selectExpr("sum(id)").collect()
+    finally:
+        session.removeProgressHandler(handler)
+    assert calls and calls[-1][3] is True
+    assert [done for *_, done in calls].count(True) == 1
+    assert len({operation_id for _, _, operation_id, _ in calls}) == 1
+    assert all(isinstance(stage, StageInfo) for stages, *_ in calls for stage 
in stages)
+    assert all(isinstance(inflight, int) for _, inflight, _, _ in calls)
+
+
+
+
 def test_apply_in_pandas_with_state_runs_as_pyspark(session, tmp_path):
     import pandas as pd
 
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index a5cdf29..cd41251 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -992,6 +992,69 @@ def test_show_checks_its_arguments_as_pyspark():
         df._show_string(1, n=2)
 
 
+def 
test_progress_handlers_are_called_with_the_progress_of_executions_as_pyspark(
+        mock_connect, capfd, monkeypatch):
+    # As PySpark Connect's: a handler is registered once, and called with the 
keyword arguments
+    # of ProgressHandler, of the StageInfo of each stage, for each progress of 
an execution that
+    # has tasks, and once more, as done, when it ends; removing a handler that 
is not registered
+    # raises a ValueError; an exception of a handler goes to 
sys.unraisablehook; and
+    # SPARK_CONNECT_PROGRESS_BAR_ENABLED prints the progress bar.
+    import sys
+
+    from pyspark.sql.connect.shell.progress import StageInfo
+
+    spark, service = mock_connect
+    # ExecutePlanResponse.execution_progress=18 -> ExecutionProgress.stages=1 
(StageInfo
+    # stage_id=1, num_tasks=2, num_completed_tasks=3, input_bytes_read=4, 
done=5),
+    # num_inflight_tasks=2.
+    stage = (varint_field(1, 7) + varint_field(2, 4) + varint_field(3, 1)
+             + varint_field(4, 1024) + varint_field(5, 0))
+    service.execute_payload = message_field(18, message_field(1, stage) + 
varint_field(2, 3))
+    calls = []
+
+    def handler(**kwargs):
+        calls.append(kwargs)
+
+    assert spark.registerProgressHandler(handler) is None
+    spark.registerProgressHandler(handler)
+    spark.range(1).collect()
+    assert [call["done"] for call in calls] == [False, True]
+    assert calls[0]["stages"] == [StageInfo(
+        stage_id=7, num_tasks=4, num_completed_tasks=1, num_bytes_read=1024, 
done=False)]
+    assert calls[1]["stages"] == calls[0]["stages"]
+    assert calls[1]["inflight_tasks"] == calls[0]["inflight_tasks"] == 3
+    assert calls[0]["operation_id"] == calls[1]["operation_id"]
+    assert len(calls[0]["operation_id"]) == 36
+
+    unraisable = []
+    monkeypatch.setattr(sys, "unraisablehook", unraisable.append)
+
+    def failing(**kwargs):
+        raise ZeroDivisionError("handler")
+
+    spark.registerProgressHandler(failing)
+    spark.range(1).collect()
+    assert unraisable and isinstance(unraisable[0].exc_value, 
ZeroDivisionError)
+    spark.removeProgressHandler(failing)
+
+    monkeypatch.setenv("SPARK_CONNECT_PROGRESS_BAR_ENABLED", "1")
+    capfd.readouterr()
+    spark.range(1).collect()
+    assert "25.00% Complete" in capfd.readouterr().out
+    monkeypatch.delenv("SPARK_CONNECT_PROGRESS_BAR_ENABLED")
+
+    spark.removeProgressHandler(handler)
+    with pytest.raises(ValueError):
+        spark.removeProgressHandler(handler)
+    calls.clear()
+    spark.registerProgressHandler(handler)
+    spark.clearProgressHandlers()
+    spark.range(1).collect()
+    assert calls == []
+
+
+
+
 def test_udfs_are_created_as_pyspark(mock_connect):
     # As PySpark's _create_py_udf: udf() without useArrow makes an 
Arrow-optimized UDF
     # (SQL_ARROW_BATCHED_UDF) when the active session enables
diff --git a/scripts/check_vendored_upstream.sh 
b/scripts/check_vendored_upstream.sh
index 875eb6d..cfdf7ce 100644
--- a/scripts/check_vendored_upstream.sh
+++ b/scripts/check_vendored_upstream.sh
@@ -85,6 +85,10 @@ VENDORED_FILES=(
   "pipelines/tests/test_decorators.py"
   "pipelines/tests/test_graph_element_registry.py"
 
+  # The progress handlers and the progress bar of the executions of a session, 
whose
+  # ExecutePlanResponse is that of the message class of 
pyspark/sql/connect/proto.py.
+  "sql/connect/shell/progress.py"
+
   # Top-level modules
   "accumulators.py"
   "memory_profiler_ext.py"


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to