This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new a556dc7  [SPARK-60858][CONNECT] Close remaining Spark 4.3.0 client API 
gaps
a556dc7 is described below

commit a556dc767df594d27ec0b8bf891685ffb342f849
Author: Liang-Chi Hsieh <[email protected]>
AuthorDate: Wed Nov 4 14:27:30 2026 +0900

    [SPARK-60858][CONNECT] Close remaining Spark 4.3.0 client API gaps
    
    ### What changes were proposed in this pull request?
    
    This closes the Spark 4.3.0 client API gaps that remain after SPARK-59305, 
in the Rust core, the PyO3 layer and the Python shim:
    
    - **Nanosecond timestamp types.** The `TimestampNTZNanos` / 
`TimestampLTZNanos` proto kinds used to decode as `TIME` and as a microsecond 
`TIMESTAMP`. They now have their own `DataType` variants, `TimestampNtzNanos` / 
`TimestampLtzNanos`, with proto, JSON (`timestamp_ntz(p)` / 
`timestamp_ltz(p)`), DDL and Arrow (nanosecond) support. In Python they appear 
as picklable `TimestampNTZNanosType` / `TimestampLTZNanosType`. These are not 
added to `pyspark.sql.types.__all__`, because PySpark  [...]
    - **Functions.** `instr(..., occurrence=k)` without `start` searches from 
position 1, and `None` means the argument is omitted, as in 4.3.0. 
`regexp_replace(..., position=None)` also treats `None` as omitted. 
`variant_delete(v)` with no paths raises `CANNOT_BE_EMPTY`.
    - **`SparkSession.__enter__` / `__exit__`.** Leaving the `with` block stops 
the session. Like PySpark, `stop()` can now be called more than once and only 
logs a warning when releasing the session fails. It also clears the active 
session and the stopped session's cached `SQLContext`.
    - **Spark Connect `SQLContext` / `HiveContext`.** These are ported from 
v4.3.0 `pyspark/sql/connect/context.py`, and 
`pyspark.sql.SQLContext.getOrCreate()` / `HiveContext.getOrCreate()` route to 
them. The cache compares sessions by session id, because 
`SparkSession.getActiveSession()` returns a new wrapper object on every call.
    - **Pipelines `create_auto_cdc_flow`.** It now supports 
`stored_as_scd_type` 2, the keyword-only `track_history_column_list` / 
`track_history_except_column_list`, and `spark_conf`, with the same client-side 
validation as 4.3.0. `api.py` is v4.3.0 plus the existing import repointing. 
The registry, PyO3 and core now send the track-history lists, `SCD_TYPE_2` and 
`sql_conf`, which used to be hardcoded empty.
    - **`operation_id`.**
      - `DataFrame.executionInfo` now returns a 
`pyspark.sql.metrics.ExecutionInfo`, including `operation_id`, instead of a 
dict. `sql/metrics.py` is vendored verbatim from v4.3.0 and added to the drift 
check.
      - Errors from an ExecutePlan carry the operation id, read by 
`SparkConnectException.operation_id`.
      - In this drop-in, `SparkConnectException` is the same class as 
`PySparkException`, so the property is defined on that shared base.
    
    ### Why are the changes needed?
    
    These 4.3.0 APIs were missing or behaved differently from PySpark 4.3.0. 
The nanosecond timestamp types were also decoded as the wrong types.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, the APIs above now exist and behave as in PySpark 4.3.0. 
`DataFrame.executionInfo` now returns an `ExecutionInfo` (or `None` before an 
action) instead of a dict.
    
    ### How was this patch tested?
    
    - New Rust unit tests: the nanos types in `types_roundtrip.rs`, `stop()` 
being safe to call twice, `operation_id` capture, and 
`SparkError::with_operation_id`.
    - New offline drop-in tests in `test_dropin_offline.py`.
    - The official v4.3.0 `pipelines/tests/test_auto_cdc_flow.py` passes (19 
tests).
    - `cargo fmt --all --check`, `cargo test -p apache-spark-connect --release 
--lib`, and all integration tests compile with and without `--features 
wasm-udf`.
    - Not run locally, because no Spark 4.3.0 Connect server was available: the 
e2e suites and the official `test_connect_context.py`, 
`test_parity_sql_context.py` and `test_parity_zip.py`. These run on CI.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #146 from viirya/spark-430-api-gaps-80fff8.
    
    Authored-by: Liang-Chi Hsieh <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/dataframe.rs                 | 199 ++++++++++++-
 crates/pyspark-rs/src/errors.rs                    |   6 +
 crates/pyspark-rs/src/lib.rs                       |   6 +-
 crates/pyspark-rs/src/pipelines.rs                 |  44 ++-
 crates/pyspark-rs/src/session.rs                   |  47 ++-
 crates/pyspark-rs/src/types.rs                     | 174 +++++++++++
 crates/spark-connect-core/src/client.rs            |   6 +-
 crates/spark-connect-core/src/error.rs             |  23 ++
 crates/spark-connect/src/dataframe.rs              |  18 ++
 crates/spark-connect/src/pipelines.rs              |   3 +-
 crates/spark-connect/src/profiler.rs               |   2 +-
 crates/spark-connect/src/session.rs                | 135 ++++++++-
 crates/spark-connect/src/types.rs                  |  83 +++++-
 crates/spark-connect/tests/types_roundtrip.rs      |  39 +++
 python/.coveragerc                                 |   1 +
 python/pyspark/errors/exceptions/__init__.py       |  16 +
 python/pyspark/pipelines/api.py                    |  97 ++++++-
 .../spark_connect_graph_element_registry.py        |  16 +-
 python/pyspark/sql/connect/client/__init__.py      |  58 ++++
 python/pyspark/sql/connect/context.py              | 255 ++++++++++++++++
 python/pyspark/sql/context.py                      |  87 +++++-
 python/pyspark/sql/functions.py                    |  30 ++
 python/pyspark/sql/metrics.py                      | 323 +++++++++++++++++++++
 python/pyspark/sql/types.py                        |   6 +
 python/tests/_mock_connect_server.py               |  62 ++++
 python/tests/test_dropin_offline.py                | 316 +++++++++++++++++++-
 scripts/check_vendored_upstream.sh                 |   2 +
 27 files changed, 1999 insertions(+), 55 deletions(-)

diff --git a/crates/pyspark-rs/src/dataframe.rs 
b/crates/pyspark-rs/src/dataframe.rs
index cf32ab2..1dfe89e 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -1230,20 +1230,48 @@ impl PyDataFrame {
         py.detach(|| self.dataframe.show(n)).to_pyerr()
     }
 
-    /// Execution metrics from the most recent action. Mirrors 
`DataFrame.executionInfo`.
+    /// Execution info from the most recent action, as a 
`pyspark.sql.metrics.ExecutionInfo`
+    /// (plan metrics, observed-metric flows, and the ExecutePlan 
`operation_id`). Mirrors
+    /// `DataFrame.executionInfo`; `None` before any action ran.
     #[getter]
     #[pyo3(name = "executionInfo")]
     fn execution_info(&self, py: Python<'_>) -> PyResult<Py<PyAny>> {
-        let info = self.dataframe.execution_info().to_pyerr()?;
-        let dict = pyo3::types::PyDict::new(py);
-        dict.set_item("has_metrics", info.metrics.is_some())?;
-        let names: Vec<String> = info
-            .observed_metrics
-            .iter()
-            .map(|m| m.name.clone())
-            .collect();
-        dict.set_item("observed_metrics", names)?;
-        Ok(dict.into_any().unbind())
+        let Ok(info) = self.dataframe.execution_info() else {
+            return Ok(py.None());
+        };
+        let metrics_mod = py.import("pyspark.sql.metrics")?;
+        let plan_metrics_cls = metrics_mod.getattr("PlanMetrics")?;
+        let metric_value_cls = metrics_mod.getattr("MetricValue")?;
+        // Mirrors SparkConnectClient._build_metrics.
+        let plan_metrics = pyo3::types::PyList::empty(py);
+        if let Some(m) = &info.metrics {
+            for x in &m.metrics {
+                let values = pyo3::types::PyList::empty(py);
+                for (k, v) in &x.execution_metrics {
+                    values.append(metric_value_cls.call1((k, v.value, 
&v.metric_type))?)?;
+                }
+                plan_metrics
+                    .append(plan_metrics_cls.call1((&x.name, x.plan_id, 
x.parent, values))?)?;
+            }
+        }
+        let observed_cls = py
+            .import("pyspark.sql.connect.client")?
+            .getattr("PlanObservedMetrics")?;
+        let observed = pyo3::types::PyList::empty(py);
+        for o in &info.observed_metrics {
+            let values = o
+                .values
+                .iter()
+                .map(|l| literal_to_py(py, l, None))
+                .collect::<PyResult<Vec<_>>>()?;
+            observed.append(observed_cls.call1((&o.name, values, 
o.keys.clone()))?)?;
+        }
+        let ei = metrics_mod.getattr("ExecutionInfo")?.call1((
+            plan_metrics,
+            observed,
+            info.operation_id.clone(),
+        ))?;
+        Ok(ei.unbind())
     }
 
     /// Convert to a pandas-on-Spark DataFrame. Mirrors `DataFrame.pandas_api`;
@@ -2491,3 +2519,152 @@ impl PyLocalRowIterator {
         }
     }
 }
+
+/// Convert an observed-metric literal to a Python value. Mirrors
+/// `LiteralExpression._to_value`, which `PlanObservedMetrics.pairs` applies: 
the kinds it
+/// decodes map to the same Python values, and for any other kind (which the 
reference
+/// rejects with `UNSUPPORTED_LITERAL`) this returns an `_UnsupportedLiteral` 
marker, so
+/// the error surfaces when `flows` is read, as in the reference, not when the
+/// `ExecutionInfo` is built.
+///
+/// `data_type` is the literal's type when the enclosing literal carries it; a 
struct's
+/// field names come from it, else from the literal's own `data_type`, else 
from the
+/// deprecated `struct_type`.
+fn literal_to_py(
+    py: Python<'_>,
+    literal: &spark_connect_proto::expression::Literal,
+    data_type: Option<&spark_connect_proto::DataType>,
+) -> PyResult<Py<PyAny>> {
+    use pyo3::IntoPyObjectExt;
+    use spark_connect_proto::data_type::Kind;
+    use spark_connect_proto::expression::literal::LiteralType;
+    let data_type = data_type.or(literal.data_type.as_ref());
+    let datetime = || py.import("datetime");
+    Ok(match &literal.literal_type {
+        None | Some(LiteralType::Null(_)) => py.None(),
+        Some(LiteralType::Boolean(b)) => b.into_py_any(py)?,
+        Some(LiteralType::Byte(v))
+        | Some(LiteralType::Short(v))
+        | Some(LiteralType::Integer(v)) => v.into_py_any(py)?,
+        Some(LiteralType::Long(v)) => v.into_py_any(py)?,
+        Some(LiteralType::Float(v)) => (*v as f64).into_py_any(py)?,
+        Some(LiteralType::Double(v)) => v.into_py_any(py)?,
+        Some(LiteralType::String(s)) => s.into_py_any(py)?,
+        Some(LiteralType::Binary(b)) => pyo3::types::PyBytes::new(py, 
b).into_any().unbind(),
+        Some(LiteralType::Decimal(d)) => py
+            .import("decimal")?
+            .getattr("Decimal")?
+            .call1((&d.value,))?
+            .unbind(),
+        // DateType.fromInternal: days since the epoch (ordinal 719163).
+        Some(LiteralType::Date(days)) => datetime()?
+            .getattr("date")?
+            .call_method1("fromordinal", (*days as i64 + 719_163,))?
+            .unbind(),
+        // TimeType.fromInternal: nanoseconds of the day, truncated to 
microseconds.
+        Some(LiteralType::Time(t)) => {
+            let nano = t.nano;
+            datetime()?
+                .getattr("time")?
+                .call1((
+                    nano.div_euclid(3_600_000_000_000),
+                    nano.rem_euclid(3_600_000_000_000) / 60_000_000_000,
+                    nano.rem_euclid(60_000_000_000) / 1_000_000_000,
+                    nano.rem_euclid(1_000_000_000) / 1_000,
+                ))?
+                .unbind()
+        }
+        // TimestampType.fromInternal: a naive local datetime, built from 
whole seconds so
+        // the microseconds are exact.
+        Some(LiteralType::Timestamp(ts)) => {
+            let kwargs = pyo3::types::PyDict::new(py);
+            kwargs.set_item("microsecond", ts.rem_euclid(1_000_000))?;
+            datetime()?
+                .getattr("datetime")?
+                .call_method1("fromtimestamp", (ts.div_euclid(1_000_000),))?
+                .call_method("replace", (), Some(&kwargs))?
+                .unbind()
+        }
+        // TimestampNTZType.fromInternal: the UTC wall clock, without a zone.
+        Some(LiteralType::TimestampNtz(ts)) => {
+            let dt = datetime()?;
+            let utc = dt.getattr("timezone")?.getattr("utc")?;
+            let kwargs = pyo3::types::PyDict::new(py);
+            kwargs.set_item("microsecond", ts.rem_euclid(1_000_000))?;
+            kwargs.set_item("tzinfo", py.None())?;
+            dt.getattr("datetime")?
+                .call_method1("fromtimestamp", (ts.div_euclid(1_000_000), 
utc))?
+                .call_method("replace", (), Some(&kwargs))?
+                .unbind()
+        }
+        // DayTimeIntervalType.fromInternal: timedelta(microseconds=v).
+        Some(LiteralType::DayTimeInterval(micros)) => {
+            let kwargs = pyo3::types::PyDict::new(py);
+            kwargs.set_item("microseconds", micros)?;
+            datetime()?
+                .getattr("timedelta")?
+                .call((), Some(&kwargs))?
+                .unbind()
+        }
+        Some(LiteralType::Array(a)) => {
+            let element_type = match data_type.and_then(|t| t.kind.as_ref()) {
+                Some(Kind::Array(t)) => t.element_type.as_deref(),
+                _ => None,
+            };
+            let items = a
+                .elements
+                .iter()
+                .map(|e| literal_to_py(py, e, element_type))
+                .collect::<PyResult<Vec<_>>>()?;
+            items.into_py_any(py)?
+        }
+        Some(LiteralType::Map(m)) => {
+            let (key_type, value_type) = match data_type.and_then(|t| 
t.kind.as_ref()) {
+                Some(Kind::Map(t)) => (t.key_type.as_deref(), 
t.value_type.as_deref()),
+                _ => (None, None),
+            };
+            let dict = pyo3::types::PyDict::new(py);
+            for (k, v) in m.keys.iter().zip(&m.values) {
+                dict.set_item(
+                    literal_to_py(py, k, key_type)?,
+                    literal_to_py(py, v, value_type)?,
+                )?;
+            }
+            dict.into_any().unbind()
+        }
+        Some(LiteralType::Struct(s)) => {
+            #[allow(deprecated)]
+            let struct_type = match data_type.and_then(|t| t.kind.as_ref()) {
+                Some(Kind::Struct(t)) => Some(t),
+                _ => match s.struct_type.as_ref().and_then(|t| 
t.kind.as_ref()) {
+                    Some(Kind::Struct(t)) => Some(t),
+                    _ => None,
+                },
+            };
+            let fields = struct_type.map(|t| 
t.fields.as_slice()).unwrap_or_default();
+            let names = (0..s.elements.len())
+                .map(|i| match fields.get(i) {
+                    Some(f) => f.name.clone(),
+                    None => format!("_{}", i + 1),
+                })
+                .collect::<Vec<_>>();
+            let values = s
+                .elements
+                .iter()
+                .enumerate()
+                .map(|(i, e)| {
+                    literal_to_py(py, e, fields.get(i).and_then(|f| 
f.data_type.as_ref()))
+                })
+                .collect::<PyResult<Vec<_>>>()?;
+            py.import("pyspark.sql.types")?
+                .getattr("_create_row")?
+                .call1((names, values))?
+                .unbind()
+        }
+        Some(_) => py
+            .import("pyspark.sql.connect.client")?
+            .getattr("_UnsupportedLiteral")?
+            .call1((format!("{literal:?}"),))?
+            .unbind(),
+    })
+}
diff --git a/crates/pyspark-rs/src/errors.rs b/crates/pyspark-rs/src/errors.rs
index c591e09..ea079cf 100644
--- a/crates/pyspark-rs/src/errors.rs
+++ b/crates/pyspark-rs/src/errors.rs
@@ -55,6 +55,7 @@ fn kind_to_pyspark_class_name(kind: SparkErrorKind) -> 
&'static str {
 /// exception based on the error kind. Falls back to PyRuntimeError if import 
fails.
 pub fn spark_error_to_py_exception(err: SparkError) -> PyErr {
     let msg = err.message();
+    let operation_id = err.operation_id.clone();
     let class_name = kind_to_pyspark_class_name(err.kind);
 
     // Try to acquire the GIL and import the exception class from 
pyspark.errors
@@ -75,6 +76,11 @@ pub fn spark_error_to_py_exception(err: SparkError) -> PyErr 
{
                 // This creates a PyObject exception instance
                 match exc_class.call1((msg.clone(),)) {
                     Ok(exc_instance) => {
+                        // Mirror the reference client, which stamps 
`_operation_id` on
+                        // errors raised for an ExecutePlan (read via 
`operation_id`).
+                        if let Some(op) = &operation_id {
+                            let _ = exc_instance.setattr("_operation_id", op);
+                        }
                         // Convert the exception instance to a PyErr.
                         // We use unsafe code to construct the PyErr from the 
exception value.
                         PyErr::from_value(exc_instance)
diff --git a/crates/pyspark-rs/src/lib.rs b/crates/pyspark-rs/src/lib.rs
index e1bd4f4..0394c64 100644
--- a/crates/pyspark-rs/src/lib.rs
+++ b/crates/pyspark-rs/src/lib.rs
@@ -84,8 +84,8 @@ use types::{
     PyDayTimeIntervalType, PyDecimalType, PyDoubleType, PyFloatType, 
PyFractionalType,
     PyGeographyType, PyGeometryType, PyIntegerType, PyIntegralType, 
PyLongType, PyMapType,
     PyNullType, PyNumericType, PyShortType, PySpatialType, PyStringType, 
PyStructField,
-    PyStructType, PyTimeType, PyTimestampNTZType, PyTimestampType, 
PyVarcharType, PyVariantType,
-    PyYearMonthIntervalType,
+    PyStructType, PyTimeType, PyTimestampLTZNanosType, 
PyTimestampNTZNanosType, PyTimestampNTZType,
+    PyTimestampType, PyVarcharType, PyVariantType, PyYearMonthIntervalType,
 };
 use window::{PyFrameBound, PyWindow, PyWindowSpec};
 
@@ -189,6 +189,8 @@ fn _pyspark(_py: Python<'_>, m: &Bound<'_, PyModule>) -> 
PyResult<()> {
     m.add_class::<PyDateType>()?;
     m.add_class::<PyTimestampType>()?;
     m.add_class::<PyTimestampNTZType>()?;
+    m.add_class::<PyTimestampNTZNanosType>()?;
+    m.add_class::<PyTimestampLTZNanosType>()?;
     m.add_class::<PyArrayType>()?;
     m.add_class::<PyMapType>()?;
     m.add_class::<PyStructField>()?;
diff --git a/crates/pyspark-rs/src/pipelines.rs 
b/crates/pyspark-rs/src/pipelines.rs
index 40c41b8..c8f5f55 100644
--- a/crates/pyspark-rs/src/pipelines.rs
+++ b/crates/pyspark-rs/src/pipelines.rs
@@ -162,12 +162,15 @@ pub fn pipeline_define_flow(
 }
 
 /// `PipelineCommand.DefineFlow` carrying an Auto-CDC flow body. 
`stored_as_scd_type` is
-/// the `SCDType` enum value (0 unspecified, 1 SCD_TYPE_1, 2 SCD_TYPE_2).
+/// the `SCDType` enum value (0 unspecified, 1 SCD_TYPE_1, 2 SCD_TYPE_2). The 
SCD2-only
+/// track-history lists are validated client-side in `pipelines/api.py`; 
`sql_conf` is
+/// forwarded as given.
 #[pyfunction]
 #[pyo3(signature = (
     session, dataflow_graph_id, flow_name, target, source, keys, sequence_by,
     apply_as_deletes=None, column_list=None, except_column_list=None,
-    stored_as_scd_type=0, scl_file=None, scl_line=None
+    stored_as_scd_type=0, scl_file=None, scl_line=None,
+    track_history_column_list=None, track_history_except_column_list=None, 
sql_conf=None
 ))]
 #[allow(clippy::too_many_arguments)]
 pub fn pipeline_define_auto_cdc_flow(
@@ -184,27 +187,28 @@ pub fn pipeline_define_auto_cdc_flow(
     stored_as_scd_type: i32,
     scl_file: Option<String>,
     scl_line: Option<i32>,
+    track_history_column_list: Option<Vec<PyRef<'_, PyColumn>>>,
+    track_history_except_column_list: Option<Vec<PyRef<'_, PyColumn>>>,
+    sql_conf: Option<HashMap<String, String>>,
 ) -> PyResult<()> {
     let to_expr = |c: &PyColumn| c.column.expression().to_proto();
+    let to_exprs = |cs: Option<Vec<PyRef<'_, PyColumn>>>| {
+        cs.unwrap_or_default()
+            .iter()
+            .map(|c| to_expr(c))
+            .collect::<Vec<_>>()
+    };
     let details = proto::pipeline_command::define_flow::AutoCdcFlowDetails {
         source: Some(source),
         keys: keys.iter().map(|c| to_expr(c)).collect(),
         sequence_by: Some(to_expr(&sequence_by)),
         apply_as_deletes: apply_as_deletes.as_ref().map(|c| to_expr(c)),
         apply_as_truncates: None,
-        column_list: column_list
-            .unwrap_or_default()
-            .iter()
-            .map(|c| to_expr(c))
-            .collect(),
-        except_column_list: except_column_list
-            .unwrap_or_default()
-            .iter()
-            .map(|c| to_expr(c))
-            .collect(),
+        column_list: to_exprs(column_list),
+        except_column_list: to_exprs(except_column_list),
         stored_as_scd_type,
-        track_history_column_list: vec![],
-        track_history_except_column_list: vec![],
+        track_history_column_list: to_exprs(track_history_column_list),
+        track_history_except_column_list: 
to_exprs(track_history_except_column_list),
         ignore_null_updates_column_list: vec![],
         ignore_null_updates_except_column_list: vec![],
     };
@@ -217,7 +221,17 @@ pub fn pipeline_define_auto_cdc_flow(
     let scl = scl(scl_file, scl_line);
     session
         .py()
-        .detach(|| pipelines::define_auto_cdc_flow(&sess, &gid, &fname, &tgt, 
details, scl))
+        .detach(|| {
+            pipelines::define_auto_cdc_flow(
+                &sess,
+                &gid,
+                &fname,
+                &tgt,
+                details,
+                sql_conf.unwrap_or_default(),
+                scl,
+            )
+        })
         .to_pyerr()
 }
 
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index f81bde1..89dd7ce 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -721,9 +721,52 @@ impl PySparkSession {
         PyStreamingQueryManager::new(self.session.streams())
     }
 
-    /// Stop this Spark session.
+    /// Stop this Spark session. Best-effort and idempotent, mirroring 
pyspark: a failure
+    /// to release the session on the server is logged, not raised. Also 
clears the
+    /// active session and the cached `SQLContext` if they belong to this 
session.
     fn stop(&self, py: Python<'_>) -> PyResult<()> {
-        py.detach(|| self.session.stop()).to_pyerr()
+        if let Err(e) = py.detach(|| self.session.stop()) {
+            let logger = py
+                .import("logging")?
+                .call_method1("getLogger", ("pyspark",))?;
+            logger.call_method1(
+                "warning",
+                (format!(
+                    "session.stop(): Session could not be released. Error: {e}"
+                ),),
+            )?;
+        }
+        let sid = self.session.session_id();
+        {
+            let mut slot = active_slot().lock().unwrap();
+            if slot.as_ref().is_some_and(|s| s.session_id() == sid) {
+                *slot = None;
+            }
+        }
+        // Only touch the SQLContext cache if the module was ever imported (as 
pyspark does).
+        let modules = py.import("sys")?.getattr("modules")?;
+        if let Ok(ctx_mod) = modules.get_item("pyspark.sql.connect.context") {
+            ctx_mod.call_method1("_reset_instantiated_context", (sid,))?;
+        }
+        Ok(())
+    }
+
+    /// Enable `with SparkSession.builder.(...).getOrCreate() as session:` 
syntax.
+    fn __enter__(slf: Py<Self>) -> Py<Self> {
+        slf
+    }
+
+    /// Stop the session when leaving a `with` block (mirrors pyspark 4.3.0).
+    #[pyo3(signature = (exc_type=None, exc_value=None, traceback=None))]
+    fn __exit__(
+        &self,
+        py: Python<'_>,
+        exc_type: Option<Py<PyAny>>,
+        exc_value: Option<Py<PyAny>>,
+        traceback: Option<Py<PyAny>>,
+    ) -> PyResult<()> {
+        let _ = (exc_type, exc_value, traceback);
+        self.stop(py)
     }
 
     /// Get the UDF registration API (`spark.udf`). Returns the
diff --git a/crates/pyspark-rs/src/types.rs b/crates/pyspark-rs/src/types.rs
index 761c1cb..935fc72 100644
--- a/crates/pyspark-rs/src/types.rs
+++ b/crates/pyspark-rs/src/types.rs
@@ -2085,6 +2085,152 @@ impl PyTimeType {
     }
 }
 
+/// Spark's `TimestampNTZNanosType(precision)` (precision 7..=9; default 9). 
PySpark 4.3.0 has no
+/// Python class for the nanosecond timestamp types, but the server reports 
them in
+/// schemas, so they materialize as this class instead of being mapped to a 
lossy type.
+#[pyclass(name = "TimestampNTZNanosType", module = "pyspark.sql.types", 
extends = PyDatetimeType)]
+pub struct PyTimestampNTZNanosType {
+    pub precision: i32,
+}
+
+#[pymethods]
+impl PyTimestampNTZNanosType {
+    #[pyo3(name = "json")]
+    fn __obj_json(slf: &Bound<'_, Self>) -> PyResult<String> {
+        Ok(py_to_data_type(slf.as_any())?.json())
+    }
+    #[pyo3(name = "jsonValue")]
+    fn __obj_json_value<'py>(
+        slf: &Bound<'py, Self>,
+        py: Python<'py>,
+    ) -> PyResult<Bound<'py, PyAny>> {
+        let v = py_to_data_type(slf.as_any())?.json_value();
+        py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+    }
+    #[pyo3(name = "needConversion")]
+    fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
+        Ok(py_to_data_type(slf.as_any())?.need_conversion())
+    }
+    #[pyo3(name = "fromInternal")]
+    fn __obj_from_internal<'py>(&self, obj: Bound<'py, PyAny>) -> Bound<'py, 
PyAny> {
+        obj
+    }
+    #[pyo3(name = "toInternal")]
+    fn __obj_to_internal<'py>(&self, obj: Bound<'py, PyAny>) -> Bound<'py, 
PyAny> {
+        obj
+    }
+    fn __reduce__(&self, py: Python<'_>) -> PyResult<(Py<PyAny>, (String,))> {
+        let dt = DataType::TimestampNtzNanos {
+            precision: self.precision,
+        };
+        type_reduce(py, &dt.json())
+    }
+    #[new]
+    #[pyo3(signature = (precision=9))]
+    fn new(precision: i32) -> PyResult<pyo3::PyClassInitializer<Self>> {
+        if !(7..=9).contains(&precision) {
+            return Err(PyErr::new::<pyo3::exceptions::PyValueError, _>(format!(
+                "TimestampNTZNanosType precision must be in 7..9, got {}",
+                precision
+            )));
+        }
+        Ok(init_chain!(
+            DataType::TimestampNtzNanos { precision },
+            PyTimestampNTZNanosType { precision },
+            [PyAtomicType, PyDatetimeType],
+            value
+        ))
+    }
+    #[getter]
+    fn precision(&self) -> i32 {
+        self.precision
+    }
+    fn __repr__(&self) -> String {
+        format!("TimestampNTZNanosType({})", self.precision)
+    }
+    #[pyo3(name = "simpleString")]
+    fn simple_string(&self) -> String {
+        format!("timestamp_ntz({})", self.precision)
+    }
+    #[pyo3(name = "typeName")]
+    fn type_name(&self) -> String {
+        format!("timestamp_ntz({})", self.precision)
+    }
+}
+
+/// Spark's `TimestampLTZNanosType(precision)` (precision 7..=9; default 9). 
PySpark 4.3.0 has no
+/// Python class for the nanosecond timestamp types, but the server reports 
them in
+/// schemas, so they materialize as this class instead of being mapped to a 
lossy type.
+#[pyclass(name = "TimestampLTZNanosType", module = "pyspark.sql.types", 
extends = PyDatetimeType)]
+pub struct PyTimestampLTZNanosType {
+    pub precision: i32,
+}
+
+#[pymethods]
+impl PyTimestampLTZNanosType {
+    #[pyo3(name = "json")]
+    fn __obj_json(slf: &Bound<'_, Self>) -> PyResult<String> {
+        Ok(py_to_data_type(slf.as_any())?.json())
+    }
+    #[pyo3(name = "jsonValue")]
+    fn __obj_json_value<'py>(
+        slf: &Bound<'py, Self>,
+        py: Python<'py>,
+    ) -> PyResult<Bound<'py, PyAny>> {
+        let v = py_to_data_type(slf.as_any())?.json_value();
+        py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+    }
+    #[pyo3(name = "needConversion")]
+    fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
+        Ok(py_to_data_type(slf.as_any())?.need_conversion())
+    }
+    #[pyo3(name = "fromInternal")]
+    fn __obj_from_internal<'py>(&self, obj: Bound<'py, PyAny>) -> Bound<'py, 
PyAny> {
+        obj
+    }
+    #[pyo3(name = "toInternal")]
+    fn __obj_to_internal<'py>(&self, obj: Bound<'py, PyAny>) -> Bound<'py, 
PyAny> {
+        obj
+    }
+    fn __reduce__(&self, py: Python<'_>) -> PyResult<(Py<PyAny>, (String,))> {
+        let dt = DataType::TimestampLtzNanos {
+            precision: self.precision,
+        };
+        type_reduce(py, &dt.json())
+    }
+    #[new]
+    #[pyo3(signature = (precision=9))]
+    fn new(precision: i32) -> PyResult<pyo3::PyClassInitializer<Self>> {
+        if !(7..=9).contains(&precision) {
+            return Err(PyErr::new::<pyo3::exceptions::PyValueError, _>(format!(
+                "TimestampLTZNanosType precision must be in 7..9, got {}",
+                precision
+            )));
+        }
+        Ok(init_chain!(
+            DataType::TimestampLtzNanos { precision },
+            PyTimestampLTZNanosType { precision },
+            [PyAtomicType, PyDatetimeType],
+            value
+        ))
+    }
+    #[getter]
+    fn precision(&self) -> i32 {
+        self.precision
+    }
+    fn __repr__(&self) -> String {
+        format!("TimestampLTZNanosType({})", self.precision)
+    }
+    #[pyo3(name = "simpleString")]
+    fn simple_string(&self) -> String {
+        format!("timestamp_ltz({})", self.precision)
+    }
+    #[pyo3(name = "typeName")]
+    fn type_name(&self) -> String {
+        format!("timestamp_ltz({})", self.precision)
+    }
+}
+
 #[pyclass(name = "CalendarIntervalType", module = "pyspark.sql.types", extends 
= PyDataType)]
 pub struct PyCalendarIntervalType;
 
@@ -2620,6 +2766,24 @@ pub(crate) fn data_type_to_py(py: Python<'_>, dt: 
&DataType) -> PyResult<Py<PyAn
                 value
             ))
         }
+        DataType::TimestampNtzNanos { precision } => {
+            let precision = *precision;
+            obj!(init_chain!(
+                DataType::TimestampNtzNanos { precision },
+                PyTimestampNTZNanosType { precision },
+                [PyAtomicType, PyDatetimeType],
+                value
+            ))
+        }
+        DataType::TimestampLtzNanos { precision } => {
+            let precision = *precision;
+            obj!(init_chain!(
+                DataType::TimestampLtzNanos { precision },
+                PyTimestampLTZNanosType { precision },
+                [PyAtomicType, PyDatetimeType],
+                value
+            ))
+        }
         DataType::CalendarInterval => obj!(init_chain!(
             DataType::CalendarInterval,
             PyCalendarIntervalType,
@@ -2744,6 +2908,16 @@ pub(crate) fn py_to_data_type(obj: &Bound<'_, PyAny>) -> 
PyResult<DataType> {
     if obj.extract::<PyRef<PyTimestampNTZType>>().is_ok() {
         return Ok(DataType::TimestampNtz);
     }
+    if let Ok(t) = obj.extract::<PyRef<PyTimestampNTZNanosType>>() {
+        return Ok(DataType::TimestampNtzNanos {
+            precision: t.precision,
+        });
+    }
+    if let Ok(t) = obj.extract::<PyRef<PyTimestampLTZNanosType>>() {
+        return Ok(DataType::TimestampLtzNanos {
+            precision: t.precision,
+        });
+    }
     if let Ok(d) = obj.extract::<PyRef<PyDecimalType>>() {
         return Ok(DataType::Decimal {
             precision: d.precision,
diff --git a/crates/spark-connect-core/src/client.rs 
b/crates/spark-connect-core/src/client.rs
index 0a71747..948a0a5 100644
--- a/crates/spark-connect-core/src/client.rs
+++ b/crates/spark-connect-core/src/client.rs
@@ -359,7 +359,8 @@ impl SparkConnectClient {
                 let mut stub = self.stub.clone();
                 async move { 
stub.execute_plan(req).await.map(Response::into_inner) }
             })
-            .await?;
+            .await
+            .map_err(|e| e.with_operation_id(iter.operation_id()))?;
         let retry_policy = self.retry_policy.clone();
         Ok(ReattachableResponseStream {
             stub: self.stub.clone(),
@@ -977,10 +978,11 @@ impl ReattachableResponseStream {
 
     /// Ends the stream with `error`, releasing the execution as the reference 
client does
     /// when an execution fails.
+    /// The error carries the ExecutePlan operation id 
(`SparkConnectException.operation_id`).
     fn fail(&mut self, error: SparkError) -> SparkError {
         self.done = true;
         self.release_all();
-        error
+        error.with_operation_id(self.iter.operation_id())
     }
 
     /// Releases the responses up to and including `response_id`, which were 
read.
diff --git a/crates/spark-connect-core/src/error.rs 
b/crates/spark-connect-core/src/error.rs
index ae5fa3a..c0f0a73 100644
--- a/crates/spark-connect-core/src/error.rs
+++ b/crates/spark-connect-core/src/error.rs
@@ -173,6 +173,9 @@ pub struct SparkError {
     /// Raw `grpc-status-details-bin` (a serialized `google.rpc.Status`), if 
present.
     /// Lets the PyO3 layer reconstruct the exact typed pyspark exception.
     pub grpc_details: Option<Vec<u8>>,
+    /// The ExecutePlan operation id the error was raised for, if known. Backs
+    /// `SparkConnectException.operation_id` (new in pyspark 4.3.0).
+    pub operation_id: Option<String>,
 }
 
 /// The PySpark exception hierarchy leaf we map to.
@@ -265,6 +268,7 @@ impl SparkError {
             server_stacktrace: None,
             grpc_code: None,
             grpc_details: None,
+            operation_id: None,
         }
     }
 
@@ -285,6 +289,7 @@ impl SparkError {
             server_stacktrace: None,
             grpc_code: None,
             grpc_details: None,
+            operation_id: None,
         }
     }
 
@@ -303,6 +308,7 @@ impl SparkError {
             server_stacktrace: None,
             grpc_code: None,
             grpc_details: None,
+            operation_id: None,
         }
     }
 
@@ -367,6 +373,7 @@ impl SparkError {
                     server_stacktrace,
                     grpc_code: None,
                     grpc_details: None,
+                    operation_id: None,
                 },
                 None => Self::connect_msg(format!("[{}] {}", status.code(), 
message)),
             }
@@ -379,6 +386,12 @@ impl SparkError {
         err
     }
 
+    /// Attach the ExecutePlan operation id this error was raised for.
+    pub fn with_operation_id(mut self, operation_id: impl Into<String>) -> 
Self {
+        self.operation_id = Some(operation_id.into());
+        self
+    }
+
     /// Get the error condition (error class).
     ///
     /// Mirrors `PySparkException.getCondition()` from PySpark.
@@ -737,6 +750,16 @@ fn parse_error_conditions_json(
 mod tests {
     use super::*;
 
+    #[test]
+    fn test_with_operation_id() {
+        let err = SparkError::connect_msg("boom");
+        assert_eq!(err.operation_id, None);
+        let err = err.with_operation_id("op-123");
+        assert_eq!(err.operation_id.as_deref(), Some("op-123"));
+        let from_status = SparkError::from_grpc_status(Status::internal("x"));
+        assert_eq!(from_status.operation_id, None);
+    }
+
     #[test]
     fn test_render_simple_message() {
         let mut params = BTreeMap::new();
diff --git a/crates/spark-connect/src/dataframe.rs 
b/crates/spark-connect/src/dataframe.rs
index 8e0a8ca..e74798c 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -1910,6 +1910,9 @@ fn capture_execution(
     info: &mut ExecutionInfo,
     session: &SparkSession,
 ) {
+    if info.operation_id.is_none() && !resp.operation_id.is_empty() {
+        info.operation_id = Some(resp.operation_id.clone());
+    }
     if let Some(metrics) = resp.metrics.take() {
         info.metrics = Some(metrics);
     }
@@ -3296,6 +3299,21 @@ mod plan_construction_tests {
     use super::*;
     use crate::session::SparkSession;
 
+    #[test]
+    fn capture_execution_records_first_operation_id() {
+        let spark = session();
+        let mut info = ExecutionInfo::default();
+        let mut resp = proto::ExecutePlanResponse::default();
+        // An empty operation id (e.g. a synthetic response) is not recorded.
+        capture_execution(&mut resp, &mut info, &spark);
+        assert_eq!(info.operation_id, None);
+        resp.operation_id = "op-1".to_string();
+        capture_execution(&mut resp, &mut info, &spark);
+        resp.operation_id = "op-2".to_string();
+        capture_execution(&mut resp, &mut info, &spark);
+        assert_eq!(info.operation_id.as_deref(), Some("op-1"));
+    }
+
     fn session() -> SparkSession {
         SparkSession::builder()
             .remote("sc://localhost:15002")
diff --git a/crates/spark-connect/src/pipelines.rs 
b/crates/spark-connect/src/pipelines.rs
index 6f5696f..7c4d492 100644
--- a/crates/spark-connect/src/pipelines.rs
+++ b/crates/spark-connect/src/pipelines.rs
@@ -180,13 +180,14 @@ pub fn define_auto_cdc_flow(
     flow_name: &str,
     target_dataset_name: &str,
     details: define_flow::AutoCdcFlowDetails,
+    sql_conf: HashMap<String, String>,
     source_code_location: Option<proto::SourceCodeLocation>,
 ) -> Result<()> {
     let inner = DefineFlow {
         dataflow_graph_id: Some(dataflow_graph_id.to_string()),
         flow_name: Some(flow_name.to_string()),
         target_dataset_name: Some(target_dataset_name.to_string()),
-        sql_conf: HashMap::new(),
+        sql_conf,
         client_id: None,
         source_code_location,
         once: None,
diff --git a/crates/spark-connect/src/profiler.rs 
b/crates/spark-connect/src/profiler.rs
index 579101a..8957e67 100644
--- a/crates/spark-connect/src/profiler.rs
+++ b/crates/spark-connect/src/profiler.rs
@@ -14,7 +14,7 @@ use spark_connect_core::error::Result;
 use spark_connect_proto as proto;
 
 /// Format a protobuf Literal value as a string.
-fn format_literal(literal: &proto::expression::Literal) -> String {
+pub fn format_literal(literal: &proto::expression::Literal) -> String {
     use proto::expression::literal::LiteralType;
 
     match &literal.literal_type {
diff --git a/crates/spark-connect/src/session.rs 
b/crates/spark-connect/src/session.rs
index 1ca69c6..7791e3c 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -31,6 +31,11 @@ pub struct ExecutionInfo {
     /// Observed metrics collected during execution (also where `observe(...)`
     /// and UDF-profiler results surface).
     pub observed_metrics: Vec<proto::execute_plan_response::ObservedMetrics>,
+    /// The ExecutePlan operation id: assigned by the client before the 
request is sent
+    /// (the request's id, or a new UUID) and echoed by the server in
+    /// `ExecutePlanResponse.operation_id`. Mirrors 
`ExecutionInfo.operation_id` (new in
+    /// pyspark 4.3.0).
+    pub operation_id: Option<String>,
 }
 
 /// A Spark Connect session for interacting with a remote Spark cluster.
@@ -543,10 +548,14 @@ impl SparkSession {
     }
 
     /// Stop this Spark session.
+    ///
+    /// Idempotent, mirroring pyspark: the session is marked stopped even if 
releasing it
+    /// on the server fails, and stopping an already stopped session is a 
no-op.
     pub fn stop(&self) -> Result<()> {
-        block_on(self.client.release_session())?;
-        self.stopped.store(true, Ordering::SeqCst);
-        Ok(())
+        if self.stopped.swap(true, Ordering::SeqCst) {
+            return Ok(());
+        }
+        block_on(self.client.release_session())
     }
 
     /// Whether `stop()` has been called on this session. Mirrors
@@ -713,6 +722,10 @@ fn datatype_to_arrow(dt: &DataType) -> 
Result<arrow::datatypes::DataType> {
         // TIMESTAMP (LTZ) and TIMESTAMP_NTZ are both micros; NTZ carries no 
zone.
         DataType::Timestamp => A::Timestamp(TimeUnit::Microsecond, 
Some("UTC".into())),
         DataType::TimestampNtz => A::Timestamp(TimeUnit::Microsecond, None),
+        DataType::TimestampLtzNanos { .. } => {
+            A::Timestamp(TimeUnit::Nanosecond, Some("UTC".into()))
+        }
+        DataType::TimestampNtzNanos { .. } => 
A::Timestamp(TimeUnit::Nanosecond, None),
         DataType::Time { .. } => A::Time64(TimeUnit::Microsecond),
         DataType::Decimal { precision, scale } => A::Decimal128(*precision as 
u8, *scale as i8),
         DataType::Array {
@@ -870,6 +883,27 @@ fn values_to_arrow(values: &[Value], dt: &DataType) -> 
Result<Arc<dyn arrow::arr
             };
             Ok(Arc::new(arr))
         }
+        // Row values are microseconds (collect truncates nanosecond columns 
to micros), so
+        // scale them up to the nanosecond unit `to_arrow_type` declares for 
these types.
+        DataType::TimestampNtzNanos { .. } | DataType::TimestampLtzNanos { .. 
} => {
+            let vals: Result<Vec<Option<i64>>> = values
+                .iter()
+                .map(|v| match v {
+                    Value::Null => Ok(None),
+                    Value::Timestamp(t) => 
t.checked_mul(1_000).map(Some).ok_or_else(|| {
+                        SparkError::connect_msg("Timestamp out of range for 
nanosecond precision")
+                    }),
+                    _ => Err(SparkError::connect_msg("Type mismatch in row 
data")),
+                })
+                .collect();
+            let arr = TimestampNanosecondArray::from(vals?);
+            let arr = if matches!(dt, DataType::TimestampNtzNanos { .. }) {
+                arr
+            } else {
+                arr.with_timezone("UTC")
+            };
+            Ok(Arc::new(arr))
+        }
         DataType::Decimal { precision, scale } => {
             let col_scale = *scale;
             let vals: Result<Vec<Option<i128>>> = values
@@ -1237,6 +1271,22 @@ mod tests {
             .expect("failed to build session")
     }
 
+    #[test]
+    fn stop_is_idempotent_and_marks_stopped() {
+        // Port 1: nothing listens, so the ReleaseSession RPC fails fast. The 
session is
+        // still marked stopped (pyspark's stop() is best-effort) and a second 
stop() is a
+        // no-op that does not retry the release.
+        let spark = SparkSession::builder()
+            .remote("sc://localhost:1")
+            .get_or_create()
+            .expect("failed to build session");
+        let other = spark.clone();
+        let _ = spark.stop();
+        assert!(spark.is_stopped());
+        assert!(other.is_stopped(), "stopped flag is shared across clones");
+        assert!(spark.stop().is_ok());
+    }
+
     #[test]
     fn session_is_not_stopped_initially() {
         let spark = session();
@@ -1411,6 +1461,85 @@ mod tests {
         );
     }
 
+    #[test]
+    fn input_ipc_scales_nanos_timestamps_and_keeps_zone() {
+        use arrow::array::{ListArray, TimestampNanosecondArray};
+        use arrow::datatypes::{DataType as A, TimeUnit};
+        let ntz = DataType::TimestampNtzNanos { precision: 9 };
+        let ltz = DataType::TimestampLtzNanos { precision: 7 };
+        let array = DataType::Array {
+            element_type: Box::new(ltz.clone()),
+            contains_null: true,
+        };
+        let schema = schema(vec![ntz, ltz, array]);
+        let rows = vec![
+            row(vec![
+                Value::Timestamp(1_700_000_000_123_456),
+                Value::Timestamp(-1),
+                Value::List(vec![Value::Timestamp(2), Value::Null]),
+            ]),
+            row(vec![Value::Null, Value::Null, Value::List(vec![])]),
+        ];
+        let batch = read_batch(&rows, &schema);
+        let column = |i: usize| {
+            batch
+                .column(i)
+                .as_any()
+                .downcast_ref::<TimestampNanosecondArray>()
+                .unwrap()
+                .clone()
+        };
+        assert_eq!(
+            batch.schema().field(0).data_type(),
+            &A::Timestamp(TimeUnit::Nanosecond, None)
+        );
+        assert_eq!(
+            batch.schema().field(1).data_type(),
+            &A::Timestamp(TimeUnit::Nanosecond, Some("UTC".into()))
+        );
+        assert_eq!(
+            column(0),
+            
TimestampNanosecondArray::from(vec![Some(1_700_000_000_123_456_000), None])
+        );
+        assert_eq!(
+            column(1),
+            TimestampNanosecondArray::from(vec![Some(-1_000), 
None]).with_timezone("UTC")
+        );
+        let list = batch
+            .column(2)
+            .as_any()
+            .downcast_ref::<ListArray>()
+            .unwrap();
+        let first = list.value(0);
+        assert_eq!(
+            first
+                .as_any()
+                .downcast_ref::<TimestampNanosecondArray>()
+                .unwrap(),
+            &TimestampNanosecondArray::from(vec![Some(2_000), 
None]).with_timezone("UTC")
+        );
+        assert_eq!(list.value_length(1), 0);
+
+        // An empty input with a nanos schema builds too, as for the micros 
types.
+        let empty = rows_to_arrow_ipc(
+            &[],
+            &schema_one(DataType::TimestampNtzNanos { precision: 9 }),
+        );
+        assert!(empty.is_ok(), "{empty:?}");
+
+        let overflow = rows_to_arrow_ipc(
+            &[row(vec![Value::Timestamp(i64::MAX)])],
+            &schema_one(DataType::TimestampNtzNanos { precision: 9 }),
+        )
+        .unwrap_err()
+        .to_string();
+        assert!(overflow.contains("out of range"), "{overflow}");
+    }
+
+    fn schema_one(dt: DataType) -> DataType {
+        schema(vec![dt])
+    }
+
     #[test]
     fn input_ipc_resolves_duplicate_names_before_numeric_conversion() {
         let nested_type = DataType::Struct {
diff --git a/crates/spark-connect/src/types.rs 
b/crates/spark-connect/src/types.rs
index fbc55a5..976520d 100644
--- a/crates/spark-connect/src/types.rs
+++ b/crates/spark-connect/src/types.rs
@@ -50,6 +50,13 @@ pub enum DataType {
     TimestampNtz,
     /// `pyspark.sql.types.TimeType`
     Time { precision: i32 },
+    /// Spark's `TimestampNTZNanosType(precision)` (precision 7..=9): a 
timestamp
+    /// without time zone with nanosecond-capable fractional seconds. PySpark 
4.3.0
+    /// has no Python class for it; this mirrors the server/proto type.
+    TimestampNtzNanos { precision: i32 },
+    /// Spark's `TimestampLTZNanosType(precision)` (precision 7..=9): a 
timestamp
+    /// with local time zone with nanosecond-capable fractional seconds.
+    TimestampLtzNanos { precision: i32 },
     /// `pyspark.sql.types.CalendarIntervalType`
     CalendarInterval,
     /// `pyspark.sql.types.YearMonthIntervalType`
@@ -191,6 +198,8 @@ impl DataType {
             DataType::Date
             | DataType::Timestamp
             | DataType::TimestampNtz
+            | DataType::TimestampNtzNanos { .. }
+            | DataType::TimestampLtzNanos { .. }
             | DataType::Time { .. }
             | DataType::DayTimeInterval { .. }
             | DataType::YearMonthInterval { .. }
@@ -229,6 +238,8 @@ impl DataType {
             DataType::Timestamp => "timestamp".to_string(),
             DataType::TimestampNtz => "timestamp_ntz".to_string(),
             DataType::Time { .. } => "time".to_string(),
+            DataType::TimestampNtzNanos { precision } => 
format!("timestamp_ntz({})", precision),
+            DataType::TimestampLtzNanos { precision } => 
format!("timestamp_ltz({})", precision),
             DataType::CalendarInterval => "interval".to_string(),
             DataType::YearMonthInterval { .. } => "interval".to_string(),
             DataType::DayTimeInterval { .. } => "interval".to_string(),
@@ -280,6 +291,8 @@ impl DataType {
             DataType::Timestamp => "timestamp".to_string(),
             DataType::TimestampNtz => "timestamp_ntz".to_string(),
             DataType::Time { precision } => format!("time({})", precision),
+            DataType::TimestampNtzNanos { precision } => 
format!("timestamp_ntz({})", precision),
+            DataType::TimestampLtzNanos { precision } => 
format!("timestamp_ltz({})", precision),
             DataType::CalendarInterval => "interval".to_string(),
             DataType::YearMonthInterval {
                 start_field,
@@ -370,6 +383,12 @@ impl DataType {
             DataType::Time { precision } => {
                 serde_json::json!(format!("time({})", precision))
             }
+            DataType::TimestampNtzNanos { precision } => {
+                serde_json::json!(format!("timestamp_ntz({})", precision))
+            }
+            DataType::TimestampLtzNanos { precision } => {
+                serde_json::json!(format!("timestamp_ltz({})", precision))
+            }
             DataType::CalendarInterval => serde_json::json!("interval"),
             DataType::YearMonthInterval {
                 start_field,
@@ -566,6 +585,22 @@ impl DataType {
                 time_type.precision = Some(*precision);
                 proto.kind = 
Some(spark_connect_proto::data_type::Kind::Time(time_type));
             }
+            DataType::TimestampNtzNanos { precision } => {
+                proto.kind = 
Some(spark_connect_proto::data_type::Kind::TimestampNtzNanos(
+                    spark_connect_proto::data_type::TimestampNtzNanos {
+                        precision: Some(*precision),
+                        ..Default::default()
+                    },
+                ));
+            }
+            DataType::TimestampLtzNanos { precision } => {
+                proto.kind = 
Some(spark_connect_proto::data_type::Kind::TimestampLtzNanos(
+                    spark_connect_proto::data_type::TimestampLtzNanos {
+                        precision: Some(*precision),
+                        ..Default::default()
+                    },
+                ));
+            }
             DataType::CalendarInterval => {
                 proto.kind = 
Some(spark_connect_proto::data_type::Kind::CalendarInterval(
                     
spark_connect_proto::data_type::CalendarInterval::default(),
@@ -802,12 +837,16 @@ impl DataType {
             Some(spark_connect_proto::data_type::Kind::Unparsed(u)) => 
Ok(DataType::Unparsed {
                 data_type_string: u.data_type_string.clone(),
             }),
+            // Precision defaults to 9 when omitted (types.proto).
             Some(spark_connect_proto::data_type::Kind::TimestampNtzNanos(tn)) 
=> {
-                let precision = tn.precision.unwrap_or(9);
-                Ok(DataType::Time { precision })
+                Ok(DataType::TimestampNtzNanos {
+                    precision: tn.precision.unwrap_or(9),
+                })
             }
-            Some(spark_connect_proto::data_type::Kind::TimestampLtzNanos(_)) 
=> {
-                Ok(DataType::Timestamp)
+            Some(spark_connect_proto::data_type::Kind::TimestampLtzNanos(tl)) 
=> {
+                Ok(DataType::TimestampLtzNanos {
+                    precision: tl.precision.unwrap_or(9),
+                })
             }
             None => Err(SparkError::connect_msg("DataType kind not set")),
         }
@@ -1097,6 +1136,9 @@ fn parse_json_type_string(s: &str) -> Result<DataType> {
         s if s.starts_with("char(") => parse_char(s),
         s if s.starts_with("varchar(") => parse_varchar(s),
         s if s.starts_with("time(") => parse_time(s),
+        s if s.starts_with("timestamp_ntz(") || 
s.starts_with("timestamp_ltz(") => {
+            parse_timestamp_nanos(s)
+        }
         s if s.starts_with("interval") => parse_interval_string(s),
         s if s.starts_with("string collate") => {
             let collation = s.trim_start_matches("string 
collate").trim().to_string();
@@ -1313,6 +1355,36 @@ fn parse_time(s: &str) -> Result<DataType> {
     Ok(DataType::Time { precision })
 }
 
+/// Parses `timestamp_ntz(p)` / `timestamp_ltz(p)` (p in 7..=9), mirroring the
+/// `TIMESTAMP_{NTZ,LTZ}_NANOS_TYPE` patterns in Scala `DataType.nameToType`.
+fn parse_timestamp_nanos(s: &str) -> Result<DataType> {
+    let (ntz, rest) = if let Some(r) = s.strip_prefix("timestamp_ntz(") {
+        (true, r)
+    } else if let Some(r) = s.strip_prefix("timestamp_ltz(") {
+        (false, r)
+    } else {
+        return Err(SparkError::value(
+            "INVALID_DATATYPE_FORMAT",
+            &[("detail", "Invalid timestamp format")],
+        ));
+    };
+    let precision = rest
+        .strip_suffix(")")
+        .and_then(|p| p.trim().parse::<i32>().ok())
+        .filter(|p| (7..=9).contains(p))
+        .ok_or_else(|| {
+            SparkError::value(
+                "INVALID_DATATYPE_FORMAT",
+                &[("detail", "Invalid timestamp precision (expected 7..9)")],
+            )
+        })?;
+    Ok(if ntz {
+        DataType::TimestampNtzNanos { precision }
+    } else {
+        DataType::TimestampLtzNanos { precision }
+    })
+}
+
 fn parse_interval_string(s: &str) -> Result<DataType> {
     if s == "interval" {
         return Ok(DataType::CalendarInterval);
@@ -1585,6 +1657,9 @@ fn parse_single_type(input: &str) -> Result<DataType> {
                 parse_varchar(&trimmed)
             } else if trimmed.starts_with("time(") {
                 parse_time(&trimmed)
+            } else if trimmed.starts_with("timestamp_ntz(") || 
trimmed.starts_with("timestamp_ltz(")
+            {
+                parse_timestamp_nanos(&trimmed)
             } else if trimmed.starts_with("interval") {
                 parse_interval_string(&trimmed)
             } else if trimmed.starts_with("string collate") {
diff --git a/crates/spark-connect/tests/types_roundtrip.rs 
b/crates/spark-connect/tests/types_roundtrip.rs
index 8b2381a..a0516fc 100644
--- a/crates/spark-connect/tests/types_roundtrip.rs
+++ b/crates/spark-connect/tests/types_roundtrip.rs
@@ -31,6 +31,8 @@ fn all_scalar_variants() -> Vec<DataType> {
         DataType::Timestamp,
         DataType::TimestampNtz,
         DataType::Time { precision: 6 },
+        DataType::TimestampNtzNanos { precision: 9 },
+        DataType::TimestampLtzNanos { precision: 7 },
         DataType::CalendarInterval,
         DataType::YearMonthInterval {
             start_field: 0,
@@ -308,3 +310,40 @@ fn from_ddl_full_grammar() {
     assert!(DataType::from_ddl("array<").is_err());
     assert!(DataType::from_ddl("struct<a:>").is_err());
 }
+
+#[test]
+fn timestamp_nanos_types() {
+    use spark_connect_proto::data_type::{Kind, TimestampLtzNanos, 
TimestampNtzNanos};
+
+    let ntz = DataType::TimestampNtzNanos { precision: 8 };
+    let ltz = DataType::TimestampLtzNanos { precision: 9 };
+    // Names mirror Scala TimestampNTZNanosType / 
TimestampLTZNanosType.typeName.
+    assert_eq!(ntz.simple_string(), "timestamp_ntz(8)");
+    assert_eq!(ntz.type_name(), "timestamp_ntz(8)");
+    assert_eq!(ltz.json(), "\"timestamp_ltz(9)\"");
+    assert!(ntz.need_conversion());
+    assert_eq!(
+        DataType::from_json_str("\"timestamp_ntz(8)\"").unwrap(),
+        ntz
+    );
+    assert_eq!(DataType::from_ddl("timestamp_ltz(9)").unwrap(), ltz);
+    // Precision outside 7..=9 is rejected.
+    assert!(DataType::from_json_str("\"timestamp_ntz(6)\"").is_err());
+
+    // Proto kinds map to the nanos variants (not TIME / micros TIMESTAMP), 
and an
+    // omitted precision defaults to 9 per types.proto.
+    let mut p = spark_connect_proto::DataType::default();
+    p.kind = Some(Kind::TimestampNtzNanos(TimestampNtzNanos::default()));
+    assert_eq!(
+        DataType::from_proto(&p).unwrap(),
+        DataType::TimestampNtzNanos { precision: 9 }
+    );
+    p.kind = Some(Kind::TimestampLtzNanos(TimestampLtzNanos {
+        precision: Some(7),
+        ..Default::default()
+    }));
+    assert_eq!(
+        DataType::from_proto(&p).unwrap(),
+        DataType::TimestampLtzNanos { precision: 7 }
+    );
+}
diff --git a/python/.coveragerc b/python/.coveragerc
index 4fe8f83..61a8ea0 100644
--- a/python/.coveragerc
+++ b/python/.coveragerc
@@ -40,6 +40,7 @@ omit =
     */pyspark/sql/variant_utils.py
     */pyspark/sql/datasource_internal.py
     */pyspark/sql/profiler.py
+    */pyspark/sql/metrics.py
     */pyspark/serializers.py
     */pyspark/loose_version.py
     */pyspark/_globals.py
diff --git a/python/pyspark/errors/exceptions/__init__.py 
b/python/pyspark/errors/exceptions/__init__.py
index 6cebf44..85add6c 100644
--- a/python/pyspark/errors/exceptions/__init__.py
+++ b/python/pyspark/errors/exceptions/__init__.py
@@ -30,6 +30,22 @@ except ImportError:
         QueryContext,
     )
 
+
+def _operation_id(self):  # type: ignore[no-untyped-def]
+    """The Spark Connect ExecutePlan operation ID, when available.
+
+    .. versionadded:: 4.3.0
+    """
+    return getattr(self, "_operation_id", None)
+
+
+# pyspark 4.3.0 adds ``SparkConnectException.operation_id``. In this drop-in 
the Connect
+# exception classes ARE the canonical base classes 
(``connect.SparkConnectException`` aliases
+# ``PySparkException``), so the property lives on that shared base; the Rust 
transport sets
+# ``_operation_id`` on errors raised for an ExecutePlan. ``base.py`` is 
vendored, hence here.
+if not hasattr(PySparkException, "operation_id"):
+    PySparkException.operation_id = property(_operation_id)  # type: 
ignore[attr-defined]
+
 __all__ = [
     "PySparkException",
     "PySparkAssertionError",
diff --git a/python/pyspark/pipelines/api.py b/python/pyspark/pipelines/api.py
index 4f734aa..33b7f0c 100644
--- a/python/pyspark/pipelines/api.py
+++ b/python/pyspark/pipelines/api.py
@@ -16,7 +16,7 @@
 #
 from typing import Callable, Dict, List, Literal, Optional, Union, overload
 
-from pyspark.errors import PySparkTypeError
+from pyspark.errors import PySparkTypeError, PySparkValueError
 from pyspark.pipelines.graph_element_registry import 
get_active_graph_element_registry
 from pyspark.pipelines.type_error_utils import 
validate_optional_list_of_str_arg
 from pyspark.pipelines.flow import AutoCdcFlow, Flow, QueryFunction
@@ -536,8 +536,12 @@ def create_auto_cdc_flow(
     apply_as_deletes: Optional[Union[str, Column]] = None,
     column_list: Optional[Union[List[str], List[Column]]] = None,
     except_column_list: Optional[Union[List[str], List[Column]]] = None,
-    stored_as_scd_type: Optional[Literal[1, "1"]] = None,
+    stored_as_scd_type: Optional[Literal[1, 2, "1", "2"]] = None,
     name: Optional[str] = None,
+    *,
+    track_history_column_list: Optional[Union[List[str], List[Column]]] = None,
+    track_history_except_column_list: Optional[Union[List[str], List[Column]]] 
= None,
+    spark_conf: Optional[Dict[str, str]] = None,
 ) -> None:
     """
     Create an Auto CDC flow into the target table from the Change Data Capture 
(CDC) source.
@@ -581,10 +585,25 @@ def create_auto_cdc_flow(
         PySpark Columns. Only one of column_list and except_column_list can be 
specified. When \
         this is specified, all columns in the `DataFrame` of the target table 
except those in \
         this list will be in the output table.
-    :param stored_as_scd_type: The SCD type for the target table. Only 1 (or 
"1") is supported. \
-        When not specified, the server default applies.
+    :param stored_as_scd_type: The SCD type for the target table. 1 (or "1") 
and 2 (or "2") are \
+        supported. When not specified, the server default applies.
+    :param track_history_column_list: SCD2-only. Columns whose value change 
opens a new history \
+        record; two consecutive upsert events for the same key are coalesced 
into the same \
+        history record when they agree on every tracked column. When not 
specified, every \
+        eligible selected user column is tracked. This should be a list of 
column identifiers \
+        without qualifiers, expressed as either Python strings or PySpark 
Columns. Only one of \
+        track_history_column_list and track_history_except_column_list can be 
specified, and \
+        both require stored_as_scd_type to be 2.
+    :param track_history_except_column_list: SCD2-only. Columns excluded from 
history tracking. \
+        This should be a list of column identifiers without qualifiers, 
expressed as either \
+        Python strings or PySpark Columns. Only one of 
track_history_column_list and \
+        track_history_except_column_list can be specified, and both require 
stored_as_scd_type \
+        to be 2.
     :param name: The name of the flow for this create_auto_cdc_flow command. 
When unspecified, \
         this will build a "default flow" with name equal to the target name.
+    :param spark_conf: A dict whose keys are the conf names and values are the 
conf values. \
+        These confs will be set when the flow is executed; they can override 
confs set for the \
+        destination, for the pipeline, or on the cluster.
     """
     # Lazy import kept to match upstream layering; in this Rust-backed drop-in 
the Connect
     # functions module is our `pyspark.sql.functions` (there is no separate 
connect variant).
@@ -626,6 +645,27 @@ def create_auto_cdc_flow(
     except_column_list = _normalize_optional_column_list(
         arg_name="except_column_list", column_list=except_column_list
     )
+    track_history_column_list = _normalize_optional_column_list(
+        arg_name="track_history_column_list", 
column_list=track_history_column_list
+    )
+    track_history_except_column_list = _normalize_optional_column_list(
+        arg_name="track_history_except_column_list",
+        column_list=track_history_except_column_list,
+    )
+
+    # An include/except pair is mutually exclusive. The server enforces this 
too, but failing
+    # here avoids a round-trip. An empty list serializes identically to an 
omitted one (an unset
+    # repeated field on the wire), so it is a no-op; gate on non-empty lists 
rather than
+    # `is not None`.
+    _reject_include_and_except_together(
+        "column_list", column_list, "except_column_list", except_column_list
+    )
+    _reject_include_and_except_together(
+        "track_history_column_list",
+        track_history_column_list,
+        "track_history_except_column_list",
+        track_history_except_column_list,
+    )
 
     if isinstance(sequence_by, str):
         sequence_by = _connect_expr(sequence_by)
@@ -651,16 +691,40 @@ def create_auto_cdc_flow(
             },
         )
 
-    if stored_as_scd_type is not None and str(stored_as_scd_type) != "1":
+    if stored_as_scd_type is not None and str(stored_as_scd_type) not in ("1", 
"2"):
         raise PySparkTypeError(
             errorClass="NOT_EXPECTED_TYPE",
             messageParameters={
                 "arg_name": "stored_as_scd_type",
-                "expected_type": "Literal[1, '1']",
+                "expected_type": "Literal[1, 2, '1', '2']",
                 "arg_type": type(stored_as_scd_type).__name__,
             },
         )
 
+    # Track-history columns describe how consecutive upserts are coalesced 
into history records,
+    # a notion that only exists under SCD2. Reject them for any other SCD 
type. The engine
+    # enforces this too, but failing here gives a clearer, client-side error.
+    #
+    # An empty list serializes identically to an omitted one (an unset 
repeated field on the
+    # wire) and is therefore a no-op, so gate on non-empty lists rather than 
`is not None`.
+    is_scd2 = stored_as_scd_type is not None and str(stored_as_scd_type) == "2"
+    supplied_track_history_args = [
+        arg_name
+        for arg_name, cols in (
+            ("track_history_column_list", track_history_column_list),
+            ("track_history_except_column_list", 
track_history_except_column_list),
+        )
+        if cols
+    ]
+    if not is_scd2 and supplied_track_history_args:
+        raise PySparkValueError(
+            errorClass="INVALID_MULTIPLE_ARGUMENT_CONDITIONS",
+            messageParameters={
+                "arg_names": ", ".join(supplied_track_history_args),
+                "condition": "specified unless stored_as_scd_type is 2 (or 
'2')",
+            },
+        )
+
     source_code_location = get_caller_source_code_location(stacklevel=1)
 
     flow = AutoCdcFlow(
@@ -673,12 +737,33 @@ def create_auto_cdc_flow(
         column_list=column_list,
         except_column_list=except_column_list,
         stored_as_scd_type=stored_as_scd_type,
+        track_history_column_list=track_history_column_list,
+        track_history_except_column_list=track_history_except_column_list,
+        spark_conf=spark_conf or {},
         source_code_location=source_code_location,
     )
 
     get_active_graph_element_registry().register_auto_cdc_flow(flow)
 
 
+def _reject_include_and_except_together(
+    include_arg_name: str,
+    include_columns: Optional[List[Column]],
+    except_arg_name: str,
+    except_columns: Optional[List[Column]],
+) -> None:
+    """Reject specifying a non-empty include list and a non-empty except list 
together.
+
+    An empty list is a no-op (it serializes to an unset repeated field, 
indistinguishable from an
+    omitted argument), so only non-empty lists count as "specified".
+    """
+    if include_columns and except_columns:
+        raise PySparkValueError(
+            errorClass="CANNOT_SET_TOGETHER",
+            messageParameters={"arg_list": f"{include_arg_name} and 
{except_arg_name}"},
+        )
+
+
 def _normalize_optional_column_list(
     arg_name: str,
     column_list: Optional[Union[List[str], List[Column]]],
diff --git a/python/pyspark/pipelines/spark_connect_graph_element_registry.py 
b/python/pyspark/pipelines/spark_connect_graph_element_registry.py
index c84e112..25c2315 100644
--- a/python/pyspark/pipelines/spark_connect_graph_element_registry.py
+++ b/python/pyspark/pipelines/spark_connect_graph_element_registry.py
@@ -53,6 +53,8 @@ _OUTPUT_TYPE_SINK = 4
 
 # spark.connect.PipelineCommand.DefineFlow.SCDType enum values.
 _SCD_TYPE_1 = 1
+_SCD_TYPE_2 = 2
+_SCD_TYPE_BY_VALUE = {"1": _SCD_TYPE_1, "2": _SCD_TYPE_2}
 
 
 def _scl(loc: SourceCodeLocation):
@@ -165,7 +167,12 @@ class 
SparkConnectGraphElementRegistry(GraphElementRegistry):
 
     def register_auto_cdc_flow(self, flow: AutoCdcFlow) -> None:
         scl_file, scl_line = _scl(flow.source_code_location)
-        stored_as_scd_type = _SCD_TYPE_1 if flow.stored_as_scd_type is not 
None else 0
+        # SCDType enum value: 0 unspecified, else SCD_TYPE_1 / SCD_TYPE_2.
+        stored_as_scd_type = (
+            _SCD_TYPE_BY_VALUE[str(flow.stored_as_scd_type)]
+            if flow.stored_as_scd_type is not None
+            else 0
+        )
         _define_auto_cdc_flow(
             self._spark,
             self._dataflow_graph_id,
@@ -180,6 +187,13 @@ class 
SparkConnectGraphElementRegistry(GraphElementRegistry):
             stored_as_scd_type,
             scl_file,
             scl_line,
+            list(flow.track_history_column_list)
+            if flow.track_history_column_list is not None
+            else None,
+            list(flow.track_history_except_column_list)
+            if flow.track_history_except_column_list is not None
+            else None,
+            dict(flow.spark_conf) if flow.spark_conf else None,
         )
 
     def register_sql(self, sql_text: str, file_path: Path) -> None:
diff --git a/python/pyspark/sql/connect/client/__init__.py 
b/python/pyspark/sql/connect/client/__init__.py
index 9cd6e43..f8b50e2 100644
--- a/python/pyspark/sql/connect/client/__init__.py
+++ b/python/pyspark/sql/connect/client/__init__.py
@@ -23,3 +23,61 @@ try:
 except ImportError:
     # If Rust backend not available, guard the export
     __all__ = []
+
+
+from typing import Any, Dict, List
+
+from pyspark.sql.metrics import ObservedMetrics
+
+
+class _UnsupportedLiteral:
+    """An observed-metric literal of a kind ``LiteralExpression._to_value`` 
rejects. The
+    Rust transport decodes eagerly, so it leaves this marker and ``pairs`` 
raises the
+    reference's ``UNSUPPORTED_LITERAL`` when read, as the reference does."""
+
+    def __init__(self, literal: str):
+        self.literal = literal
+
+    def __repr__(self) -> str:
+        return f"_UnsupportedLiteral({self.literal})"
+
+
+class PlanObservedMetrics(ObservedMetrics):
+    """Observed metrics of one ``observe(...)`` node, as carried by
+    ``DataFrame.executionInfo.flows``. Mirrors 
``pyspark.sql.connect.client.core
+    .PlanObservedMetrics``; the Rust transport has already decoded the literal 
values."""
+
+    def __init__(self, name: str, metrics: List[Any], keys: List[str]):
+        self._name = name
+        self._metrics = metrics
+        self._keys = keys if keys else [f"observed_metric_{i}" for i in 
range(len(metrics))]
+
+    def __repr__(self) -> str:
+        return f"Plan observed({self._name}={self._metrics})"
+
+    @property
+    def name(self) -> str:
+        return self._name
+
+    @property
+    def metrics(self) -> List[Any]:
+        return self._metrics
+
+    @property
+    def pairs(self) -> Dict[str, Any]:
+        for metric in self._metrics:
+            if isinstance(metric, _UnsupportedLiteral):
+                from pyspark.errors import PySparkTypeError
+
+                raise PySparkTypeError(
+                    errorClass="UNSUPPORTED_LITERAL",
+                    messageParameters={"literal": metric.literal},
+                )
+        return {self._keys[i]: self._metrics[i] for i in 
range(len(self._metrics))}
+
+    @property
+    def keys(self) -> List[str]:
+        return self._keys
+
+    def to_dict(self) -> Dict[str, Any]:
+        return {"name": self._name, "keys": self._keys, "pairs": self.pairs}
diff --git a/python/pyspark/sql/connect/context.py 
b/python/pyspark/sql/connect/context.py
new file mode 100644
index 0000000..2015eae
--- /dev/null
+++ b/python/pyspark/sql/connect/context.py
@@ -0,0 +1,255 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Spark Connect SQLContext/HiveContext, ported from the v4.3.0
+``pyspark/sql/connect/context.py``. Fork-adapted (not vendored): it wraps the 
Rust-backed
+:class:`SparkSession` and imports from the drop-in's own modules instead of the
+gRPC-client internals."""
+import warnings
+from typing import Any, Callable, ClassVar, List, Optional, Union
+
+from pyspark._globals import _NoValue, _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.types import BooleanType, DataType, StringType, StructField, 
StructType
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
+    """The entry point for working with structured data (rows and columns) in 
Spark, in Spark 1.x.
+
+    As of Spark 2.0, this is replaced by :class:`SparkSession`. This Spark 
Connect
+    implementation wraps a :class:`SparkSession` directly and needs no 
``SparkContext``.
+
+    .. deprecated:: 4.3.0
+        Use :func:`SparkSession.builder.getOrCreate()` instead.
+    """
+
+    _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+    def __init__(self, sparkSession: Any) -> None:
+        warnings.warn(
+            "Deprecated in 4.3.0. Use SparkSession.builder.getOrCreate() 
instead.",
+            FutureWarning,
+            stacklevel=2,
+        )
+        self.sparkSession = sparkSession
+        if type(self)._instantiatedContext is None:
+            type(self)._instantiatedContext = self
+
+    @classmethod
+    def _from_session(cls, sparkSession: Any) -> "SQLContext":
+        """Create a new instance without emitting a deprecation warning."""
+        ctx = object.__new__(cls)
+        ctx.sparkSession = sparkSession
+        return ctx
+
+    @classmethod
+    def _get_or_create_from_session(cls, sparkSession: Any) -> "SQLContext":
+        """Return the cached instance, or create one when the cached context 
wraps a
+        different session (e.g. the previous one was stopped and a new one 
started)."""
+        if cls._instantiatedContext is None or not _same_session(
+            cls._instantiatedContext.sparkSession, sparkSession
+        ):
+            cls._instantiatedContext = cls._from_session(sparkSession)
+        return cls._instantiatedContext
+
+    def newSession(self) -> "SQLContext":
+        """Returns a new SQLContext as a new session with empty state."""
+        return self._from_session(self.sparkSession.newSession())
+
+    def setConf(self, key: str, value: Union[bool, int, str]) -> None:
+        """Sets the given Spark SQL configuration property."""
+        self.sparkSession.conf.set(key, value)
+
+    def getConf(
+        self, key: str, defaultValue: Union[Optional[str], _NoValueType] = 
_NoValue
+    ) -> Optional[str]:
+        """Returns the value of Spark SQL configuration property for the given 
key."""
+        if defaultValue is _NoValue:
+            return self.sparkSession.conf.get(key)
+        return self.sparkSession.conf.get(key, defaultValue)
+
+    @property
+    def udf(self) -> Any:
+        """Returns a :class:`UDFRegistration` for UDF registration."""
+        return self.sparkSession.udf
+
+    @property
+    def udtf(self) -> Any:
+        """Returns a :class:`UDTFRegistration` for UDTF registration."""
+        return self.sparkSession.udtf
+
+    def range(
+        self,
+        start: int,
+        end: Optional[int] = None,
+        step: int = 1,
+        numPartitions: Optional[int] = None,
+    ) -> Any:
+        """Create a :class:`DataFrame` with a single LongType column named 
``id``."""
+        return self.sparkSession.range(start, end, step, numPartitions)
+
+    def registerFunction(
+        self, name: str, f: Callable[..., Any], returnType: Optional[DataType] 
= None
+    ) -> Any:
+        """An alias for :func:`spark.udf.register`.
+
+        .. deprecated:: 4.3.0
+            Use :func:`spark.udf.register` instead.
+        """
+        warnings.warn("Deprecated in 4.3.0. Use spark.udf.register instead.", 
FutureWarning)
+        return self.sparkSession.udf.register(name, f, returnType)
+
+    def registerJavaFunction(
+        self, name: str, javaClassName: str, returnType: Optional[DataType] = 
None
+    ) -> None:
+        """Not supported in Spark Connect."""
+        raise PySparkNotImplementedError(
+            errorClass="NOT_IMPLEMENTED",
+            messageParameters={"feature": "registerJavaFunction"},
+        )
+
+    def createDataFrame(
+        self,
+        data: Any,
+        schema: Any = None,
+        samplingRatio: Optional[float] = None,
+        verifySchema: Optional[bool] = None,
+    ) -> Any:
+        """Creates a :class:`DataFrame` from local data."""
+        return self.sparkSession.createDataFrame(data, schema, samplingRatio, 
verifySchema)
+
+    def registerDataFrameAsTable(self, df: Any, tableName: str) -> None:
+        """Registers the given :class:`DataFrame` as a temporary table in the 
catalog."""
+        df.createOrReplaceTempView(tableName)
+
+    def dropTempTable(self, tableName: str) -> None:
+        """Remove the temporary table from catalog."""
+        self.sparkSession.catalog.dropTempView(tableName)
+
+    def createExternalTable(
+        self,
+        tableName: str,
+        path: Optional[str] = None,
+        source: Optional[str] = None,
+        schema: Optional[StructType] = None,
+        **options: str,
+    ) -> Any:
+        """Creates an external table based on the dataset in a data source."""
+        return self.sparkSession.catalog.createExternalTable(
+            tableName, path, source, schema, **options
+        )
+
+    def sql(self, sqlQuery: str) -> Any:
+        """Returns a :class:`DataFrame` representing the result of the given 
query."""
+        return self.sparkSession.sql(sqlQuery)
+
+    def table(self, tableName: str) -> Any:
+        """Returns the specified table or view as a :class:`DataFrame`."""
+        return self.sparkSession.table(tableName)
+
+    def tables(self, dbName: Optional[str] = None) -> Any:
+        """Returns a :class:`DataFrame` of (namespace, tableName, isTemporary) 
for the tables
+        in the given (or current) database."""
+        schema = StructType(
+            [
+                StructField("namespace", StringType(), nullable=True),
+                StructField("tableName", StringType(), nullable=True),
+                StructField("isTemporary", BooleanType(), nullable=False),
+            ]
+        )
+        # catalog.listTables() rather than SHOW TABLES keeps the column names 
fixed, and the
+        # full namespace is joined ("a.b") to match classic SHOW TABLES.
+        rows = [
+            (".".join(t.namespace) if t.namespace else "", t.name, 
t.isTemporary)
+            for t in self.sparkSession.catalog.listTables(dbName)
+        ]
+        return self.sparkSession.createDataFrame(rows, schema)
+
+    def tableNames(self, dbName: Optional[str] = None) -> List[str]:
+        """Returns a list of names of tables in the database ``dbName``."""
+        return [t.name for t in self.sparkSession.catalog.listTables(dbName)]
+
+    def cacheTable(self, tableName: str) -> None:
+        """Caches the specified table in-memory."""
+        self.sparkSession.catalog.cacheTable(tableName)
+
+    def uncacheTable(self, tableName: str) -> None:
+        """Removes the specified table from the in-memory cache."""
+        self.sparkSession.catalog.uncacheTable(tableName)
+
+    def clearCache(self) -> None:
+        """Removes all cached tables from the in-memory cache."""
+        self.sparkSession.catalog.clearCache()
+
+    @property
+    def read(self) -> Any:
+        """Returns a :class:`DataFrameReader`."""
+        return self.sparkSession.read
+
+    @property
+    def readStream(self) -> Any:
+        """Returns a :class:`DataStreamReader`."""
+        return self.sparkSession.readStream
+
+    @property
+    def streams(self) -> Any:
+        """Returns a :class:`StreamingQueryManager`."""
+        return self.sparkSession.streams
+
+
+class HiveContext(SQLContext):
+    """Not supported in Spark Connect.
+
+    .. deprecated:: 4.3.0
+        Use SparkSession.builder.enableHiveSupport().getOrCreate().
+    """
+
+    # Override to prevent inheriting SQLContext's cached instance, which would 
cause
+    # _get_or_create_from_session to skip _from_session and return the wrong 
type.
+    _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+    def __init__(self, sparkSession: Any) -> None:
+        raise PySparkNotImplementedError(
+            errorClass="NOT_IMPLEMENTED",
+            messageParameters={"feature": "HiveContext"},
+        )
+
+    @classmethod
+    def _from_session(cls, sparkSession: Any) -> "SQLContext":
+        raise PySparkNotImplementedError(
+            errorClass="NOT_IMPLEMENTED",
+            messageParameters={"feature": "HiveContext"},
+        )
+
+
+def _same_session(a: Any, b: Any) -> bool:
+    """Session identity. ``SparkSession.getActiveSession()`` returns a fresh 
Rust-backed
+    wrapper per call, so fall back to comparing session ids when the objects 
differ."""
+    if a is b:
+        return True
+    sid = getattr(a, "session_id", None)
+    return sid is not None and sid == getattr(b, "session_id", None)
+
+
+def _reset_instantiated_context(session_id: str) -> None:
+    """Called from ``SparkSession.stop()``: drop the cached context if it 
wraps the stopped
+    session. Compared by session id since the Rust-backed session objects are 
wrappers."""
+    for cls in (SQLContext, HiveContext):
+        ctx = cls.__dict__.get("_instantiatedContext")
+        if ctx is not None and getattr(ctx.sparkSession, "session_id", None) 
== session_id:
+            cls._instantiatedContext = None
diff --git a/python/pyspark/sql/context.py b/python/pyspark/sql/context.py
index b67ce3f..7f5787b 100644
--- a/python/pyspark/sql/context.py
+++ b/python/pyspark/sql/context.py
@@ -15,7 +15,17 @@
 # limitations under the License.
 #
 """Legacy SQLContext/HiveContext plus the UDF/UDTF registration re-exports that
-pyspark.sql.__init__ imports."""
+pyspark.sql.__init__ imports.
+
+As in pyspark 4.3.0, these are the classic entry points; in Spark Connect mode 
(always,
+for this client) ``getOrCreate()`` routes to the SparkSession-backed 
counterparts in
+:mod:`pyspark.sql.connect.context`. Constructing them from a ``SparkContext`` 
is not
+possible here, since a Connect client has no SparkContext.
+"""
+import warnings
+from typing import Any, ClassVar, Optional
+
+from pyspark.errors import PySparkNotImplementedError
 from pyspark.sql.udf import UDFRegistration
 from pyspark.sql.udtf import UDTFRegistration
 
@@ -23,13 +33,78 @@ __all__ = ["SQLContext", "HiveContext", "UDFRegistration", 
"UDTFRegistration"]
 
 
 class SQLContext:
-    """Deprecated. Retained for import compatibility; use SparkSession."""
+    """The entry point for working with structured data in Spark 1.x.
+
+    .. deprecated:: 3.0.0
+        Use :func:`SparkSession.builder.getOrCreate()` instead.
+    """
+
+    _instantiatedContext: ClassVar[Optional[Any]] = None
 
-    def __init__(self, *args, **kwargs) -> None:
-        raise NotImplementedError(
-            "SQLContext is not supported in the Spark Connect client; use 
SparkSession."
+    def __init__(self, sparkContext: Any, sparkSession: Any = None, 
jsqlContext: Any = None):
+        raise PySparkNotImplementedError(
+            errorClass="NOT_IMPLEMENTED",
+            messageParameters={
+                "feature": f"{type(self).__name__}(sparkContext) in Spark 
Connect; "
+                f"use {type(self).__name__}.getOrCreate()"
+            },
         )
 
+    @classmethod
+    def getOrCreate(cls, sc: Any = None) -> Any:
+        """Get the existing SQLContext or create one for the active 
SparkSession.
+
+        In Spark Connect mode ``sc`` is not required and the active
+        :class:`SparkSession` is used.
+
+        .. deprecated:: 3.0.0
+            Use :func:`SparkSession.builder.getOrCreate()` instead.
+        """
+        from pyspark.sql.utils import is_remote
+
+        warnings.warn(
+            "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() 
instead.",
+            FutureWarning,
+        )
+        if not is_remote():  # pragma: no cover - this client is always remote
+            raise PySparkNotImplementedError(
+                errorClass="NOT_IMPLEMENTED",
+                messageParameters={"feature": "SQLContext outside Spark 
Connect"},
+            )
+        from pyspark.sql.connect import context as _connect_context
+
+        session = _get_active_session_or_create()
+        # Route to the Connect counterpart by name so subclasses (HiveContext) 
are handled:
+        # the Connect HiveContext._from_session raises 
PySparkNotImplementedError.
+        connect_cls = getattr(_connect_context, cls.__name__, None)
+        if connect_cls is None:
+            # A user-defined subclass has no Connect counterpart; fail loudly 
rather than
+            # return a base context missing the subclass's attributes.
+            raise PySparkNotImplementedError(
+                errorClass="NOT_IMPLEMENTED",
+                messageParameters={"feature": f"{cls.__name__}.getOrCreate in 
Spark Connect"},
+            )
+        return connect_cls._get_or_create_from_session(session)
+
 
 class HiveContext(SQLContext):
-    """Deprecated. Retained for import compatibility; use SparkSession."""
+    """A variant of Spark SQL that integrates with data stored in Hive.
+
+    .. deprecated:: 2.0.0
+        Use SparkSession.builder.enableHiveSupport().getOrCreate().
+    """
+
+
+def _get_active_session_or_create() -> Any:
+    """Mirror ``SparkSession._getActiveSessionOrCreate`` for the Rust-backed 
session: the
+    active session, else a new one against ``SPARK_REMOTE``."""
+    import os
+
+    from pyspark.sql.session import SparkSession
+
+    session = SparkSession.getActiveSession()
+    if session is None:
+        session = SparkSession.builder.remote(
+            os.environ.get("SPARK_REMOTE", "sc://localhost")
+        ).getOrCreate()
+    return session
diff --git a/python/pyspark/sql/functions.py b/python/pyspark/sql/functions.py
index cf847aa..0e75123 100644
--- a/python/pyspark/sql/functions.py
+++ b/python/pyspark/sql/functions.py
@@ -373,6 +373,13 @@ def variant_delete(v, *paths):
         v: a variant column
         *paths: one or more path expressions (Column or str)
     """
+    if len(paths) == 0:
+        from pyspark.errors import PySparkValueError
+
+        raise PySparkValueError(
+            errorClass="CANNOT_BE_EMPTY",
+            messageParameters={"item": "paths"},
+        )
     _a = [v]
     _a.extend(paths)
     return _dispatch("variant_delete", _a)
@@ -513,6 +520,29 @@ def lead(col, offset=1, default=None):
     return _wrap(_call_function("lead", _to_col(col), _pyfunc_lit(offset), 
_pyfunc_lit(default)))
 
 
+def instr(str, substr, start=None, occurrence=None):
+    """instr(str, substr, start=None, occurrence=None): 1-based position of 
substr.
+
+    Mirrors pyspark.sql.connect.functions.instr: ``occurrence`` without 
``start``
+    searches from position 1.
+    """
+    # Non-column args stay raw; the core turns them into literals (as `lit` 
would).
+    args = [str, substr]
+    if start is not None or occurrence is not None:
+        args.append(start if start is not None else 1)
+    if occurrence is not None:
+        args.append(occurrence)
+    return _dispatch("instr", args)
+
+
+def regexp_replace(string, pattern, replacement, position=None):
+    """regexp_replace(string, pattern, replacement, position=None)."""
+    args = [string, pattern, replacement]
+    if position is not None:
+        args.append(position)
+    return _dispatch("regexp_replace", args)
+
+
 def json_tuple(col, *fields):
     """json_tuple(col, *fields): extract the given JSON fields (literal names) 
as columns."""
     if not fields:
diff --git a/python/pyspark/sql/metrics.py b/python/pyspark/sql/metrics.py
new file mode 100644
index 0000000..7af2386
--- /dev/null
+++ b/python/pyspark/sql/metrics.py
@@ -0,0 +1,323 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+import abc
+import dataclasses
+from typing import Optional, List, Tuple, Dict, Any, Union, TYPE_CHECKING, 
Sequence
+
+from pyspark.errors import PySparkValueError
+
+if TYPE_CHECKING:
+    try:
+        import graphviz  # type: ignore
+    except ImportError:
+        pass
+
+
+class ObservedMetrics(abc.ABC):
+    @property
+    @abc.abstractmethod
+    def name(self) -> str: ...
+
+    @property
+    @abc.abstractmethod
+    def pairs(self) -> Dict[str, Any]: ...
+
+    @property
+    @abc.abstractmethod
+    def keys(self) -> List[str]: ...
+
+
+class MetricValue:
+    """The metric values is the Python representation of a plan metric value 
from the JVM.
+    However, it does not have any reference to the original value."""
+
+    def __init__(self, name: str, value: Union[int, float], type: str):
+        self._name = name
+        self._type = type
+        self._value = value
+
+    def __repr__(self) -> str:
+        return f"<{self._name}={self._value} ({self._type})>"
+
+    @property
+    def name(self) -> str:
+        return self._name
+
+    @property
+    def value(self) -> Union[int, float]:
+        return self._value
+
+    @property
+    def metric_type(self) -> str:
+        return self._type
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Return a JSON-serializable dictionary representation of this metric 
value.
+
+        Returns
+        -------
+        dict
+            A dictionary with keys 'name', 'value', and 'type'.
+        """
+        return {
+            "name": self._name,
+            "value": self._value,
+            "type": self._type,
+        }
+
+
+class PlanMetrics:
+    """Represents a particular plan node and the associated metrics of this 
node."""
+
+    def __init__(self, name: str, id: int, parent: int, metrics: 
List[MetricValue]):
+        self._name = name
+        self._id = id
+        self._parent_id = parent
+        self._metrics = metrics
+
+    def __repr__(self) -> str:
+        return f"Plan({self._name}: 
{self._id}->{self._parent_id})={self._metrics}"
+
+    @property
+    def name(self) -> str:
+        return self._name
+
+    @property
+    def plan_id(self) -> int:
+        return self._id
+
+    @property
+    def parent_plan_id(self) -> int:
+        return self._parent_id
+
+    @property
+    def metrics(self) -> List[MetricValue]:
+        return self._metrics
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Return a JSON-serializable dictionary representation of this plan 
metrics.
+
+        Returns
+        -------
+        dict
+            A dictionary with keys 'name', 'plan_id', 'parent_plan_id', and 
'metrics'.
+        """
+        return {
+            "name": self._name,
+            "plan_id": self._id,
+            "parent_plan_id": self._parent_id,
+            "metrics": [m.to_dict() for m in self._metrics],
+        }
+
+
+class CollectedMetrics:
+    @dataclasses.dataclass
+    class Node:
+        id: int
+        name: str = dataclasses.field(default="")
+        metrics: List[MetricValue] = dataclasses.field(default_factory=list)
+        children: List[int] = dataclasses.field(default_factory=list)
+
+    def text(self, current: "Node", graph: Dict[int, "Node"], prefix: str = 
"") -> str:
+        """
+        Converts the current node and its children into a textual 
representation. This is used
+        to provide a usable output for the command line or other text-based 
interfaces. However,
+        it is recommended to use the Graphviz representation for a more visual 
representation.
+
+        Parameters
+        ----------
+        current: Node
+            Current node in the graph.
+        graph: dict
+            A dictionary representing the full graph mapping from node ID 
(int) to the node itself.
+            The node is an instance of :class:`CollectedMetrics:Node`.
+        prefix: str
+            String prefix used for generating the output buffer.
+
+        Returns
+        -------
+        The full string representation of the current node as root.
+        """
+        base_metrics = set(["numPartitions", "peakMemory", "numOutputRows", 
"spillSize"])
+
+        # Format the metrics of this node:
+        metric_buffer = []
+        for m in current.metrics:
+            if m.name in base_metrics:
+                metric_buffer.append(f"{m.name}: {m.value} ({m.metric_type})")
+
+        buffer = f"{prefix}+- {current.name}({','.join(metric_buffer)})\n"
+        for i, child in enumerate(current.children):
+            c = graph[child]
+            new_prefix = prefix + "  " if i == len(c.children) - 1 else prefix
+            if current.id != c.id:
+                buffer += self.text(c, graph, new_prefix)
+        return buffer
+
+    def __init__(self, metrics: List[PlanMetrics]):
+        # Sort the input list
+        self._metrics = sorted(metrics, key=lambda x: x._parent_id, 
reverse=False)
+
+    def extract_graph(self) -> Tuple[int, Dict[int, "CollectedMetrics.Node"]]:
+        """
+        Builds the graph of the query plan. The graph is represented as a 
dictionary where the key
+        is the node ID and the value is the node itself. The root node is the 
node that has no
+        parent.
+
+        Returns
+        -------
+        The root node ID and the graph of all nodes.
+        """
+        all_nodes: Dict[int, CollectedMetrics.Node] = {}
+
+        for m in self._metrics:
+            # Add yourself to the list if you have to.
+            if m.plan_id not in all_nodes:
+                all_nodes[m.plan_id] = CollectedMetrics.Node(m.plan_id, 
m.name, m.metrics)
+            else:
+                all_nodes[m.plan_id].name = m.name
+                all_nodes[m.plan_id].metrics = m.metrics
+
+            # Now check for the parent of this node if it's in
+            if m.parent_plan_id not in all_nodes:
+                all_nodes[m.parent_plan_id] = 
CollectedMetrics.Node(m.parent_plan_id)
+
+            all_nodes[m.parent_plan_id].children.append(m.plan_id)
+
+        # Next step is to find all the root nodes. Root nodes are never used 
in children.
+        # So we start with all node ids as candidates.
+        candidates = set(all_nodes.keys())
+        for k, v in all_nodes.items():
+            for c in v.children:
+                if c in candidates and c != k:
+                    candidates.remove(c)
+
+        assert len(candidates) == 1, f"Expected 1 root node, found 
{len(candidates)}"
+        return candidates.pop(), all_nodes
+
+    def toText(self) -> str:
+        """
+        Converts the execution graph from a graph into a textual representation
+        that can be read at the command line for example.
+
+        Returns
+        -------
+        A string representation of the collected metrics.
+        """
+        root, graph = self.extract_graph()
+        return self.text(graph[root], graph)
+
+    def toDot(self, filename: Optional[str] = None, out_format: str = "png") 
-> "graphviz.Digraph":
+        """
+        Converts the collected metrics into a dot representation. Since the 
graphviz Digraph
+        implementation provides the ability to render the result graph 
directory in a
+        notebook, we return the graph object directly.
+
+        If the graphviz package is not available, a PACKAGE_NOT_INSTALLED 
error is raised.
+
+        Parameters
+        ----------
+        filename : str, optional
+            The filename to save the graph to given an output format. The path 
can be
+            relative or absolute.
+
+        out_format : str
+            The output format of the graph. The default is 'png'.
+
+        Returns
+        -------
+        An instance of the graphviz.Digraph object.
+        """
+        try:
+            import graphviz
+
+            dot = graphviz.Digraph(
+                comment="Query Plan",
+                node_attr={
+                    "shape": "box",
+                    "font-size": "10pt",
+                },
+            )
+
+            root, graph = self.extract_graph()
+            for k, v in graph.items():
+                # Build table rows for the metrics
+                rows = "\n".join(
+                    [
+                        (
+                            f'<TR><TD><FONT 
POINT-SIZE="8">{x.name}</FONT></TD><TD>'
+                            f'<FONT POINT-SIZE="8">{x.value} 
({x.metric_type})</FONT></TD></TR>'
+                        )
+                        for x in v.metrics
+                    ]
+                )
+
+                dot.node(
+                    str(k),
+                    """<<TABLE BORDER="0" CELLBORDER="1" CELLSPACING="0">
+                    <TR>
+                        <TD COLSPAN="2" BGCOLOR="lightgrey">
+                            <FONT POINT-SIZE=\"10\">{}</FONT>
+                        </TD>
+                    </TR>
+                    <TR><TD COLSPAN="2"><FONT 
POINT-SIZE=\"10\">Metrics</FONT></TD></TR>
+                    {}
+                    </TABLE>>""".format(v.name, rows),
+                )
+                for c in v.children:
+                    dot.edge(str(k), str(c))
+
+            if filename:
+                dot.render(filename, format=out_format, cleanup=True)
+            return dot
+
+        except ImportError:
+            raise PySparkValueError(
+                errorClass="PACKAGE_NOT_INSTALLED",
+                messageParameters={"package_name": "graphviz", 
"minimum_version": "0.20"},
+            )
+
+
+class ExecutionInfo:
+    """The ExecutionInfo class allows users to inspect the query execution of 
this particular
+    data frame. This value is only set in the data frame if it was executed."""
+
+    def __init__(
+        self,
+        metrics: Optional[list[PlanMetrics]],
+        obs: Optional[Sequence[ObservedMetrics]],
+        operation_id: Optional[str] = None,
+    ):
+        self._metrics = CollectedMetrics(metrics) if metrics else None
+        self._observations = obs if obs else []
+        self._operation_id = operation_id
+
+    @property
+    def metrics(self) -> Optional[CollectedMetrics]:
+        return self._metrics
+
+    @property
+    def flows(self) -> List[Tuple[str, Dict[str, Any]]]:
+        return [(f.name, f.pairs) for f in self._observations]
+
+    @property
+    def operation_id(self) -> Optional[str]:
+        """The Spark Connect ExecutePlan operation ID, when available.
+
+        .. versionadded:: 4.3.0
+        """
+        return self._operation_id
diff --git a/python/pyspark/sql/types.py b/python/pyspark/sql/types.py
index ce52e5b..3e8a051 100644
--- a/python/pyspark/sql/types.py
+++ b/python/pyspark/sql/types.py
@@ -17,6 +17,8 @@ try:
         AnyTimeType, AnsiIntervalType, SpatialType,
         GeometryType, GeographyType,
         VariantVal, Geometry, Geography,
+        # Spark's nanosecond timestamp types (no PySpark 4.3.0 class; not in 
__all__).
+        TimestampNTZNanosType, TimestampLTZNanosType,
     )
     # Row lives in the Rust core; upstream code imports it from 
pyspark.sql.types too.
     from pyspark._pyspark import Row
@@ -313,6 +315,10 @@ def _parse_datatype_json_value(v):
         m = _re.match(r"time\((\d+)\)$", v)
         if m:
             return TimeType(int(m.group(1)))
+        m = _re.match(r"timestamp_(ntz|ltz)\(\s*(\d+)\s*\)$", v)
+        if m:
+            cls = TimestampNTZNanosType if m.group(1) == "ntz" else 
TimestampLTZNanosType
+            return cls(int(m.group(2)))
         if v == "calendarinterval":
             return CalendarIntervalType()
         _ym = {"year": 0, "month": 1}
diff --git a/python/tests/_mock_connect_server.py 
b/python/tests/_mock_connect_server.py
index 555f829..72a6bed 100644
--- a/python/tests/_mock_connect_server.py
+++ b/python/tests/_mock_connect_server.py
@@ -39,6 +39,44 @@ def message_field(field, value):
     return varint(field * 8 + 2) + varint(len(value)) + value
 
 
+def varint_field(field, value):
+    return varint(field * 8) + varint(value & (2**64 - 1))
+
+
+def parse_message(data):
+    """Decode protobuf wire bytes into {field: [value, ...]}: varints as ints 
and
+    length-delimited fields as bytes (decode nested messages with another 
call)."""
+    fields, i = {}, 0
+
+    def read_varint():
+        nonlocal i
+        shift = result = 0
+        while True:
+            byte = data[i]
+            i += 1
+            result |= (byte & 127) << shift
+            shift += 7
+            if byte < 128:
+                return result
+
+    while i < len(data):
+        key = read_varint()
+        field, wire = key >> 3, key & 7
+        if wire == 0:
+            value = read_varint()
+        elif wire == 2:
+            size = read_varint()
+            value, i = data[i : i + size], i + size
+        elif wire == 1:
+            value, i = data[i : i + 8], i + 8
+        elif wire == 5:
+            value, i = data[i : i + 4], i + 4
+        else:
+            raise ValueError(f"unsupported wire type {wire}")
+        fields.setdefault(field, []).append(value)
+    return fields
+
+
 def struct_schema(name="id"):
     # types.proto: DataType.long=7, .struct=21; Struct.fields=1;
     # StructField: name=1, data_type=2, nullable=3, metadata=4.
@@ -81,12 +119,36 @@ def serve(connection):
             # base.proto: AnalyzePlanResponse.schema=2, Schema.schema=1.
             return message_field(2, message_field(1, self.data_type))
 
+        # ExecutePlan records each request and streams one response carrying
+        # `execute_payload` (raw ExecutePlanResponse fields), then 
ResultComplete; with
+        # `execute_failure` set it aborts instead. Responses echo the request's
+        # operation id, as the server does.
+        execute_requests = []
+        execute_payload = b""
+        execute_failure = None
+
+        def execute(self, request, context):
+            self.execute_requests.append(request)
+            if self.execute_failure:
+                context.abort(grpc.StatusCode.INVALID_ARGUMENT, 
self.execute_failure)
+            # ExecutePlanRequest.operation_id=6; 
ExecutePlanResponse.operation_id=12,
+            # response_id=13, result_complete=14.
+            operation_id = parse_message(request).get(6, [b""])[0]
+            header = message_field(12, operation_id)
+            yield header + message_field(13, "r1") + self.execute_payload
+            yield header + message_field(13, "r2") + message_field(14, b"")
+
     service = Service()
     with ThreadPoolExecutor(max_workers=4) as executor:
         server = grpc.server(executor)
         handlers = {
             "AnalyzePlan": 
grpc.unary_unary_rpc_method_handler(service.analyze),
             "ReleaseSession": grpc.unary_unary_rpc_method_handler(lambda 
request, context: b""),
+            "ExecutePlan": 
grpc.unary_stream_rpc_method_handler(service.execute),
+            "ReattachExecute": grpc.unary_stream_rpc_method_handler(
+                lambda request, context: iter([message_field(14, b"")])
+            ),
+            "ReleaseExecute": grpc.unary_unary_rpc_method_handler(lambda 
request, context: b""),
         }
         server.add_generic_rpc_handlers(
             
(grpc.method_handlers_generic_handler("spark.connect.SparkConnectService", 
handlers),)
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 2a027bb..209a4b6 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -13,7 +13,13 @@ from threading import Barrier
 
 import pytest
 
-from _mock_connect_server import message_field, mock_connect_session, 
struct_schema
+from _mock_connect_server import (
+    message_field,
+    mock_connect_session,
+    parse_message,
+    struct_schema,
+    varint_field,
+)
 from pyspark.sql import SparkSession, types as T
 
 # Collected here so the existing offline CI suite exercises real Arrow results.
@@ -3228,3 +3234,311 @@ def 
test_schema_cache_metadata_wait_releases_gil(mock_connect, accessor):
     else:
         assert result == [("id", "bigint")]
     assert service.calls == 1
+
+
+# --------------------------------------------------------------------------- 
4.3.0 gaps
+
+
+def test_timestamp_nanos_types_roundtrip_and_pickle():
+    # Spark's TimestampNTZNanosType/TimestampLTZNanosType (precision 7..9) 
materialize as
+    # their own types, not as TimeType / a microsecond TimestampType.
+    for cls, name in ((T.TimestampNTZNanosType, "timestamp_ntz"),
+                      (T.TimestampLTZNanosType, "timestamp_ltz")):
+        t = cls(8)
+        assert t.precision == 8
+        assert t.simpleString() == t.typeName() == f"{name}(8)"
+        assert t.json() == f'"{name}(8)"'
+        assert cls().precision == 9
+        assert isinstance(t, T.DatetimeType)
+        assert repr(pickle.loads(pickle.dumps(t))) == repr(t)
+        assert repr(T._parse_datatype_json_string(f'"{name}(7)"')) == 
repr(cls(7))
+        with pytest.raises(ValueError):
+            cls(6)
+    nested = T.StructType([T.StructField("a", 
T.ArrayType(T.TimestampLTZNanosType(9)))])
+    assert pickle.loads(pickle.dumps(nested)).simpleString() == 
"struct<a:array<timestamp_ltz(9)>>"
+
+
+def test_instr_regexp_replace_optional_args():
+    from pyspark.sql import functions as F
+
+    captured = []
+    orig = F._call_function
+    F._call_function = lambda fname, *args: captured.append((fname, args)) or 
orig(fname, *args)
+    try:
+        F.instr("s", "a")
+        assert captured[-1][1][1:] == ("a",)
+        # occurrence without start searches from position 1 (pyspark 4.3.0).
+        F.instr("s", "a", occurrence=2)
+        assert captured[-1][1][1:] == ("a", 1, 2)
+        F.instr("s", "a", start=None, occurrence=None)
+        assert len(captured[-1][1]) == 2
+        F.instr("s", "a", 3)
+        assert captured[-1][1][1:] == ("a", 3)
+        F.regexp_replace("s", "a", "b")
+        assert len(captured[-1][1]) == 3
+        F.regexp_replace("s", "a", "b", position=None)
+        assert len(captured[-1][1]) == 3
+        F.regexp_replace("s", "a", "b", 4)
+        assert captured[-1][1][1:] == ("a", "b", 4)
+    finally:
+        F._call_function = orig
+
+
+def test_variant_delete_requires_paths():
+    from pyspark.errors import PySparkValueError
+    from pyspark.sql import functions as F
+
+    with pytest.raises(PySparkValueError, match="CANNOT_BE_EMPTY"):
+        F.variant_delete("v")
+
+
+def _offline_session():
+    # Port 1: nothing listens; the channel is lazy, so the session builds 
offline.
+    return SparkSession.builder.remote("sc://localhost:1").getOrCreate()
+
+
+def test_session_context_manager_stops_session():
+    spark = _offline_session()
+    with spark as s:
+        assert s is spark
+        assert not s.is_stopped
+    # __exit__ stops it; stop() is best-effort (the release RPC fails offline) 
and
+    # idempotent.
+    assert spark.is_stopped
+    spark.stop()
+    assert SparkSession.getActiveSession() is None
+
+
+def test_sql_context_connect_backed():
+    import warnings
+
+    from pyspark.errors import PySparkNotImplementedError
+    from pyspark.sql import HiveContext, SQLContext
+    from pyspark.sql.connect import context as cctx
+
+    spark = _offline_session()
+    try:
+        with warnings.catch_warnings(record=True) as caught:
+            warnings.simplefilter("always")
+            ctx = SQLContext.getOrCreate()
+        assert any(issubclass(w.category, FutureWarning) for w in caught)
+        assert isinstance(ctx, cctx.SQLContext)
+        assert ctx.sparkSession.session_id == spark.session_id
+        # Cached per session (getActiveSession returns a fresh wrapper each 
call).
+        assert SQLContext.getOrCreate() is ctx
+        assert cctx.SQLContext._get_or_create_from_session(spark) is ctx
+        with pytest.raises(PySparkNotImplementedError):
+            ctx.registerJavaFunction("f", "com.example.F")
+        with pytest.raises(PySparkNotImplementedError):
+            HiveContext.getOrCreate()
+        with pytest.raises(PySparkNotImplementedError):
+            cctx.HiveContext(spark)
+        with pytest.raises(PySparkNotImplementedError):
+            SQLContext(None)
+        with warnings.catch_warnings(record=True) as caught:
+            warnings.simplefilter("always")
+            cctx.SQLContext(spark)
+        assert any(issubclass(w.category, FutureWarning) for w in caught)
+        assert ctx.udf is not None and ctx.read is not None
+        # Stopping a different session leaves the cache; stopping its own 
clears it.
+        other = spark.newSession()
+        other.stop()
+        assert cctx.SQLContext._instantiatedContext is ctx
+        spark.stop()
+        assert cctx.SQLContext._instantiatedContext is None
+    finally:
+        cctx.SQLContext._instantiatedContext = None
+        spark.stop()
+
+
+def test_execution_info_and_operation_id():
+    from pyspark.errors import AnalysisException
+    from pyspark.errors.exceptions.connect import SparkConnectException
+    from pyspark.sql.connect.client import PlanObservedMetrics
+    from pyspark.sql.metrics import ExecutionInfo
+
+    spark = _offline_session()
+    try:
+        # No action ran yet.
+        assert spark.range(1).executionInfo is None
+    finally:
+        spark.stop()
+    ei = ExecutionInfo([], [PlanObservedMetrics("o", [1, 2], [])], "op-1")
+    assert ei.operation_id == "op-1"
+    assert ei.flows == [("o", {"observed_metric_0": 1, "observed_metric_1": 
2})]
+    assert ei.metrics is None
+    # SparkConnectException.operation_id (4.3.0): None unless the transport 
stamped it.
+    err = AnalysisException("boom")
+    assert isinstance(err, SparkConnectException)
+    assert err.operation_id is None
+    err._operation_id = "op-2"
+    assert err.operation_id == "op-2"
+
+
+def test_create_auto_cdc_flow_scd2_track_history():
+    from pyspark import pipelines as dp
+    from pyspark.errors import PySparkValueError
+    from pyspark.pipelines.graph_element_registry import 
graph_element_registration_context
+    from pyspark.pipelines.tests.local_graph_element_registry import 
LocalGraphElementRegistry
+    from pyspark.sql.functions import col
+
+    registry = LocalGraphElementRegistry()
+    with graph_element_registration_context(registry):
+        dp.create_streaming_table("t")
+        dp.create_auto_cdc_flow(
+            target="t", source="s", keys=["k"], sequence_by="seq",
+            stored_as_scd_type=2, track_history_column_list=["a", col("b")],
+            spark_conf={"k": "v"},
+        )
+        with pytest.raises(PySparkValueError, match="CANNOT_SET_TOGETHER"):
+            dp.create_auto_cdc_flow(
+                target="t", source="s", keys=["k"], sequence_by="seq",
+                stored_as_scd_type=2, track_history_column_list=["a"],
+                track_history_except_column_list=["b"],
+            )
+        with pytest.raises(PySparkValueError, 
match="INVALID_MULTIPLE_ARGUMENT_CONDITIONS"):
+            dp.create_auto_cdc_flow(
+                target="t", source="s", keys=["k"], sequence_by="seq",
+                stored_as_scd_type=1, track_history_column_list=["a"],
+            )
+    flow = registry.auto_cdc_flows[0]
+    assert flow.stored_as_scd_type == 2
+    assert len(flow.track_history_column_list) == 2
+    assert flow.spark_conf == {"k": "v"}
+
+
+# --------------------------------------------------------------------------- 
ExecutePlan (mock)
+
+
+def _request_operation_id(request):
+    # ExecutePlanRequest.operation_id=6.
+    return parse_message(request)[6][0].decode()
+
+
+def 
test_execution_info_decodes_observed_metrics_and_operation_id(mock_connect):
+    import datetime
+
+    spark, service = mock_connect
+    ts = 1_700_000_000_123_456
+    # expressions.proto Literal: integer=6, string=13, date=16, timestamp=17,
+    # day_time_interval=21, map=23 (keys=3, values=4), struct=24 (elements=2),
+    # data_type=100; types.proto DataType.struct=21 (fields=1; 
StructField.name=1).
+    struct_type = message_field(
+        21, message_field(1, message_field(1, "a")) + message_field(1, 
message_field(1, "b"))
+    )
+    struct_literal = message_field(
+        24, message_field(2, varint_field(6, 7)) + message_field(2, 
message_field(13, "x"))
+    ) + message_field(100, struct_type)
+    map_literal = message_field(
+        23, message_field(3, message_field(13, "k")) + message_field(4, 
varint_field(16, 1))
+    )
+    # base.proto ObservedMetrics: name=1, values=2, keys=3; 
ExecutePlanResponse.observed_metrics=6.
+    observed = message_field(1, "o")
+    for literal in (
+        varint_field(17, ts),
+        struct_literal,
+        map_literal,
+        varint_field(21, 90_061_000_001),
+    ):
+        observed += message_field(2, literal)
+    for key in ("ts", "s", "m", "dt"):
+        observed += message_field(3, key)
+    service.execute_payload = message_field(6, observed)
+
+    assert spark.range(1).executionInfo is None
+    spark.range(1).collect()
+    info = spark.range(1).executionInfo
+    assert info.operation_id == 
_request_operation_id(service.execute_requests[-1])
+    ((name, pairs),) = info.flows
+    assert name == "o"
+    assert pairs["ts"] == datetime.datetime.fromtimestamp(ts // 10**6).replace(
+        microsecond=ts % 10**6
+    )
+    assert (pairs["s"].a, pairs["s"].b) == (7, "x")
+    assert pairs["m"] == {"k": datetime.date(1970, 1, 2)}
+    assert pairs["dt"] == datetime.timedelta(microseconds=90_061_000_001)
+
+
+def test_execution_info_unsupported_literal_raises_on_flows(mock_connect):
+    from pyspark.errors import PySparkTypeError
+
+    spark, service = mock_connect
+    # Literal.calendar_interval=19 (months=1): the reference rejects it in 
_to_value.
+    observed = message_field(1, "o") + message_field(2, message_field(19, 
varint_field(1, 2)))
+    service.execute_payload = message_field(6, observed)
+    spark.range(1).collect()
+    info = spark.range(1).executionInfo
+    assert info.operation_id == 
_request_operation_id(service.execute_requests[-1])
+    with pytest.raises(PySparkTypeError, match="UNSUPPORTED_LITERAL"):
+        info.flows
+
+
+def test_execute_plan_error_carries_operation_id(mock_connect):
+    from pyspark.errors import PySparkException
+
+    spark, service = mock_connect
+    service.execute_failure = "plan rejected"
+    with pytest.raises(PySparkException, match="plan rejected") as error:
+        spark.range(1).collect()
+    assert error.value.operation_id == 
_request_operation_id(service.execute_requests[-1])
+
+
+def _sent_define_flow(request):
+    """The DefineFlow in a captured ExecutePlanRequest: plan=3 -> command=2 ->
+    pipeline_command=19 -> define_flow=3."""
+    message = parse_message(request)
+    for field in (3, 2, 19, 3):
+        message = parse_message(message[field][0])
+    return message
+
+
+def _unparsed_identifiers(expressions):
+    # Expression.unresolved_attribute=2, 
UnresolvedAttribute.unparsed_identifier=1.
+    return [
+        parse_message(parse_message(e)[2][0])[1][0].decode() for e in 
expressions
+    ]
+
+
+def test_connect_registry_sends_auto_cdc_define_flow(mock_connect):
+    from pyspark import pipelines as dp
+    from pyspark.pipelines.graph_element_registry import 
graph_element_registration_context
+    from pyspark.pipelines.spark_connect_graph_element_registry import (
+        SparkConnectGraphElementRegistry,
+    )
+
+    spark, service = mock_connect
+    with 
graph_element_registration_context(SparkConnectGraphElementRegistry(spark, 
"g")):
+        dp.create_auto_cdc_flow(
+            target="t", source="s", keys=["k"], sequence_by="seq",
+            stored_as_scd_type=2, name="scd2",
+            track_history_column_list=["a", "b"], spark_conf={"c": "v"},
+        )
+        dp.create_auto_cdc_flow(
+            target="t", source="s", keys=["k"], sequence_by="seq",
+            stored_as_scd_type="2", name="scd2_except",
+            track_history_except_column_list=["z"],
+        )
+        dp.create_auto_cdc_flow(
+            target="t", source="s", keys=["k"], sequence_by="seq", name="plain"
+        )
+    scd2, scd2_except, plain = (_sent_define_flow(r) for r in 
service.execute_requests[-3:])
+
+    # DefineFlow: flow_name=2, sql_conf=4 (entries key=1, value=2), 
auto_cdc_flow_details=10;
+    # AutoCdcFlowDetails: stored_as_scd_type=10, track_history_column_list=11,
+    # track_history_except_column_list=12.
+    assert scd2[2] == [b"scd2"]
+    assert [parse_message(e) for e in scd2[4]] == [{1: [b"c"], 2: [b"v"]}]
+    details = parse_message(scd2[10][0])
+    assert details[10] == [2]
+    assert _unparsed_identifiers(details[11]) == ["a", "b"]
+    assert 12 not in details
+
+    details = parse_message(scd2_except[10][0])
+    assert details[10] == [2]
+    assert 11 not in details
+    assert _unparsed_identifiers(details[12]) == ["z"]
+
+    details = parse_message(plain[10][0])
+    assert 4 not in plain
+    assert 10 not in details and 11 not in details and 12 not in details
+
diff --git a/scripts/check_vendored_upstream.sh 
b/scripts/check_vendored_upstream.sh
index 0f4ba59..2e8fd61 100644
--- a/scripts/check_vendored_upstream.sh
+++ b/scripts/check_vendored_upstream.sh
@@ -143,6 +143,8 @@ VENDORED_FILES=(
   "sql/datasource_internal.py"
   "sql/internal.py"
   "sql/profiler.py"
+  # ExecutionInfo / PlanMetrics / MetricValue (DataFrame.executionInfo).
+  "sql/metrics.py"
 
   # Streaming (excluding adapted context, dstream, util)
   "streaming/__init__.py"


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to