This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new a556dc7 [SPARK-60858][CONNECT] Close remaining Spark 4.3.0 client API
gaps
a556dc7 is described below
commit a556dc767df594d27ec0b8bf891685ffb342f849
Author: Liang-Chi Hsieh <[email protected]>
AuthorDate: Wed Nov 4 14:27:30 2026 +0900
[SPARK-60858][CONNECT] Close remaining Spark 4.3.0 client API gaps
### What changes were proposed in this pull request?
This closes the Spark 4.3.0 client API gaps that remain after SPARK-59305,
in the Rust core, the PyO3 layer and the Python shim:
- **Nanosecond timestamp types.** The `TimestampNTZNanos` /
`TimestampLTZNanos` proto kinds used to decode as `TIME` and as a microsecond
`TIMESTAMP`. They now have their own `DataType` variants, `TimestampNtzNanos` /
`TimestampLtzNanos`, with proto, JSON (`timestamp_ntz(p)` /
`timestamp_ltz(p)`), DDL and Arrow (nanosecond) support. In Python they appear
as picklable `TimestampNTZNanosType` / `TimestampLTZNanosType`. These are not
added to `pyspark.sql.types.__all__`, because PySpark [...]
- **Functions.** `instr(..., occurrence=k)` without `start` searches from
position 1, and `None` means the argument is omitted, as in 4.3.0.
`regexp_replace(..., position=None)` also treats `None` as omitted.
`variant_delete(v)` with no paths raises `CANNOT_BE_EMPTY`.
- **`SparkSession.__enter__` / `__exit__`.** Leaving the `with` block stops
the session. Like PySpark, `stop()` can now be called more than once and only
logs a warning when releasing the session fails. It also clears the active
session and the stopped session's cached `SQLContext`.
- **Spark Connect `SQLContext` / `HiveContext`.** These are ported from
v4.3.0 `pyspark/sql/connect/context.py`, and
`pyspark.sql.SQLContext.getOrCreate()` / `HiveContext.getOrCreate()` route to
them. The cache compares sessions by session id, because
`SparkSession.getActiveSession()` returns a new wrapper object on every call.
- **Pipelines `create_auto_cdc_flow`.** It now supports
`stored_as_scd_type` 2, the keyword-only `track_history_column_list` /
`track_history_except_column_list`, and `spark_conf`, with the same client-side
validation as 4.3.0. `api.py` is v4.3.0 plus the existing import repointing.
The registry, PyO3 and core now send the track-history lists, `SCD_TYPE_2` and
`sql_conf`, which used to be hardcoded empty.
- **`operation_id`.**
- `DataFrame.executionInfo` now returns a
`pyspark.sql.metrics.ExecutionInfo`, including `operation_id`, instead of a
dict. `sql/metrics.py` is vendored verbatim from v4.3.0 and added to the drift
check.
- Errors from an ExecutePlan carry the operation id, read by
`SparkConnectException.operation_id`.
- In this drop-in, `SparkConnectException` is the same class as
`PySparkException`, so the property is defined on that shared base.
### Why are the changes needed?
These 4.3.0 APIs were missing or behaved differently from PySpark 4.3.0.
The nanosecond timestamp types were also decoded as the wrong types.
### Does this PR introduce _any_ user-facing change?
Yes, the APIs above now exist and behave as in PySpark 4.3.0.
`DataFrame.executionInfo` now returns an `ExecutionInfo` (or `None` before an
action) instead of a dict.
### How was this patch tested?
- New Rust unit tests: the nanos types in `types_roundtrip.rs`, `stop()`
being safe to call twice, `operation_id` capture, and
`SparkError::with_operation_id`.
- New offline drop-in tests in `test_dropin_offline.py`.
- The official v4.3.0 `pipelines/tests/test_auto_cdc_flow.py` passes (19
tests).
- `cargo fmt --all --check`, `cargo test -p apache-spark-connect --release
--lib`, and all integration tests compile with and without `--features
wasm-udf`.
- Not run locally, because no Spark 4.3.0 Connect server was available: the
e2e suites and the official `test_connect_context.py`,
`test_parity_sql_context.py` and `test_parity_zip.py`. These run on CI.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #146 from viirya/spark-430-api-gaps-80fff8.
Authored-by: Liang-Chi Hsieh <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/dataframe.rs | 199 ++++++++++++-
crates/pyspark-rs/src/errors.rs | 6 +
crates/pyspark-rs/src/lib.rs | 6 +-
crates/pyspark-rs/src/pipelines.rs | 44 ++-
crates/pyspark-rs/src/session.rs | 47 ++-
crates/pyspark-rs/src/types.rs | 174 +++++++++++
crates/spark-connect-core/src/client.rs | 6 +-
crates/spark-connect-core/src/error.rs | 23 ++
crates/spark-connect/src/dataframe.rs | 18 ++
crates/spark-connect/src/pipelines.rs | 3 +-
crates/spark-connect/src/profiler.rs | 2 +-
crates/spark-connect/src/session.rs | 135 ++++++++-
crates/spark-connect/src/types.rs | 83 +++++-
crates/spark-connect/tests/types_roundtrip.rs | 39 +++
python/.coveragerc | 1 +
python/pyspark/errors/exceptions/__init__.py | 16 +
python/pyspark/pipelines/api.py | 97 ++++++-
.../spark_connect_graph_element_registry.py | 16 +-
python/pyspark/sql/connect/client/__init__.py | 58 ++++
python/pyspark/sql/connect/context.py | 255 ++++++++++++++++
python/pyspark/sql/context.py | 87 +++++-
python/pyspark/sql/functions.py | 30 ++
python/pyspark/sql/metrics.py | 323 +++++++++++++++++++++
python/pyspark/sql/types.py | 6 +
python/tests/_mock_connect_server.py | 62 ++++
python/tests/test_dropin_offline.py | 316 +++++++++++++++++++-
scripts/check_vendored_upstream.sh | 2 +
27 files changed, 1999 insertions(+), 55 deletions(-)
diff --git a/crates/pyspark-rs/src/dataframe.rs
b/crates/pyspark-rs/src/dataframe.rs
index cf32ab2..1dfe89e 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -1230,20 +1230,48 @@ impl PyDataFrame {
py.detach(|| self.dataframe.show(n)).to_pyerr()
}
- /// Execution metrics from the most recent action. Mirrors
`DataFrame.executionInfo`.
+ /// Execution info from the most recent action, as a
`pyspark.sql.metrics.ExecutionInfo`
+ /// (plan metrics, observed-metric flows, and the ExecutePlan
`operation_id`). Mirrors
+ /// `DataFrame.executionInfo`; `None` before any action ran.
#[getter]
#[pyo3(name = "executionInfo")]
fn execution_info(&self, py: Python<'_>) -> PyResult<Py<PyAny>> {
- let info = self.dataframe.execution_info().to_pyerr()?;
- let dict = pyo3::types::PyDict::new(py);
- dict.set_item("has_metrics", info.metrics.is_some())?;
- let names: Vec<String> = info
- .observed_metrics
- .iter()
- .map(|m| m.name.clone())
- .collect();
- dict.set_item("observed_metrics", names)?;
- Ok(dict.into_any().unbind())
+ let Ok(info) = self.dataframe.execution_info() else {
+ return Ok(py.None());
+ };
+ let metrics_mod = py.import("pyspark.sql.metrics")?;
+ let plan_metrics_cls = metrics_mod.getattr("PlanMetrics")?;
+ let metric_value_cls = metrics_mod.getattr("MetricValue")?;
+ // Mirrors SparkConnectClient._build_metrics.
+ let plan_metrics = pyo3::types::PyList::empty(py);
+ if let Some(m) = &info.metrics {
+ for x in &m.metrics {
+ let values = pyo3::types::PyList::empty(py);
+ for (k, v) in &x.execution_metrics {
+ values.append(metric_value_cls.call1((k, v.value,
&v.metric_type))?)?;
+ }
+ plan_metrics
+ .append(plan_metrics_cls.call1((&x.name, x.plan_id,
x.parent, values))?)?;
+ }
+ }
+ let observed_cls = py
+ .import("pyspark.sql.connect.client")?
+ .getattr("PlanObservedMetrics")?;
+ let observed = pyo3::types::PyList::empty(py);
+ for o in &info.observed_metrics {
+ let values = o
+ .values
+ .iter()
+ .map(|l| literal_to_py(py, l, None))
+ .collect::<PyResult<Vec<_>>>()?;
+ observed.append(observed_cls.call1((&o.name, values,
o.keys.clone()))?)?;
+ }
+ let ei = metrics_mod.getattr("ExecutionInfo")?.call1((
+ plan_metrics,
+ observed,
+ info.operation_id.clone(),
+ ))?;
+ Ok(ei.unbind())
}
/// Convert to a pandas-on-Spark DataFrame. Mirrors `DataFrame.pandas_api`;
@@ -2491,3 +2519,152 @@ impl PyLocalRowIterator {
}
}
}
+
+/// Convert an observed-metric literal to a Python value. Mirrors
+/// `LiteralExpression._to_value`, which `PlanObservedMetrics.pairs` applies:
the kinds it
+/// decodes map to the same Python values, and for any other kind (which the
reference
+/// rejects with `UNSUPPORTED_LITERAL`) this returns an `_UnsupportedLiteral`
marker, so
+/// the error surfaces when `flows` is read, as in the reference, not when the
+/// `ExecutionInfo` is built.
+///
+/// `data_type` is the literal's type when the enclosing literal carries it; a
struct's
+/// field names come from it, else from the literal's own `data_type`, else
from the
+/// deprecated `struct_type`.
+fn literal_to_py(
+ py: Python<'_>,
+ literal: &spark_connect_proto::expression::Literal,
+ data_type: Option<&spark_connect_proto::DataType>,
+) -> PyResult<Py<PyAny>> {
+ use pyo3::IntoPyObjectExt;
+ use spark_connect_proto::data_type::Kind;
+ use spark_connect_proto::expression::literal::LiteralType;
+ let data_type = data_type.or(literal.data_type.as_ref());
+ let datetime = || py.import("datetime");
+ Ok(match &literal.literal_type {
+ None | Some(LiteralType::Null(_)) => py.None(),
+ Some(LiteralType::Boolean(b)) => b.into_py_any(py)?,
+ Some(LiteralType::Byte(v))
+ | Some(LiteralType::Short(v))
+ | Some(LiteralType::Integer(v)) => v.into_py_any(py)?,
+ Some(LiteralType::Long(v)) => v.into_py_any(py)?,
+ Some(LiteralType::Float(v)) => (*v as f64).into_py_any(py)?,
+ Some(LiteralType::Double(v)) => v.into_py_any(py)?,
+ Some(LiteralType::String(s)) => s.into_py_any(py)?,
+ Some(LiteralType::Binary(b)) => pyo3::types::PyBytes::new(py,
b).into_any().unbind(),
+ Some(LiteralType::Decimal(d)) => py
+ .import("decimal")?
+ .getattr("Decimal")?
+ .call1((&d.value,))?
+ .unbind(),
+ // DateType.fromInternal: days since the epoch (ordinal 719163).
+ Some(LiteralType::Date(days)) => datetime()?
+ .getattr("date")?
+ .call_method1("fromordinal", (*days as i64 + 719_163,))?
+ .unbind(),
+ // TimeType.fromInternal: nanoseconds of the day, truncated to
microseconds.
+ Some(LiteralType::Time(t)) => {
+ let nano = t.nano;
+ datetime()?
+ .getattr("time")?
+ .call1((
+ nano.div_euclid(3_600_000_000_000),
+ nano.rem_euclid(3_600_000_000_000) / 60_000_000_000,
+ nano.rem_euclid(60_000_000_000) / 1_000_000_000,
+ nano.rem_euclid(1_000_000_000) / 1_000,
+ ))?
+ .unbind()
+ }
+ // TimestampType.fromInternal: a naive local datetime, built from
whole seconds so
+ // the microseconds are exact.
+ Some(LiteralType::Timestamp(ts)) => {
+ let kwargs = pyo3::types::PyDict::new(py);
+ kwargs.set_item("microsecond", ts.rem_euclid(1_000_000))?;
+ datetime()?
+ .getattr("datetime")?
+ .call_method1("fromtimestamp", (ts.div_euclid(1_000_000),))?
+ .call_method("replace", (), Some(&kwargs))?
+ .unbind()
+ }
+ // TimestampNTZType.fromInternal: the UTC wall clock, without a zone.
+ Some(LiteralType::TimestampNtz(ts)) => {
+ let dt = datetime()?;
+ let utc = dt.getattr("timezone")?.getattr("utc")?;
+ let kwargs = pyo3::types::PyDict::new(py);
+ kwargs.set_item("microsecond", ts.rem_euclid(1_000_000))?;
+ kwargs.set_item("tzinfo", py.None())?;
+ dt.getattr("datetime")?
+ .call_method1("fromtimestamp", (ts.div_euclid(1_000_000),
utc))?
+ .call_method("replace", (), Some(&kwargs))?
+ .unbind()
+ }
+ // DayTimeIntervalType.fromInternal: timedelta(microseconds=v).
+ Some(LiteralType::DayTimeInterval(micros)) => {
+ let kwargs = pyo3::types::PyDict::new(py);
+ kwargs.set_item("microseconds", micros)?;
+ datetime()?
+ .getattr("timedelta")?
+ .call((), Some(&kwargs))?
+ .unbind()
+ }
+ Some(LiteralType::Array(a)) => {
+ let element_type = match data_type.and_then(|t| t.kind.as_ref()) {
+ Some(Kind::Array(t)) => t.element_type.as_deref(),
+ _ => None,
+ };
+ let items = a
+ .elements
+ .iter()
+ .map(|e| literal_to_py(py, e, element_type))
+ .collect::<PyResult<Vec<_>>>()?;
+ items.into_py_any(py)?
+ }
+ Some(LiteralType::Map(m)) => {
+ let (key_type, value_type) = match data_type.and_then(|t|
t.kind.as_ref()) {
+ Some(Kind::Map(t)) => (t.key_type.as_deref(),
t.value_type.as_deref()),
+ _ => (None, None),
+ };
+ let dict = pyo3::types::PyDict::new(py);
+ for (k, v) in m.keys.iter().zip(&m.values) {
+ dict.set_item(
+ literal_to_py(py, k, key_type)?,
+ literal_to_py(py, v, value_type)?,
+ )?;
+ }
+ dict.into_any().unbind()
+ }
+ Some(LiteralType::Struct(s)) => {
+ #[allow(deprecated)]
+ let struct_type = match data_type.and_then(|t| t.kind.as_ref()) {
+ Some(Kind::Struct(t)) => Some(t),
+ _ => match s.struct_type.as_ref().and_then(|t|
t.kind.as_ref()) {
+ Some(Kind::Struct(t)) => Some(t),
+ _ => None,
+ },
+ };
+ let fields = struct_type.map(|t|
t.fields.as_slice()).unwrap_or_default();
+ let names = (0..s.elements.len())
+ .map(|i| match fields.get(i) {
+ Some(f) => f.name.clone(),
+ None => format!("_{}", i + 1),
+ })
+ .collect::<Vec<_>>();
+ let values = s
+ .elements
+ .iter()
+ .enumerate()
+ .map(|(i, e)| {
+ literal_to_py(py, e, fields.get(i).and_then(|f|
f.data_type.as_ref()))
+ })
+ .collect::<PyResult<Vec<_>>>()?;
+ py.import("pyspark.sql.types")?
+ .getattr("_create_row")?
+ .call1((names, values))?
+ .unbind()
+ }
+ Some(_) => py
+ .import("pyspark.sql.connect.client")?
+ .getattr("_UnsupportedLiteral")?
+ .call1((format!("{literal:?}"),))?
+ .unbind(),
+ })
+}
diff --git a/crates/pyspark-rs/src/errors.rs b/crates/pyspark-rs/src/errors.rs
index c591e09..ea079cf 100644
--- a/crates/pyspark-rs/src/errors.rs
+++ b/crates/pyspark-rs/src/errors.rs
@@ -55,6 +55,7 @@ fn kind_to_pyspark_class_name(kind: SparkErrorKind) ->
&'static str {
/// exception based on the error kind. Falls back to PyRuntimeError if import
fails.
pub fn spark_error_to_py_exception(err: SparkError) -> PyErr {
let msg = err.message();
+ let operation_id = err.operation_id.clone();
let class_name = kind_to_pyspark_class_name(err.kind);
// Try to acquire the GIL and import the exception class from
pyspark.errors
@@ -75,6 +76,11 @@ pub fn spark_error_to_py_exception(err: SparkError) -> PyErr
{
// This creates a PyObject exception instance
match exc_class.call1((msg.clone(),)) {
Ok(exc_instance) => {
+ // Mirror the reference client, which stamps
`_operation_id` on
+ // errors raised for an ExecutePlan (read via
`operation_id`).
+ if let Some(op) = &operation_id {
+ let _ = exc_instance.setattr("_operation_id", op);
+ }
// Convert the exception instance to a PyErr.
// We use unsafe code to construct the PyErr from the
exception value.
PyErr::from_value(exc_instance)
diff --git a/crates/pyspark-rs/src/lib.rs b/crates/pyspark-rs/src/lib.rs
index e1bd4f4..0394c64 100644
--- a/crates/pyspark-rs/src/lib.rs
+++ b/crates/pyspark-rs/src/lib.rs
@@ -84,8 +84,8 @@ use types::{
PyDayTimeIntervalType, PyDecimalType, PyDoubleType, PyFloatType,
PyFractionalType,
PyGeographyType, PyGeometryType, PyIntegerType, PyIntegralType,
PyLongType, PyMapType,
PyNullType, PyNumericType, PyShortType, PySpatialType, PyStringType,
PyStructField,
- PyStructType, PyTimeType, PyTimestampNTZType, PyTimestampType,
PyVarcharType, PyVariantType,
- PyYearMonthIntervalType,
+ PyStructType, PyTimeType, PyTimestampLTZNanosType,
PyTimestampNTZNanosType, PyTimestampNTZType,
+ PyTimestampType, PyVarcharType, PyVariantType, PyYearMonthIntervalType,
};
use window::{PyFrameBound, PyWindow, PyWindowSpec};
@@ -189,6 +189,8 @@ fn _pyspark(_py: Python<'_>, m: &Bound<'_, PyModule>) ->
PyResult<()> {
m.add_class::<PyDateType>()?;
m.add_class::<PyTimestampType>()?;
m.add_class::<PyTimestampNTZType>()?;
+ m.add_class::<PyTimestampNTZNanosType>()?;
+ m.add_class::<PyTimestampLTZNanosType>()?;
m.add_class::<PyArrayType>()?;
m.add_class::<PyMapType>()?;
m.add_class::<PyStructField>()?;
diff --git a/crates/pyspark-rs/src/pipelines.rs
b/crates/pyspark-rs/src/pipelines.rs
index 40c41b8..c8f5f55 100644
--- a/crates/pyspark-rs/src/pipelines.rs
+++ b/crates/pyspark-rs/src/pipelines.rs
@@ -162,12 +162,15 @@ pub fn pipeline_define_flow(
}
/// `PipelineCommand.DefineFlow` carrying an Auto-CDC flow body.
`stored_as_scd_type` is
-/// the `SCDType` enum value (0 unspecified, 1 SCD_TYPE_1, 2 SCD_TYPE_2).
+/// the `SCDType` enum value (0 unspecified, 1 SCD_TYPE_1, 2 SCD_TYPE_2). The
SCD2-only
+/// track-history lists are validated client-side in `pipelines/api.py`;
`sql_conf` is
+/// forwarded as given.
#[pyfunction]
#[pyo3(signature = (
session, dataflow_graph_id, flow_name, target, source, keys, sequence_by,
apply_as_deletes=None, column_list=None, except_column_list=None,
- stored_as_scd_type=0, scl_file=None, scl_line=None
+ stored_as_scd_type=0, scl_file=None, scl_line=None,
+ track_history_column_list=None, track_history_except_column_list=None,
sql_conf=None
))]
#[allow(clippy::too_many_arguments)]
pub fn pipeline_define_auto_cdc_flow(
@@ -184,27 +187,28 @@ pub fn pipeline_define_auto_cdc_flow(
stored_as_scd_type: i32,
scl_file: Option<String>,
scl_line: Option<i32>,
+ track_history_column_list: Option<Vec<PyRef<'_, PyColumn>>>,
+ track_history_except_column_list: Option<Vec<PyRef<'_, PyColumn>>>,
+ sql_conf: Option<HashMap<String, String>>,
) -> PyResult<()> {
let to_expr = |c: &PyColumn| c.column.expression().to_proto();
+ let to_exprs = |cs: Option<Vec<PyRef<'_, PyColumn>>>| {
+ cs.unwrap_or_default()
+ .iter()
+ .map(|c| to_expr(c))
+ .collect::<Vec<_>>()
+ };
let details = proto::pipeline_command::define_flow::AutoCdcFlowDetails {
source: Some(source),
keys: keys.iter().map(|c| to_expr(c)).collect(),
sequence_by: Some(to_expr(&sequence_by)),
apply_as_deletes: apply_as_deletes.as_ref().map(|c| to_expr(c)),
apply_as_truncates: None,
- column_list: column_list
- .unwrap_or_default()
- .iter()
- .map(|c| to_expr(c))
- .collect(),
- except_column_list: except_column_list
- .unwrap_or_default()
- .iter()
- .map(|c| to_expr(c))
- .collect(),
+ column_list: to_exprs(column_list),
+ except_column_list: to_exprs(except_column_list),
stored_as_scd_type,
- track_history_column_list: vec![],
- track_history_except_column_list: vec![],
+ track_history_column_list: to_exprs(track_history_column_list),
+ track_history_except_column_list:
to_exprs(track_history_except_column_list),
ignore_null_updates_column_list: vec![],
ignore_null_updates_except_column_list: vec![],
};
@@ -217,7 +221,17 @@ pub fn pipeline_define_auto_cdc_flow(
let scl = scl(scl_file, scl_line);
session
.py()
- .detach(|| pipelines::define_auto_cdc_flow(&sess, &gid, &fname, &tgt,
details, scl))
+ .detach(|| {
+ pipelines::define_auto_cdc_flow(
+ &sess,
+ &gid,
+ &fname,
+ &tgt,
+ details,
+ sql_conf.unwrap_or_default(),
+ scl,
+ )
+ })
.to_pyerr()
}
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index f81bde1..89dd7ce 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -721,9 +721,52 @@ impl PySparkSession {
PyStreamingQueryManager::new(self.session.streams())
}
- /// Stop this Spark session.
+ /// Stop this Spark session. Best-effort and idempotent, mirroring
pyspark: a failure
+ /// to release the session on the server is logged, not raised. Also
clears the
+ /// active session and the cached `SQLContext` if they belong to this
session.
fn stop(&self, py: Python<'_>) -> PyResult<()> {
- py.detach(|| self.session.stop()).to_pyerr()
+ if let Err(e) = py.detach(|| self.session.stop()) {
+ let logger = py
+ .import("logging")?
+ .call_method1("getLogger", ("pyspark",))?;
+ logger.call_method1(
+ "warning",
+ (format!(
+ "session.stop(): Session could not be released. Error: {e}"
+ ),),
+ )?;
+ }
+ let sid = self.session.session_id();
+ {
+ let mut slot = active_slot().lock().unwrap();
+ if slot.as_ref().is_some_and(|s| s.session_id() == sid) {
+ *slot = None;
+ }
+ }
+ // Only touch the SQLContext cache if the module was ever imported (as
pyspark does).
+ let modules = py.import("sys")?.getattr("modules")?;
+ if let Ok(ctx_mod) = modules.get_item("pyspark.sql.connect.context") {
+ ctx_mod.call_method1("_reset_instantiated_context", (sid,))?;
+ }
+ Ok(())
+ }
+
+ /// Enable `with SparkSession.builder.(...).getOrCreate() as session:`
syntax.
+ fn __enter__(slf: Py<Self>) -> Py<Self> {
+ slf
+ }
+
+ /// Stop the session when leaving a `with` block (mirrors pyspark 4.3.0).
+ #[pyo3(signature = (exc_type=None, exc_value=None, traceback=None))]
+ fn __exit__(
+ &self,
+ py: Python<'_>,
+ exc_type: Option<Py<PyAny>>,
+ exc_value: Option<Py<PyAny>>,
+ traceback: Option<Py<PyAny>>,
+ ) -> PyResult<()> {
+ let _ = (exc_type, exc_value, traceback);
+ self.stop(py)
}
/// Get the UDF registration API (`spark.udf`). Returns the
diff --git a/crates/pyspark-rs/src/types.rs b/crates/pyspark-rs/src/types.rs
index 761c1cb..935fc72 100644
--- a/crates/pyspark-rs/src/types.rs
+++ b/crates/pyspark-rs/src/types.rs
@@ -2085,6 +2085,152 @@ impl PyTimeType {
}
}
+/// Spark's `TimestampNTZNanosType(precision)` (precision 7..=9; default 9).
PySpark 4.3.0 has no
+/// Python class for the nanosecond timestamp types, but the server reports
them in
+/// schemas, so they materialize as this class instead of being mapped to a
lossy type.
+#[pyclass(name = "TimestampNTZNanosType", module = "pyspark.sql.types",
extends = PyDatetimeType)]
+pub struct PyTimestampNTZNanosType {
+ pub precision: i32,
+}
+
+#[pymethods]
+impl PyTimestampNTZNanosType {
+ #[pyo3(name = "json")]
+ fn __obj_json(slf: &Bound<'_, Self>) -> PyResult<String> {
+ Ok(py_to_data_type(slf.as_any())?.json())
+ }
+ #[pyo3(name = "jsonValue")]
+ fn __obj_json_value<'py>(
+ slf: &Bound<'py, Self>,
+ py: Python<'py>,
+ ) -> PyResult<Bound<'py, PyAny>> {
+ let v = py_to_data_type(slf.as_any())?.json_value();
+ py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+ }
+ #[pyo3(name = "needConversion")]
+ fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
+ Ok(py_to_data_type(slf.as_any())?.need_conversion())
+ }
+ #[pyo3(name = "fromInternal")]
+ fn __obj_from_internal<'py>(&self, obj: Bound<'py, PyAny>) -> Bound<'py,
PyAny> {
+ obj
+ }
+ #[pyo3(name = "toInternal")]
+ fn __obj_to_internal<'py>(&self, obj: Bound<'py, PyAny>) -> Bound<'py,
PyAny> {
+ obj
+ }
+ fn __reduce__(&self, py: Python<'_>) -> PyResult<(Py<PyAny>, (String,))> {
+ let dt = DataType::TimestampNtzNanos {
+ precision: self.precision,
+ };
+ type_reduce(py, &dt.json())
+ }
+ #[new]
+ #[pyo3(signature = (precision=9))]
+ fn new(precision: i32) -> PyResult<pyo3::PyClassInitializer<Self>> {
+ if !(7..=9).contains(&precision) {
+ return Err(PyErr::new::<pyo3::exceptions::PyValueError, _>(format!(
+ "TimestampNTZNanosType precision must be in 7..9, got {}",
+ precision
+ )));
+ }
+ Ok(init_chain!(
+ DataType::TimestampNtzNanos { precision },
+ PyTimestampNTZNanosType { precision },
+ [PyAtomicType, PyDatetimeType],
+ value
+ ))
+ }
+ #[getter]
+ fn precision(&self) -> i32 {
+ self.precision
+ }
+ fn __repr__(&self) -> String {
+ format!("TimestampNTZNanosType({})", self.precision)
+ }
+ #[pyo3(name = "simpleString")]
+ fn simple_string(&self) -> String {
+ format!("timestamp_ntz({})", self.precision)
+ }
+ #[pyo3(name = "typeName")]
+ fn type_name(&self) -> String {
+ format!("timestamp_ntz({})", self.precision)
+ }
+}
+
+/// Spark's `TimestampLTZNanosType(precision)` (precision 7..=9; default 9).
PySpark 4.3.0 has no
+/// Python class for the nanosecond timestamp types, but the server reports
them in
+/// schemas, so they materialize as this class instead of being mapped to a
lossy type.
+#[pyclass(name = "TimestampLTZNanosType", module = "pyspark.sql.types",
extends = PyDatetimeType)]
+pub struct PyTimestampLTZNanosType {
+ pub precision: i32,
+}
+
+#[pymethods]
+impl PyTimestampLTZNanosType {
+ #[pyo3(name = "json")]
+ fn __obj_json(slf: &Bound<'_, Self>) -> PyResult<String> {
+ Ok(py_to_data_type(slf.as_any())?.json())
+ }
+ #[pyo3(name = "jsonValue")]
+ fn __obj_json_value<'py>(
+ slf: &Bound<'py, Self>,
+ py: Python<'py>,
+ ) -> PyResult<Bound<'py, PyAny>> {
+ let v = py_to_data_type(slf.as_any())?.json_value();
+ py.import("json")?.getattr("loads")?.call1((v.to_string(),))
+ }
+ #[pyo3(name = "needConversion")]
+ fn __obj_need_conversion(slf: &Bound<'_, Self>) -> PyResult<bool> {
+ Ok(py_to_data_type(slf.as_any())?.need_conversion())
+ }
+ #[pyo3(name = "fromInternal")]
+ fn __obj_from_internal<'py>(&self, obj: Bound<'py, PyAny>) -> Bound<'py,
PyAny> {
+ obj
+ }
+ #[pyo3(name = "toInternal")]
+ fn __obj_to_internal<'py>(&self, obj: Bound<'py, PyAny>) -> Bound<'py,
PyAny> {
+ obj
+ }
+ fn __reduce__(&self, py: Python<'_>) -> PyResult<(Py<PyAny>, (String,))> {
+ let dt = DataType::TimestampLtzNanos {
+ precision: self.precision,
+ };
+ type_reduce(py, &dt.json())
+ }
+ #[new]
+ #[pyo3(signature = (precision=9))]
+ fn new(precision: i32) -> PyResult<pyo3::PyClassInitializer<Self>> {
+ if !(7..=9).contains(&precision) {
+ return Err(PyErr::new::<pyo3::exceptions::PyValueError, _>(format!(
+ "TimestampLTZNanosType precision must be in 7..9, got {}",
+ precision
+ )));
+ }
+ Ok(init_chain!(
+ DataType::TimestampLtzNanos { precision },
+ PyTimestampLTZNanosType { precision },
+ [PyAtomicType, PyDatetimeType],
+ value
+ ))
+ }
+ #[getter]
+ fn precision(&self) -> i32 {
+ self.precision
+ }
+ fn __repr__(&self) -> String {
+ format!("TimestampLTZNanosType({})", self.precision)
+ }
+ #[pyo3(name = "simpleString")]
+ fn simple_string(&self) -> String {
+ format!("timestamp_ltz({})", self.precision)
+ }
+ #[pyo3(name = "typeName")]
+ fn type_name(&self) -> String {
+ format!("timestamp_ltz({})", self.precision)
+ }
+}
+
#[pyclass(name = "CalendarIntervalType", module = "pyspark.sql.types", extends
= PyDataType)]
pub struct PyCalendarIntervalType;
@@ -2620,6 +2766,24 @@ pub(crate) fn data_type_to_py(py: Python<'_>, dt:
&DataType) -> PyResult<Py<PyAn
value
))
}
+ DataType::TimestampNtzNanos { precision } => {
+ let precision = *precision;
+ obj!(init_chain!(
+ DataType::TimestampNtzNanos { precision },
+ PyTimestampNTZNanosType { precision },
+ [PyAtomicType, PyDatetimeType],
+ value
+ ))
+ }
+ DataType::TimestampLtzNanos { precision } => {
+ let precision = *precision;
+ obj!(init_chain!(
+ DataType::TimestampLtzNanos { precision },
+ PyTimestampLTZNanosType { precision },
+ [PyAtomicType, PyDatetimeType],
+ value
+ ))
+ }
DataType::CalendarInterval => obj!(init_chain!(
DataType::CalendarInterval,
PyCalendarIntervalType,
@@ -2744,6 +2908,16 @@ pub(crate) fn py_to_data_type(obj: &Bound<'_, PyAny>) ->
PyResult<DataType> {
if obj.extract::<PyRef<PyTimestampNTZType>>().is_ok() {
return Ok(DataType::TimestampNtz);
}
+ if let Ok(t) = obj.extract::<PyRef<PyTimestampNTZNanosType>>() {
+ return Ok(DataType::TimestampNtzNanos {
+ precision: t.precision,
+ });
+ }
+ if let Ok(t) = obj.extract::<PyRef<PyTimestampLTZNanosType>>() {
+ return Ok(DataType::TimestampLtzNanos {
+ precision: t.precision,
+ });
+ }
if let Ok(d) = obj.extract::<PyRef<PyDecimalType>>() {
return Ok(DataType::Decimal {
precision: d.precision,
diff --git a/crates/spark-connect-core/src/client.rs
b/crates/spark-connect-core/src/client.rs
index 0a71747..948a0a5 100644
--- a/crates/spark-connect-core/src/client.rs
+++ b/crates/spark-connect-core/src/client.rs
@@ -359,7 +359,8 @@ impl SparkConnectClient {
let mut stub = self.stub.clone();
async move {
stub.execute_plan(req).await.map(Response::into_inner) }
})
- .await?;
+ .await
+ .map_err(|e| e.with_operation_id(iter.operation_id()))?;
let retry_policy = self.retry_policy.clone();
Ok(ReattachableResponseStream {
stub: self.stub.clone(),
@@ -977,10 +978,11 @@ impl ReattachableResponseStream {
/// Ends the stream with `error`, releasing the execution as the reference
client does
/// when an execution fails.
+ /// The error carries the ExecutePlan operation id
(`SparkConnectException.operation_id`).
fn fail(&mut self, error: SparkError) -> SparkError {
self.done = true;
self.release_all();
- error
+ error.with_operation_id(self.iter.operation_id())
}
/// Releases the responses up to and including `response_id`, which were
read.
diff --git a/crates/spark-connect-core/src/error.rs
b/crates/spark-connect-core/src/error.rs
index ae5fa3a..c0f0a73 100644
--- a/crates/spark-connect-core/src/error.rs
+++ b/crates/spark-connect-core/src/error.rs
@@ -173,6 +173,9 @@ pub struct SparkError {
/// Raw `grpc-status-details-bin` (a serialized `google.rpc.Status`), if
present.
/// Lets the PyO3 layer reconstruct the exact typed pyspark exception.
pub grpc_details: Option<Vec<u8>>,
+ /// The ExecutePlan operation id the error was raised for, if known. Backs
+ /// `SparkConnectException.operation_id` (new in pyspark 4.3.0).
+ pub operation_id: Option<String>,
}
/// The PySpark exception hierarchy leaf we map to.
@@ -265,6 +268,7 @@ impl SparkError {
server_stacktrace: None,
grpc_code: None,
grpc_details: None,
+ operation_id: None,
}
}
@@ -285,6 +289,7 @@ impl SparkError {
server_stacktrace: None,
grpc_code: None,
grpc_details: None,
+ operation_id: None,
}
}
@@ -303,6 +308,7 @@ impl SparkError {
server_stacktrace: None,
grpc_code: None,
grpc_details: None,
+ operation_id: None,
}
}
@@ -367,6 +373,7 @@ impl SparkError {
server_stacktrace,
grpc_code: None,
grpc_details: None,
+ operation_id: None,
},
None => Self::connect_msg(format!("[{}] {}", status.code(),
message)),
}
@@ -379,6 +386,12 @@ impl SparkError {
err
}
+ /// Attach the ExecutePlan operation id this error was raised for.
+ pub fn with_operation_id(mut self, operation_id: impl Into<String>) ->
Self {
+ self.operation_id = Some(operation_id.into());
+ self
+ }
+
/// Get the error condition (error class).
///
/// Mirrors `PySparkException.getCondition()` from PySpark.
@@ -737,6 +750,16 @@ fn parse_error_conditions_json(
mod tests {
use super::*;
+ #[test]
+ fn test_with_operation_id() {
+ let err = SparkError::connect_msg("boom");
+ assert_eq!(err.operation_id, None);
+ let err = err.with_operation_id("op-123");
+ assert_eq!(err.operation_id.as_deref(), Some("op-123"));
+ let from_status = SparkError::from_grpc_status(Status::internal("x"));
+ assert_eq!(from_status.operation_id, None);
+ }
+
#[test]
fn test_render_simple_message() {
let mut params = BTreeMap::new();
diff --git a/crates/spark-connect/src/dataframe.rs
b/crates/spark-connect/src/dataframe.rs
index 8e0a8ca..e74798c 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -1910,6 +1910,9 @@ fn capture_execution(
info: &mut ExecutionInfo,
session: &SparkSession,
) {
+ if info.operation_id.is_none() && !resp.operation_id.is_empty() {
+ info.operation_id = Some(resp.operation_id.clone());
+ }
if let Some(metrics) = resp.metrics.take() {
info.metrics = Some(metrics);
}
@@ -3296,6 +3299,21 @@ mod plan_construction_tests {
use super::*;
use crate::session::SparkSession;
+ #[test]
+ fn capture_execution_records_first_operation_id() {
+ let spark = session();
+ let mut info = ExecutionInfo::default();
+ let mut resp = proto::ExecutePlanResponse::default();
+ // An empty operation id (e.g. a synthetic response) is not recorded.
+ capture_execution(&mut resp, &mut info, &spark);
+ assert_eq!(info.operation_id, None);
+ resp.operation_id = "op-1".to_string();
+ capture_execution(&mut resp, &mut info, &spark);
+ resp.operation_id = "op-2".to_string();
+ capture_execution(&mut resp, &mut info, &spark);
+ assert_eq!(info.operation_id.as_deref(), Some("op-1"));
+ }
+
fn session() -> SparkSession {
SparkSession::builder()
.remote("sc://localhost:15002")
diff --git a/crates/spark-connect/src/pipelines.rs
b/crates/spark-connect/src/pipelines.rs
index 6f5696f..7c4d492 100644
--- a/crates/spark-connect/src/pipelines.rs
+++ b/crates/spark-connect/src/pipelines.rs
@@ -180,13 +180,14 @@ pub fn define_auto_cdc_flow(
flow_name: &str,
target_dataset_name: &str,
details: define_flow::AutoCdcFlowDetails,
+ sql_conf: HashMap<String, String>,
source_code_location: Option<proto::SourceCodeLocation>,
) -> Result<()> {
let inner = DefineFlow {
dataflow_graph_id: Some(dataflow_graph_id.to_string()),
flow_name: Some(flow_name.to_string()),
target_dataset_name: Some(target_dataset_name.to_string()),
- sql_conf: HashMap::new(),
+ sql_conf,
client_id: None,
source_code_location,
once: None,
diff --git a/crates/spark-connect/src/profiler.rs
b/crates/spark-connect/src/profiler.rs
index 579101a..8957e67 100644
--- a/crates/spark-connect/src/profiler.rs
+++ b/crates/spark-connect/src/profiler.rs
@@ -14,7 +14,7 @@ use spark_connect_core::error::Result;
use spark_connect_proto as proto;
/// Format a protobuf Literal value as a string.
-fn format_literal(literal: &proto::expression::Literal) -> String {
+pub fn format_literal(literal: &proto::expression::Literal) -> String {
use proto::expression::literal::LiteralType;
match &literal.literal_type {
diff --git a/crates/spark-connect/src/session.rs
b/crates/spark-connect/src/session.rs
index 1ca69c6..7791e3c 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -31,6 +31,11 @@ pub struct ExecutionInfo {
/// Observed metrics collected during execution (also where `observe(...)`
/// and UDF-profiler results surface).
pub observed_metrics: Vec<proto::execute_plan_response::ObservedMetrics>,
+ /// The ExecutePlan operation id: assigned by the client before the
request is sent
+ /// (the request's id, or a new UUID) and echoed by the server in
+ /// `ExecutePlanResponse.operation_id`. Mirrors
`ExecutionInfo.operation_id` (new in
+ /// pyspark 4.3.0).
+ pub operation_id: Option<String>,
}
/// A Spark Connect session for interacting with a remote Spark cluster.
@@ -543,10 +548,14 @@ impl SparkSession {
}
/// Stop this Spark session.
+ ///
+ /// Idempotent, mirroring pyspark: the session is marked stopped even if
releasing it
+ /// on the server fails, and stopping an already stopped session is a
no-op.
pub fn stop(&self) -> Result<()> {
- block_on(self.client.release_session())?;
- self.stopped.store(true, Ordering::SeqCst);
- Ok(())
+ if self.stopped.swap(true, Ordering::SeqCst) {
+ return Ok(());
+ }
+ block_on(self.client.release_session())
}
/// Whether `stop()` has been called on this session. Mirrors
@@ -713,6 +722,10 @@ fn datatype_to_arrow(dt: &DataType) ->
Result<arrow::datatypes::DataType> {
// TIMESTAMP (LTZ) and TIMESTAMP_NTZ are both micros; NTZ carries no
zone.
DataType::Timestamp => A::Timestamp(TimeUnit::Microsecond,
Some("UTC".into())),
DataType::TimestampNtz => A::Timestamp(TimeUnit::Microsecond, None),
+ DataType::TimestampLtzNanos { .. } => {
+ A::Timestamp(TimeUnit::Nanosecond, Some("UTC".into()))
+ }
+ DataType::TimestampNtzNanos { .. } =>
A::Timestamp(TimeUnit::Nanosecond, None),
DataType::Time { .. } => A::Time64(TimeUnit::Microsecond),
DataType::Decimal { precision, scale } => A::Decimal128(*precision as
u8, *scale as i8),
DataType::Array {
@@ -870,6 +883,27 @@ fn values_to_arrow(values: &[Value], dt: &DataType) ->
Result<Arc<dyn arrow::arr
};
Ok(Arc::new(arr))
}
+ // Row values are microseconds (collect truncates nanosecond columns
to micros), so
+ // scale them up to the nanosecond unit `to_arrow_type` declares for
these types.
+ DataType::TimestampNtzNanos { .. } | DataType::TimestampLtzNanos { ..
} => {
+ let vals: Result<Vec<Option<i64>>> = values
+ .iter()
+ .map(|v| match v {
+ Value::Null => Ok(None),
+ Value::Timestamp(t) =>
t.checked_mul(1_000).map(Some).ok_or_else(|| {
+ SparkError::connect_msg("Timestamp out of range for
nanosecond precision")
+ }),
+ _ => Err(SparkError::connect_msg("Type mismatch in row
data")),
+ })
+ .collect();
+ let arr = TimestampNanosecondArray::from(vals?);
+ let arr = if matches!(dt, DataType::TimestampNtzNanos { .. }) {
+ arr
+ } else {
+ arr.with_timezone("UTC")
+ };
+ Ok(Arc::new(arr))
+ }
DataType::Decimal { precision, scale } => {
let col_scale = *scale;
let vals: Result<Vec<Option<i128>>> = values
@@ -1237,6 +1271,22 @@ mod tests {
.expect("failed to build session")
}
+ #[test]
+ fn stop_is_idempotent_and_marks_stopped() {
+ // Port 1: nothing listens, so the ReleaseSession RPC fails fast. The
session is
+ // still marked stopped (pyspark's stop() is best-effort) and a second
stop() is a
+ // no-op that does not retry the release.
+ let spark = SparkSession::builder()
+ .remote("sc://localhost:1")
+ .get_or_create()
+ .expect("failed to build session");
+ let other = spark.clone();
+ let _ = spark.stop();
+ assert!(spark.is_stopped());
+ assert!(other.is_stopped(), "stopped flag is shared across clones");
+ assert!(spark.stop().is_ok());
+ }
+
#[test]
fn session_is_not_stopped_initially() {
let spark = session();
@@ -1411,6 +1461,85 @@ mod tests {
);
}
+ #[test]
+ fn input_ipc_scales_nanos_timestamps_and_keeps_zone() {
+ use arrow::array::{ListArray, TimestampNanosecondArray};
+ use arrow::datatypes::{DataType as A, TimeUnit};
+ let ntz = DataType::TimestampNtzNanos { precision: 9 };
+ let ltz = DataType::TimestampLtzNanos { precision: 7 };
+ let array = DataType::Array {
+ element_type: Box::new(ltz.clone()),
+ contains_null: true,
+ };
+ let schema = schema(vec![ntz, ltz, array]);
+ let rows = vec![
+ row(vec![
+ Value::Timestamp(1_700_000_000_123_456),
+ Value::Timestamp(-1),
+ Value::List(vec![Value::Timestamp(2), Value::Null]),
+ ]),
+ row(vec![Value::Null, Value::Null, Value::List(vec![])]),
+ ];
+ let batch = read_batch(&rows, &schema);
+ let column = |i: usize| {
+ batch
+ .column(i)
+ .as_any()
+ .downcast_ref::<TimestampNanosecondArray>()
+ .unwrap()
+ .clone()
+ };
+ assert_eq!(
+ batch.schema().field(0).data_type(),
+ &A::Timestamp(TimeUnit::Nanosecond, None)
+ );
+ assert_eq!(
+ batch.schema().field(1).data_type(),
+ &A::Timestamp(TimeUnit::Nanosecond, Some("UTC".into()))
+ );
+ assert_eq!(
+ column(0),
+
TimestampNanosecondArray::from(vec![Some(1_700_000_000_123_456_000), None])
+ );
+ assert_eq!(
+ column(1),
+ TimestampNanosecondArray::from(vec![Some(-1_000),
None]).with_timezone("UTC")
+ );
+ let list = batch
+ .column(2)
+ .as_any()
+ .downcast_ref::<ListArray>()
+ .unwrap();
+ let first = list.value(0);
+ assert_eq!(
+ first
+ .as_any()
+ .downcast_ref::<TimestampNanosecondArray>()
+ .unwrap(),
+ &TimestampNanosecondArray::from(vec![Some(2_000),
None]).with_timezone("UTC")
+ );
+ assert_eq!(list.value_length(1), 0);
+
+ // An empty input with a nanos schema builds too, as for the micros
types.
+ let empty = rows_to_arrow_ipc(
+ &[],
+ &schema_one(DataType::TimestampNtzNanos { precision: 9 }),
+ );
+ assert!(empty.is_ok(), "{empty:?}");
+
+ let overflow = rows_to_arrow_ipc(
+ &[row(vec![Value::Timestamp(i64::MAX)])],
+ &schema_one(DataType::TimestampNtzNanos { precision: 9 }),
+ )
+ .unwrap_err()
+ .to_string();
+ assert!(overflow.contains("out of range"), "{overflow}");
+ }
+
+ fn schema_one(dt: DataType) -> DataType {
+ schema(vec![dt])
+ }
+
#[test]
fn input_ipc_resolves_duplicate_names_before_numeric_conversion() {
let nested_type = DataType::Struct {
diff --git a/crates/spark-connect/src/types.rs
b/crates/spark-connect/src/types.rs
index fbc55a5..976520d 100644
--- a/crates/spark-connect/src/types.rs
+++ b/crates/spark-connect/src/types.rs
@@ -50,6 +50,13 @@ pub enum DataType {
TimestampNtz,
/// `pyspark.sql.types.TimeType`
Time { precision: i32 },
+ /// Spark's `TimestampNTZNanosType(precision)` (precision 7..=9): a
timestamp
+ /// without time zone with nanosecond-capable fractional seconds. PySpark
4.3.0
+ /// has no Python class for it; this mirrors the server/proto type.
+ TimestampNtzNanos { precision: i32 },
+ /// Spark's `TimestampLTZNanosType(precision)` (precision 7..=9): a
timestamp
+ /// with local time zone with nanosecond-capable fractional seconds.
+ TimestampLtzNanos { precision: i32 },
/// `pyspark.sql.types.CalendarIntervalType`
CalendarInterval,
/// `pyspark.sql.types.YearMonthIntervalType`
@@ -191,6 +198,8 @@ impl DataType {
DataType::Date
| DataType::Timestamp
| DataType::TimestampNtz
+ | DataType::TimestampNtzNanos { .. }
+ | DataType::TimestampLtzNanos { .. }
| DataType::Time { .. }
| DataType::DayTimeInterval { .. }
| DataType::YearMonthInterval { .. }
@@ -229,6 +238,8 @@ impl DataType {
DataType::Timestamp => "timestamp".to_string(),
DataType::TimestampNtz => "timestamp_ntz".to_string(),
DataType::Time { .. } => "time".to_string(),
+ DataType::TimestampNtzNanos { precision } =>
format!("timestamp_ntz({})", precision),
+ DataType::TimestampLtzNanos { precision } =>
format!("timestamp_ltz({})", precision),
DataType::CalendarInterval => "interval".to_string(),
DataType::YearMonthInterval { .. } => "interval".to_string(),
DataType::DayTimeInterval { .. } => "interval".to_string(),
@@ -280,6 +291,8 @@ impl DataType {
DataType::Timestamp => "timestamp".to_string(),
DataType::TimestampNtz => "timestamp_ntz".to_string(),
DataType::Time { precision } => format!("time({})", precision),
+ DataType::TimestampNtzNanos { precision } =>
format!("timestamp_ntz({})", precision),
+ DataType::TimestampLtzNanos { precision } =>
format!("timestamp_ltz({})", precision),
DataType::CalendarInterval => "interval".to_string(),
DataType::YearMonthInterval {
start_field,
@@ -370,6 +383,12 @@ impl DataType {
DataType::Time { precision } => {
serde_json::json!(format!("time({})", precision))
}
+ DataType::TimestampNtzNanos { precision } => {
+ serde_json::json!(format!("timestamp_ntz({})", precision))
+ }
+ DataType::TimestampLtzNanos { precision } => {
+ serde_json::json!(format!("timestamp_ltz({})", precision))
+ }
DataType::CalendarInterval => serde_json::json!("interval"),
DataType::YearMonthInterval {
start_field,
@@ -566,6 +585,22 @@ impl DataType {
time_type.precision = Some(*precision);
proto.kind =
Some(spark_connect_proto::data_type::Kind::Time(time_type));
}
+ DataType::TimestampNtzNanos { precision } => {
+ proto.kind =
Some(spark_connect_proto::data_type::Kind::TimestampNtzNanos(
+ spark_connect_proto::data_type::TimestampNtzNanos {
+ precision: Some(*precision),
+ ..Default::default()
+ },
+ ));
+ }
+ DataType::TimestampLtzNanos { precision } => {
+ proto.kind =
Some(spark_connect_proto::data_type::Kind::TimestampLtzNanos(
+ spark_connect_proto::data_type::TimestampLtzNanos {
+ precision: Some(*precision),
+ ..Default::default()
+ },
+ ));
+ }
DataType::CalendarInterval => {
proto.kind =
Some(spark_connect_proto::data_type::Kind::CalendarInterval(
spark_connect_proto::data_type::CalendarInterval::default(),
@@ -802,12 +837,16 @@ impl DataType {
Some(spark_connect_proto::data_type::Kind::Unparsed(u)) =>
Ok(DataType::Unparsed {
data_type_string: u.data_type_string.clone(),
}),
+ // Precision defaults to 9 when omitted (types.proto).
Some(spark_connect_proto::data_type::Kind::TimestampNtzNanos(tn))
=> {
- let precision = tn.precision.unwrap_or(9);
- Ok(DataType::Time { precision })
+ Ok(DataType::TimestampNtzNanos {
+ precision: tn.precision.unwrap_or(9),
+ })
}
- Some(spark_connect_proto::data_type::Kind::TimestampLtzNanos(_))
=> {
- Ok(DataType::Timestamp)
+ Some(spark_connect_proto::data_type::Kind::TimestampLtzNanos(tl))
=> {
+ Ok(DataType::TimestampLtzNanos {
+ precision: tl.precision.unwrap_or(9),
+ })
}
None => Err(SparkError::connect_msg("DataType kind not set")),
}
@@ -1097,6 +1136,9 @@ fn parse_json_type_string(s: &str) -> Result<DataType> {
s if s.starts_with("char(") => parse_char(s),
s if s.starts_with("varchar(") => parse_varchar(s),
s if s.starts_with("time(") => parse_time(s),
+ s if s.starts_with("timestamp_ntz(") ||
s.starts_with("timestamp_ltz(") => {
+ parse_timestamp_nanos(s)
+ }
s if s.starts_with("interval") => parse_interval_string(s),
s if s.starts_with("string collate") => {
let collation = s.trim_start_matches("string
collate").trim().to_string();
@@ -1313,6 +1355,36 @@ fn parse_time(s: &str) -> Result<DataType> {
Ok(DataType::Time { precision })
}
+/// Parses `timestamp_ntz(p)` / `timestamp_ltz(p)` (p in 7..=9), mirroring the
+/// `TIMESTAMP_{NTZ,LTZ}_NANOS_TYPE` patterns in Scala `DataType.nameToType`.
+fn parse_timestamp_nanos(s: &str) -> Result<DataType> {
+ let (ntz, rest) = if let Some(r) = s.strip_prefix("timestamp_ntz(") {
+ (true, r)
+ } else if let Some(r) = s.strip_prefix("timestamp_ltz(") {
+ (false, r)
+ } else {
+ return Err(SparkError::value(
+ "INVALID_DATATYPE_FORMAT",
+ &[("detail", "Invalid timestamp format")],
+ ));
+ };
+ let precision = rest
+ .strip_suffix(")")
+ .and_then(|p| p.trim().parse::<i32>().ok())
+ .filter(|p| (7..=9).contains(p))
+ .ok_or_else(|| {
+ SparkError::value(
+ "INVALID_DATATYPE_FORMAT",
+ &[("detail", "Invalid timestamp precision (expected 7..9)")],
+ )
+ })?;
+ Ok(if ntz {
+ DataType::TimestampNtzNanos { precision }
+ } else {
+ DataType::TimestampLtzNanos { precision }
+ })
+}
+
fn parse_interval_string(s: &str) -> Result<DataType> {
if s == "interval" {
return Ok(DataType::CalendarInterval);
@@ -1585,6 +1657,9 @@ fn parse_single_type(input: &str) -> Result<DataType> {
parse_varchar(&trimmed)
} else if trimmed.starts_with("time(") {
parse_time(&trimmed)
+ } else if trimmed.starts_with("timestamp_ntz(") ||
trimmed.starts_with("timestamp_ltz(")
+ {
+ parse_timestamp_nanos(&trimmed)
} else if trimmed.starts_with("interval") {
parse_interval_string(&trimmed)
} else if trimmed.starts_with("string collate") {
diff --git a/crates/spark-connect/tests/types_roundtrip.rs
b/crates/spark-connect/tests/types_roundtrip.rs
index 8b2381a..a0516fc 100644
--- a/crates/spark-connect/tests/types_roundtrip.rs
+++ b/crates/spark-connect/tests/types_roundtrip.rs
@@ -31,6 +31,8 @@ fn all_scalar_variants() -> Vec<DataType> {
DataType::Timestamp,
DataType::TimestampNtz,
DataType::Time { precision: 6 },
+ DataType::TimestampNtzNanos { precision: 9 },
+ DataType::TimestampLtzNanos { precision: 7 },
DataType::CalendarInterval,
DataType::YearMonthInterval {
start_field: 0,
@@ -308,3 +310,40 @@ fn from_ddl_full_grammar() {
assert!(DataType::from_ddl("array<").is_err());
assert!(DataType::from_ddl("struct<a:>").is_err());
}
+
+#[test]
+fn timestamp_nanos_types() {
+ use spark_connect_proto::data_type::{Kind, TimestampLtzNanos,
TimestampNtzNanos};
+
+ let ntz = DataType::TimestampNtzNanos { precision: 8 };
+ let ltz = DataType::TimestampLtzNanos { precision: 9 };
+ // Names mirror Scala TimestampNTZNanosType /
TimestampLTZNanosType.typeName.
+ assert_eq!(ntz.simple_string(), "timestamp_ntz(8)");
+ assert_eq!(ntz.type_name(), "timestamp_ntz(8)");
+ assert_eq!(ltz.json(), "\"timestamp_ltz(9)\"");
+ assert!(ntz.need_conversion());
+ assert_eq!(
+ DataType::from_json_str("\"timestamp_ntz(8)\"").unwrap(),
+ ntz
+ );
+ assert_eq!(DataType::from_ddl("timestamp_ltz(9)").unwrap(), ltz);
+ // Precision outside 7..=9 is rejected.
+ assert!(DataType::from_json_str("\"timestamp_ntz(6)\"").is_err());
+
+ // Proto kinds map to the nanos variants (not TIME / micros TIMESTAMP),
and an
+ // omitted precision defaults to 9 per types.proto.
+ let mut p = spark_connect_proto::DataType::default();
+ p.kind = Some(Kind::TimestampNtzNanos(TimestampNtzNanos::default()));
+ assert_eq!(
+ DataType::from_proto(&p).unwrap(),
+ DataType::TimestampNtzNanos { precision: 9 }
+ );
+ p.kind = Some(Kind::TimestampLtzNanos(TimestampLtzNanos {
+ precision: Some(7),
+ ..Default::default()
+ }));
+ assert_eq!(
+ DataType::from_proto(&p).unwrap(),
+ DataType::TimestampLtzNanos { precision: 7 }
+ );
+}
diff --git a/python/.coveragerc b/python/.coveragerc
index 4fe8f83..61a8ea0 100644
--- a/python/.coveragerc
+++ b/python/.coveragerc
@@ -40,6 +40,7 @@ omit =
*/pyspark/sql/variant_utils.py
*/pyspark/sql/datasource_internal.py
*/pyspark/sql/profiler.py
+ */pyspark/sql/metrics.py
*/pyspark/serializers.py
*/pyspark/loose_version.py
*/pyspark/_globals.py
diff --git a/python/pyspark/errors/exceptions/__init__.py
b/python/pyspark/errors/exceptions/__init__.py
index 6cebf44..85add6c 100644
--- a/python/pyspark/errors/exceptions/__init__.py
+++ b/python/pyspark/errors/exceptions/__init__.py
@@ -30,6 +30,22 @@ except ImportError:
QueryContext,
)
+
+def _operation_id(self): # type: ignore[no-untyped-def]
+ """The Spark Connect ExecutePlan operation ID, when available.
+
+ .. versionadded:: 4.3.0
+ """
+ return getattr(self, "_operation_id", None)
+
+
+# pyspark 4.3.0 adds ``SparkConnectException.operation_id``. In this drop-in
the Connect
+# exception classes ARE the canonical base classes
(``connect.SparkConnectException`` aliases
+# ``PySparkException``), so the property lives on that shared base; the Rust
transport sets
+# ``_operation_id`` on errors raised for an ExecutePlan. ``base.py`` is
vendored, hence here.
+if not hasattr(PySparkException, "operation_id"):
+ PySparkException.operation_id = property(_operation_id) # type:
ignore[attr-defined]
+
__all__ = [
"PySparkException",
"PySparkAssertionError",
diff --git a/python/pyspark/pipelines/api.py b/python/pyspark/pipelines/api.py
index 4f734aa..33b7f0c 100644
--- a/python/pyspark/pipelines/api.py
+++ b/python/pyspark/pipelines/api.py
@@ -16,7 +16,7 @@
#
from typing import Callable, Dict, List, Literal, Optional, Union, overload
-from pyspark.errors import PySparkTypeError
+from pyspark.errors import PySparkTypeError, PySparkValueError
from pyspark.pipelines.graph_element_registry import
get_active_graph_element_registry
from pyspark.pipelines.type_error_utils import
validate_optional_list_of_str_arg
from pyspark.pipelines.flow import AutoCdcFlow, Flow, QueryFunction
@@ -536,8 +536,12 @@ def create_auto_cdc_flow(
apply_as_deletes: Optional[Union[str, Column]] = None,
column_list: Optional[Union[List[str], List[Column]]] = None,
except_column_list: Optional[Union[List[str], List[Column]]] = None,
- stored_as_scd_type: Optional[Literal[1, "1"]] = None,
+ stored_as_scd_type: Optional[Literal[1, 2, "1", "2"]] = None,
name: Optional[str] = None,
+ *,
+ track_history_column_list: Optional[Union[List[str], List[Column]]] = None,
+ track_history_except_column_list: Optional[Union[List[str], List[Column]]]
= None,
+ spark_conf: Optional[Dict[str, str]] = None,
) -> None:
"""
Create an Auto CDC flow into the target table from the Change Data Capture
(CDC) source.
@@ -581,10 +585,25 @@ def create_auto_cdc_flow(
PySpark Columns. Only one of column_list and except_column_list can be
specified. When \
this is specified, all columns in the `DataFrame` of the target table
except those in \
this list will be in the output table.
- :param stored_as_scd_type: The SCD type for the target table. Only 1 (or
"1") is supported. \
- When not specified, the server default applies.
+ :param stored_as_scd_type: The SCD type for the target table. 1 (or "1")
and 2 (or "2") are \
+ supported. When not specified, the server default applies.
+ :param track_history_column_list: SCD2-only. Columns whose value change
opens a new history \
+ record; two consecutive upsert events for the same key are coalesced
into the same \
+ history record when they agree on every tracked column. When not
specified, every \
+ eligible selected user column is tracked. This should be a list of
column identifiers \
+ without qualifiers, expressed as either Python strings or PySpark
Columns. Only one of \
+ track_history_column_list and track_history_except_column_list can be
specified, and \
+ both require stored_as_scd_type to be 2.
+ :param track_history_except_column_list: SCD2-only. Columns excluded from
history tracking. \
+ This should be a list of column identifiers without qualifiers,
expressed as either \
+ Python strings or PySpark Columns. Only one of
track_history_column_list and \
+ track_history_except_column_list can be specified, and both require
stored_as_scd_type \
+ to be 2.
:param name: The name of the flow for this create_auto_cdc_flow command.
When unspecified, \
this will build a "default flow" with name equal to the target name.
+ :param spark_conf: A dict whose keys are the conf names and values are the
conf values. \
+ These confs will be set when the flow is executed; they can override
confs set for the \
+ destination, for the pipeline, or on the cluster.
"""
# Lazy import kept to match upstream layering; in this Rust-backed drop-in
the Connect
# functions module is our `pyspark.sql.functions` (there is no separate
connect variant).
@@ -626,6 +645,27 @@ def create_auto_cdc_flow(
except_column_list = _normalize_optional_column_list(
arg_name="except_column_list", column_list=except_column_list
)
+ track_history_column_list = _normalize_optional_column_list(
+ arg_name="track_history_column_list",
column_list=track_history_column_list
+ )
+ track_history_except_column_list = _normalize_optional_column_list(
+ arg_name="track_history_except_column_list",
+ column_list=track_history_except_column_list,
+ )
+
+ # An include/except pair is mutually exclusive. The server enforces this
too, but failing
+ # here avoids a round-trip. An empty list serializes identically to an
omitted one (an unset
+ # repeated field on the wire), so it is a no-op; gate on non-empty lists
rather than
+ # `is not None`.
+ _reject_include_and_except_together(
+ "column_list", column_list, "except_column_list", except_column_list
+ )
+ _reject_include_and_except_together(
+ "track_history_column_list",
+ track_history_column_list,
+ "track_history_except_column_list",
+ track_history_except_column_list,
+ )
if isinstance(sequence_by, str):
sequence_by = _connect_expr(sequence_by)
@@ -651,16 +691,40 @@ def create_auto_cdc_flow(
},
)
- if stored_as_scd_type is not None and str(stored_as_scd_type) != "1":
+ if stored_as_scd_type is not None and str(stored_as_scd_type) not in ("1",
"2"):
raise PySparkTypeError(
errorClass="NOT_EXPECTED_TYPE",
messageParameters={
"arg_name": "stored_as_scd_type",
- "expected_type": "Literal[1, '1']",
+ "expected_type": "Literal[1, 2, '1', '2']",
"arg_type": type(stored_as_scd_type).__name__,
},
)
+ # Track-history columns describe how consecutive upserts are coalesced
into history records,
+ # a notion that only exists under SCD2. Reject them for any other SCD
type. The engine
+ # enforces this too, but failing here gives a clearer, client-side error.
+ #
+ # An empty list serializes identically to an omitted one (an unset
repeated field on the
+ # wire) and is therefore a no-op, so gate on non-empty lists rather than
`is not None`.
+ is_scd2 = stored_as_scd_type is not None and str(stored_as_scd_type) == "2"
+ supplied_track_history_args = [
+ arg_name
+ for arg_name, cols in (
+ ("track_history_column_list", track_history_column_list),
+ ("track_history_except_column_list",
track_history_except_column_list),
+ )
+ if cols
+ ]
+ if not is_scd2 and supplied_track_history_args:
+ raise PySparkValueError(
+ errorClass="INVALID_MULTIPLE_ARGUMENT_CONDITIONS",
+ messageParameters={
+ "arg_names": ", ".join(supplied_track_history_args),
+ "condition": "specified unless stored_as_scd_type is 2 (or
'2')",
+ },
+ )
+
source_code_location = get_caller_source_code_location(stacklevel=1)
flow = AutoCdcFlow(
@@ -673,12 +737,33 @@ def create_auto_cdc_flow(
column_list=column_list,
except_column_list=except_column_list,
stored_as_scd_type=stored_as_scd_type,
+ track_history_column_list=track_history_column_list,
+ track_history_except_column_list=track_history_except_column_list,
+ spark_conf=spark_conf or {},
source_code_location=source_code_location,
)
get_active_graph_element_registry().register_auto_cdc_flow(flow)
+def _reject_include_and_except_together(
+ include_arg_name: str,
+ include_columns: Optional[List[Column]],
+ except_arg_name: str,
+ except_columns: Optional[List[Column]],
+) -> None:
+ """Reject specifying a non-empty include list and a non-empty except list
together.
+
+ An empty list is a no-op (it serializes to an unset repeated field,
indistinguishable from an
+ omitted argument), so only non-empty lists count as "specified".
+ """
+ if include_columns and except_columns:
+ raise PySparkValueError(
+ errorClass="CANNOT_SET_TOGETHER",
+ messageParameters={"arg_list": f"{include_arg_name} and
{except_arg_name}"},
+ )
+
+
def _normalize_optional_column_list(
arg_name: str,
column_list: Optional[Union[List[str], List[Column]]],
diff --git a/python/pyspark/pipelines/spark_connect_graph_element_registry.py
b/python/pyspark/pipelines/spark_connect_graph_element_registry.py
index c84e112..25c2315 100644
--- a/python/pyspark/pipelines/spark_connect_graph_element_registry.py
+++ b/python/pyspark/pipelines/spark_connect_graph_element_registry.py
@@ -53,6 +53,8 @@ _OUTPUT_TYPE_SINK = 4
# spark.connect.PipelineCommand.DefineFlow.SCDType enum values.
_SCD_TYPE_1 = 1
+_SCD_TYPE_2 = 2
+_SCD_TYPE_BY_VALUE = {"1": _SCD_TYPE_1, "2": _SCD_TYPE_2}
def _scl(loc: SourceCodeLocation):
@@ -165,7 +167,12 @@ class
SparkConnectGraphElementRegistry(GraphElementRegistry):
def register_auto_cdc_flow(self, flow: AutoCdcFlow) -> None:
scl_file, scl_line = _scl(flow.source_code_location)
- stored_as_scd_type = _SCD_TYPE_1 if flow.stored_as_scd_type is not
None else 0
+ # SCDType enum value: 0 unspecified, else SCD_TYPE_1 / SCD_TYPE_2.
+ stored_as_scd_type = (
+ _SCD_TYPE_BY_VALUE[str(flow.stored_as_scd_type)]
+ if flow.stored_as_scd_type is not None
+ else 0
+ )
_define_auto_cdc_flow(
self._spark,
self._dataflow_graph_id,
@@ -180,6 +187,13 @@ class
SparkConnectGraphElementRegistry(GraphElementRegistry):
stored_as_scd_type,
scl_file,
scl_line,
+ list(flow.track_history_column_list)
+ if flow.track_history_column_list is not None
+ else None,
+ list(flow.track_history_except_column_list)
+ if flow.track_history_except_column_list is not None
+ else None,
+ dict(flow.spark_conf) if flow.spark_conf else None,
)
def register_sql(self, sql_text: str, file_path: Path) -> None:
diff --git a/python/pyspark/sql/connect/client/__init__.py
b/python/pyspark/sql/connect/client/__init__.py
index 9cd6e43..f8b50e2 100644
--- a/python/pyspark/sql/connect/client/__init__.py
+++ b/python/pyspark/sql/connect/client/__init__.py
@@ -23,3 +23,61 @@ try:
except ImportError:
# If Rust backend not available, guard the export
__all__ = []
+
+
+from typing import Any, Dict, List
+
+from pyspark.sql.metrics import ObservedMetrics
+
+
+class _UnsupportedLiteral:
+ """An observed-metric literal of a kind ``LiteralExpression._to_value``
rejects. The
+ Rust transport decodes eagerly, so it leaves this marker and ``pairs``
raises the
+ reference's ``UNSUPPORTED_LITERAL`` when read, as the reference does."""
+
+ def __init__(self, literal: str):
+ self.literal = literal
+
+ def __repr__(self) -> str:
+ return f"_UnsupportedLiteral({self.literal})"
+
+
+class PlanObservedMetrics(ObservedMetrics):
+ """Observed metrics of one ``observe(...)`` node, as carried by
+ ``DataFrame.executionInfo.flows``. Mirrors
``pyspark.sql.connect.client.core
+ .PlanObservedMetrics``; the Rust transport has already decoded the literal
values."""
+
+ def __init__(self, name: str, metrics: List[Any], keys: List[str]):
+ self._name = name
+ self._metrics = metrics
+ self._keys = keys if keys else [f"observed_metric_{i}" for i in
range(len(metrics))]
+
+ def __repr__(self) -> str:
+ return f"Plan observed({self._name}={self._metrics})"
+
+ @property
+ def name(self) -> str:
+ return self._name
+
+ @property
+ def metrics(self) -> List[Any]:
+ return self._metrics
+
+ @property
+ def pairs(self) -> Dict[str, Any]:
+ for metric in self._metrics:
+ if isinstance(metric, _UnsupportedLiteral):
+ from pyspark.errors import PySparkTypeError
+
+ raise PySparkTypeError(
+ errorClass="UNSUPPORTED_LITERAL",
+ messageParameters={"literal": metric.literal},
+ )
+ return {self._keys[i]: self._metrics[i] for i in
range(len(self._metrics))}
+
+ @property
+ def keys(self) -> List[str]:
+ return self._keys
+
+ def to_dict(self) -> Dict[str, Any]:
+ return {"name": self._name, "keys": self._keys, "pairs": self.pairs}
diff --git a/python/pyspark/sql/connect/context.py
b/python/pyspark/sql/connect/context.py
new file mode 100644
index 0000000..2015eae
--- /dev/null
+++ b/python/pyspark/sql/connect/context.py
@@ -0,0 +1,255 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Spark Connect SQLContext/HiveContext, ported from the v4.3.0
+``pyspark/sql/connect/context.py``. Fork-adapted (not vendored): it wraps the
Rust-backed
+:class:`SparkSession` and imports from the drop-in's own modules instead of the
+gRPC-client internals."""
+import warnings
+from typing import Any, Callable, ClassVar, List, Optional, Union
+
+from pyspark._globals import _NoValue, _NoValueType
+from pyspark.errors import PySparkNotImplementedError
+from pyspark.sql.types import BooleanType, DataType, StringType, StructField,
StructType
+
+__all__ = ["SQLContext", "HiveContext"]
+
+
+class SQLContext:
+ """The entry point for working with structured data (rows and columns) in
Spark, in Spark 1.x.
+
+ As of Spark 2.0, this is replaced by :class:`SparkSession`. This Spark
Connect
+ implementation wraps a :class:`SparkSession` directly and needs no
``SparkContext``.
+
+ .. deprecated:: 4.3.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+ """
+
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: Any) -> None:
+ warnings.warn(
+ "Deprecated in 4.3.0. Use SparkSession.builder.getOrCreate()
instead.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.sparkSession = sparkSession
+ if type(self)._instantiatedContext is None:
+ type(self)._instantiatedContext = self
+
+ @classmethod
+ def _from_session(cls, sparkSession: Any) -> "SQLContext":
+ """Create a new instance without emitting a deprecation warning."""
+ ctx = object.__new__(cls)
+ ctx.sparkSession = sparkSession
+ return ctx
+
+ @classmethod
+ def _get_or_create_from_session(cls, sparkSession: Any) -> "SQLContext":
+ """Return the cached instance, or create one when the cached context
wraps a
+ different session (e.g. the previous one was stopped and a new one
started)."""
+ if cls._instantiatedContext is None or not _same_session(
+ cls._instantiatedContext.sparkSession, sparkSession
+ ):
+ cls._instantiatedContext = cls._from_session(sparkSession)
+ return cls._instantiatedContext
+
+ def newSession(self) -> "SQLContext":
+ """Returns a new SQLContext as a new session with empty state."""
+ return self._from_session(self.sparkSession.newSession())
+
+ def setConf(self, key: str, value: Union[bool, int, str]) -> None:
+ """Sets the given Spark SQL configuration property."""
+ self.sparkSession.conf.set(key, value)
+
+ def getConf(
+ self, key: str, defaultValue: Union[Optional[str], _NoValueType] =
_NoValue
+ ) -> Optional[str]:
+ """Returns the value of Spark SQL configuration property for the given
key."""
+ if defaultValue is _NoValue:
+ return self.sparkSession.conf.get(key)
+ return self.sparkSession.conf.get(key, defaultValue)
+
+ @property
+ def udf(self) -> Any:
+ """Returns a :class:`UDFRegistration` for UDF registration."""
+ return self.sparkSession.udf
+
+ @property
+ def udtf(self) -> Any:
+ """Returns a :class:`UDTFRegistration` for UDTF registration."""
+ return self.sparkSession.udtf
+
+ def range(
+ self,
+ start: int,
+ end: Optional[int] = None,
+ step: int = 1,
+ numPartitions: Optional[int] = None,
+ ) -> Any:
+ """Create a :class:`DataFrame` with a single LongType column named
``id``."""
+ return self.sparkSession.range(start, end, step, numPartitions)
+
+ def registerFunction(
+ self, name: str, f: Callable[..., Any], returnType: Optional[DataType]
= None
+ ) -> Any:
+ """An alias for :func:`spark.udf.register`.
+
+ .. deprecated:: 4.3.0
+ Use :func:`spark.udf.register` instead.
+ """
+ warnings.warn("Deprecated in 4.3.0. Use spark.udf.register instead.",
FutureWarning)
+ return self.sparkSession.udf.register(name, f, returnType)
+
+ def registerJavaFunction(
+ self, name: str, javaClassName: str, returnType: Optional[DataType] =
None
+ ) -> None:
+ """Not supported in Spark Connect."""
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={"feature": "registerJavaFunction"},
+ )
+
+ def createDataFrame(
+ self,
+ data: Any,
+ schema: Any = None,
+ samplingRatio: Optional[float] = None,
+ verifySchema: Optional[bool] = None,
+ ) -> Any:
+ """Creates a :class:`DataFrame` from local data."""
+ return self.sparkSession.createDataFrame(data, schema, samplingRatio,
verifySchema)
+
+ def registerDataFrameAsTable(self, df: Any, tableName: str) -> None:
+ """Registers the given :class:`DataFrame` as a temporary table in the
catalog."""
+ df.createOrReplaceTempView(tableName)
+
+ def dropTempTable(self, tableName: str) -> None:
+ """Remove the temporary table from catalog."""
+ self.sparkSession.catalog.dropTempView(tableName)
+
+ def createExternalTable(
+ self,
+ tableName: str,
+ path: Optional[str] = None,
+ source: Optional[str] = None,
+ schema: Optional[StructType] = None,
+ **options: str,
+ ) -> Any:
+ """Creates an external table based on the dataset in a data source."""
+ return self.sparkSession.catalog.createExternalTable(
+ tableName, path, source, schema, **options
+ )
+
+ def sql(self, sqlQuery: str) -> Any:
+ """Returns a :class:`DataFrame` representing the result of the given
query."""
+ return self.sparkSession.sql(sqlQuery)
+
+ def table(self, tableName: str) -> Any:
+ """Returns the specified table or view as a :class:`DataFrame`."""
+ return self.sparkSession.table(tableName)
+
+ def tables(self, dbName: Optional[str] = None) -> Any:
+ """Returns a :class:`DataFrame` of (namespace, tableName, isTemporary)
for the tables
+ in the given (or current) database."""
+ schema = StructType(
+ [
+ StructField("namespace", StringType(), nullable=True),
+ StructField("tableName", StringType(), nullable=True),
+ StructField("isTemporary", BooleanType(), nullable=False),
+ ]
+ )
+ # catalog.listTables() rather than SHOW TABLES keeps the column names
fixed, and the
+ # full namespace is joined ("a.b") to match classic SHOW TABLES.
+ rows = [
+ (".".join(t.namespace) if t.namespace else "", t.name,
t.isTemporary)
+ for t in self.sparkSession.catalog.listTables(dbName)
+ ]
+ return self.sparkSession.createDataFrame(rows, schema)
+
+ def tableNames(self, dbName: Optional[str] = None) -> List[str]:
+ """Returns a list of names of tables in the database ``dbName``."""
+ return [t.name for t in self.sparkSession.catalog.listTables(dbName)]
+
+ def cacheTable(self, tableName: str) -> None:
+ """Caches the specified table in-memory."""
+ self.sparkSession.catalog.cacheTable(tableName)
+
+ def uncacheTable(self, tableName: str) -> None:
+ """Removes the specified table from the in-memory cache."""
+ self.sparkSession.catalog.uncacheTable(tableName)
+
+ def clearCache(self) -> None:
+ """Removes all cached tables from the in-memory cache."""
+ self.sparkSession.catalog.clearCache()
+
+ @property
+ def read(self) -> Any:
+ """Returns a :class:`DataFrameReader`."""
+ return self.sparkSession.read
+
+ @property
+ def readStream(self) -> Any:
+ """Returns a :class:`DataStreamReader`."""
+ return self.sparkSession.readStream
+
+ @property
+ def streams(self) -> Any:
+ """Returns a :class:`StreamingQueryManager`."""
+ return self.sparkSession.streams
+
+
+class HiveContext(SQLContext):
+ """Not supported in Spark Connect.
+
+ .. deprecated:: 4.3.0
+ Use SparkSession.builder.enableHiveSupport().getOrCreate().
+ """
+
+ # Override to prevent inheriting SQLContext's cached instance, which would
cause
+ # _get_or_create_from_session to skip _from_session and return the wrong
type.
+ _instantiatedContext: ClassVar[Optional["SQLContext"]] = None
+
+ def __init__(self, sparkSession: Any) -> None:
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={"feature": "HiveContext"},
+ )
+
+ @classmethod
+ def _from_session(cls, sparkSession: Any) -> "SQLContext":
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={"feature": "HiveContext"},
+ )
+
+
+def _same_session(a: Any, b: Any) -> bool:
+ """Session identity. ``SparkSession.getActiveSession()`` returns a fresh
Rust-backed
+ wrapper per call, so fall back to comparing session ids when the objects
differ."""
+ if a is b:
+ return True
+ sid = getattr(a, "session_id", None)
+ return sid is not None and sid == getattr(b, "session_id", None)
+
+
+def _reset_instantiated_context(session_id: str) -> None:
+ """Called from ``SparkSession.stop()``: drop the cached context if it
wraps the stopped
+ session. Compared by session id since the Rust-backed session objects are
wrappers."""
+ for cls in (SQLContext, HiveContext):
+ ctx = cls.__dict__.get("_instantiatedContext")
+ if ctx is not None and getattr(ctx.sparkSession, "session_id", None)
== session_id:
+ cls._instantiatedContext = None
diff --git a/python/pyspark/sql/context.py b/python/pyspark/sql/context.py
index b67ce3f..7f5787b 100644
--- a/python/pyspark/sql/context.py
+++ b/python/pyspark/sql/context.py
@@ -15,7 +15,17 @@
# limitations under the License.
#
"""Legacy SQLContext/HiveContext plus the UDF/UDTF registration re-exports that
-pyspark.sql.__init__ imports."""
+pyspark.sql.__init__ imports.
+
+As in pyspark 4.3.0, these are the classic entry points; in Spark Connect mode
(always,
+for this client) ``getOrCreate()`` routes to the SparkSession-backed
counterparts in
+:mod:`pyspark.sql.connect.context`. Constructing them from a ``SparkContext``
is not
+possible here, since a Connect client has no SparkContext.
+"""
+import warnings
+from typing import Any, ClassVar, Optional
+
+from pyspark.errors import PySparkNotImplementedError
from pyspark.sql.udf import UDFRegistration
from pyspark.sql.udtf import UDTFRegistration
@@ -23,13 +33,78 @@ __all__ = ["SQLContext", "HiveContext", "UDFRegistration",
"UDTFRegistration"]
class SQLContext:
- """Deprecated. Retained for import compatibility; use SparkSession."""
+ """The entry point for working with structured data in Spark 1.x.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+ """
+
+ _instantiatedContext: ClassVar[Optional[Any]] = None
- def __init__(self, *args, **kwargs) -> None:
- raise NotImplementedError(
- "SQLContext is not supported in the Spark Connect client; use
SparkSession."
+ def __init__(self, sparkContext: Any, sparkSession: Any = None,
jsqlContext: Any = None):
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={
+ "feature": f"{type(self).__name__}(sparkContext) in Spark
Connect; "
+ f"use {type(self).__name__}.getOrCreate()"
+ },
)
+ @classmethod
+ def getOrCreate(cls, sc: Any = None) -> Any:
+ """Get the existing SQLContext or create one for the active
SparkSession.
+
+ In Spark Connect mode ``sc`` is not required and the active
+ :class:`SparkSession` is used.
+
+ .. deprecated:: 3.0.0
+ Use :func:`SparkSession.builder.getOrCreate()` instead.
+ """
+ from pyspark.sql.utils import is_remote
+
+ warnings.warn(
+ "Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate()
instead.",
+ FutureWarning,
+ )
+ if not is_remote(): # pragma: no cover - this client is always remote
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={"feature": "SQLContext outside Spark
Connect"},
+ )
+ from pyspark.sql.connect import context as _connect_context
+
+ session = _get_active_session_or_create()
+ # Route to the Connect counterpart by name so subclasses (HiveContext)
are handled:
+ # the Connect HiveContext._from_session raises
PySparkNotImplementedError.
+ connect_cls = getattr(_connect_context, cls.__name__, None)
+ if connect_cls is None:
+ # A user-defined subclass has no Connect counterpart; fail loudly
rather than
+ # return a base context missing the subclass's attributes.
+ raise PySparkNotImplementedError(
+ errorClass="NOT_IMPLEMENTED",
+ messageParameters={"feature": f"{cls.__name__}.getOrCreate in
Spark Connect"},
+ )
+ return connect_cls._get_or_create_from_session(session)
+
class HiveContext(SQLContext):
- """Deprecated. Retained for import compatibility; use SparkSession."""
+ """A variant of Spark SQL that integrates with data stored in Hive.
+
+ .. deprecated:: 2.0.0
+ Use SparkSession.builder.enableHiveSupport().getOrCreate().
+ """
+
+
+def _get_active_session_or_create() -> Any:
+ """Mirror ``SparkSession._getActiveSessionOrCreate`` for the Rust-backed
session: the
+ active session, else a new one against ``SPARK_REMOTE``."""
+ import os
+
+ from pyspark.sql.session import SparkSession
+
+ session = SparkSession.getActiveSession()
+ if session is None:
+ session = SparkSession.builder.remote(
+ os.environ.get("SPARK_REMOTE", "sc://localhost")
+ ).getOrCreate()
+ return session
diff --git a/python/pyspark/sql/functions.py b/python/pyspark/sql/functions.py
index cf847aa..0e75123 100644
--- a/python/pyspark/sql/functions.py
+++ b/python/pyspark/sql/functions.py
@@ -373,6 +373,13 @@ def variant_delete(v, *paths):
v: a variant column
*paths: one or more path expressions (Column or str)
"""
+ if len(paths) == 0:
+ from pyspark.errors import PySparkValueError
+
+ raise PySparkValueError(
+ errorClass="CANNOT_BE_EMPTY",
+ messageParameters={"item": "paths"},
+ )
_a = [v]
_a.extend(paths)
return _dispatch("variant_delete", _a)
@@ -513,6 +520,29 @@ def lead(col, offset=1, default=None):
return _wrap(_call_function("lead", _to_col(col), _pyfunc_lit(offset),
_pyfunc_lit(default)))
+def instr(str, substr, start=None, occurrence=None):
+ """instr(str, substr, start=None, occurrence=None): 1-based position of
substr.
+
+ Mirrors pyspark.sql.connect.functions.instr: ``occurrence`` without
``start``
+ searches from position 1.
+ """
+ # Non-column args stay raw; the core turns them into literals (as `lit`
would).
+ args = [str, substr]
+ if start is not None or occurrence is not None:
+ args.append(start if start is not None else 1)
+ if occurrence is not None:
+ args.append(occurrence)
+ return _dispatch("instr", args)
+
+
+def regexp_replace(string, pattern, replacement, position=None):
+ """regexp_replace(string, pattern, replacement, position=None)."""
+ args = [string, pattern, replacement]
+ if position is not None:
+ args.append(position)
+ return _dispatch("regexp_replace", args)
+
+
def json_tuple(col, *fields):
"""json_tuple(col, *fields): extract the given JSON fields (literal names)
as columns."""
if not fields:
diff --git a/python/pyspark/sql/metrics.py b/python/pyspark/sql/metrics.py
new file mode 100644
index 0000000..7af2386
--- /dev/null
+++ b/python/pyspark/sql/metrics.py
@@ -0,0 +1,323 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+import abc
+import dataclasses
+from typing import Optional, List, Tuple, Dict, Any, Union, TYPE_CHECKING,
Sequence
+
+from pyspark.errors import PySparkValueError
+
+if TYPE_CHECKING:
+ try:
+ import graphviz # type: ignore
+ except ImportError:
+ pass
+
+
+class ObservedMetrics(abc.ABC):
+ @property
+ @abc.abstractmethod
+ def name(self) -> str: ...
+
+ @property
+ @abc.abstractmethod
+ def pairs(self) -> Dict[str, Any]: ...
+
+ @property
+ @abc.abstractmethod
+ def keys(self) -> List[str]: ...
+
+
+class MetricValue:
+ """The metric values is the Python representation of a plan metric value
from the JVM.
+ However, it does not have any reference to the original value."""
+
+ def __init__(self, name: str, value: Union[int, float], type: str):
+ self._name = name
+ self._type = type
+ self._value = value
+
+ def __repr__(self) -> str:
+ return f"<{self._name}={self._value} ({self._type})>"
+
+ @property
+ def name(self) -> str:
+ return self._name
+
+ @property
+ def value(self) -> Union[int, float]:
+ return self._value
+
+ @property
+ def metric_type(self) -> str:
+ return self._type
+
+ def to_dict(self) -> Dict[str, Any]:
+ """Return a JSON-serializable dictionary representation of this metric
value.
+
+ Returns
+ -------
+ dict
+ A dictionary with keys 'name', 'value', and 'type'.
+ """
+ return {
+ "name": self._name,
+ "value": self._value,
+ "type": self._type,
+ }
+
+
+class PlanMetrics:
+ """Represents a particular plan node and the associated metrics of this
node."""
+
+ def __init__(self, name: str, id: int, parent: int, metrics:
List[MetricValue]):
+ self._name = name
+ self._id = id
+ self._parent_id = parent
+ self._metrics = metrics
+
+ def __repr__(self) -> str:
+ return f"Plan({self._name}:
{self._id}->{self._parent_id})={self._metrics}"
+
+ @property
+ def name(self) -> str:
+ return self._name
+
+ @property
+ def plan_id(self) -> int:
+ return self._id
+
+ @property
+ def parent_plan_id(self) -> int:
+ return self._parent_id
+
+ @property
+ def metrics(self) -> List[MetricValue]:
+ return self._metrics
+
+ def to_dict(self) -> Dict[str, Any]:
+ """Return a JSON-serializable dictionary representation of this plan
metrics.
+
+ Returns
+ -------
+ dict
+ A dictionary with keys 'name', 'plan_id', 'parent_plan_id', and
'metrics'.
+ """
+ return {
+ "name": self._name,
+ "plan_id": self._id,
+ "parent_plan_id": self._parent_id,
+ "metrics": [m.to_dict() for m in self._metrics],
+ }
+
+
+class CollectedMetrics:
+ @dataclasses.dataclass
+ class Node:
+ id: int
+ name: str = dataclasses.field(default="")
+ metrics: List[MetricValue] = dataclasses.field(default_factory=list)
+ children: List[int] = dataclasses.field(default_factory=list)
+
+ def text(self, current: "Node", graph: Dict[int, "Node"], prefix: str =
"") -> str:
+ """
+ Converts the current node and its children into a textual
representation. This is used
+ to provide a usable output for the command line or other text-based
interfaces. However,
+ it is recommended to use the Graphviz representation for a more visual
representation.
+
+ Parameters
+ ----------
+ current: Node
+ Current node in the graph.
+ graph: dict
+ A dictionary representing the full graph mapping from node ID
(int) to the node itself.
+ The node is an instance of :class:`CollectedMetrics:Node`.
+ prefix: str
+ String prefix used for generating the output buffer.
+
+ Returns
+ -------
+ The full string representation of the current node as root.
+ """
+ base_metrics = set(["numPartitions", "peakMemory", "numOutputRows",
"spillSize"])
+
+ # Format the metrics of this node:
+ metric_buffer = []
+ for m in current.metrics:
+ if m.name in base_metrics:
+ metric_buffer.append(f"{m.name}: {m.value} ({m.metric_type})")
+
+ buffer = f"{prefix}+- {current.name}({','.join(metric_buffer)})\n"
+ for i, child in enumerate(current.children):
+ c = graph[child]
+ new_prefix = prefix + " " if i == len(c.children) - 1 else prefix
+ if current.id != c.id:
+ buffer += self.text(c, graph, new_prefix)
+ return buffer
+
+ def __init__(self, metrics: List[PlanMetrics]):
+ # Sort the input list
+ self._metrics = sorted(metrics, key=lambda x: x._parent_id,
reverse=False)
+
+ def extract_graph(self) -> Tuple[int, Dict[int, "CollectedMetrics.Node"]]:
+ """
+ Builds the graph of the query plan. The graph is represented as a
dictionary where the key
+ is the node ID and the value is the node itself. The root node is the
node that has no
+ parent.
+
+ Returns
+ -------
+ The root node ID and the graph of all nodes.
+ """
+ all_nodes: Dict[int, CollectedMetrics.Node] = {}
+
+ for m in self._metrics:
+ # Add yourself to the list if you have to.
+ if m.plan_id not in all_nodes:
+ all_nodes[m.plan_id] = CollectedMetrics.Node(m.plan_id,
m.name, m.metrics)
+ else:
+ all_nodes[m.plan_id].name = m.name
+ all_nodes[m.plan_id].metrics = m.metrics
+
+ # Now check for the parent of this node if it's in
+ if m.parent_plan_id not in all_nodes:
+ all_nodes[m.parent_plan_id] =
CollectedMetrics.Node(m.parent_plan_id)
+
+ all_nodes[m.parent_plan_id].children.append(m.plan_id)
+
+ # Next step is to find all the root nodes. Root nodes are never used
in children.
+ # So we start with all node ids as candidates.
+ candidates = set(all_nodes.keys())
+ for k, v in all_nodes.items():
+ for c in v.children:
+ if c in candidates and c != k:
+ candidates.remove(c)
+
+ assert len(candidates) == 1, f"Expected 1 root node, found
{len(candidates)}"
+ return candidates.pop(), all_nodes
+
+ def toText(self) -> str:
+ """
+ Converts the execution graph from a graph into a textual representation
+ that can be read at the command line for example.
+
+ Returns
+ -------
+ A string representation of the collected metrics.
+ """
+ root, graph = self.extract_graph()
+ return self.text(graph[root], graph)
+
+ def toDot(self, filename: Optional[str] = None, out_format: str = "png")
-> "graphviz.Digraph":
+ """
+ Converts the collected metrics into a dot representation. Since the
graphviz Digraph
+ implementation provides the ability to render the result graph
directory in a
+ notebook, we return the graph object directly.
+
+ If the graphviz package is not available, a PACKAGE_NOT_INSTALLED
error is raised.
+
+ Parameters
+ ----------
+ filename : str, optional
+ The filename to save the graph to given an output format. The path
can be
+ relative or absolute.
+
+ out_format : str
+ The output format of the graph. The default is 'png'.
+
+ Returns
+ -------
+ An instance of the graphviz.Digraph object.
+ """
+ try:
+ import graphviz
+
+ dot = graphviz.Digraph(
+ comment="Query Plan",
+ node_attr={
+ "shape": "box",
+ "font-size": "10pt",
+ },
+ )
+
+ root, graph = self.extract_graph()
+ for k, v in graph.items():
+ # Build table rows for the metrics
+ rows = "\n".join(
+ [
+ (
+ f'<TR><TD><FONT
POINT-SIZE="8">{x.name}</FONT></TD><TD>'
+ f'<FONT POINT-SIZE="8">{x.value}
({x.metric_type})</FONT></TD></TR>'
+ )
+ for x in v.metrics
+ ]
+ )
+
+ dot.node(
+ str(k),
+ """<<TABLE BORDER="0" CELLBORDER="1" CELLSPACING="0">
+ <TR>
+ <TD COLSPAN="2" BGCOLOR="lightgrey">
+ <FONT POINT-SIZE=\"10\">{}</FONT>
+ </TD>
+ </TR>
+ <TR><TD COLSPAN="2"><FONT
POINT-SIZE=\"10\">Metrics</FONT></TD></TR>
+ {}
+ </TABLE>>""".format(v.name, rows),
+ )
+ for c in v.children:
+ dot.edge(str(k), str(c))
+
+ if filename:
+ dot.render(filename, format=out_format, cleanup=True)
+ return dot
+
+ except ImportError:
+ raise PySparkValueError(
+ errorClass="PACKAGE_NOT_INSTALLED",
+ messageParameters={"package_name": "graphviz",
"minimum_version": "0.20"},
+ )
+
+
+class ExecutionInfo:
+ """The ExecutionInfo class allows users to inspect the query execution of
this particular
+ data frame. This value is only set in the data frame if it was executed."""
+
+ def __init__(
+ self,
+ metrics: Optional[list[PlanMetrics]],
+ obs: Optional[Sequence[ObservedMetrics]],
+ operation_id: Optional[str] = None,
+ ):
+ self._metrics = CollectedMetrics(metrics) if metrics else None
+ self._observations = obs if obs else []
+ self._operation_id = operation_id
+
+ @property
+ def metrics(self) -> Optional[CollectedMetrics]:
+ return self._metrics
+
+ @property
+ def flows(self) -> List[Tuple[str, Dict[str, Any]]]:
+ return [(f.name, f.pairs) for f in self._observations]
+
+ @property
+ def operation_id(self) -> Optional[str]:
+ """The Spark Connect ExecutePlan operation ID, when available.
+
+ .. versionadded:: 4.3.0
+ """
+ return self._operation_id
diff --git a/python/pyspark/sql/types.py b/python/pyspark/sql/types.py
index ce52e5b..3e8a051 100644
--- a/python/pyspark/sql/types.py
+++ b/python/pyspark/sql/types.py
@@ -17,6 +17,8 @@ try:
AnyTimeType, AnsiIntervalType, SpatialType,
GeometryType, GeographyType,
VariantVal, Geometry, Geography,
+ # Spark's nanosecond timestamp types (no PySpark 4.3.0 class; not in
__all__).
+ TimestampNTZNanosType, TimestampLTZNanosType,
)
# Row lives in the Rust core; upstream code imports it from
pyspark.sql.types too.
from pyspark._pyspark import Row
@@ -313,6 +315,10 @@ def _parse_datatype_json_value(v):
m = _re.match(r"time\((\d+)\)$", v)
if m:
return TimeType(int(m.group(1)))
+ m = _re.match(r"timestamp_(ntz|ltz)\(\s*(\d+)\s*\)$", v)
+ if m:
+ cls = TimestampNTZNanosType if m.group(1) == "ntz" else
TimestampLTZNanosType
+ return cls(int(m.group(2)))
if v == "calendarinterval":
return CalendarIntervalType()
_ym = {"year": 0, "month": 1}
diff --git a/python/tests/_mock_connect_server.py
b/python/tests/_mock_connect_server.py
index 555f829..72a6bed 100644
--- a/python/tests/_mock_connect_server.py
+++ b/python/tests/_mock_connect_server.py
@@ -39,6 +39,44 @@ def message_field(field, value):
return varint(field * 8 + 2) + varint(len(value)) + value
+def varint_field(field, value):
+ return varint(field * 8) + varint(value & (2**64 - 1))
+
+
+def parse_message(data):
+ """Decode protobuf wire bytes into {field: [value, ...]}: varints as ints
and
+ length-delimited fields as bytes (decode nested messages with another
call)."""
+ fields, i = {}, 0
+
+ def read_varint():
+ nonlocal i
+ shift = result = 0
+ while True:
+ byte = data[i]
+ i += 1
+ result |= (byte & 127) << shift
+ shift += 7
+ if byte < 128:
+ return result
+
+ while i < len(data):
+ key = read_varint()
+ field, wire = key >> 3, key & 7
+ if wire == 0:
+ value = read_varint()
+ elif wire == 2:
+ size = read_varint()
+ value, i = data[i : i + size], i + size
+ elif wire == 1:
+ value, i = data[i : i + 8], i + 8
+ elif wire == 5:
+ value, i = data[i : i + 4], i + 4
+ else:
+ raise ValueError(f"unsupported wire type {wire}")
+ fields.setdefault(field, []).append(value)
+ return fields
+
+
def struct_schema(name="id"):
# types.proto: DataType.long=7, .struct=21; Struct.fields=1;
# StructField: name=1, data_type=2, nullable=3, metadata=4.
@@ -81,12 +119,36 @@ def serve(connection):
# base.proto: AnalyzePlanResponse.schema=2, Schema.schema=1.
return message_field(2, message_field(1, self.data_type))
+ # ExecutePlan records each request and streams one response carrying
+ # `execute_payload` (raw ExecutePlanResponse fields), then
ResultComplete; with
+ # `execute_failure` set it aborts instead. Responses echo the request's
+ # operation id, as the server does.
+ execute_requests = []
+ execute_payload = b""
+ execute_failure = None
+
+ def execute(self, request, context):
+ self.execute_requests.append(request)
+ if self.execute_failure:
+ context.abort(grpc.StatusCode.INVALID_ARGUMENT,
self.execute_failure)
+ # ExecutePlanRequest.operation_id=6;
ExecutePlanResponse.operation_id=12,
+ # response_id=13, result_complete=14.
+ operation_id = parse_message(request).get(6, [b""])[0]
+ header = message_field(12, operation_id)
+ yield header + message_field(13, "r1") + self.execute_payload
+ yield header + message_field(13, "r2") + message_field(14, b"")
+
service = Service()
with ThreadPoolExecutor(max_workers=4) as executor:
server = grpc.server(executor)
handlers = {
"AnalyzePlan":
grpc.unary_unary_rpc_method_handler(service.analyze),
"ReleaseSession": grpc.unary_unary_rpc_method_handler(lambda
request, context: b""),
+ "ExecutePlan":
grpc.unary_stream_rpc_method_handler(service.execute),
+ "ReattachExecute": grpc.unary_stream_rpc_method_handler(
+ lambda request, context: iter([message_field(14, b"")])
+ ),
+ "ReleaseExecute": grpc.unary_unary_rpc_method_handler(lambda
request, context: b""),
}
server.add_generic_rpc_handlers(
(grpc.method_handlers_generic_handler("spark.connect.SparkConnectService",
handlers),)
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 2a027bb..209a4b6 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -13,7 +13,13 @@ from threading import Barrier
import pytest
-from _mock_connect_server import message_field, mock_connect_session,
struct_schema
+from _mock_connect_server import (
+ message_field,
+ mock_connect_session,
+ parse_message,
+ struct_schema,
+ varint_field,
+)
from pyspark.sql import SparkSession, types as T
# Collected here so the existing offline CI suite exercises real Arrow results.
@@ -3228,3 +3234,311 @@ def
test_schema_cache_metadata_wait_releases_gil(mock_connect, accessor):
else:
assert result == [("id", "bigint")]
assert service.calls == 1
+
+
+# ---------------------------------------------------------------------------
4.3.0 gaps
+
+
+def test_timestamp_nanos_types_roundtrip_and_pickle():
+ # Spark's TimestampNTZNanosType/TimestampLTZNanosType (precision 7..9)
materialize as
+ # their own types, not as TimeType / a microsecond TimestampType.
+ for cls, name in ((T.TimestampNTZNanosType, "timestamp_ntz"),
+ (T.TimestampLTZNanosType, "timestamp_ltz")):
+ t = cls(8)
+ assert t.precision == 8
+ assert t.simpleString() == t.typeName() == f"{name}(8)"
+ assert t.json() == f'"{name}(8)"'
+ assert cls().precision == 9
+ assert isinstance(t, T.DatetimeType)
+ assert repr(pickle.loads(pickle.dumps(t))) == repr(t)
+ assert repr(T._parse_datatype_json_string(f'"{name}(7)"')) ==
repr(cls(7))
+ with pytest.raises(ValueError):
+ cls(6)
+ nested = T.StructType([T.StructField("a",
T.ArrayType(T.TimestampLTZNanosType(9)))])
+ assert pickle.loads(pickle.dumps(nested)).simpleString() ==
"struct<a:array<timestamp_ltz(9)>>"
+
+
+def test_instr_regexp_replace_optional_args():
+ from pyspark.sql import functions as F
+
+ captured = []
+ orig = F._call_function
+ F._call_function = lambda fname, *args: captured.append((fname, args)) or
orig(fname, *args)
+ try:
+ F.instr("s", "a")
+ assert captured[-1][1][1:] == ("a",)
+ # occurrence without start searches from position 1 (pyspark 4.3.0).
+ F.instr("s", "a", occurrence=2)
+ assert captured[-1][1][1:] == ("a", 1, 2)
+ F.instr("s", "a", start=None, occurrence=None)
+ assert len(captured[-1][1]) == 2
+ F.instr("s", "a", 3)
+ assert captured[-1][1][1:] == ("a", 3)
+ F.regexp_replace("s", "a", "b")
+ assert len(captured[-1][1]) == 3
+ F.regexp_replace("s", "a", "b", position=None)
+ assert len(captured[-1][1]) == 3
+ F.regexp_replace("s", "a", "b", 4)
+ assert captured[-1][1][1:] == ("a", "b", 4)
+ finally:
+ F._call_function = orig
+
+
+def test_variant_delete_requires_paths():
+ from pyspark.errors import PySparkValueError
+ from pyspark.sql import functions as F
+
+ with pytest.raises(PySparkValueError, match="CANNOT_BE_EMPTY"):
+ F.variant_delete("v")
+
+
+def _offline_session():
+ # Port 1: nothing listens; the channel is lazy, so the session builds
offline.
+ return SparkSession.builder.remote("sc://localhost:1").getOrCreate()
+
+
+def test_session_context_manager_stops_session():
+ spark = _offline_session()
+ with spark as s:
+ assert s is spark
+ assert not s.is_stopped
+ # __exit__ stops it; stop() is best-effort (the release RPC fails offline)
and
+ # idempotent.
+ assert spark.is_stopped
+ spark.stop()
+ assert SparkSession.getActiveSession() is None
+
+
+def test_sql_context_connect_backed():
+ import warnings
+
+ from pyspark.errors import PySparkNotImplementedError
+ from pyspark.sql import HiveContext, SQLContext
+ from pyspark.sql.connect import context as cctx
+
+ spark = _offline_session()
+ try:
+ with warnings.catch_warnings(record=True) as caught:
+ warnings.simplefilter("always")
+ ctx = SQLContext.getOrCreate()
+ assert any(issubclass(w.category, FutureWarning) for w in caught)
+ assert isinstance(ctx, cctx.SQLContext)
+ assert ctx.sparkSession.session_id == spark.session_id
+ # Cached per session (getActiveSession returns a fresh wrapper each
call).
+ assert SQLContext.getOrCreate() is ctx
+ assert cctx.SQLContext._get_or_create_from_session(spark) is ctx
+ with pytest.raises(PySparkNotImplementedError):
+ ctx.registerJavaFunction("f", "com.example.F")
+ with pytest.raises(PySparkNotImplementedError):
+ HiveContext.getOrCreate()
+ with pytest.raises(PySparkNotImplementedError):
+ cctx.HiveContext(spark)
+ with pytest.raises(PySparkNotImplementedError):
+ SQLContext(None)
+ with warnings.catch_warnings(record=True) as caught:
+ warnings.simplefilter("always")
+ cctx.SQLContext(spark)
+ assert any(issubclass(w.category, FutureWarning) for w in caught)
+ assert ctx.udf is not None and ctx.read is not None
+ # Stopping a different session leaves the cache; stopping its own
clears it.
+ other = spark.newSession()
+ other.stop()
+ assert cctx.SQLContext._instantiatedContext is ctx
+ spark.stop()
+ assert cctx.SQLContext._instantiatedContext is None
+ finally:
+ cctx.SQLContext._instantiatedContext = None
+ spark.stop()
+
+
+def test_execution_info_and_operation_id():
+ from pyspark.errors import AnalysisException
+ from pyspark.errors.exceptions.connect import SparkConnectException
+ from pyspark.sql.connect.client import PlanObservedMetrics
+ from pyspark.sql.metrics import ExecutionInfo
+
+ spark = _offline_session()
+ try:
+ # No action ran yet.
+ assert spark.range(1).executionInfo is None
+ finally:
+ spark.stop()
+ ei = ExecutionInfo([], [PlanObservedMetrics("o", [1, 2], [])], "op-1")
+ assert ei.operation_id == "op-1"
+ assert ei.flows == [("o", {"observed_metric_0": 1, "observed_metric_1":
2})]
+ assert ei.metrics is None
+ # SparkConnectException.operation_id (4.3.0): None unless the transport
stamped it.
+ err = AnalysisException("boom")
+ assert isinstance(err, SparkConnectException)
+ assert err.operation_id is None
+ err._operation_id = "op-2"
+ assert err.operation_id == "op-2"
+
+
+def test_create_auto_cdc_flow_scd2_track_history():
+ from pyspark import pipelines as dp
+ from pyspark.errors import PySparkValueError
+ from pyspark.pipelines.graph_element_registry import
graph_element_registration_context
+ from pyspark.pipelines.tests.local_graph_element_registry import
LocalGraphElementRegistry
+ from pyspark.sql.functions import col
+
+ registry = LocalGraphElementRegistry()
+ with graph_element_registration_context(registry):
+ dp.create_streaming_table("t")
+ dp.create_auto_cdc_flow(
+ target="t", source="s", keys=["k"], sequence_by="seq",
+ stored_as_scd_type=2, track_history_column_list=["a", col("b")],
+ spark_conf={"k": "v"},
+ )
+ with pytest.raises(PySparkValueError, match="CANNOT_SET_TOGETHER"):
+ dp.create_auto_cdc_flow(
+ target="t", source="s", keys=["k"], sequence_by="seq",
+ stored_as_scd_type=2, track_history_column_list=["a"],
+ track_history_except_column_list=["b"],
+ )
+ with pytest.raises(PySparkValueError,
match="INVALID_MULTIPLE_ARGUMENT_CONDITIONS"):
+ dp.create_auto_cdc_flow(
+ target="t", source="s", keys=["k"], sequence_by="seq",
+ stored_as_scd_type=1, track_history_column_list=["a"],
+ )
+ flow = registry.auto_cdc_flows[0]
+ assert flow.stored_as_scd_type == 2
+ assert len(flow.track_history_column_list) == 2
+ assert flow.spark_conf == {"k": "v"}
+
+
+# ---------------------------------------------------------------------------
ExecutePlan (mock)
+
+
+def _request_operation_id(request):
+ # ExecutePlanRequest.operation_id=6.
+ return parse_message(request)[6][0].decode()
+
+
+def
test_execution_info_decodes_observed_metrics_and_operation_id(mock_connect):
+ import datetime
+
+ spark, service = mock_connect
+ ts = 1_700_000_000_123_456
+ # expressions.proto Literal: integer=6, string=13, date=16, timestamp=17,
+ # day_time_interval=21, map=23 (keys=3, values=4), struct=24 (elements=2),
+ # data_type=100; types.proto DataType.struct=21 (fields=1;
StructField.name=1).
+ struct_type = message_field(
+ 21, message_field(1, message_field(1, "a")) + message_field(1,
message_field(1, "b"))
+ )
+ struct_literal = message_field(
+ 24, message_field(2, varint_field(6, 7)) + message_field(2,
message_field(13, "x"))
+ ) + message_field(100, struct_type)
+ map_literal = message_field(
+ 23, message_field(3, message_field(13, "k")) + message_field(4,
varint_field(16, 1))
+ )
+ # base.proto ObservedMetrics: name=1, values=2, keys=3;
ExecutePlanResponse.observed_metrics=6.
+ observed = message_field(1, "o")
+ for literal in (
+ varint_field(17, ts),
+ struct_literal,
+ map_literal,
+ varint_field(21, 90_061_000_001),
+ ):
+ observed += message_field(2, literal)
+ for key in ("ts", "s", "m", "dt"):
+ observed += message_field(3, key)
+ service.execute_payload = message_field(6, observed)
+
+ assert spark.range(1).executionInfo is None
+ spark.range(1).collect()
+ info = spark.range(1).executionInfo
+ assert info.operation_id ==
_request_operation_id(service.execute_requests[-1])
+ ((name, pairs),) = info.flows
+ assert name == "o"
+ assert pairs["ts"] == datetime.datetime.fromtimestamp(ts // 10**6).replace(
+ microsecond=ts % 10**6
+ )
+ assert (pairs["s"].a, pairs["s"].b) == (7, "x")
+ assert pairs["m"] == {"k": datetime.date(1970, 1, 2)}
+ assert pairs["dt"] == datetime.timedelta(microseconds=90_061_000_001)
+
+
+def test_execution_info_unsupported_literal_raises_on_flows(mock_connect):
+ from pyspark.errors import PySparkTypeError
+
+ spark, service = mock_connect
+ # Literal.calendar_interval=19 (months=1): the reference rejects it in
_to_value.
+ observed = message_field(1, "o") + message_field(2, message_field(19,
varint_field(1, 2)))
+ service.execute_payload = message_field(6, observed)
+ spark.range(1).collect()
+ info = spark.range(1).executionInfo
+ assert info.operation_id ==
_request_operation_id(service.execute_requests[-1])
+ with pytest.raises(PySparkTypeError, match="UNSUPPORTED_LITERAL"):
+ info.flows
+
+
+def test_execute_plan_error_carries_operation_id(mock_connect):
+ from pyspark.errors import PySparkException
+
+ spark, service = mock_connect
+ service.execute_failure = "plan rejected"
+ with pytest.raises(PySparkException, match="plan rejected") as error:
+ spark.range(1).collect()
+ assert error.value.operation_id ==
_request_operation_id(service.execute_requests[-1])
+
+
+def _sent_define_flow(request):
+ """The DefineFlow in a captured ExecutePlanRequest: plan=3 -> command=2 ->
+ pipeline_command=19 -> define_flow=3."""
+ message = parse_message(request)
+ for field in (3, 2, 19, 3):
+ message = parse_message(message[field][0])
+ return message
+
+
+def _unparsed_identifiers(expressions):
+ # Expression.unresolved_attribute=2,
UnresolvedAttribute.unparsed_identifier=1.
+ return [
+ parse_message(parse_message(e)[2][0])[1][0].decode() for e in
expressions
+ ]
+
+
+def test_connect_registry_sends_auto_cdc_define_flow(mock_connect):
+ from pyspark import pipelines as dp
+ from pyspark.pipelines.graph_element_registry import
graph_element_registration_context
+ from pyspark.pipelines.spark_connect_graph_element_registry import (
+ SparkConnectGraphElementRegistry,
+ )
+
+ spark, service = mock_connect
+ with
graph_element_registration_context(SparkConnectGraphElementRegistry(spark,
"g")):
+ dp.create_auto_cdc_flow(
+ target="t", source="s", keys=["k"], sequence_by="seq",
+ stored_as_scd_type=2, name="scd2",
+ track_history_column_list=["a", "b"], spark_conf={"c": "v"},
+ )
+ dp.create_auto_cdc_flow(
+ target="t", source="s", keys=["k"], sequence_by="seq",
+ stored_as_scd_type="2", name="scd2_except",
+ track_history_except_column_list=["z"],
+ )
+ dp.create_auto_cdc_flow(
+ target="t", source="s", keys=["k"], sequence_by="seq", name="plain"
+ )
+ scd2, scd2_except, plain = (_sent_define_flow(r) for r in
service.execute_requests[-3:])
+
+ # DefineFlow: flow_name=2, sql_conf=4 (entries key=1, value=2),
auto_cdc_flow_details=10;
+ # AutoCdcFlowDetails: stored_as_scd_type=10, track_history_column_list=11,
+ # track_history_except_column_list=12.
+ assert scd2[2] == [b"scd2"]
+ assert [parse_message(e) for e in scd2[4]] == [{1: [b"c"], 2: [b"v"]}]
+ details = parse_message(scd2[10][0])
+ assert details[10] == [2]
+ assert _unparsed_identifiers(details[11]) == ["a", "b"]
+ assert 12 not in details
+
+ details = parse_message(scd2_except[10][0])
+ assert details[10] == [2]
+ assert 11 not in details
+ assert _unparsed_identifiers(details[12]) == ["z"]
+
+ details = parse_message(plain[10][0])
+ assert 4 not in plain
+ assert 10 not in details and 11 not in details and 12 not in details
+
diff --git a/scripts/check_vendored_upstream.sh
b/scripts/check_vendored_upstream.sh
index 0f4ba59..2e8fd61 100644
--- a/scripts/check_vendored_upstream.sh
+++ b/scripts/check_vendored_upstream.sh
@@ -143,6 +143,8 @@ VENDORED_FILES=(
"sql/datasource_internal.py"
"sql/internal.py"
"sql/profiler.py"
+ # ExecutionInfo / PlanMetrics / MetricValue (DataFrame.executionInfo).
+ "sql/metrics.py"
# Streaming (excluding adapted context, dstream, util)
"streaming/__init__.py"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]