This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 66e17ce [SPARK-61000][CONNECT][PYTHON] Convert the columns of
toPandas as PySpark does
66e17ce is described below
commit 66e17ce2a0a1b60ab67d83c1f43918071ef42dd7
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 18:10:42 2026 +0900
[SPARK-61000][CONNECT][PYTHON] Convert the columns of toPandas as PySpark
does
### What changes were proposed in this pull request?
[SPARK-61000](https://issues.apache.org/jira/browse/SPARK-61000):
`DataFrame.toPandas()` now converts each column of the Arrow result by its
Spark type, as PySpark's `SparkConnectClient.to_pandas` does.
- It first reads three settings in one request: the session time zone,
`spark.sql.execution.pandas.structHandlingMode`, and
`spark.sql.execution.arrow.pyspark.selfDestruct.enabled`. The
`pandasStructHandlingMode` of `_to_pandas` overrides the struct handling mode,
as in PySpark.
- It converts the result with
`pyspark.sql.pandas.conversion._convert_arrow_table_to_pandas`, of the schema
of the result that the server sends with it, through the new
`DataFrame::collect_record_batches_with_schema` of the Rust core.
- The metrics and observed metrics of the execution are in `pdf.attrs`, as
in PySpark.
So timestamps are naive in the session time zone, maps are dicts, and
values of user-defined types are deserialized.
`pyspark.sql.pandas.conversion`, vendored from Apache Spark 4.3.0, did not
import, so now:
- `pyspark/errors/exceptions/captured.py` and `pyspark/traceback_utils.py`
are vendored from Apache Spark 4.3.0, and checked by the drift check.
`captured.py` imports py4j only where it uses the JVM.
- `pyspark/util.py` has PySpark's local socket helpers, verbatim, which it
imports.
### Why are the changes needed?
`toPandas()` returned `pyarrow.Table.to_pandas()` of the result. Against a
Spark 4.3.0 server:
| Column | PySpark 4.3.0 and now | Before |
| --- | --- | --- |
| `TIMESTAMP'2024-01-02 03:04:05'` | `datetime64[ns]`,
`Timestamp('2024-01-02 03:04:05')` in the session time zone | `datetime64[us,
UTC]`, `Timestamp('2024-01-02 03:04:05+0000', tz='UTC')` |
| `map('k', 1)` | `{'k': 1}` | `[('k', 1)]` |
| a `VectorUDT` | `DenseVector([1.5, 2.0])` | `{'type': 1, 'size': None,
'indices': None, 'values': array([1.5, 2.])}` |
| `pdf.attrs` | `{'metrics': [PlanMetrics, ...]}` | `{}` |
### Does this PR introduce _any_ user-facing change?
Yes, `toPandas()` gives the pandas DataFrames of PySpark 4.3.0.
### How was this patch tested?
- Against a local Spark 4.3.0 server, `toPandas()` gives the same dtypes
and values as official PySpark 4.3.0 for timestamps, NTZ timestamps, structs,
maps, arrays, decimals, dates, nulls, binaries and vectors. So do its `attrs`,
with and without observed metrics, an empty result, `_to_pandas` with each
struct handling mode, and a session time zone of `Asia/Seoul` with a client in
`America/Los_Angeles`. Before this change, the timestamps, maps, vectors and
`attrs` differed.
- A new live test checks these. It also passes with official PySpark 4.3.0,
and fails without this change. A new offline test checks that
`pyspark.sql.pandas.conversion` imports.
- The Arrow handoff test of `toPandas` now expects PySpark's conversion of
the same Arrow table, and its mock server serves the settings.
- The lib tests and every integration test, e2e included, against the local
server (660 passed, and 123 of the transport crate), the build of all tests
with `wasm-udf`, the offline Python suites (887 passed, 9 skipped), and the
live drop-in and Arrow suites (82 passed) and live cases (10 passed). In the
changed lines, clippy reports only the `result_large_err` of the `py.detach`
closures, which every action of the Python `DataFrame` has.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #200 from HyukjinKwon/SPARK-61000-to-pandas.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/dataframe.rs | 134 ++++++--
crates/spark-connect/src/dataframe.rs | 17 +-
python/pyspark/errors/exceptions/captured.py | 465 ++++++++++++++++++++++++++
python/pyspark/sql/connect/client/__init__.py | 32 ++
python/pyspark/traceback_utils.py | 80 +++++
python/pyspark/util.py | 196 +++++++++++
python/tests/test_arrow_handoff.py | 35 +-
python/tests/test_dropin_e2e.py | 32 ++
python/tests/test_dropin_offline.py | 14 +
scripts/check_vendored_upstream.sh | 9 +-
10 files changed, 973 insertions(+), 41 deletions(-)
diff --git a/crates/pyspark-rs/src/dataframe.rs
b/crates/pyspark-rs/src/dataframe.rs
index 54264e9..11a760b 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -50,6 +50,53 @@ pub struct PyDataFrame {
}
impl PyDataFrame {
+ /// The pandas DataFrame of the result, as `SparkConnectClient.to_pandas`
makes it: the
+ /// settings of the conversion first, in one request; then the Arrow
result with the schema
+ /// that the server sends, which `pyspark.sql.connect.client._to_pandas`
converts.
+ fn to_pandas(
+ &self,
+ py: Python<'_>,
+ struct_handling_mode: Option<Bound<'_, PyAny>>,
+ ) -> PyResult<Py<PyAny>> {
+ let settings = py
+ .detach(|| {
+ self.dataframe.spark_session().conf().get_many(&[
+ "spark.sql.session.timeZone",
+ "spark.sql.execution.pandas.structHandlingMode",
+ "spark.sql.execution.arrow.pyspark.selfDestruct.enabled",
+ ])
+ })
+ .to_pyerr()?;
+ let (batches, schema) = py
+ .detach(|| self.dataframe.collect_record_batches_with_schema())
+ .to_pyerr()?;
+ let table = crate::pyarrow_export::record_batches_to_pyarrow(py,
batches)?;
+ let schema = match &schema {
+ Some(schema) => crate::types::data_type_to_py(py, schema)?,
+ None => py.None(),
+ };
+ let (metrics, observed) = match self.dataframe.execution_info() {
+ Ok(info) => execution_metrics(py, &info)?,
+ Err(_) => (
+ pyo3::types::PyList::empty(py),
+ pyo3::types::PyList::empty(py),
+ ),
+ };
+ let kwargs = pyo3::types::PyDict::new(py);
+ kwargs.set_item("timezone", &settings[0])?;
+ match struct_handling_mode {
+ Some(mode) => kwargs.set_item("struct_handling_mode",
mode.str()?)?,
+ None => kwargs.set_item("struct_handling_mode", &settings[1])?,
+ }
+ kwargs.set_item("self_destruct", &settings[2])?;
+ let pdf = py.import("pyspark.sql.connect.client")?.call_method(
+ "_to_pandas",
+ (table, schema, metrics, observed),
+ Some(&kwargs),
+ )?;
+ Ok(pdf.unbind())
+ }
+
/// The table of `show` and `_show_string`, for the arguments of
`function`.
fn show_table(
&self,
@@ -1273,24 +1320,27 @@ impl PyDataFrame {
/// Collect the DataFrame into a pandas DataFrame.
///
- /// Mirrors `pyspark.sql.DataFrame.toPandas()`: decodes the result via
Arrow
- /// (`toArrow`) and converts the `pyarrow.Table` with `to_pandas()`.
+ /// Mirrors `pyspark.sql.DataFrame.toPandas()`, as
`SparkConnectClient.to_pandas` converts
+ /// the Arrow result: each column by its Spark type, in the session time
zone.
fn toPandas(&self, py: Python<'_>) -> PyResult<Py<PyAny>> {
- let table = self.to_arrow(py)?;
- let df = table.bind(py).call_method0("to_pandas")?;
- Ok(df.unbind())
+ self.to_pandas(py, None)
}
/// Internal Arrow-based pandas conversion used by pandas-on-Spark
- /// (`pyspark.sql.connect.dataframe.DataFrame._to_pandas`). Accepts and
ignores
- /// keyword options such as `pandasStructHandlingMode`; delegates to
`toPandas`.
- #[pyo3(signature = (**_kwargs))]
+ /// (`pyspark.sql.connect.dataframe.DataFrame._to_pandas`): `toPandas`,
with
+ /// `pandasStructHandlingMode` overriding
`spark.sql.execution.pandas.structHandlingMode`,
+ /// as in `SparkConnectClient.to_pandas`.
+ #[pyo3(signature = (**kwargs))]
fn _to_pandas(
&self,
py: Python<'_>,
- _kwargs: Option<&Bound<'_, pyo3::types::PyDict>>,
+ kwargs: Option<&Bound<'_, pyo3::types::PyDict>>,
) -> PyResult<Py<PyAny>> {
- self.toPandas(py)
+ let mode = match kwargs {
+ Some(kwargs) => kwargs.get_item("pandasStructHandlingMode")?,
+ None => None,
+ };
+ self.to_pandas(py, mode)
}
/// Get the count of rows.
@@ -1336,32 +1386,7 @@ impl PyDataFrame {
return Ok(py.None());
};
let metrics_mod = py.import("pyspark.sql.metrics")?;
- let plan_metrics_cls = metrics_mod.getattr("PlanMetrics")?;
- let metric_value_cls = metrics_mod.getattr("MetricValue")?;
- // Mirrors SparkConnectClient._build_metrics.
- let plan_metrics = pyo3::types::PyList::empty(py);
- if let Some(m) = &info.metrics {
- for x in &m.metrics {
- let values = pyo3::types::PyList::empty(py);
- for (k, v) in &x.execution_metrics {
- values.append(metric_value_cls.call1((k, v.value,
&v.metric_type))?)?;
- }
- plan_metrics
- .append(plan_metrics_cls.call1((&x.name, x.plan_id,
x.parent, values))?)?;
- }
- }
- let observed_cls = py
- .import("pyspark.sql.connect.client")?
- .getattr("PlanObservedMetrics")?;
- let observed = pyo3::types::PyList::empty(py);
- for o in &info.observed_metrics {
- let values = o
- .values
- .iter()
- .map(|l| literal_to_py(py, l, None))
- .collect::<PyResult<Vec<_>>>()?;
- observed.append(observed_cls.call1((&o.name, values,
o.keys.clone()))?)?;
- }
+ let (plan_metrics, observed) = execution_metrics(py, &info)?;
let ei = metrics_mod.getattr("ExecutionInfo")?.call1((
plan_metrics,
observed,
@@ -2838,6 +2863,43 @@ impl PyLocalRowIterator {
}
}
+/// The metrics and observed metrics of an execution, as the lists of
`PlanMetrics` and
+/// `PlanObservedMetrics` that `SparkConnectClient._build_metrics` makes.
+fn execution_metrics<'py>(
+ py: Python<'py>,
+ info: &spark_connect::session::ExecutionInfo,
+) -> PyResult<(
+ Bound<'py, pyo3::types::PyList>,
+ Bound<'py, pyo3::types::PyList>,
+)> {
+ let metrics_mod = py.import("pyspark.sql.metrics")?;
+ let plan_metrics_cls = metrics_mod.getattr("PlanMetrics")?;
+ let metric_value_cls = metrics_mod.getattr("MetricValue")?;
+ let plan_metrics = pyo3::types::PyList::empty(py);
+ if let Some(m) = &info.metrics {
+ for x in &m.metrics {
+ let values = pyo3::types::PyList::empty(py);
+ for (k, v) in &x.execution_metrics {
+ values.append(metric_value_cls.call1((k, v.value,
&v.metric_type))?)?;
+ }
+ plan_metrics.append(plan_metrics_cls.call1((&x.name, x.plan_id,
x.parent, values))?)?;
+ }
+ }
+ let observed_cls = py
+ .import("pyspark.sql.connect.client")?
+ .getattr("PlanObservedMetrics")?;
+ let observed = pyo3::types::PyList::empty(py);
+ for o in &info.observed_metrics {
+ let values = o
+ .values
+ .iter()
+ .map(|l| literal_to_py(py, l, None))
+ .collect::<PyResult<Vec<_>>>()?;
+ observed.append(observed_cls.call1((&o.name, values,
o.keys.clone()))?)?;
+ }
+ Ok((plan_metrics, observed))
+}
+
/// Convert an observed-metric literal to a Python value. Mirrors
/// `LiteralExpression._to_value`, which `PlanObservedMetrics.pairs` applies:
the kinds it
/// decodes map to the same Python values, and for any other kind (which the
reference
diff --git a/crates/spark-connect/src/dataframe.rs
b/crates/spark-connect/src/dataframe.rs
index dc69cb9..b04de08 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -821,10 +821,22 @@ impl DataFrame {
/// returning the raw `RecordBatch`es without converting to Rows. This is
the
/// foundation for `to_datafusion()` and `to_polars()` conversions.
pub fn collect_record_batches(&self) ->
Result<Vec<arrow::record_batch::RecordBatch>> {
+ self.collect_record_batches_with_schema()
+ .map(|(batches, _)| batches)
+ }
+
+ /// Collect all data as Arrow RecordBatches, with the schema of the result
if the server
+ /// sends it (`ExecutePlanResponse.schema`), as the reference client's
`to_pandas` reads
+ /// it: unlike the Arrow schema, it has the Spark types of the columns,
such as their
+ /// user-defined types.
+ pub fn collect_record_batches_with_schema(
+ &self,
+ ) -> Result<(Vec<arrow::record_batch::RecordBatch>, Option<DataType>)> {
let request = self.build_execute_request()?;
let mut stream =
block_on(self.session.client().execute_plan_reattachable(request))?;
let mut batches = vec![];
+ let mut schema = None;
let mut info = ExecutionInfo::default();
loop {
let resp = block_on(stream.message())?;
@@ -832,6 +844,9 @@ impl DataFrame {
break;
};
capture_execution(&mut resp, &mut info, &self.session);
+ if let Some(data_type) = resp.schema.take() {
+ schema = Some(DataType::from_proto(&data_type)?);
+ }
if let
Some(proto::execute_plan_response::ResponseType::ArrowBatch(batch)) =
resp.response_type
{
@@ -841,7 +856,7 @@ impl DataFrame {
}
self.record_execution(info);
- Ok(batches)
+ Ok((batches, schema))
}
/// Get the count of rows.
diff --git a/python/pyspark/errors/exceptions/captured.py
b/python/pyspark/errors/exceptions/captured.py
new file mode 100644
index 0000000..da755de
--- /dev/null
+++ b/python/pyspark/errors/exceptions/captured.py
@@ -0,0 +1,465 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+import warnings
+from contextlib import contextmanager
+from typing import Any, Callable, Dict, Iterator, Optional, cast, List,
TYPE_CHECKING
+
+from pyspark.errors.exceptions.base import (
+ AnalysisException as BaseAnalysisException,
+ IllegalArgumentException as BaseIllegalArgumentException,
+ ArithmeticException as BaseArithmeticException,
+ UnsupportedOperationException as BaseUnsupportedOperationException,
+ ArrayIndexOutOfBoundsException as BaseArrayIndexOutOfBoundsException,
+ DateTimeException as BaseDateTimeException,
+ NumberFormatException as BaseNumberFormatException,
+ ParseException as BaseParseException,
+ PySparkException,
+ PythonException as BasePythonException,
+ QueryExecutionException as BaseQueryExecutionException,
+ SparkRuntimeException as BaseSparkRuntimeException,
+ SparkUpgradeException as BaseSparkUpgradeException,
+ SparkNoSuchElementException as BaseNoSuchElementException,
+ StreamingQueryException as BaseStreamingQueryException,
+ UnknownException as BaseUnknownException,
+ QueryContext as BaseQueryContext,
+ QueryContextType,
+ recover_python_exception,
+)
+
+if TYPE_CHECKING:
+ from py4j.protocol import Py4JJavaError
+ from py4j.java_gateway import JavaObject
+
+
+class CapturedException(PySparkException):
+ def __init__(
+ self,
+ desc: Optional[str] = None,
+ stackTrace: Optional[str] = None,
+ cause: Optional["Py4JJavaError"] = None,
+ origin: Optional["Py4JJavaError"] = None,
+ ):
+ from pyspark import SparkContext
+ from py4j.protocol import Py4JJavaError
+
+ # desc & stackTrace vs origin are mutually exclusive.
+ # cause is optional.
+ assert (origin is not None and desc is None and stackTrace is None) or
(
+ origin is None and desc is not None and stackTrace is not None
+ )
+
+ self._desc = desc if desc is not None else cast(Py4JJavaError,
origin).getMessage()
+ if self._desc is None:
+ self._desc = ""
+ assert SparkContext._jvm is not None
+ self._stackTrace = (
+ stackTrace
+ if stackTrace is not None
+ else (getattr(SparkContext._jvm,
"org.apache.spark.util.Utils").exceptionString(origin))
+ )
+ self._cause = convert_exception(cause) if cause is not None else None
+ if self._cause is None and origin is not None and origin.getCause() is
not None:
+ self._cause = convert_exception(origin.getCause())
+ self._origin = origin
+ self._log_exception()
+
+ def __str__(self) -> str:
+ from pyspark import SparkContext
+
+ assert SparkContext._jvm is not None
+
+ jvm = SparkContext._jvm
+
+ # SPARK-42752: default to True to see issues with initialization
+ debug_enabled = True
+ try:
+ sql_conf = getattr(jvm,
"org.apache.spark.sql.internal.SQLConf").get()
+ debug_enabled = sql_conf.pysparkJVMStacktraceEnabled()
+ except BaseException:
+ pass
+
+ desc = self._desc
+ if debug_enabled:
+ desc = desc + "\n\nJVM stacktrace:\n%s" % self._stackTrace
+ return str(desc)
+
+ def getCondition(self) -> Optional[str]:
+ from pyspark import SparkContext
+ from py4j.java_gateway import is_instance_of
+
+ assert SparkContext._gateway is not None
+ assert SparkContext._jvm is not None
+
+ gw = SparkContext._gateway
+ if self._origin is not None and is_instance_of(
+ gw, self._origin, "org.apache.spark.SparkThrowable"
+ ):
+ utils = SparkContext._jvm.PythonErrorUtils
+ return utils.getCondition(self._origin)
+ else:
+ return None
+
+ def getErrorClass(self) -> Optional[str]:
+ warnings.warn("Deprecated in 4.0.0, use getCondition instead.",
FutureWarning)
+ return self.getCondition()
+
+ def getMessageParameters(self) -> Optional[Dict[str, str]]:
+ from pyspark import SparkContext
+ from py4j.java_gateway import is_instance_of
+
+ assert SparkContext._gateway is not None
+ assert SparkContext._jvm is not None
+
+ gw = SparkContext._gateway
+ if self._origin is not None and is_instance_of(
+ gw, self._origin, "org.apache.spark.SparkThrowable"
+ ):
+ utils = SparkContext._jvm.PythonErrorUtils
+ return dict(utils.getMessageParameters(self._origin))
+ else:
+ return None
+
+ def getSqlState(self) -> Optional[str]:
+ from pyspark import SparkContext
+ from py4j.java_gateway import is_instance_of
+
+ assert SparkContext._gateway is not None
+ assert SparkContext._jvm is not None
+ gw = SparkContext._gateway
+ if self._origin is not None and is_instance_of(
+ gw, self._origin, "org.apache.spark.SparkThrowable"
+ ):
+ utils = SparkContext._jvm.PythonErrorUtils
+ return utils.getSqlState(self._origin)
+ else:
+ return None
+
+ def getMessage(self) -> str:
+ from pyspark import SparkContext
+ from py4j.java_gateway import is_instance_of
+
+ assert SparkContext._gateway is not None
+ assert SparkContext._jvm is not None
+ gw = SparkContext._gateway
+
+ if self._origin is not None and is_instance_of(
+ gw, self._origin, "org.apache.spark.SparkThrowable"
+ ):
+ utils = SparkContext._jvm.PythonErrorUtils
+ errorClass = utils.getCondition(self._origin)
+ messageParameters = utils.getMessageParameters(self._origin)
+
+ error_message = getattr(gw.jvm,
"org.apache.spark.SparkThrowableHelper").getMessage(
+ errorClass, messageParameters
+ )
+
+ return error_message
+ else:
+ return ""
+
+ def getQueryContext(self) -> List[BaseQueryContext]:
+ from pyspark import SparkContext
+ from py4j.java_gateway import is_instance_of
+
+ assert SparkContext._gateway is not None
+ assert SparkContext._jvm is not None
+
+ gw = SparkContext._gateway
+ if self._origin is not None and is_instance_of(
+ gw, self._origin, "org.apache.spark.SparkThrowable"
+ ):
+ contexts: List[BaseQueryContext] = []
+ utils = SparkContext._jvm.PythonErrorUtils
+ for q in utils.getQueryContext(self._origin):
+ if q.contextType().toString() == "SQL":
+ contexts.append(SQLQueryContext(q))
+ else:
+ contexts.append(DataFrameQueryContext(q))
+
+ return contexts
+ else:
+ return []
+
+
+def convert_exception(e: "Py4JJavaError") -> CapturedException:
+ converted = _convert_exception(e)
+ return recover_python_exception(converted)
+
+
+def _convert_exception(e: "Py4JJavaError") -> CapturedException:
+ from pyspark import SparkContext
+ from py4j.java_gateway import is_instance_of
+
+ assert e is not None
+ assert SparkContext._jvm is not None
+ assert SparkContext._gateway is not None
+
+ jvm = SparkContext._jvm
+ gw = SparkContext._gateway
+
+ if is_instance_of(gw, e,
"org.apache.spark.sql.catalyst.parser.ParseException"):
+ return ParseException(origin=e)
+ # Order matters. ParseException inherits AnalysisException.
+ elif is_instance_of(gw, e, "org.apache.spark.sql.AnalysisException"):
+ return AnalysisException(origin=e)
+ elif is_instance_of(gw, e,
"org.apache.spark.sql.streaming.StreamingQueryException"):
+ return StreamingQueryException(origin=e)
+ elif is_instance_of(gw, e,
"org.apache.spark.sql.execution.QueryExecutionException"):
+ return QueryExecutionException(origin=e)
+ # Order matters. NumberFormatException inherits IllegalArgumentException.
+ elif is_instance_of(gw, e, "java.lang.NumberFormatException"):
+ return NumberFormatException(origin=e)
+ elif is_instance_of(gw, e, "java.lang.IllegalArgumentException"):
+ return IllegalArgumentException(origin=e)
+ elif is_instance_of(gw, e, "java.lang.ArithmeticException"):
+ return ArithmeticException(origin=e)
+ elif is_instance_of(gw, e, "java.lang.UnsupportedOperationException"):
+ return UnsupportedOperationException(origin=e)
+ elif is_instance_of(gw, e, "java.lang.ArrayIndexOutOfBoundsException"):
+ return ArrayIndexOutOfBoundsException(origin=e)
+ elif is_instance_of(gw, e, "java.time.DateTimeException"):
+ return DateTimeException(origin=e)
+ elif is_instance_of(gw, e, "org.apache.spark.SparkRuntimeException"):
+ return SparkRuntimeException(origin=e)
+ elif is_instance_of(gw, e, "org.apache.spark.SparkUpgradeException"):
+ return SparkUpgradeException(origin=e)
+ elif is_instance_of(gw, e, "org.apache.spark.SparkNoSuchElementException"):
+ return SparkNoSuchElementException(origin=e)
+ elif is_instance_of(gw, e, "org.apache.spark.api.python.PythonException"):
+ return PythonException(origin=e)
+ return UnknownException(
+ desc=e.toString(),
+ stackTrace=getattr(jvm,
"org.apache.spark.util.Utils").exceptionString(e),
+ cause=e.getCause(),
+ )
+
+
+def capture_sql_exception(f: Callable[..., Any]) -> Callable[..., Any]:
+ def deco(*a: Any, **kw: Any) -> Any:
+ from py4j.protocol import Py4JJavaError
+
+ try:
+ return f(*a, **kw)
+ except Py4JJavaError as e:
+ converted = convert_exception(e.java_exception)
+ if not isinstance(converted, UnknownException):
+ # Hide where the exception came from that shows a non-Pythonic
+ # JVM exception message.
+ raise converted from None
+ else:
+ raise
+
+ return deco
+
+
+@contextmanager
+def unwrap_spark_exception() -> Iterator[Any]:
+ from pyspark import SparkContext
+ from py4j.protocol import Py4JJavaError
+ from py4j.java_gateway import is_instance_of
+
+ assert SparkContext._gateway is not None
+
+ gw = SparkContext._gateway
+ try:
+ yield
+ except Py4JJavaError as e:
+ je: "Py4JJavaError" = e.java_exception
+ if je is not None and is_instance_of(gw, je,
"org.apache.spark.SparkException"):
+ converted = convert_exception(je.getCause())
+ if not isinstance(converted, UnknownException):
+ raise converted from None
+ raise
+
+
+def install_exception_handler() -> None:
+ """
+ Hook an exception handler into Py4j, which could capture some SQL
exceptions in Java.
+
+ When calling Java API, it will call `get_return_value` to parse the
returned object.
+ If any exception happened in JVM, the result will be Java exception
object, it raise
+ py4j.protocol.Py4JJavaError. We replace the original `get_return_value`
with one that
+ could capture the Java exception and throw a Python one (with the same
error message).
+
+ It's idempotent, could be called multiple times.
+ """
+ import py4j
+
+ original = py4j.protocol.get_return_value
+ # The original `get_return_value` is not patched, it's idempotent.
+ patched = capture_sql_exception(original)
+ # only patch the one used in py4j.java_gateway (call Java API)
+ py4j.java_gateway.get_return_value = patched
+
+
+class AnalysisException(CapturedException, BaseAnalysisException):
+ """
+ Failed to analyze a SQL query plan.
+ """
+
+
+class ParseException(AnalysisException, BaseParseException):
+ """
+ Failed to parse a SQL command.
+ """
+
+
+class IllegalArgumentException(CapturedException,
BaseIllegalArgumentException):
+ """
+ Passed an illegal or inappropriate argument.
+ """
+
+
+class StreamingQueryException(CapturedException, BaseStreamingQueryException):
+ """
+ Exception that stopped a :class:`StreamingQuery`.
+ """
+
+
+class QueryExecutionException(CapturedException, BaseQueryExecutionException):
+ """
+ Failed to execute a query.
+ """
+
+
+class PythonException(CapturedException, BasePythonException):
+ """
+ Exceptions thrown from Python workers.
+ """
+
+ def __str__(self) -> str:
+ messageParameters = self.getMessageParameters()
+
+ if (
+ messageParameters is None
+ or "msg" not in messageParameters
+ or "traceback" not in messageParameters
+ ):
+ return super().__str__()
+ return
f"{messageParameters['msg']}:\n{messageParameters['traceback'].strip()}"
+
+
+class ArithmeticException(CapturedException, BaseArithmeticException):
+ """
+ Arithmetic exception.
+ """
+
+
+class UnsupportedOperationException(CapturedException,
BaseUnsupportedOperationException):
+ """
+ Unsupported operation exception.
+ """
+
+
+class ArrayIndexOutOfBoundsException(CapturedException,
BaseArrayIndexOutOfBoundsException):
+ """
+ Array index out of bounds exception.
+ """
+
+
+class DateTimeException(CapturedException, BaseDateTimeException):
+ """
+ Datetime exception.
+ """
+
+
+class NumberFormatException(IllegalArgumentException,
BaseNumberFormatException):
+ """
+ Number format exception.
+ """
+
+
+class SparkRuntimeException(CapturedException, BaseSparkRuntimeException):
+ """
+ Runtime exception.
+ """
+
+
+class SparkUpgradeException(CapturedException, BaseSparkUpgradeException):
+ """
+ Exception thrown because of Spark upgrade.
+ """
+
+
+class SparkNoSuchElementException(CapturedException,
BaseNoSuchElementException):
+ """
+ No such element exception.
+ """
+
+
+class UnknownException(CapturedException, BaseUnknownException):
+ """
+ None of the other exceptions.
+ """
+
+
+class SQLQueryContext(BaseQueryContext):
+ def __init__(self, q: "JavaObject"):
+ self._q = q
+
+ def contextType(self) -> QueryContextType:
+ return QueryContextType.SQL
+
+ def objectType(self) -> str:
+ return str(self._q.objectType())
+
+ def objectName(self) -> str:
+ return str(self._q.objectName())
+
+ def startIndex(self) -> int:
+ return int(self._q.startIndex())
+
+ def stopIndex(self) -> int:
+ return int(self._q.stopIndex())
+
+ def fragment(self) -> str:
+ return str(self._q.fragment())
+
+ def callSite(self) -> str:
+ return str(self._q.callSite())
+
+ def summary(self) -> str:
+ return str(self._q.summary())
+
+
+class DataFrameQueryContext(BaseQueryContext):
+ def __init__(self, q: "JavaObject"):
+ self._q = q
+
+ def contextType(self) -> QueryContextType:
+ return QueryContextType.DataFrame
+
+ def objectType(self) -> str:
+ return str(self._q.objectType())
+
+ def objectName(self) -> str:
+ return str(self._q.objectName())
+
+ def startIndex(self) -> int:
+ return int(self._q.startIndex())
+
+ def stopIndex(self) -> int:
+ return int(self._q.stopIndex())
+
+ def fragment(self) -> str:
+ return str(self._q.fragment())
+
+ def callSite(self) -> str:
+ return str(self._q.callSite())
+
+ def summary(self) -> str:
+ return str(self._q.summary())
diff --git a/python/pyspark/sql/connect/client/__init__.py
b/python/pyspark/sql/connect/client/__init__.py
index f8b50e2..96c383c 100644
--- a/python/pyspark/sql/connect/client/__init__.py
+++ b/python/pyspark/sql/connect/client/__init__.py
@@ -81,3 +81,35 @@ class PlanObservedMetrics(ObservedMetrics):
def to_dict(self) -> Dict[str, Any]:
return {"name": self._name, "keys": self._keys, "pairs": self.pairs}
+
+
+def _to_pandas(
+ table: Any,
+ schema: Any,
+ metrics: List[Any],
+ observed_metrics: List[Any],
+ *,
+ timezone: Any,
+ struct_handling_mode: Any,
+ self_destruct: Any,
+) -> Any:
+ """The pandas DataFrame of the Arrow ``table`` of a result, of ``schema``
if the server sent
+ it, as ``SparkConnectClient.to_pandas`` converts it: each column by its
Spark type, in the
+ session time zone, with the metrics of the execution in ``attrs``."""
+ from pyspark.sql.pandas.conversion import _convert_arrow_table_to_pandas
+ from pyspark.sql.pandas.types import from_arrow_schema
+
+ schema = schema or from_arrow_schema(table.schema,
prefer_timestamp_ntz=True)
+ pdf = _convert_arrow_table_to_pandas(
+ arrow_table=table,
+ schema=schema,
+ timezone=timezone,
+ struct_handling_mode=struct_handling_mode,
+ date_as_object=False,
+ self_destruct=self_destruct == "true",
+ )
+ if len(metrics) > 0:
+ pdf.attrs["metrics"] = metrics
+ if len(observed_metrics) > 0:
+ pdf.attrs["observed_metrics"] = observed_metrics
+ return pdf
diff --git a/python/pyspark/traceback_utils.py
b/python/pyspark/traceback_utils.py
new file mode 100644
index 0000000..fcc388b
--- /dev/null
+++ b/python/pyspark/traceback_utils.py
@@ -0,0 +1,80 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from collections import namedtuple
+import os
+import traceback
+
+CallSite = namedtuple("CallSite", "function file linenum")
+
+
+def first_spark_call():
+ """
+ Return a CallSite representing the first Spark call in the current call
stack.
+ """
+ tb = traceback.extract_stack()
+ if len(tb) == 0:
+ return None
+ file, line, module, what = tb[len(tb) - 1]
+ sparkpath = os.path.dirname(file)
+ first_spark_frame = len(tb) - 1
+ for i in range(0, len(tb)):
+ file, line, fun, what = tb[i]
+ if file.startswith(sparkpath):
+ first_spark_frame = i
+ break
+ if first_spark_frame == 0:
+ file, line, fun, what = tb[0]
+ return CallSite(function=fun, file=file, linenum=line)
+ sfile, sline, sfun, swhat = tb[first_spark_frame]
+ ufile, uline, ufun, uwhat = tb[first_spark_frame - 1]
+ return CallSite(function=sfun, file=ufile, linenum=uline)
+
+
+class SCCallSiteSync:
+ """
+ Helper for setting the spark context call site.
+
+ Example usage:
+ from pyspark.core.context import SCCallSiteSync
+ with SCCallSiteSync(<relevant SparkContext>) as css:
+ <a Spark call>
+ """
+
+ _spark_stack_depth = 0
+
+ def __init__(self, sc):
+ call_site = first_spark_call()
+ if call_site is not None:
+ self._call_site = "%s at %s:%s" % (
+ call_site.function,
+ call_site.file,
+ call_site.linenum,
+ )
+ else:
+ self._call_site = "Error! Could not extract traceback info"
+ self._context = sc
+
+ def __enter__(self):
+ if SCCallSiteSync._spark_stack_depth == 0:
+ self._context._jsc.setCallSite(self._call_site)
+ SCCallSiteSync._spark_stack_depth += 1
+
+ def __exit__(self, type, value, tb):
+ SCCallSiteSync._spark_stack_depth -= 1
+ if SCCallSiteSync._spark_stack_depth == 0:
+ self._context._jsc.setCallSite(None)
diff --git a/python/pyspark/util.py b/python/pyspark/util.py
index 84bc316..e74aecb 100644
--- a/python/pyspark/util.py
+++ b/python/pyspark/util.py
@@ -299,3 +299,199 @@
________________________________________________________________________________
"spark_version": spark_version,
}
)
+
+
+# Verbatim from PySpark: the local socket helpers of the JVM gateway. Spark
Connect does not
+# use them, but the modules of PySpark that the client vendors, such as
+# pyspark.sql.pandas.conversion, import them.
+import io # noqa: E402
+import socket # noqa: E402
+from typing import Union # noqa: E402
+
+from pyspark.errors import PySparkRuntimeError # noqa: E402
+from pyspark.serializers import UTF8Deserializer # noqa: E402
+
+
+def _create_local_socket(sock_info: "JavaArray") -> Tuple["io.BufferedRWPair",
"socket.socket"]:
+ """
+ Create a local socket that can be used to load deserialized data from the
JVM
+
+ Parameters
+ ----------
+ sock_info : tuple
+ Tuple containing port number and authentication secret for a local
socket.
+
+ Returns
+ -------
+ sockfile file descriptor of the local socket
+ """
+ sockfile: "io.BufferedRWPair"
+ sock: "socket.socket"
+ conn_info: int = sock_info[0]
+ auth_secret: str = sock_info[1]
+ sockfile, sock = local_connect_and_auth(conn_info, auth_secret)
+ # The RDD materialization time is unpredictable, if we set a timeout for
socket reading
+ # operation, it will very possibly fail. See SPARK-18281.
+ sock.settimeout(None)
+ return sockfile, sock
+
+
+@contextmanager
+def _load_from_socket(sock_info: "JavaArray", serializer: "Serializer") ->
Iterator[Any]:
+ """
+ Connect to a local socket described by sock_info and use the given
serializer to yield data
+
+ Parameters
+ ----------
+ sock_info : tuple
+ Tuple containing port number and authentication secret for a local
socket.
+ serializer : class:`Serializer`
+ The PySpark serializer to use
+
+ Returns
+ -------
+ result of meth:`Serializer.load_stream`,
+ usually a generator that yields deserialized data
+ """
+ try:
+ sockfile, sock = _create_local_socket(sock_info)
+ yield serializer.load_stream(sockfile)
+ finally:
+ sockfile.close()
+ sock.close()
+
+
+def _local_iterator_from_socket(sock_info: "JavaArray", serializer:
"Serializer") -> Iterator[Any]:
+ class PyLocalIterable:
+ """Create a synchronous local iterable over a socket"""
+
+ def __init__(self, _sock_info: "JavaArray", _serializer: "Serializer"):
+ port: int
+ auth_secret: str
+ self.jsocket_auth_server: "JavaObject"
+ port, auth_secret, self.jsocket_auth_server = _sock_info
+ self._sockfile, self._sock = _create_local_socket((port,
auth_secret))
+ self._serializer = _serializer
+ self._read_iter: Iterator[Any] = iter([]) # Initialize as empty
iterator
+ self._read_status = 1
+
+ def __iter__(self) -> Iterator[Any]:
+ while self._read_status == 1:
+ # Request next partition data from Java
+ write_int(1, self._sockfile)
+ self._sockfile.flush()
+
+ # If response is 1 then there is a partition to read, if 0
then fully consumed
+ self._read_status = read_int(self._sockfile)
+ if self._read_status == 1:
+ # Load the partition data as a stream and read each item
+ self._read_iter =
self._serializer.load_stream(self._sockfile)
+ for item in self._read_iter:
+ yield item
+
+ # An error occurred, join serving thread and raise any
exceptions from the JVM
+ elif self._read_status == -1:
+ self.jsocket_auth_server.getResult()
+
+ def __del__(self) -> None:
+ # If local iterator is not fully consumed,
+ if self._read_status == 1:
+ try:
+ # Finish consuming partition data stream
+ for _ in self._read_iter:
+ pass
+ # Tell Java to stop sending data and close connection
+ write_int(0, self._sockfile)
+ self._sockfile.flush()
+ except Exception:
+ # Ignore any errors, socket is automatically closed when
garbage-collected
+ pass
+ self._sockfile.close()
+ self._sock.close()
+
+ return iter(PyLocalIterable(sock_info, serializer))
+
+
+def local_connect_and_auth(
+ conn_info: Optional[Union[str, int]], auth_secret: Optional[str]
+) -> Tuple:
+ """
+ Connect to local host, authenticate with it, and return a (sockfile,sock)
for that connection.
+ Handles IPV4 & IPV6, does some error handling.
+
+ Parameters
+ ----------
+ port : str or int, optional
+ auth_secret : str, optional
+
+ Returns
+ -------
+ tuple
+ with (sockfile, sock)
+ """
+ is_unix_domain_socket = isinstance(conn_info, str) and auth_secret is None
+ if is_unix_domain_socket:
+ sock_path = conn_info
+ assert isinstance(sock_path, str)
+ sock = None
+ try:
+ sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+ sock.settimeout(int(os.environ.get("SPARK_AUTH_SOCKET_TIMEOUT",
15)))
+ sock.connect(sock_path)
+ sockfile = sock.makefile("rwb",
int(os.environ.get("SPARK_BUFFER_SIZE", 65536)))
+ return (sockfile, sock)
+ except socket.error as e:
+ if sock is not None:
+ sock.close()
+ raise PySparkRuntimeError(
+ errorClass="CANNOT_OPEN_SOCKET",
+ messageParameters={
+ "errors": "tried to connect to %s, but an error occurred:
%s"
+ % (sock_path, str(e)),
+ },
+ )
+
+ sock = None
+ errors = []
+ # Support for both IPv4 and IPv6.
+ addr = "127.0.0.1"
+ if os.environ.get("SPARK_PREFER_IPV6", "false").lower() == "true":
+ addr = "::1"
+ for res in socket.getaddrinfo(addr, conn_info, socket.AF_UNSPEC,
socket.SOCK_STREAM):
+ af, socktype, proto, _, sa = res
+ try:
+ sock = socket.socket(af, socktype, proto)
+ sock.settimeout(int(os.environ.get("SPARK_AUTH_SOCKET_TIMEOUT",
15)))
+ sock.connect(sa)
+ sockfile = sock.makefile("rwb",
int(os.environ.get("SPARK_BUFFER_SIZE", 65536)))
+ assert isinstance(auth_secret, str)
+ _do_server_auth(sockfile, auth_secret)
+ return (sockfile, sock)
+ except socket.error as e:
+ emsg = str(e)
+ errors.append("tried to connect to %s, but an error occurred: %s"
% (sa, emsg))
+ if sock is not None:
+ sock.close()
+ sock = None
+ raise PySparkRuntimeError(
+ errorClass="CANNOT_OPEN_SOCKET",
+ messageParameters={
+ "errors": str(errors),
+ },
+ )
+
+
+def _do_server_auth(conn: "io.IOBase", auth_secret: str) -> None:
+ """
+ Performs the authentication protocol defined by the SocketAuthHelper class
on the given
+ file-like object 'conn'.
+ """
+ write_with_length(auth_secret.encode("utf-8"), conn)
+ conn.flush()
+ reply = UTF8Deserializer().loads(conn)
+ if reply != "ok":
+ conn.close()
+ raise PySparkRuntimeError(
+ errorClass="UNEXPECTED_RESPONSE_FROM_SERVER",
+ messageParameters={},
+ )
diff --git a/python/tests/test_arrow_handoff.py
b/python/tests/test_arrow_handoff.py
index 5f8a74b..68af777 100644
--- a/python/tests/test_arrow_handoff.py
+++ b/python/tests/test_arrow_handoff.py
@@ -129,6 +129,23 @@ def _dataframe(batches, *, protocol, payloads=None,
calls=None):
server_side_session_id=server_id,
).SerializeToString()
+ def config(request, context):
+ # The settings that toPandas reads first, as
SparkConnectClient.to_pandas does.
+ if calls is not None:
+ calls["config"] = calls.get("config", 0) + 1
+ request = protocol("ConfigRequest").FromString(request)
+ settings = {
+ "spark.sql.session.timeZone": "UTC",
+ "spark.sql.execution.pandas.structHandlingMode": "legacy",
+ "spark.sql.execution.arrow.pyspark.selfDestruct.enabled": "false",
+ }
+ return protocol(
+ "ConfigResponse",
+ session_id=request.session_id,
+ server_side_session_id=server_id,
+ pairs=[dict(key=key, value=settings[key]) for key in
request.operation.get.keys],
+ ).SerializeToString()
+
def release_session(request, context):
request = protocol("ReleaseSessionRequest").FromString(request)
return protocol(
@@ -147,6 +164,7 @@ def _dataframe(batches, *, protocol, payloads=None,
calls=None):
"ExecutePlan":
grpc.unary_stream_rpc_method_handler(execute),
"ReleaseExecute":
grpc.unary_unary_rpc_method_handler(release_execute),
"ReleaseSession":
grpc.unary_unary_rpc_method_handler(release_session),
+ "Config": grpc.unary_unary_rpc_method_handler(config),
},
)
]
@@ -288,12 +306,25 @@ def test_to_pandas_uses_the_same_handoff(protocol):
arrays = [batch.column(index) for index in indices]
selected = pa.RecordBatch.from_arrays(arrays, schema=schema)
batches = [selected.slice(0, 2), selected.slice(2, 2)]
- expected = pa.Table.from_batches(batches).to_pandas()
+ # As SparkConnectClient.to_pandas converts the same table: of the Spark
types of its Arrow
+ # schema (the server sends no schema here), in the session time zone.
+ from pyspark.sql.pandas.conversion import _convert_arrow_table_to_pandas
+ from pyspark.sql.pandas.types import from_arrow_schema
+
+ table = pa.Table.from_batches(batches)
+ expected = _convert_arrow_table_to_pandas(
+ arrow_table=table,
+ schema=from_arrow_schema(table.schema, prefer_timestamp_ntz=True),
+ timezone="UTC",
+ struct_handling_mode="legacy",
+ date_as_object=False,
+ self_destruct=False,
+ )
calls = {}
with _dataframe(protocol=protocol, batches=batches, calls=calls) as
(frame, _):
actual = frame.toPandas()
pd.testing.assert_frame_equal(actual, expected)
- assert calls == {"execute": 1}
+ assert calls == {"config": 1, "execute": 1}
@pytest.mark.parametrize(
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 570b9a4..2896f8e 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -793,6 +793,38 @@ def
test_column_alias_names_every_column_and_keeps_its_metadata(session):
assert df.schema.fields[0].metadata == {"k": 1, "n": [1, 2], "s": "v"}
+def test_to_pandas_converts_the_columns_as_pyspark(session):
+ # As official PySpark 4.3.0's SparkConnectClient.to_pandas: each column by
its Spark type,
+ # in the session time zone, with the metrics of the execution in attrs.
+ from decimal import Decimal
+
+ import pandas as pd
+ from pyspark.ml.functions import array_to_vector
+ from pyspark.ml.linalg import DenseVector
+
+ df = session.sql(
+ "SELECT TIMESTAMP'2024-01-02 03:04:05' AS ts, map('k', 1) AS m, "
+ "named_struct('a', 1, 'b', 'x') AS st, 1.5BD AS d, array(1.5D, 2.0D)
AS av"
+ ).withColumn("vec", array_to_vector("av")).drop("av")
+ pdf = df.toPandas()
+ assert str(pdf.dtypes["ts"]) == "datetime64[ns]"
+ assert pdf["ts"][0] == pd.Timestamp("2024-01-02 03:04:05")
+ assert pdf["m"][0] == {"k": 1}
+ assert pdf["st"][0] == {"a": 1, "b": "x"}
+ assert pdf["d"][0] == Decimal("1.5")
+ assert pdf["vec"][0] == DenseVector([1.5, 2.0])
+ assert sorted(pdf.attrs) == ["metrics"]
+ assert type(pdf.attrs["metrics"][0]).__name__ == "PlanMetrics"
+ struct =
df.select("st")._to_pandas(pandasStructHandlingMode="row")["st"][0]
+ assert type(struct).__name__ == "Row" and struct == Row(a=1, b="x")
+ session.conf.set("spark.sql.session.timeZone", "Asia/Seoul")
+ try:
+ local = session.sql("SELECT TIMESTAMP'2024-01-02 03:04:05' AS
ts").toPandas()
+ assert local["ts"][0] == pd.Timestamp("2024-01-02 03:04:05")
+ finally:
+ session.conf.unset("spark.sql.session.timeZone")
+
+
def test_window_order_by_a_column_sorts_nulls_first_as_pyspark(session):
from pyspark.sql import Window
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index d8c9453..bcfaeec 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -6051,6 +6051,20 @@ def
test_execution_info_unsupported_literal_raises_on_flows(mock_connect):
info.flows
+def test_pandas_conversion_and_its_imports_are_pyspark_modules():
+ # Vendored from PySpark: toPandas converts with
pyspark.sql.pandas.conversion, which
+ # imports the classic error conversion and the gateway's socket helpers,
as PySpark does.
+ import inspect
+
+ import pyspark.errors.exceptions.captured as captured
+ import pyspark.traceback_utils # noqa: F401
+ import pyspark.util
+ from pyspark.sql.pandas.conversion import _convert_arrow_table_to_pandas
+
+ assert callable(_convert_arrow_table_to_pandas) and
callable(captured.convert_exception)
+ assert
inspect.isgeneratorfunction(pyspark.util._load_from_socket.__wrapped__)
+
+
def test_execute_plan_error_carries_operation_id(mock_connect):
from pyspark.errors import PySparkException
diff --git a/scripts/check_vendored_upstream.sh
b/scripts/check_vendored_upstream.sh
index 108600c..00923bd 100644
--- a/scripts/check_vendored_upstream.sh
+++ b/scripts/check_vendored_upstream.sh
@@ -47,9 +47,13 @@ VENDORED_DIRS=(
# Individual files under python/pyspark vendored verbatim from upstream.
Listed per-file
# (not per-dir) because their parent dirs also hold fork-adapted files that
intentionally
# differ (e.g. pipelines/api.py's connect->functions import repointing, and
the Rust-backed
-# errors/exceptions/__init__.py; upstream's
errors/exceptions/{captured,connect}.py are
-# omitted here as they need py4j/grpc). Keep these byte-identical to the tag.
+# errors/exceptions/__init__.py; upstream's errors/exceptions/connect.py is
omitted here
+# as it needs grpc, and captured.py imports py4j only where the JVM is used).
Keep these
+# byte-identical to the tag.
VENDORED_FILES=(
+ # Call sites of the JVM gateway, which pyspark.sql.pandas.conversion imports.
+ "traceback_utils.py"
+
# Logging
"logger/__init__.py"
"logger/logger.py"
@@ -61,6 +65,7 @@ VENDORED_FILES=(
"errors/error-conditions.json"
"errors/utils.py"
"errors/exceptions/base.py"
+ "errors/exceptions/captured.py"
"errors/exceptions/tblib.py"
# Pipelines
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]