This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 66e17ce  [SPARK-61000][CONNECT][PYTHON] Convert the columns of 
toPandas as PySpark does
66e17ce is described below

commit 66e17ce2a0a1b60ab67d83c1f43918071ef42dd7
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 18:10:42 2026 +0900

    [SPARK-61000][CONNECT][PYTHON] Convert the columns of toPandas as PySpark 
does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61000](https://issues.apache.org/jira/browse/SPARK-61000): 
`DataFrame.toPandas()` now converts each column of the Arrow result by its 
Spark type, as PySpark's `SparkConnectClient.to_pandas` does.
    
    - It first reads three settings in one request: the session time zone, 
`spark.sql.execution.pandas.structHandlingMode`, and 
`spark.sql.execution.arrow.pyspark.selfDestruct.enabled`. The 
`pandasStructHandlingMode` of `_to_pandas` overrides the struct handling mode, 
as in PySpark.
    - It converts the result with 
`pyspark.sql.pandas.conversion._convert_arrow_table_to_pandas`, of the schema 
of the result that the server sends with it, through the new 
`DataFrame::collect_record_batches_with_schema` of the Rust core.
    - The metrics and observed metrics of the execution are in `pdf.attrs`, as 
in PySpark.
    
    So timestamps are naive in the session time zone, maps are dicts, and 
values of user-defined types are deserialized.
    
    `pyspark.sql.pandas.conversion`, vendored from Apache Spark 4.3.0, did not 
import, so now:
    
    - `pyspark/errors/exceptions/captured.py` and `pyspark/traceback_utils.py` 
are vendored from Apache Spark 4.3.0, and checked by the drift check. 
`captured.py` imports py4j only where it uses the JVM.
    - `pyspark/util.py` has PySpark's local socket helpers, verbatim, which it 
imports.
    
    ### Why are the changes needed?
    
    `toPandas()` returned `pyarrow.Table.to_pandas()` of the result. Against a 
Spark 4.3.0 server:
    
    | Column | PySpark 4.3.0 and now | Before |
    | --- | --- | --- |
    | `TIMESTAMP'2024-01-02 03:04:05'` | `datetime64[ns]`, 
`Timestamp('2024-01-02 03:04:05')` in the session time zone | `datetime64[us, 
UTC]`, `Timestamp('2024-01-02 03:04:05+0000', tz='UTC')` |
    | `map('k', 1)` | `{'k': 1}` | `[('k', 1)]` |
    | a `VectorUDT` | `DenseVector([1.5, 2.0])` | `{'type': 1, 'size': None, 
'indices': None, 'values': array([1.5, 2.])}` |
    | `pdf.attrs` | `{'metrics': [PlanMetrics, ...]}` | `{}` |
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, `toPandas()` gives the pandas DataFrames of PySpark 4.3.0.
    
    ### How was this patch tested?
    
    - Against a local Spark 4.3.0 server, `toPandas()` gives the same dtypes 
and values as official PySpark 4.3.0 for timestamps, NTZ timestamps, structs, 
maps, arrays, decimals, dates, nulls, binaries and vectors. So do its `attrs`, 
with and without observed metrics, an empty result, `_to_pandas` with each 
struct handling mode, and a session time zone of `Asia/Seoul` with a client in 
`America/Los_Angeles`. Before this change, the timestamps, maps, vectors and 
`attrs` differed.
    - A new live test checks these. It also passes with official PySpark 4.3.0, 
and fails without this change. A new offline test checks that 
`pyspark.sql.pandas.conversion` imports.
    - The Arrow handoff test of `toPandas` now expects PySpark's conversion of 
the same Arrow table, and its mock server serves the settings.
    - The lib tests and every integration test, e2e included, against the local 
server (660 passed, and 123 of the transport crate), the build of all tests 
with `wasm-udf`, the offline Python suites (887 passed, 9 skipped), and the 
live drop-in and Arrow suites (82 passed) and live cases (10 passed). In the 
changed lines, clippy reports only the `result_large_err` of the `py.detach` 
closures, which every action of the Python `DataFrame` has.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #200 from HyukjinKwon/SPARK-61000-to-pandas.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/dataframe.rs            | 134 ++++++--
 crates/spark-connect/src/dataframe.rs         |  17 +-
 python/pyspark/errors/exceptions/captured.py  | 465 ++++++++++++++++++++++++++
 python/pyspark/sql/connect/client/__init__.py |  32 ++
 python/pyspark/traceback_utils.py             |  80 +++++
 python/pyspark/util.py                        | 196 +++++++++++
 python/tests/test_arrow_handoff.py            |  35 +-
 python/tests/test_dropin_e2e.py               |  32 ++
 python/tests/test_dropin_offline.py           |  14 +
 scripts/check_vendored_upstream.sh            |   9 +-
 10 files changed, 973 insertions(+), 41 deletions(-)

diff --git a/crates/pyspark-rs/src/dataframe.rs 
b/crates/pyspark-rs/src/dataframe.rs
index 54264e9..11a760b 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -50,6 +50,53 @@ pub struct PyDataFrame {
 }
 
 impl PyDataFrame {
+    /// The pandas DataFrame of the result, as `SparkConnectClient.to_pandas` 
makes it: the
+    /// settings of the conversion first, in one request; then the Arrow 
result with the schema
+    /// that the server sends, which `pyspark.sql.connect.client._to_pandas` 
converts.
+    fn to_pandas(
+        &self,
+        py: Python<'_>,
+        struct_handling_mode: Option<Bound<'_, PyAny>>,
+    ) -> PyResult<Py<PyAny>> {
+        let settings = py
+            .detach(|| {
+                self.dataframe.spark_session().conf().get_many(&[
+                    "spark.sql.session.timeZone",
+                    "spark.sql.execution.pandas.structHandlingMode",
+                    "spark.sql.execution.arrow.pyspark.selfDestruct.enabled",
+                ])
+            })
+            .to_pyerr()?;
+        let (batches, schema) = py
+            .detach(|| self.dataframe.collect_record_batches_with_schema())
+            .to_pyerr()?;
+        let table = crate::pyarrow_export::record_batches_to_pyarrow(py, 
batches)?;
+        let schema = match &schema {
+            Some(schema) => crate::types::data_type_to_py(py, schema)?,
+            None => py.None(),
+        };
+        let (metrics, observed) = match self.dataframe.execution_info() {
+            Ok(info) => execution_metrics(py, &info)?,
+            Err(_) => (
+                pyo3::types::PyList::empty(py),
+                pyo3::types::PyList::empty(py),
+            ),
+        };
+        let kwargs = pyo3::types::PyDict::new(py);
+        kwargs.set_item("timezone", &settings[0])?;
+        match struct_handling_mode {
+            Some(mode) => kwargs.set_item("struct_handling_mode", 
mode.str()?)?,
+            None => kwargs.set_item("struct_handling_mode", &settings[1])?,
+        }
+        kwargs.set_item("self_destruct", &settings[2])?;
+        let pdf = py.import("pyspark.sql.connect.client")?.call_method(
+            "_to_pandas",
+            (table, schema, metrics, observed),
+            Some(&kwargs),
+        )?;
+        Ok(pdf.unbind())
+    }
+
     /// The table of `show` and `_show_string`, for the arguments of 
`function`.
     fn show_table(
         &self,
@@ -1273,24 +1320,27 @@ impl PyDataFrame {
 
     /// Collect the DataFrame into a pandas DataFrame.
     ///
-    /// Mirrors `pyspark.sql.DataFrame.toPandas()`: decodes the result via 
Arrow
-    /// (`toArrow`) and converts the `pyarrow.Table` with `to_pandas()`.
+    /// Mirrors `pyspark.sql.DataFrame.toPandas()`, as 
`SparkConnectClient.to_pandas` converts
+    /// the Arrow result: each column by its Spark type, in the session time 
zone.
     fn toPandas(&self, py: Python<'_>) -> PyResult<Py<PyAny>> {
-        let table = self.to_arrow(py)?;
-        let df = table.bind(py).call_method0("to_pandas")?;
-        Ok(df.unbind())
+        self.to_pandas(py, None)
     }
 
     /// Internal Arrow-based pandas conversion used by pandas-on-Spark
-    /// (`pyspark.sql.connect.dataframe.DataFrame._to_pandas`). Accepts and 
ignores
-    /// keyword options such as `pandasStructHandlingMode`; delegates to 
`toPandas`.
-    #[pyo3(signature = (**_kwargs))]
+    /// (`pyspark.sql.connect.dataframe.DataFrame._to_pandas`): `toPandas`, 
with
+    /// `pandasStructHandlingMode` overriding 
`spark.sql.execution.pandas.structHandlingMode`,
+    /// as in `SparkConnectClient.to_pandas`.
+    #[pyo3(signature = (**kwargs))]
     fn _to_pandas(
         &self,
         py: Python<'_>,
-        _kwargs: Option<&Bound<'_, pyo3::types::PyDict>>,
+        kwargs: Option<&Bound<'_, pyo3::types::PyDict>>,
     ) -> PyResult<Py<PyAny>> {
-        self.toPandas(py)
+        let mode = match kwargs {
+            Some(kwargs) => kwargs.get_item("pandasStructHandlingMode")?,
+            None => None,
+        };
+        self.to_pandas(py, mode)
     }
 
     /// Get the count of rows.
@@ -1336,32 +1386,7 @@ impl PyDataFrame {
             return Ok(py.None());
         };
         let metrics_mod = py.import("pyspark.sql.metrics")?;
-        let plan_metrics_cls = metrics_mod.getattr("PlanMetrics")?;
-        let metric_value_cls = metrics_mod.getattr("MetricValue")?;
-        // Mirrors SparkConnectClient._build_metrics.
-        let plan_metrics = pyo3::types::PyList::empty(py);
-        if let Some(m) = &info.metrics {
-            for x in &m.metrics {
-                let values = pyo3::types::PyList::empty(py);
-                for (k, v) in &x.execution_metrics {
-                    values.append(metric_value_cls.call1((k, v.value, 
&v.metric_type))?)?;
-                }
-                plan_metrics
-                    .append(plan_metrics_cls.call1((&x.name, x.plan_id, 
x.parent, values))?)?;
-            }
-        }
-        let observed_cls = py
-            .import("pyspark.sql.connect.client")?
-            .getattr("PlanObservedMetrics")?;
-        let observed = pyo3::types::PyList::empty(py);
-        for o in &info.observed_metrics {
-            let values = o
-                .values
-                .iter()
-                .map(|l| literal_to_py(py, l, None))
-                .collect::<PyResult<Vec<_>>>()?;
-            observed.append(observed_cls.call1((&o.name, values, 
o.keys.clone()))?)?;
-        }
+        let (plan_metrics, observed) = execution_metrics(py, &info)?;
         let ei = metrics_mod.getattr("ExecutionInfo")?.call1((
             plan_metrics,
             observed,
@@ -2838,6 +2863,43 @@ impl PyLocalRowIterator {
     }
 }
 
+/// The metrics and observed metrics of an execution, as the lists of 
`PlanMetrics` and
+/// `PlanObservedMetrics` that `SparkConnectClient._build_metrics` makes.
+fn execution_metrics<'py>(
+    py: Python<'py>,
+    info: &spark_connect::session::ExecutionInfo,
+) -> PyResult<(
+    Bound<'py, pyo3::types::PyList>,
+    Bound<'py, pyo3::types::PyList>,
+)> {
+    let metrics_mod = py.import("pyspark.sql.metrics")?;
+    let plan_metrics_cls = metrics_mod.getattr("PlanMetrics")?;
+    let metric_value_cls = metrics_mod.getattr("MetricValue")?;
+    let plan_metrics = pyo3::types::PyList::empty(py);
+    if let Some(m) = &info.metrics {
+        for x in &m.metrics {
+            let values = pyo3::types::PyList::empty(py);
+            for (k, v) in &x.execution_metrics {
+                values.append(metric_value_cls.call1((k, v.value, 
&v.metric_type))?)?;
+            }
+            plan_metrics.append(plan_metrics_cls.call1((&x.name, x.plan_id, 
x.parent, values))?)?;
+        }
+    }
+    let observed_cls = py
+        .import("pyspark.sql.connect.client")?
+        .getattr("PlanObservedMetrics")?;
+    let observed = pyo3::types::PyList::empty(py);
+    for o in &info.observed_metrics {
+        let values = o
+            .values
+            .iter()
+            .map(|l| literal_to_py(py, l, None))
+            .collect::<PyResult<Vec<_>>>()?;
+        observed.append(observed_cls.call1((&o.name, values, 
o.keys.clone()))?)?;
+    }
+    Ok((plan_metrics, observed))
+}
+
 /// Convert an observed-metric literal to a Python value. Mirrors
 /// `LiteralExpression._to_value`, which `PlanObservedMetrics.pairs` applies: 
the kinds it
 /// decodes map to the same Python values, and for any other kind (which the 
reference
diff --git a/crates/spark-connect/src/dataframe.rs 
b/crates/spark-connect/src/dataframe.rs
index dc69cb9..b04de08 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -821,10 +821,22 @@ impl DataFrame {
     /// returning the raw `RecordBatch`es without converting to Rows. This is 
the
     /// foundation for `to_datafusion()` and `to_polars()` conversions.
     pub fn collect_record_batches(&self) -> 
Result<Vec<arrow::record_batch::RecordBatch>> {
+        self.collect_record_batches_with_schema()
+            .map(|(batches, _)| batches)
+    }
+
+    /// Collect all data as Arrow RecordBatches, with the schema of the result 
if the server
+    /// sends it (`ExecutePlanResponse.schema`), as the reference client's 
`to_pandas` reads
+    /// it: unlike the Arrow schema, it has the Spark types of the columns, 
such as their
+    /// user-defined types.
+    pub fn collect_record_batches_with_schema(
+        &self,
+    ) -> Result<(Vec<arrow::record_batch::RecordBatch>, Option<DataType>)> {
         let request = self.build_execute_request()?;
         let mut stream = 
block_on(self.session.client().execute_plan_reattachable(request))?;
 
         let mut batches = vec![];
+        let mut schema = None;
         let mut info = ExecutionInfo::default();
         loop {
             let resp = block_on(stream.message())?;
@@ -832,6 +844,9 @@ impl DataFrame {
                 break;
             };
             capture_execution(&mut resp, &mut info, &self.session);
+            if let Some(data_type) = resp.schema.take() {
+                schema = Some(DataType::from_proto(&data_type)?);
+            }
             if let 
Some(proto::execute_plan_response::ResponseType::ArrowBatch(batch)) =
                 resp.response_type
             {
@@ -841,7 +856,7 @@ impl DataFrame {
         }
         self.record_execution(info);
 
-        Ok(batches)
+        Ok((batches, schema))
     }
 
     /// Get the count of rows.
diff --git a/python/pyspark/errors/exceptions/captured.py 
b/python/pyspark/errors/exceptions/captured.py
new file mode 100644
index 0000000..da755de
--- /dev/null
+++ b/python/pyspark/errors/exceptions/captured.py
@@ -0,0 +1,465 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+import warnings
+from contextlib import contextmanager
+from typing import Any, Callable, Dict, Iterator, Optional, cast, List, 
TYPE_CHECKING
+
+from pyspark.errors.exceptions.base import (
+    AnalysisException as BaseAnalysisException,
+    IllegalArgumentException as BaseIllegalArgumentException,
+    ArithmeticException as BaseArithmeticException,
+    UnsupportedOperationException as BaseUnsupportedOperationException,
+    ArrayIndexOutOfBoundsException as BaseArrayIndexOutOfBoundsException,
+    DateTimeException as BaseDateTimeException,
+    NumberFormatException as BaseNumberFormatException,
+    ParseException as BaseParseException,
+    PySparkException,
+    PythonException as BasePythonException,
+    QueryExecutionException as BaseQueryExecutionException,
+    SparkRuntimeException as BaseSparkRuntimeException,
+    SparkUpgradeException as BaseSparkUpgradeException,
+    SparkNoSuchElementException as BaseNoSuchElementException,
+    StreamingQueryException as BaseStreamingQueryException,
+    UnknownException as BaseUnknownException,
+    QueryContext as BaseQueryContext,
+    QueryContextType,
+    recover_python_exception,
+)
+
+if TYPE_CHECKING:
+    from py4j.protocol import Py4JJavaError
+    from py4j.java_gateway import JavaObject
+
+
+class CapturedException(PySparkException):
+    def __init__(
+        self,
+        desc: Optional[str] = None,
+        stackTrace: Optional[str] = None,
+        cause: Optional["Py4JJavaError"] = None,
+        origin: Optional["Py4JJavaError"] = None,
+    ):
+        from pyspark import SparkContext
+        from py4j.protocol import Py4JJavaError
+
+        # desc & stackTrace vs origin are mutually exclusive.
+        # cause is optional.
+        assert (origin is not None and desc is None and stackTrace is None) or 
(
+            origin is None and desc is not None and stackTrace is not None
+        )
+
+        self._desc = desc if desc is not None else cast(Py4JJavaError, 
origin).getMessage()
+        if self._desc is None:
+            self._desc = ""
+        assert SparkContext._jvm is not None
+        self._stackTrace = (
+            stackTrace
+            if stackTrace is not None
+            else (getattr(SparkContext._jvm, 
"org.apache.spark.util.Utils").exceptionString(origin))
+        )
+        self._cause = convert_exception(cause) if cause is not None else None
+        if self._cause is None and origin is not None and origin.getCause() is 
not None:
+            self._cause = convert_exception(origin.getCause())
+        self._origin = origin
+        self._log_exception()
+
+    def __str__(self) -> str:
+        from pyspark import SparkContext
+
+        assert SparkContext._jvm is not None
+
+        jvm = SparkContext._jvm
+
+        # SPARK-42752: default to True to see issues with initialization
+        debug_enabled = True
+        try:
+            sql_conf = getattr(jvm, 
"org.apache.spark.sql.internal.SQLConf").get()
+            debug_enabled = sql_conf.pysparkJVMStacktraceEnabled()
+        except BaseException:
+            pass
+
+        desc = self._desc
+        if debug_enabled:
+            desc = desc + "\n\nJVM stacktrace:\n%s" % self._stackTrace
+        return str(desc)
+
+    def getCondition(self) -> Optional[str]:
+        from pyspark import SparkContext
+        from py4j.java_gateway import is_instance_of
+
+        assert SparkContext._gateway is not None
+        assert SparkContext._jvm is not None
+
+        gw = SparkContext._gateway
+        if self._origin is not None and is_instance_of(
+            gw, self._origin, "org.apache.spark.SparkThrowable"
+        ):
+            utils = SparkContext._jvm.PythonErrorUtils
+            return utils.getCondition(self._origin)
+        else:
+            return None
+
+    def getErrorClass(self) -> Optional[str]:
+        warnings.warn("Deprecated in 4.0.0, use getCondition instead.", 
FutureWarning)
+        return self.getCondition()
+
+    def getMessageParameters(self) -> Optional[Dict[str, str]]:
+        from pyspark import SparkContext
+        from py4j.java_gateway import is_instance_of
+
+        assert SparkContext._gateway is not None
+        assert SparkContext._jvm is not None
+
+        gw = SparkContext._gateway
+        if self._origin is not None and is_instance_of(
+            gw, self._origin, "org.apache.spark.SparkThrowable"
+        ):
+            utils = SparkContext._jvm.PythonErrorUtils
+            return dict(utils.getMessageParameters(self._origin))
+        else:
+            return None
+
+    def getSqlState(self) -> Optional[str]:
+        from pyspark import SparkContext
+        from py4j.java_gateway import is_instance_of
+
+        assert SparkContext._gateway is not None
+        assert SparkContext._jvm is not None
+        gw = SparkContext._gateway
+        if self._origin is not None and is_instance_of(
+            gw, self._origin, "org.apache.spark.SparkThrowable"
+        ):
+            utils = SparkContext._jvm.PythonErrorUtils
+            return utils.getSqlState(self._origin)
+        else:
+            return None
+
+    def getMessage(self) -> str:
+        from pyspark import SparkContext
+        from py4j.java_gateway import is_instance_of
+
+        assert SparkContext._gateway is not None
+        assert SparkContext._jvm is not None
+        gw = SparkContext._gateway
+
+        if self._origin is not None and is_instance_of(
+            gw, self._origin, "org.apache.spark.SparkThrowable"
+        ):
+            utils = SparkContext._jvm.PythonErrorUtils
+            errorClass = utils.getCondition(self._origin)
+            messageParameters = utils.getMessageParameters(self._origin)
+
+            error_message = getattr(gw.jvm, 
"org.apache.spark.SparkThrowableHelper").getMessage(
+                errorClass, messageParameters
+            )
+
+            return error_message
+        else:
+            return ""
+
+    def getQueryContext(self) -> List[BaseQueryContext]:
+        from pyspark import SparkContext
+        from py4j.java_gateway import is_instance_of
+
+        assert SparkContext._gateway is not None
+        assert SparkContext._jvm is not None
+
+        gw = SparkContext._gateway
+        if self._origin is not None and is_instance_of(
+            gw, self._origin, "org.apache.spark.SparkThrowable"
+        ):
+            contexts: List[BaseQueryContext] = []
+            utils = SparkContext._jvm.PythonErrorUtils
+            for q in utils.getQueryContext(self._origin):
+                if q.contextType().toString() == "SQL":
+                    contexts.append(SQLQueryContext(q))
+                else:
+                    contexts.append(DataFrameQueryContext(q))
+
+            return contexts
+        else:
+            return []
+
+
+def convert_exception(e: "Py4JJavaError") -> CapturedException:
+    converted = _convert_exception(e)
+    return recover_python_exception(converted)
+
+
+def _convert_exception(e: "Py4JJavaError") -> CapturedException:
+    from pyspark import SparkContext
+    from py4j.java_gateway import is_instance_of
+
+    assert e is not None
+    assert SparkContext._jvm is not None
+    assert SparkContext._gateway is not None
+
+    jvm = SparkContext._jvm
+    gw = SparkContext._gateway
+
+    if is_instance_of(gw, e, 
"org.apache.spark.sql.catalyst.parser.ParseException"):
+        return ParseException(origin=e)
+    # Order matters. ParseException inherits AnalysisException.
+    elif is_instance_of(gw, e, "org.apache.spark.sql.AnalysisException"):
+        return AnalysisException(origin=e)
+    elif is_instance_of(gw, e, 
"org.apache.spark.sql.streaming.StreamingQueryException"):
+        return StreamingQueryException(origin=e)
+    elif is_instance_of(gw, e, 
"org.apache.spark.sql.execution.QueryExecutionException"):
+        return QueryExecutionException(origin=e)
+    # Order matters. NumberFormatException inherits IllegalArgumentException.
+    elif is_instance_of(gw, e, "java.lang.NumberFormatException"):
+        return NumberFormatException(origin=e)
+    elif is_instance_of(gw, e, "java.lang.IllegalArgumentException"):
+        return IllegalArgumentException(origin=e)
+    elif is_instance_of(gw, e, "java.lang.ArithmeticException"):
+        return ArithmeticException(origin=e)
+    elif is_instance_of(gw, e, "java.lang.UnsupportedOperationException"):
+        return UnsupportedOperationException(origin=e)
+    elif is_instance_of(gw, e, "java.lang.ArrayIndexOutOfBoundsException"):
+        return ArrayIndexOutOfBoundsException(origin=e)
+    elif is_instance_of(gw, e, "java.time.DateTimeException"):
+        return DateTimeException(origin=e)
+    elif is_instance_of(gw, e, "org.apache.spark.SparkRuntimeException"):
+        return SparkRuntimeException(origin=e)
+    elif is_instance_of(gw, e, "org.apache.spark.SparkUpgradeException"):
+        return SparkUpgradeException(origin=e)
+    elif is_instance_of(gw, e, "org.apache.spark.SparkNoSuchElementException"):
+        return SparkNoSuchElementException(origin=e)
+    elif is_instance_of(gw, e, "org.apache.spark.api.python.PythonException"):
+        return PythonException(origin=e)
+    return UnknownException(
+        desc=e.toString(),
+        stackTrace=getattr(jvm, 
"org.apache.spark.util.Utils").exceptionString(e),
+        cause=e.getCause(),
+    )
+
+
+def capture_sql_exception(f: Callable[..., Any]) -> Callable[..., Any]:
+    def deco(*a: Any, **kw: Any) -> Any:
+        from py4j.protocol import Py4JJavaError
+
+        try:
+            return f(*a, **kw)
+        except Py4JJavaError as e:
+            converted = convert_exception(e.java_exception)
+            if not isinstance(converted, UnknownException):
+                # Hide where the exception came from that shows a non-Pythonic
+                # JVM exception message.
+                raise converted from None
+            else:
+                raise
+
+    return deco
+
+
+@contextmanager
+def unwrap_spark_exception() -> Iterator[Any]:
+    from pyspark import SparkContext
+    from py4j.protocol import Py4JJavaError
+    from py4j.java_gateway import is_instance_of
+
+    assert SparkContext._gateway is not None
+
+    gw = SparkContext._gateway
+    try:
+        yield
+    except Py4JJavaError as e:
+        je: "Py4JJavaError" = e.java_exception
+        if je is not None and is_instance_of(gw, je, 
"org.apache.spark.SparkException"):
+            converted = convert_exception(je.getCause())
+            if not isinstance(converted, UnknownException):
+                raise converted from None
+        raise
+
+
+def install_exception_handler() -> None:
+    """
+    Hook an exception handler into Py4j, which could capture some SQL 
exceptions in Java.
+
+    When calling Java API, it will call `get_return_value` to parse the 
returned object.
+    If any exception happened in JVM, the result will be Java exception 
object, it raise
+    py4j.protocol.Py4JJavaError. We replace the original `get_return_value` 
with one that
+    could capture the Java exception and throw a Python one (with the same 
error message).
+
+    It's idempotent, could be called multiple times.
+    """
+    import py4j
+
+    original = py4j.protocol.get_return_value
+    # The original `get_return_value` is not patched, it's idempotent.
+    patched = capture_sql_exception(original)
+    # only patch the one used in py4j.java_gateway (call Java API)
+    py4j.java_gateway.get_return_value = patched
+
+
+class AnalysisException(CapturedException, BaseAnalysisException):
+    """
+    Failed to analyze a SQL query plan.
+    """
+
+
+class ParseException(AnalysisException, BaseParseException):
+    """
+    Failed to parse a SQL command.
+    """
+
+
+class IllegalArgumentException(CapturedException, 
BaseIllegalArgumentException):
+    """
+    Passed an illegal or inappropriate argument.
+    """
+
+
+class StreamingQueryException(CapturedException, BaseStreamingQueryException):
+    """
+    Exception that stopped a :class:`StreamingQuery`.
+    """
+
+
+class QueryExecutionException(CapturedException, BaseQueryExecutionException):
+    """
+    Failed to execute a query.
+    """
+
+
+class PythonException(CapturedException, BasePythonException):
+    """
+    Exceptions thrown from Python workers.
+    """
+
+    def __str__(self) -> str:
+        messageParameters = self.getMessageParameters()
+
+        if (
+            messageParameters is None
+            or "msg" not in messageParameters
+            or "traceback" not in messageParameters
+        ):
+            return super().__str__()
+        return 
f"{messageParameters['msg']}:\n{messageParameters['traceback'].strip()}"
+
+
+class ArithmeticException(CapturedException, BaseArithmeticException):
+    """
+    Arithmetic exception.
+    """
+
+
+class UnsupportedOperationException(CapturedException, 
BaseUnsupportedOperationException):
+    """
+    Unsupported operation exception.
+    """
+
+
+class ArrayIndexOutOfBoundsException(CapturedException, 
BaseArrayIndexOutOfBoundsException):
+    """
+    Array index out of bounds exception.
+    """
+
+
+class DateTimeException(CapturedException, BaseDateTimeException):
+    """
+    Datetime exception.
+    """
+
+
+class NumberFormatException(IllegalArgumentException, 
BaseNumberFormatException):
+    """
+    Number format exception.
+    """
+
+
+class SparkRuntimeException(CapturedException, BaseSparkRuntimeException):
+    """
+    Runtime exception.
+    """
+
+
+class SparkUpgradeException(CapturedException, BaseSparkUpgradeException):
+    """
+    Exception thrown because of Spark upgrade.
+    """
+
+
+class SparkNoSuchElementException(CapturedException, 
BaseNoSuchElementException):
+    """
+    No such element exception.
+    """
+
+
+class UnknownException(CapturedException, BaseUnknownException):
+    """
+    None of the other exceptions.
+    """
+
+
+class SQLQueryContext(BaseQueryContext):
+    def __init__(self, q: "JavaObject"):
+        self._q = q
+
+    def contextType(self) -> QueryContextType:
+        return QueryContextType.SQL
+
+    def objectType(self) -> str:
+        return str(self._q.objectType())
+
+    def objectName(self) -> str:
+        return str(self._q.objectName())
+
+    def startIndex(self) -> int:
+        return int(self._q.startIndex())
+
+    def stopIndex(self) -> int:
+        return int(self._q.stopIndex())
+
+    def fragment(self) -> str:
+        return str(self._q.fragment())
+
+    def callSite(self) -> str:
+        return str(self._q.callSite())
+
+    def summary(self) -> str:
+        return str(self._q.summary())
+
+
+class DataFrameQueryContext(BaseQueryContext):
+    def __init__(self, q: "JavaObject"):
+        self._q = q
+
+    def contextType(self) -> QueryContextType:
+        return QueryContextType.DataFrame
+
+    def objectType(self) -> str:
+        return str(self._q.objectType())
+
+    def objectName(self) -> str:
+        return str(self._q.objectName())
+
+    def startIndex(self) -> int:
+        return int(self._q.startIndex())
+
+    def stopIndex(self) -> int:
+        return int(self._q.stopIndex())
+
+    def fragment(self) -> str:
+        return str(self._q.fragment())
+
+    def callSite(self) -> str:
+        return str(self._q.callSite())
+
+    def summary(self) -> str:
+        return str(self._q.summary())
diff --git a/python/pyspark/sql/connect/client/__init__.py 
b/python/pyspark/sql/connect/client/__init__.py
index f8b50e2..96c383c 100644
--- a/python/pyspark/sql/connect/client/__init__.py
+++ b/python/pyspark/sql/connect/client/__init__.py
@@ -81,3 +81,35 @@ class PlanObservedMetrics(ObservedMetrics):
 
     def to_dict(self) -> Dict[str, Any]:
         return {"name": self._name, "keys": self._keys, "pairs": self.pairs}
+
+
+def _to_pandas(
+    table: Any,
+    schema: Any,
+    metrics: List[Any],
+    observed_metrics: List[Any],
+    *,
+    timezone: Any,
+    struct_handling_mode: Any,
+    self_destruct: Any,
+) -> Any:
+    """The pandas DataFrame of the Arrow ``table`` of a result, of ``schema`` 
if the server sent
+    it, as ``SparkConnectClient.to_pandas`` converts it: each column by its 
Spark type, in the
+    session time zone, with the metrics of the execution in ``attrs``."""
+    from pyspark.sql.pandas.conversion import _convert_arrow_table_to_pandas
+    from pyspark.sql.pandas.types import from_arrow_schema
+
+    schema = schema or from_arrow_schema(table.schema, 
prefer_timestamp_ntz=True)
+    pdf = _convert_arrow_table_to_pandas(
+        arrow_table=table,
+        schema=schema,
+        timezone=timezone,
+        struct_handling_mode=struct_handling_mode,
+        date_as_object=False,
+        self_destruct=self_destruct == "true",
+    )
+    if len(metrics) > 0:
+        pdf.attrs["metrics"] = metrics
+    if len(observed_metrics) > 0:
+        pdf.attrs["observed_metrics"] = observed_metrics
+    return pdf
diff --git a/python/pyspark/traceback_utils.py 
b/python/pyspark/traceback_utils.py
new file mode 100644
index 0000000..fcc388b
--- /dev/null
+++ b/python/pyspark/traceback_utils.py
@@ -0,0 +1,80 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from collections import namedtuple
+import os
+import traceback
+
+CallSite = namedtuple("CallSite", "function file linenum")
+
+
+def first_spark_call():
+    """
+    Return a CallSite representing the first Spark call in the current call 
stack.
+    """
+    tb = traceback.extract_stack()
+    if len(tb) == 0:
+        return None
+    file, line, module, what = tb[len(tb) - 1]
+    sparkpath = os.path.dirname(file)
+    first_spark_frame = len(tb) - 1
+    for i in range(0, len(tb)):
+        file, line, fun, what = tb[i]
+        if file.startswith(sparkpath):
+            first_spark_frame = i
+            break
+    if first_spark_frame == 0:
+        file, line, fun, what = tb[0]
+        return CallSite(function=fun, file=file, linenum=line)
+    sfile, sline, sfun, swhat = tb[first_spark_frame]
+    ufile, uline, ufun, uwhat = tb[first_spark_frame - 1]
+    return CallSite(function=sfun, file=ufile, linenum=uline)
+
+
+class SCCallSiteSync:
+    """
+    Helper for setting the spark context call site.
+
+    Example usage:
+    from pyspark.core.context import SCCallSiteSync
+    with SCCallSiteSync(<relevant SparkContext>) as css:
+        <a Spark call>
+    """
+
+    _spark_stack_depth = 0
+
+    def __init__(self, sc):
+        call_site = first_spark_call()
+        if call_site is not None:
+            self._call_site = "%s at %s:%s" % (
+                call_site.function,
+                call_site.file,
+                call_site.linenum,
+            )
+        else:
+            self._call_site = "Error! Could not extract traceback info"
+        self._context = sc
+
+    def __enter__(self):
+        if SCCallSiteSync._spark_stack_depth == 0:
+            self._context._jsc.setCallSite(self._call_site)
+        SCCallSiteSync._spark_stack_depth += 1
+
+    def __exit__(self, type, value, tb):
+        SCCallSiteSync._spark_stack_depth -= 1
+        if SCCallSiteSync._spark_stack_depth == 0:
+            self._context._jsc.setCallSite(None)
diff --git a/python/pyspark/util.py b/python/pyspark/util.py
index 84bc316..e74aecb 100644
--- a/python/pyspark/util.py
+++ b/python/pyspark/util.py
@@ -299,3 +299,199 @@ 
________________________________________________________________________________
             "spark_version": spark_version,
         }
     )
+
+
+# Verbatim from PySpark: the local socket helpers of the JVM gateway. Spark 
Connect does not
+# use them, but the modules of PySpark that the client vendors, such as
+# pyspark.sql.pandas.conversion, import them.
+import io  # noqa: E402
+import socket  # noqa: E402
+from typing import Union  # noqa: E402
+
+from pyspark.errors import PySparkRuntimeError  # noqa: E402
+from pyspark.serializers import UTF8Deserializer  # noqa: E402
+
+
+def _create_local_socket(sock_info: "JavaArray") -> Tuple["io.BufferedRWPair", 
"socket.socket"]:
+    """
+    Create a local socket that can be used to load deserialized data from the 
JVM
+
+    Parameters
+    ----------
+    sock_info : tuple
+        Tuple containing port number and authentication secret for a local 
socket.
+
+    Returns
+    -------
+    sockfile file descriptor of the local socket
+    """
+    sockfile: "io.BufferedRWPair"
+    sock: "socket.socket"
+    conn_info: int = sock_info[0]
+    auth_secret: str = sock_info[1]
+    sockfile, sock = local_connect_and_auth(conn_info, auth_secret)
+    # The RDD materialization time is unpredictable, if we set a timeout for 
socket reading
+    # operation, it will very possibly fail. See SPARK-18281.
+    sock.settimeout(None)
+    return sockfile, sock
+
+
+@contextmanager
+def _load_from_socket(sock_info: "JavaArray", serializer: "Serializer") -> 
Iterator[Any]:
+    """
+    Connect to a local socket described by sock_info and use the given 
serializer to yield data
+
+    Parameters
+    ----------
+    sock_info : tuple
+        Tuple containing port number and authentication secret for a local 
socket.
+    serializer : class:`Serializer`
+        The PySpark serializer to use
+
+    Returns
+    -------
+    result of meth:`Serializer.load_stream`,
+    usually a generator that yields deserialized data
+    """
+    try:
+        sockfile, sock = _create_local_socket(sock_info)
+        yield serializer.load_stream(sockfile)
+    finally:
+        sockfile.close()
+        sock.close()
+
+
+def _local_iterator_from_socket(sock_info: "JavaArray", serializer: 
"Serializer") -> Iterator[Any]:
+    class PyLocalIterable:
+        """Create a synchronous local iterable over a socket"""
+
+        def __init__(self, _sock_info: "JavaArray", _serializer: "Serializer"):
+            port: int
+            auth_secret: str
+            self.jsocket_auth_server: "JavaObject"
+            port, auth_secret, self.jsocket_auth_server = _sock_info
+            self._sockfile, self._sock = _create_local_socket((port, 
auth_secret))
+            self._serializer = _serializer
+            self._read_iter: Iterator[Any] = iter([])  # Initialize as empty 
iterator
+            self._read_status = 1
+
+        def __iter__(self) -> Iterator[Any]:
+            while self._read_status == 1:
+                # Request next partition data from Java
+                write_int(1, self._sockfile)
+                self._sockfile.flush()
+
+                # If response is 1 then there is a partition to read, if 0 
then fully consumed
+                self._read_status = read_int(self._sockfile)
+                if self._read_status == 1:
+                    # Load the partition data as a stream and read each item
+                    self._read_iter = 
self._serializer.load_stream(self._sockfile)
+                    for item in self._read_iter:
+                        yield item
+
+                # An error occurred, join serving thread and raise any 
exceptions from the JVM
+                elif self._read_status == -1:
+                    self.jsocket_auth_server.getResult()
+
+        def __del__(self) -> None:
+            # If local iterator is not fully consumed,
+            if self._read_status == 1:
+                try:
+                    # Finish consuming partition data stream
+                    for _ in self._read_iter:
+                        pass
+                    # Tell Java to stop sending data and close connection
+                    write_int(0, self._sockfile)
+                    self._sockfile.flush()
+                except Exception:
+                    # Ignore any errors, socket is automatically closed when 
garbage-collected
+                    pass
+            self._sockfile.close()
+            self._sock.close()
+
+    return iter(PyLocalIterable(sock_info, serializer))
+
+
+def local_connect_and_auth(
+    conn_info: Optional[Union[str, int]], auth_secret: Optional[str]
+) -> Tuple:
+    """
+    Connect to local host, authenticate with it, and return a (sockfile,sock) 
for that connection.
+    Handles IPV4 & IPV6, does some error handling.
+
+    Parameters
+    ----------
+    port : str or int, optional
+    auth_secret : str, optional
+
+    Returns
+    -------
+    tuple
+        with (sockfile, sock)
+    """
+    is_unix_domain_socket = isinstance(conn_info, str) and auth_secret is None
+    if is_unix_domain_socket:
+        sock_path = conn_info
+        assert isinstance(sock_path, str)
+        sock = None
+        try:
+            sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+            sock.settimeout(int(os.environ.get("SPARK_AUTH_SOCKET_TIMEOUT", 
15)))
+            sock.connect(sock_path)
+            sockfile = sock.makefile("rwb", 
int(os.environ.get("SPARK_BUFFER_SIZE", 65536)))
+            return (sockfile, sock)
+        except socket.error as e:
+            if sock is not None:
+                sock.close()
+            raise PySparkRuntimeError(
+                errorClass="CANNOT_OPEN_SOCKET",
+                messageParameters={
+                    "errors": "tried to connect to %s, but an error occurred: 
%s"
+                    % (sock_path, str(e)),
+                },
+            )
+
+    sock = None
+    errors = []
+    # Support for both IPv4 and IPv6.
+    addr = "127.0.0.1"
+    if os.environ.get("SPARK_PREFER_IPV6", "false").lower() == "true":
+        addr = "::1"
+    for res in socket.getaddrinfo(addr, conn_info, socket.AF_UNSPEC, 
socket.SOCK_STREAM):
+        af, socktype, proto, _, sa = res
+        try:
+            sock = socket.socket(af, socktype, proto)
+            sock.settimeout(int(os.environ.get("SPARK_AUTH_SOCKET_TIMEOUT", 
15)))
+            sock.connect(sa)
+            sockfile = sock.makefile("rwb", 
int(os.environ.get("SPARK_BUFFER_SIZE", 65536)))
+            assert isinstance(auth_secret, str)
+            _do_server_auth(sockfile, auth_secret)
+            return (sockfile, sock)
+        except socket.error as e:
+            emsg = str(e)
+            errors.append("tried to connect to %s, but an error occurred: %s" 
% (sa, emsg))
+            if sock is not None:
+                sock.close()
+                sock = None
+    raise PySparkRuntimeError(
+        errorClass="CANNOT_OPEN_SOCKET",
+        messageParameters={
+            "errors": str(errors),
+        },
+    )
+
+
+def _do_server_auth(conn: "io.IOBase", auth_secret: str) -> None:
+    """
+    Performs the authentication protocol defined by the SocketAuthHelper class 
on the given
+    file-like object 'conn'.
+    """
+    write_with_length(auth_secret.encode("utf-8"), conn)
+    conn.flush()
+    reply = UTF8Deserializer().loads(conn)
+    if reply != "ok":
+        conn.close()
+        raise PySparkRuntimeError(
+            errorClass="UNEXPECTED_RESPONSE_FROM_SERVER",
+            messageParameters={},
+        )
diff --git a/python/tests/test_arrow_handoff.py 
b/python/tests/test_arrow_handoff.py
index 5f8a74b..68af777 100644
--- a/python/tests/test_arrow_handoff.py
+++ b/python/tests/test_arrow_handoff.py
@@ -129,6 +129,23 @@ def _dataframe(batches, *, protocol, payloads=None, 
calls=None):
             server_side_session_id=server_id,
         ).SerializeToString()
 
+    def config(request, context):
+        # The settings that toPandas reads first, as 
SparkConnectClient.to_pandas does.
+        if calls is not None:
+            calls["config"] = calls.get("config", 0) + 1
+        request = protocol("ConfigRequest").FromString(request)
+        settings = {
+            "spark.sql.session.timeZone": "UTC",
+            "spark.sql.execution.pandas.structHandlingMode": "legacy",
+            "spark.sql.execution.arrow.pyspark.selfDestruct.enabled": "false",
+        }
+        return protocol(
+            "ConfigResponse",
+            session_id=request.session_id,
+            server_side_session_id=server_id,
+            pairs=[dict(key=key, value=settings[key]) for key in 
request.operation.get.keys],
+        ).SerializeToString()
+
     def release_session(request, context):
         request = protocol("ReleaseSessionRequest").FromString(request)
         return protocol(
@@ -147,6 +164,7 @@ def _dataframe(batches, *, protocol, payloads=None, 
calls=None):
                         "ExecutePlan": 
grpc.unary_stream_rpc_method_handler(execute),
                         "ReleaseExecute": 
grpc.unary_unary_rpc_method_handler(release_execute),
                         "ReleaseSession": 
grpc.unary_unary_rpc_method_handler(release_session),
+                        "Config": grpc.unary_unary_rpc_method_handler(config),
                     },
                 )
             ]
@@ -288,12 +306,25 @@ def test_to_pandas_uses_the_same_handoff(protocol):
     arrays = [batch.column(index) for index in indices]
     selected = pa.RecordBatch.from_arrays(arrays, schema=schema)
     batches = [selected.slice(0, 2), selected.slice(2, 2)]
-    expected = pa.Table.from_batches(batches).to_pandas()
+    # As SparkConnectClient.to_pandas converts the same table: of the Spark 
types of its Arrow
+    # schema (the server sends no schema here), in the session time zone.
+    from pyspark.sql.pandas.conversion import _convert_arrow_table_to_pandas
+    from pyspark.sql.pandas.types import from_arrow_schema
+
+    table = pa.Table.from_batches(batches)
+    expected = _convert_arrow_table_to_pandas(
+        arrow_table=table,
+        schema=from_arrow_schema(table.schema, prefer_timestamp_ntz=True),
+        timezone="UTC",
+        struct_handling_mode="legacy",
+        date_as_object=False,
+        self_destruct=False,
+    )
     calls = {}
     with _dataframe(protocol=protocol, batches=batches, calls=calls) as 
(frame, _):
         actual = frame.toPandas()
     pd.testing.assert_frame_equal(actual, expected)
-    assert calls == {"execute": 1}
+    assert calls == {"config": 1, "execute": 1}
 
 
 @pytest.mark.parametrize(
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 570b9a4..2896f8e 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -793,6 +793,38 @@ def 
test_column_alias_names_every_column_and_keeps_its_metadata(session):
     assert df.schema.fields[0].metadata == {"k": 1, "n": [1, 2], "s": "v"}
 
 
+def test_to_pandas_converts_the_columns_as_pyspark(session):
+    # As official PySpark 4.3.0's SparkConnectClient.to_pandas: each column by 
its Spark type,
+    # in the session time zone, with the metrics of the execution in attrs.
+    from decimal import Decimal
+
+    import pandas as pd
+    from pyspark.ml.functions import array_to_vector
+    from pyspark.ml.linalg import DenseVector
+
+    df = session.sql(
+        "SELECT TIMESTAMP'2024-01-02 03:04:05' AS ts, map('k', 1) AS m, "
+        "named_struct('a', 1, 'b', 'x') AS st, 1.5BD AS d, array(1.5D, 2.0D) 
AS av"
+    ).withColumn("vec", array_to_vector("av")).drop("av")
+    pdf = df.toPandas()
+    assert str(pdf.dtypes["ts"]) == "datetime64[ns]"
+    assert pdf["ts"][0] == pd.Timestamp("2024-01-02 03:04:05")
+    assert pdf["m"][0] == {"k": 1}
+    assert pdf["st"][0] == {"a": 1, "b": "x"}
+    assert pdf["d"][0] == Decimal("1.5")
+    assert pdf["vec"][0] == DenseVector([1.5, 2.0])
+    assert sorted(pdf.attrs) == ["metrics"]
+    assert type(pdf.attrs["metrics"][0]).__name__ == "PlanMetrics"
+    struct = 
df.select("st")._to_pandas(pandasStructHandlingMode="row")["st"][0]
+    assert type(struct).__name__ == "Row" and struct == Row(a=1, b="x")
+    session.conf.set("spark.sql.session.timeZone", "Asia/Seoul")
+    try:
+        local = session.sql("SELECT TIMESTAMP'2024-01-02 03:04:05' AS 
ts").toPandas()
+        assert local["ts"][0] == pd.Timestamp("2024-01-02 03:04:05")
+    finally:
+        session.conf.unset("spark.sql.session.timeZone")
+
+
 def test_window_order_by_a_column_sorts_nulls_first_as_pyspark(session):
     from pyspark.sql import Window
 
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index d8c9453..bcfaeec 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -6051,6 +6051,20 @@ def 
test_execution_info_unsupported_literal_raises_on_flows(mock_connect):
         info.flows
 
 
+def test_pandas_conversion_and_its_imports_are_pyspark_modules():
+    # Vendored from PySpark: toPandas converts with 
pyspark.sql.pandas.conversion, which
+    # imports the classic error conversion and the gateway's socket helpers, 
as PySpark does.
+    import inspect
+
+    import pyspark.errors.exceptions.captured as captured
+    import pyspark.traceback_utils  # noqa: F401
+    import pyspark.util
+    from pyspark.sql.pandas.conversion import _convert_arrow_table_to_pandas
+
+    assert callable(_convert_arrow_table_to_pandas) and 
callable(captured.convert_exception)
+    assert 
inspect.isgeneratorfunction(pyspark.util._load_from_socket.__wrapped__)
+
+
 def test_execute_plan_error_carries_operation_id(mock_connect):
     from pyspark.errors import PySparkException
 
diff --git a/scripts/check_vendored_upstream.sh 
b/scripts/check_vendored_upstream.sh
index 108600c..00923bd 100644
--- a/scripts/check_vendored_upstream.sh
+++ b/scripts/check_vendored_upstream.sh
@@ -47,9 +47,13 @@ VENDORED_DIRS=(
 # Individual files under python/pyspark vendored verbatim from upstream. 
Listed per-file
 # (not per-dir) because their parent dirs also hold fork-adapted files that 
intentionally
 # differ (e.g. pipelines/api.py's connect->functions import repointing, and 
the Rust-backed
-# errors/exceptions/__init__.py; upstream's 
errors/exceptions/{captured,connect}.py are
-# omitted here as they need py4j/grpc). Keep these byte-identical to the tag.
+# errors/exceptions/__init__.py; upstream's errors/exceptions/connect.py is 
omitted here
+# as it needs grpc, and captured.py imports py4j only where the JVM is used). 
Keep these
+# byte-identical to the tag.
 VENDORED_FILES=(
+  # Call sites of the JVM gateway, which pyspark.sql.pandas.conversion imports.
+  "traceback_utils.py"
+
   # Logging
   "logger/__init__.py"
   "logger/logger.py"
@@ -61,6 +65,7 @@ VENDORED_FILES=(
   "errors/error-conditions.json"
   "errors/utils.py"
   "errors/exceptions/base.py"
+  "errors/exceptions/captured.py"
   "errors/exceptions/tblib.py"
 
   # Pipelines


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to