This is an automated email from the ASF dual-hosted git repository.
gaogaotiantian pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new e312611241ad [SPARK-60141][PYTHON] Correct SparkContext profiler
collector type annotation
e312611241ad is described below
commit e312611241ad5c5e47f87cd8de2176c66c4aa43b
Author: Matthew M.M <[email protected]>
AuthorDate: Sun Oct 11 00:39:24 2026 -0700
[SPARK-60141][PYTHON] Correct SparkContext profiler collector type
annotation
### What changes were proposed in this pull request?
This PR updates `SparkContext.profiler_collector` to
`Optional[ProfilerCollector]` to reflect its actual runtime value. Also adds
type narrowing assertions at the RDD and UDF profiler call sites.
### Why are the changes needed?
When Python profiling is disabled, `SparkContext.profiler_collector` is
set to `None`. However, it was annotated as always being a `ProfilerCollector`,
which does not reflect the actual runtime state and requires a `type:
ignore[assignment]` comment.
The updated annotation correctly represents both possible values and
allows the assignment type ignore to be removed.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
The following checks were run:
- `mypy --config-file python/mypy.ini python/pyspark`
- `dev/lint-python --mypy --ruff`
- `python/run-tests --testnames pyspark.tests.test_profiler`
- `python/run-tests --testnames pyspark.sql.tests.test_udf_profiler`
- `python/run-tests --testnames pyspark.tests.test_memory_profiler`
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex GPT5.6Sol
help understand code
Closes #59350 from Horse-MA00/fix-profiler-collector-type.
Authored-by: Matthew M.M <[email protected]>
Signed-off-by: Tian Gao <[email protected]>
(cherry picked from commit 2c17e77999bcdb0e969e5208ae5469f50145a2e3)
Signed-off-by: Tian Gao <[email protected]>
---
python/pyspark/core/context.py | 4 ++--
python/pyspark/core/rdd.py | 1 +
python/pyspark/sql/udf.py | 1 +
3 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/python/pyspark/core/context.py b/python/pyspark/core/context.py
index 10e4fe14ab9a..178c307ff4bf 100644
--- a/python/pyspark/core/context.py
+++ b/python/pyspark/core/context.py
@@ -164,7 +164,7 @@ class SparkContext:
None # zip and egg files that need to be added to PYTHONPATH
)
serializer: Serializer
- profiler_collector: ProfilerCollector
+ profiler_collector: Optional[ProfilerCollector]
PACKAGE_EXTENSIONS: Iterable[str] = (".zip", ".egg", ".jar")
@@ -399,7 +399,7 @@ class SparkContext:
profiler_cls, udf_profiler_cls, memory_profiler_cls, dump_path
)
else:
- self.profiler_collector = None # type: ignore[assignment]
+ self.profiler_collector = None
# create a signal handler which would be invoked on receiving SIGINT
def signal_handler(signal: Any, frame: Any) -> NoReturn:
diff --git a/python/pyspark/core/rdd.py b/python/pyspark/core/rdd.py
index 629a1bc93e6b..b36dc6b94101 100644
--- a/python/pyspark/core/rdd.py
+++ b/python/pyspark/core/rdd.py
@@ -5337,6 +5337,7 @@ class PipelinedRDD(RDD[U], Generic[T, U]):
if profiler:
assert self._jrdd_val is not None
+ assert self.ctx.profiler_collector is not None
self._id = self._jrdd_val.id()
self.ctx.profiler_collector.add_profiler(self._id, profiler)
return self._jrdd_val
diff --git a/python/pyspark/sql/udf.py b/python/pyspark/sql/udf.py
index e0ff7281ef12..9168aaef0788 100644
--- a/python/pyspark/sql/udf.py
+++ b/python/pyspark/sql/udf.py
@@ -623,6 +623,7 @@ class UserDefinedFunction:
# replace it with a function the worker cannot drive (it has no
``zero``/``reduce``/
# ``bufferSchema``), and the memory profiler's
``inspect.getsourcelines(f.__code__)``
# fails on the driver because an ``Aggregator`` instance has no
``__code__``.
+ assert sc.profiler_collector is not None
if self.evalType ==
PythonEvalType.SQL_GROUPED_AGG_ARROW_INCREMENTAL_FINAL_UDF:
warnings.warn(
"Profiling incremental Python aggregators is not
supported.",
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
