This is an automated email from the ASF dual-hosted git repository.
ruifengz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 0101cf5a94aa [SPARK-54572][PYTHON] Support VSCode breakpoints for
pyspark
0101cf5a94aa is described below
commit 0101cf5a94aa98e2cadddfd7986129ccd448a549
Author: Tian Gao <[email protected]>
AuthorDate: Thu Dec 4 09:48:42 2025 +0800
[SPARK-54572][PYTHON] Support VSCode breakpoints for pyspark
### What changes were proposed in this pull request?
This PR adds a script and a supporting directory for users to have native
debugging experience with VSCode (Cursor) for not only driver code, but
UDF/workers/daemon.
<img width="2324" height="1010" alt="image"
src="https://github.com/user-attachments/assets/57d1bbae-53c6-48e9-8910-25a47e4f0e24";
/>
### Why are the changes needed?
So user can debug their (local) code with VSCode debugger.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Locally works. This does not touch spark code, it's just a dev tool.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #53289 from gaogaotiantian/support-vscode-breakpoint.
Authored-by: Tian Gao <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
---
python/conf_vscode/README.md | 36 ++++++++++++++++++++++++++
python/conf_vscode/sitecustomize.py | 30 ++++++++++++++++++++++
python/run-with-vscode-breakpoints | 50 +++++++++++++++++++++++++++++++++++++
3 files changed, 116 insertions(+)
diff --git a/python/conf_vscode/README.md b/python/conf_vscode/README.md
new file mode 100644
index 000000000000..23b366be1087
--- /dev/null
+++ b/python/conf_vscode/README.md
@@ -0,0 +1,36 @@
+# VSCode Debugger Support
+
+This directory is to support [VSCode Python
debugger](https://code.visualstudio.com/docs/python/debugging)
+for PySpark.
+
+## Usage
+
+First of all, you need to install `debugpy`.
+
+```
+pip install debugpy
+```
+
+Set a breakpoint in VSCode normally by clicking the red dot on the left of the
line number. You can
+set the breakpoint in user code, UDF code or PySpark engine code.
+
+Then, instead of run your script with
+
+```
+python your_script.py
+```
+
+Run it with the script wrapper we provided
+
+```
+python/run-with-vscode-breakpoints python your_script.py
+```
+
+Your VSCode debugger should be brought up automatically when the breakpoint is
hit.
+
+This also works for `run-tests` or any other scripts - under the hood it just
sets two environment variables.
+If you want, you can set them manually.
+
+```
+python/run-with-vscode-breakpoints python/run-tests --testnames="..."
+```
diff --git a/python/conf_vscode/sitecustomize.py
b/python/conf_vscode/sitecustomize.py
new file mode 100644
index 000000000000..cc5def39dda4
--- /dev/null
+++ b/python/conf_vscode/sitecustomize.py
@@ -0,0 +1,30 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+try:
+ import debugpy
+ import sys
+ import os
+ if "DEBUGPY_ADAPTER_ENDPOINTS" in os.environ and not any("debugpy" in arg
for arg in sys.argv):
+ debugpy.listen(0)
+ debugpy.wait_for_client()
+ try:
+ os.remove(os.getenv("DEBUGPY_ADAPTER_ENDPOINTS"))
+ except Exception:
+ pass
+except ImportError:
+ pass
diff --git a/python/run-with-vscode-breakpoints
b/python/run-with-vscode-breakpoints
new file mode 100755
index 000000000000..b6f702bfe2d9
--- /dev/null
+++ b/python/run-with-vscode-breakpoints
@@ -0,0 +1,50 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -o pipefail
+set -e
+
+FWDIR="$(cd "`dirname $0`"; pwd)"
+
+# Function to display help message
+function usage {
+ cat <<EOF
+Usage:
+ $(basename $0) your_original_commands
+
+Requirements:
+ - debugpy must be installed (pip install debugpy)
+EOF
+ exit 0
+}
+
+# Check for help flags or no arguments
+if [ $# -eq 0 ] || [ "$1" = "-h" ] || [ "$1" = "--help" ]; then
+ usage
+fi
+
+if ! python3 -c "import debugpy" 2>/dev/null; then
+ echo "Error: debugpy is not installed. Please install it using 'pip install
debugpy'."
+ exit 1
+fi
+
+export DEBUGPY_ADAPTER_ENDPOINTS=$VSCODE_DEBUGPY_ADAPTER_ENDPOINTS
+export PYTHONPATH="$FWDIR/conf_vscode:$PYTHONPATH"
+
+exec "$@"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
