This is an automated email from the ASF dual-hosted git repository.
ruifengz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new b5bf0a7adabf [SPARK-54572][PYTHON][FOLLOWUP] Add a file lock for the
VSCode magic file
b5bf0a7adabf is described below
commit b5bf0a7adabfb1b990c0dd281ad0a961f31a72f9
Author: Tian Gao <[email protected]>
AuthorDate: Thu Dec 4 18:02:03 2025 +0800
[SPARK-54572][PYTHON][FOLLOWUP] Add a file lock for the VSCode magic file
### What changes were proposed in this pull request?
Add a lock to access the VSCode magic file so only one process is allowed
to write it and wait for connection from VSCode.
Also we do not track daemon by default because fork could cause issues. We
add an option `--hook-daemon` when the user needs to.
### Why are the changes needed?
If multiple processes are writing the file at the same time, VSCode will
fail to establish the connection to the debugger server.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Manually confirmed that `run-tests` works after the patch.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #53318 from gaogaotiantian/fix-vscode-support.
Authored-by: Tian Gao <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
---
python/conf_vscode/README.md | 7 +++++++
python/conf_vscode/sitecustomize.py | 35 +++++++++++++++++++++++++++--------
python/run-with-vscode-breakpoints | 16 ++++++++++++++++
3 files changed, 50 insertions(+), 8 deletions(-)
diff --git a/python/conf_vscode/README.md b/python/conf_vscode/README.md
index 23b366be1087..94f8c3418201 100644
--- a/python/conf_vscode/README.md
+++ b/python/conf_vscode/README.md
@@ -34,3 +34,10 @@ If you want, you can set them manually.
```
python/run-with-vscode-breakpoints python/run-tests --testnames="..."
```
+
+`daemon` is not tracked by default because of potential issues for fork.
However if you need to debug `daemon`
+you can do
+
+```
+python/run-with-vscode-breakpoints --hook-daemon your_script.py
+```
diff --git a/python/conf_vscode/sitecustomize.py
b/python/conf_vscode/sitecustomize.py
index cc5def39dda4..3c3abe69c27b 100644
--- a/python/conf_vscode/sitecustomize.py
+++ b/python/conf_vscode/sitecustomize.py
@@ -16,15 +16,34 @@
#
try:
- import debugpy
- import sys
import os
+ import sys
+
if "DEBUGPY_ADAPTER_ENDPOINTS" in os.environ and not any("debugpy" in arg
for arg in sys.argv):
- debugpy.listen(0)
- debugpy.wait_for_client()
- try:
- os.remove(os.getenv("DEBUGPY_ADAPTER_ENDPOINTS"))
- except Exception:
- pass
+
+ def install_debugpy():
+ import debugpy
+ import fcntl
+
+ lock_file = os.getenv("DEBUGPY_ADAPTER_ENDPOINTS") + ".lock"
+ try:
+ fd = os.open(lock_file, os.O_CREAT | os.O_RDWR, 0o600)
+ fcntl.flock(fd, fcntl.LOCK_EX)
+ debugpy.listen(0)
+ debugpy.wait_for_client()
+ try:
+ os.remove(os.getenv("DEBUGPY_ADAPTER_ENDPOINTS"))
+ except Exception:
+ pass
+ finally:
+ fcntl.flock(fd, fcntl.LOCK_UN)
+ os.close(fd)
+
+ if "pyspark.daemon" in sys.orig_argv and "PYSPARK_DEBUGPY_HOOK_DAEMON"
not in os.environ:
+ # Hooking daemon could be potentially dangerous because we need to
fork later
+ os.register_at_fork(after_in_child=install_debugpy)
+ else:
+ install_debugpy()
+
except ImportError:
pass
diff --git a/python/run-with-vscode-breakpoints
b/python/run-with-vscode-breakpoints
index b6f702bfe2d9..11c821fa8323 100755
--- a/python/run-with-vscode-breakpoints
+++ b/python/run-with-vscode-breakpoints
@@ -44,6 +44,22 @@ if ! python3 -c "import debugpy" 2>/dev/null; then
exit 1
fi
+# If --hook-daemon is in the arguments, set the env variable to hook the
daemon process
+# Daemon is not hooked by default to avoid potential fork issues
+new_args=()
+for arg in "$@"; do
+ case "$arg" in
+ --hook-daemon)
+ export PYSPARK_DEBUGPY_HOOK_DAEMON=1
+ # skip this arg
+ ;;
+ *)
+ new_args+=("$arg")
+ ;;
+ esac
+done
+set -- "${new_args[@]}"
+
export DEBUGPY_ADAPTER_ENDPOINTS=$VSCODE_DEBUGPY_ADAPTER_ENDPOINTS
export PYTHONPATH="$FWDIR/conf_vscode:$PYTHONPATH"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
