codeant-ai-for-open-source[bot] commented on code in PR #40084:
URL: https://github.com/apache/superset/pull/40084#discussion_r3523814845
##########
superset/extensions/local_extensions_watcher.py:
##########
@@ -29,37 +29,172 @@
logger = logging.getLogger(__name__)
+# Sentinel file Flask watches via --extra-files. Touching it on a real change
+# triggers a server reload without depending on cwd or the location of any
+# Python source file.
+RELOAD_TRIGGER = Path(__file__).resolve().parent / ".reload_trigger"
+
# Guard to prevent multiple initializations
_watcher_initialized = False
_watcher_lock = threading.Lock()
-def _get_file_handler_class() -> Any:
+def _get_file_handler_class() -> Any: # noqa: C901
"""Get the file handler class, importing watchdog only when needed."""
try:
- from watchdog.events import FileSystemEventHandler
+ import hashlib
+
+ from watchdog.events import (
+ FileCreatedEvent,
+ FileModifiedEvent,
+ FileMovedEvent,
+ FileSystemEventHandler,
+ )
class LocalExtensionFileHandler(FileSystemEventHandler):
- """Custom file system event handler for LOCAL_EXTENSIONS
directories."""
+ """Custom file system event handler for LOCAL_EXTENSIONS
directories.
+
+ Only reacts to genuine content changes (create / modify / move) in
the
+ dist directory, verified by comparing a SHA-256 of the file's
content.
+ This avoids the Docker VirtioFS / osxfs problem where reading a
file
+ generates inotify events that watchdog surfaces as modifications.
+ """
+
+ def __init__(self) -> None:
+ super().__init__()
+ # sha256 of last-seen content, keyed by absolute path.
Populated
+ # from existing files in watched `dist` dirs at startup (see
+ # `prime_baseline`) so that startup-noise inotify events from
+ # Docker VirtioFS reads don't get treated as the first real
edit.
+ self._file_hashes: dict[str, str] = {}
+ self._lock = threading.Lock()
+ # Trailing debounce: schedule a single reload after a quiet
+ # window so simultaneous webpack writes coalesce into one
+ # restart that fires *after* the build settles.
+ self._debounce_seconds: float = 1.0
+ self._pending_timer: threading.Timer | None = None
+
+ # ── helpers ──────────────────────────────────────────────────────
+
+ @staticmethod
+ def _sha256(path: str) -> str | None:
+ try:
+ with open(path, "rb") as fh:
+ return hashlib.sha256(fh.read()).hexdigest()
+ except OSError:
+ return None
+
+ def prime_baseline(self, watch_dirs: set[str]) -> None:
+ """Pre-populate content hashes for existing files in watched
+ `dist` directories. Called once at watcher startup so a
+ developer's first real edit registers as a content change
+ rather than as the file's 'first observation'."""
+ for root_dir in watch_dirs:
+ root = Path(root_dir)
+ for path in root.rglob("*"):
+ if not path.is_file():
+ continue
+ if "dist" not in path.parts:
+ continue
+ digest = self._sha256(str(path))
+ if digest is not None:
+ self._file_hashes[str(path)] = digest
+
+ def _content_changed(self, path: str) -> bool:
+ """Return True when the file's content differs from last seen.
+
+ With `prime_baseline` called at startup, the baseline reflects
+ what was on disk when the watcher started. A first observation
+ that differs (or doesn't exist in baseline) is treated as a
+ genuine change.
+ """
+ digest = self._sha256(path)
+ if digest is None:
+ return False
+ old_digest = self._file_hashes.get(path)
+ self._file_hashes[path] = digest
+ # New file (not in baseline) is a real change; otherwise
compare.
+ return old_digest != digest
+
+ def _trigger_reload(self, source_path: str) -> None:
+ """Touch the reload-trigger sentinel; Flask's --extra-files
+ watcher reloads on its mtime change."""
+ logger.info("File change settled in LOCAL_EXTENSIONS: %s",
source_path)
+ logger.info("Triggering restart by touching %s",
RELOAD_TRIGGER)
+ try:
+ os.utime(RELOAD_TRIGGER, (time.time(), time.time()))
+ except OSError as e:
+ logger.warning(
+ "Failed to touch reload trigger %s: %s",
RELOAD_TRIGGER, e
+ )
+
+ def _schedule_reload(self, source_path: str) -> None:
+ """Trailing-debounce: cancel any pending reload and schedule a
+ new one for `_debounce_seconds` from now. Each new event resets
+ the timer, so the reload fires only after a quiet window."""
+ with self._lock:
+ if self._pending_timer is not None:
+ self._pending_timer.cancel()
+ timer = threading.Timer(
+ self._debounce_seconds,
+ self._trigger_reload,
+ args=(source_path,),
+ )
+ timer.daemon = True
+ self._pending_timer = timer
+ timer.start()
Review Comment:
**Suggestion:** The debounce logic can still fire multiple reloads under
timing races: `cancel()` does not stop a timer callback that has already
started, so a near-simultaneous event can allow the old callback and the new
timer to both trigger restarts. Add a generation/token check (or
clear-and-validate under lock in the callback) so only the most recent
scheduled timer is allowed to execute `_trigger_reload`. [race condition]
<details>
<summary><b>Severity Level:</b> Major ⚠️</summary>
```mdx
- ⚠️ LOCAL_EXTENSIONS changes can still cause double reloads.
- ⚠️ Extra restarts slow Docker extension development feedback loop.
- ⚠️ Debounce semantics less effective under bursty file writes.
```
</details>
<details>
<summary><b>Steps of Reproduction ✅ </b></summary>
```mdx
1. Start Superset in debug mode so
`start_local_extensions_watcher_thread(app)` in
`superset/app.py:23-25` launches the watcher and
`setup_local_extensions_watcher()` in
`superset/extensions/local_extensions_watcher.py:205-285` registers
`LocalExtensionFileHandler` with watchdog for the configured
`LOCAL_EXTENSIONS`
directories.
2. Trigger a first write to a file under a `dist/` directory (for example,
have webpack
update `local_extensions/my_ext/dist/bundle.js`); watchdog emits a
`FileModifiedEvent`,
`on_any_event()` at `local_extensions_watcher.py:149-197` passes the type
check at lines
154-157, confirms `"dist"` in the path at 187-189,
`_content_changed(target)` returns
`True` at 191-195, and `_schedule_reload(target)` is called at line 197,
creating
`Timer-1` in `_schedule_reload()` at lines 135-145 to invoke
`_trigger_reload()` after
`self._debounce_seconds`.
3. Just before `Timer-1` fires (i.e., very close to `self._debounce_seconds`
seconds
later), trigger a second write to the same or another `dist` file (for
example, another
webpack output); `on_any_event()` calls `_schedule_reload()` again at line
197 for this
second event, `with self._lock:` at 135 acquires the lock,
`self._pending_timer.cancel()`
at 137 attempts to cancel `Timer-1`, and then `Timer-2` is created and
started at lines
138-145.
4. If the second event arrives after `Timer-1`’s internal thread has already
begun
executing `_trigger_reload()` (defined at
`local_extensions_watcher.py:119-129`),
`cancel()` cannot stop it, so both `Timer-1` and `Timer-2` run
`_trigger_reload()` without
any generation check; each call touches `RELOAD_TRIGGER` via `os.utime()` at
125, leading
to two Flask reload cycles for what should be a single debounced change
burst.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=9bd60496e81f4bbebf87b7e7548a0018&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
[](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=9bd60496e81f4bbebf87b7e7548a0018&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/extensions/local_extensions_watcher.py
**Line:** 135:145
**Comment:**
*Race Condition: The debounce logic can still fire multiple reloads
under timing races: `cancel()` does not stop a timer callback that has already
started, so a near-simultaneous event can allow the old callback and the new
timer to both trigger restarts. Add a generation/token check (or
clear-and-validate under lock in the callback) so only the most recent
scheduled timer is allowed to execute `_trigger_reload`.
Validate the correctness of the flagged issue. If correct, How can I resolve
this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask
user if the user wants to fix the rest of the comments as well. if said yes,
then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a
href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F40084&comment_hash=af4d9e9e77e2efc70d2cb6c4e9950f407a9f5bfb86e33073d2e070f34ceaf6ab&reaction=like'>👍</a>
| <a
href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F40084&comment_hash=af4d9e9e77e2efc70d2cb6c4e9950f407a9f5bfb86e33073d2e070f34ceaf6ab&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
