Branch: refs/heads/main
Home: https://github.com/WebKit/WebKit
Commit: 6131f147456111c0c430607b0c94c11fdb53eaf6
https://github.com/WebKit/WebKit/commit/6131f147456111c0c430607b0c94c11fdb53eaf6
Author: Carlos Alberto Lopez Perez <[email protected]>
Date: 2026-08-19 (Wed, 19 Aug 2026)
Changed paths:
M Tools/Scripts/webkitpy/port/linux_get_crash_log.py
M Tools/Scripts/webkitpy/port/linux_get_crash_log_unittest.py
Log Message:
-----------
[Tools] linux_get_crash_log: run GDB with a maximum timeout of 30 minutes and
improve checking of DEBUGINFOD_URLS
https://bugs.webkit.org/show_bug.cgi?id=321783
Reviewed by Nikolas Zimmermann.
GDB can spend days generating a Linux layout-test crash log on the Debug bots.
We are not sure what causes that. A hypothesis is related to the DEBUGINFOD
servers which on Ubuntu are kind of unstable, and when those don't reply on time
they can stall GDB waiting for a reply. However, if this was the issue we should
see the problem in Release build as well. A possible problem is that we
currently
check if those servers are online at the moment of starting the test run, but we
no longer check if the servers are still online in the middle of the run. And a
Debug build is slower by nature (several hours), so it has more chances of
hitting
the corner case of a DEBUGINFOD going offline in the middle of the run.
This patch addresses that hypothetical corner case. The DEBUGINFOD servers are
now
checked immediately before each GDB process starts instead of once per run.
Because
probing an offline server is slow, the result is cached for five minutes and
shared
between workers through a file keyed by an MD5 of the server list, so
concurrent runs
with different DEBUGINFOD_URLS do not share a cache entry. The check no longer
rewrites
DEBUGINFOD_URLS in os.environ: each GDB invocation gets its own environment.
Since we are not certain that stalled DEBUGINFOD servers are the cause, the
patch
adds a second, independent layer of protection against long-lived GDB processes.
A new TaskPooledProcessRunner class bounds any process by two limits: an
execution
timeout, and a request from the TaskPool for the worker to end. Once either
fires,
the target is sent SIGTERM, given a grace period, then sent SIGKILL, and only
as a
last resort is the whole process group killed. This guarantees GDB cannot run
beyond
its timeout, and that a run interrupted with SIGTERM or SIGINT terminates GDB
and
returns control to the worker in time for the TaskPool coordinator.
The GDB execution timeout defaults to 30 minutes and can be changed with env var
WEBKIT_CRASHLOG_GDB_EXECUTION_TIMEOUT. When it expires, the generated crash log
starts with the reason it was truncated, the DEBUGINFOD_URLS that were in use,
and
the name of that variable, so a truncated backtrace on a bot explains itself.
While
GDB runs, a debug line is logged every 60 seconds naming the process and how
much
of its budget remains.
On top of that, several improvements are made to make the code more robust,
like the
handling of the GNU "time" wrapper: SIGKILL is first sent to GDB rather than to
the
wrapper, so "time" survives to write its resource statistics even when GDB had
to
be killed, and its output file is created inside the crash-log temporary
directory
and removed afterwards.
* Tools/Scripts/webkitpy/port/linux_get_crash_log.py:
(CrashLogEnvVars):
(CrashLogUtils):
(CrashLogUtils.get_gdb_execution_timeout):
(CrashLogUtils.is_taskpool_worker_terminating):
(CrashLogUtils._get_gdb_lock_configuration):
(CrashLogUtils.get_crashlog_temp_dir):
(CrashLogUtils.get_thread_data_dir):
(CrashLogUtils.get_debuginfod_cache_dir):
(CrashLogUtils.core_pattern_has_pid_format_string):
(TaskPooledProcessRunner):
(TaskPooledProcessRunner.__init__):
(TaskPooledProcessRunner._communicate):
(TaskPooledProcessRunner._signal_process_group):
(TaskPooledProcessRunner._get_target_pid_and_name):
(TaskPooledProcessRunner._get_child_pids):
(TaskPooledProcessRunner._signal_target):
(TaskPooledProcessRunner._force_kill_and_reap):
(TaskPooledProcessRunner._terminate_after_interruption):
(TaskPooledProcessRunner.run):
(DebuginfodServerCache):
(DebuginfodServerCache._cache_path):
(DebuginfodServerCache._is_server_available):
(DebuginfodServerCache._probe):
(DebuginfodServerCache._read):
(DebuginfodServerCache._write):
(DebuginfodServerCache._get_working_servers):
(DebuginfodServerCache.environment_for_gdb):
(ThreadNamesCrashLogCapturer.handle_coredump):
(GDBCrashLogStartupHandler.__init__):
(GDBCrashLogStartupHandler._maybe_remove_file_if_old):
(GDBCrashLogStartupHandler.clean_old_debuginfod_server_caches):
(GDBCrashLogGenerator._get_gdb_output):
(GDBCrashLogStartupHandler._is_debuginfod_server_available): Deleted.
(GDBCrashLogStartupHandler._check_debuginfod_servers): Deleted.
* Tools/Scripts/webkitpy/port/linux_get_crash_log_unittest.py:
(CrashLogTemporaryDirectoriesTest):
(CrashLogTemporaryDirectoriesTest.test_thread_and_debuginfod_directories_share_crashlog_root):
(CleanOldCrashLogFilesTest):
(CleanOldCrashLogFilesTest.setUp):
(CleanOldCrashLogFilesTest.test_removes_only_old_cache_and_temporary_files):
(CleanOldCrashLogFilesTest.test_keeps_cache_directory_when_last_cache_is_removed):
(CleanOldCrashLogFilesTest.test_list_failure_does_not_abort_cleanup):
(DebuginfodServerCacheTest):
(DebuginfodServerCacheTest.setUp):
(DebuginfodServerCacheTest._age_cache):
(DebuginfodServerCacheTest.test_recent_result_is_shared_without_reprobing):
(DebuginfodServerCacheTest.test_environment_canonicalizes_reordered_and_duplicate_servers):
(DebuginfodServerCacheTest.test_probe_keeps_file_urls_and_drops_unsupported_urls):
(DebuginfodServerCacheTest.test_http_availability_probe):
(DebuginfodServerCacheTest.test_concurrent_refresh_is_coalesced_by_file_lock):
(DebuginfodServerCacheTest.test_concurrent_refresh_is_coalesced_by_file_lock.slow_probe):
(DebuginfodServerCacheTest.test_concurrent_refresh_is_coalesced_by_file_lock.worker):
(DebuginfodServerCacheTest.test_environment_for_gdb_filters_copy_without_changing_process_environment):
(DebuginfodServerCacheTest.test_environment_for_gdb_filters_copy_without_changing_process_environment.only_first_server_works):
(DebuginfodServerCacheTest.test_server_offline_during_first_crash_can_recover):
(DebuginfodServerCacheTest.test_cache_path_failure_falls_back_to_uncached_probe):
(DebuginfodServerCacheTest.test_cache_write_failure_does_not_repeat_probe_or_raise):
(DebuginfodServerCacheTest.test_corrupted_cache_is_ignored):
(DebuginfodServerCacheTest.test_different_server_lists_use_independent_cached_results):
(DebuginfodServerCacheTest.test_no_configured_servers_does_not_probe_or_set_gdb_environment):
(GDBExecutionTimeoutConfigurationTest):
(GDBExecutionTimeoutConfigurationTest._timeout_with_value):
(GDBExecutionTimeoutConfigurationTest.test_timeout_configuration):
(FakeGDBProcess):
(FakeGDBProcess.__init__):
(FakeGDBProcess.communicate):
(FakeGDBProcess.send_signal):
(FakeGDBProcess.wait):
(FakeGDBExecutive):
(FakeGDBExecutive.__init__):
(FakeGDBExecutive.popen):
(SubprocessExecutive):
(SubprocessExecutive.popen):
(_run_process):
(TaskPooledProcessRunnerCommunicateTest):
(TaskPooledProcessRunnerCommunicateTest._runner):
(TaskPooledProcessRunnerCommunicateTest.test_retries_with_bounded_timeouts_until_process_exits):
(TaskPooledProcessRunnerCommunicateTest.test_raises_when_overall_deadline_expires):
(TaskPooledProcessRunnerCommunicateTest.test_termination_request_stops_communicating):
(TaskPooledProcessRunnerTest):
(TaskPooledProcessRunnerTest.setUp):
(TaskPooledProcessRunnerTest.setUp.communicate):
(TaskPooledProcessRunnerTest.test_success_uses_timeout_environment_and_new_session):
(TaskPooledProcessRunnerTest.test_termination_requested_before_run_prevents_process_start):
(TaskPooledProcessRunnerTest.test_timeout_escalates_to_sigkill_and_preserves_partial_output):
(TaskPooledProcessRunnerTest.test_sigterm_grace_period_collects_additional_output_without_sigkill):
(TaskPooledProcessRunnerTest.test_gdb_sigkill_allows_time_wrapper_to_return_output):
(TaskPooledProcessRunnerTest.test_gdb_sigkill_allows_time_wrapper_to_return_output.fake_open):
(TaskPooledProcessRunnerTest.test_unreapable_process_does_not_make_cleanup_unbounded):
(TaskPooledProcessRunnerTest.test_killpg_failure_uses_direct_process_fallback):
(TaskPooledProcessRunnerTest.test_reap_failure_does_not_mask_grace_period_keyboard_interrupt):
(TaskPooledProcessRunnerTest.test_termination_request_terminates_process):
(TaskPooledProcessRunnerTest.test_termination_request_terminates_process.request_termination):
(TaskPooledProcessRunnerTest.test_interrupted_process_ignoring_sigterm_is_force_killed):
(TaskPooledProcessRunnerTest.test_interrupted_process_ignoring_sigterm_is_force_killed.interrupt_then_timeout):
(TaskPooledProcessRunnerTest.test_real_wrapper_and_child_are_both_killed_after_grace_period):
(TaskPooledProcessRunnerTest.test_real_wrapper_and_child_are_both_killed_after_grace_period.child_is_running):
(TaskPooledProcessRunnerTest.test_gdb_sigkill_allows_real_wrapper_to_write_output):
(TaskPooledProcessRunnerTest.test_taskpool_sigint_handler_interrupts_gdb):
(GDBTimeoutCrashLogTest):
(GDBTimeoutCrashLogTest.test_worker_termination_after_lock_prevents_gdb_start):
(GDBTimeoutCrashLogTest.test_worker_termination_after_lock_prevents_gdb_start.gdb_lock):
(GDBTimeoutCrashLogTest.test_worker_termination_while_gdb_runs_discards_output):
(GDBTimeoutCrashLogTest.test_worker_termination_while_gdb_runs_discards_output.run_process):
(GDBTimeoutCrashLogTest.test_time_output_setup_failure_does_not_abort_gdb):
(GDBTimeoutCrashLogTest.test_time_output_uses_crashlog_directory_and_is_removed_on_failure):
(GDBTimeoutCrashLogTest.test_time_output_uses_crashlog_directory_and_is_removed_on_failure.fail_after_checking_time_output):
(GDBTimeoutCrashLogTest.test_timeout_is_reported_and_filtered_environment_reaches_process):
(GDBTimeoutCrashLogTest.test_timeout_is_reported_and_filtered_environment_reaches_process.gdb_lock):
(GDBTimeoutCrashLogTest.test_timeout_is_reported_and_filtered_environment_reaches_process.environment_for_gdb):
(GDBTimeoutCrashLogTest.test_timeout_is_reported_and_filtered_environment_reaches_process.run_process):
(CleanOldCoredumpsTest): Deleted.
(CleanOldCoredumpsTest.setUp): Deleted.
(CleanOldCoredumpsTest._touch): Deleted.
(CleanOldCoredumpsTest.test_reaps_old_pattern_and_claimed_keeps_fresh_and_unrelated):
Deleted.
Canonical link: https://commits.webkit.org/319483@main
To unsubscribe from these emails, change your notification settings at
https://github.com/WebKit/WebKit/settings/notifications
