mrproliu opened a new pull request, #211:
URL: https://github.com/apache/skywalking-rover/pull/211
## Background
When running the access log module on a node with multi-threaded workloads
(JVM, Go,
etc.), `rover` keeps logging the **same process being "detected" and
"recognized as
dead" over and over**, every few seconds, for the whole lifetime of the
process.
The affected processes are never actually restarted — they are healthy and
long-lived.
The churn produces a large amount of misleading `info` logs, makes the
process entities
flap on the backend (their `id` never stabilizes), and pollutes the eBPF
`process_monitor_control` map.
## Symptom (logs)
The same PID is added by the eBPF path and immediately deleted by the
periodic `/proc`
scan, in a tight loop:
```
time="2026-06-27T01:54:40Z" level=info msg="detected new process by add
process: pid: 814309, entity:
{\"Layer\":\"K8S_SERVICE\",\"ServiceName\":\"skywalking-showcase::demo-oap.skywalking-showcase\",\"InstanceName\":\"demo-oap-84c6bc7b44-8v6cn\",\"ProcessName\":\"java\",\"Labels\":[\"k8s-service\"]}"
module=process.finder
time="2026-06-27T01:54:45Z" level=info msg="the process has been recognized
as dead, so deleted. pid: 814309, entity: {...\"ProcessName\":\"java\"...}, id:
" module=process.finder
time="2026-06-27T01:54:49Z" level=info msg="detected new process by add
process: pid: 814309, entity: {...\"ProcessName\":\"java\"...}"
module=process.finder
time="2026-06-27T01:54:50Z" level=info msg="the process has been recognized
as dead, so deleted. pid: 814309, ... id: " module=process.finder
...repeats every ~5s...
```
Aggregated over a single rover instance, the "java" entries appear **0
times** via the
periodic scan (`by sync all`) but are re-added **continuously** by the eBPF
path
(`by add process`), while being marked dead on every scan tick:
```
detected new process by add process : 1031
detected new process by sync all : 0 (for java)
recognized as dead : 152 (for java)
```
Note the dead entries carry an **empty `id`** — the entity dies (from
rover's view)
before it can complete backend registration.
## Root cause
The two process-discovery paths disagree about what a "process" is:
1. **Periodic scan** (`pkg/process/finders/kubernetes/finder.go`, every 5s)
enumerates
processes via gopsutil `process.Processes()`, which does a
`readdir(/proc)` and
therefore only returns **TGIDs** (thread-group leaders / real processes).
Anything
not in this set is marked dead by `SyncAllProcessInFinder`.
2. **eBPF path** feeds PIDs from the `sched_process_fork` tracepoint
(`bpf/accesslog/process/process.c`) into `ShouldMonitor()` →
`process.NewProcess()`,
which validates a PID with `os.Stat(/proc/<pid>)` — and `/proc/<tid>` is
accessible
for **threads** too, so a thread is happily accepted as a "process".
The bug is in the tracepoint:
```c
SEC("tracepoint/sched/sched_process_fork")
int tracepoint_sched_process_fork(struct trace_event_raw_sched_process_fork*
ctx) {
__u32 tgid = ctx->parent_pid; // <-- parent_pid is a TID, not a TGID
...
event.pid = tgid; // reported to user space as if it were
a process id
}
```
The `sched_process_fork` tracepoint format only exposes thread ids:
```
field:pid_t parent_pid; // = parent->pid (kernel task->pid == TID)
field:pid_t child_pid; // = child->pid (TID)
```
In the kernel, `task->pid` is the **thread id (TID)** and `task->tgid` is
the **process
id**. A multi-threaded app constantly creates threads, and any worker thread
that calls
`clone()`/`fork()` makes the tracepoint fire with `parent_pid` = that worker
thread's
TID. rover then registers the **thread** as a process. The 5s scan
(TGID-only) never
sees that TID, so it deletes it; the next thread activity re-adds it →
infinite churn.
Verified on a live node — the "processes" being churned are JVM threads:
```
tid comm Tgid (real process) in `ls /proc` (readdir)
state
817781 prometheus-http 815492 NO (it is a thread)
ALIVE 4.7h
818295 prometheus-http 815492 NO
ALIVE
822290 grpc-nio-worker 821882 NO
ALIVE
910081 java 887536 NO
ALIVE
815492 (main java) 815492 (itself) YES (a real process)
never churned
```
The stable main JVMs (TGIDs) are tracked correctly and never marked dead;
only the
threads (TIDs) flip.
## Fix
Report the real process **TGID** instead of the tracepoint's `parent_pid` (a
TID):
```c
SEC("tracepoint/sched/sched_process_fork")
int tracepoint_sched_process_fork(struct trace_event_raw_sched_process_fork*
ctx) {
- __u32 tgid = ctx->parent_pid;
+ // ctx->parent_pid is the forking task's TID (thread id), not the
process id.
+ // For multi-threaded apps (e.g. JVM) a worker thread forking would
report its
+ // thread TID, which the periodic /proc scan never lists, causing the
process to
+ // flip between detected and dead. Use the real TGID instead.
+ __u32 tgid = bpf_get_current_pid_tgid() >> 32;
__u32 v = 1;
bpf_map_update_elem(&process_monitor_control, &tgid, &v, 0);
struct process_execute_event event = {};
event.pid = tgid;
bpf_perf_event_output(ctx, &process_execute_queue, BPF_F_CURRENT_CPU,
&event, sizeof(event));
return 0;
}
```
This also fixes a latent bug: the consumer `tgid_should_trace()` looks up
`process_monitor_control` **by TGID**, but the producer was inserting
**TIDs**, so those
entries could never match and just wasted map space.
`sched_process_fork` is invoked as `trace_sched_process_fork(current,
child)` from
`copy_process()`, so `current` is the forking **parent** task.
`bpf_get_current_pid_tgid()
>> 32` therefore yields the parent's TGID — semantically equivalent to the
original
`parent_pid`'s process, just corrected from TID to TGID.
## Is `bpf_get_current_pid_tgid()` valid inside a tracepoint program?
Yes.
- It runs in process context here (the fork syscall path), so `current` is a
valid user
task and the helper returns its `pid`/`tgid` (lower/upper 32 bits
respectively).
- `BPF_PROG_TYPE_TRACEPOINT` is among the supported program types for this
helper — see
the official helper reference:
<https://docs.ebpf.io/linux/helper-function/bpf_get_current_pid_tgid/>
and the kernel `bpf-helpers(7)` man page:
<https://man7.org/linux/man-pages/man7/bpf-helpers.7.html>
- It is already used by many existing tracepoint programs in this repository
(`bpf/accesslog/syscalls/transfer.c`, `connect.c`, `close.c`,
`bpf/accesslog/l24/write_l4.c`, ...), confirming it loads and works for
tracepoint
programs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
