aglinxinyuan opened a new issue, #6922: URL: https://github.com/apache/texera/issues/6922
### What happened? Every **successful** region teardown logs a batch of alarming WARNs that read like crashes, scaling with worker count: | log line | where | when it fires on a healthy teardown | |---|---|---| | `actor X is not reachable anymore, it might have crashed. old ref = ...` | `PekkoActorRefMappingService.scala:76-81` (`removeActorRef`) | once **per worker** — `RegionExecutionManager.scala:198` calls `removeActorRef` unconditionally for every worker of every completed region | | same WARN again | via the dead-letter branch `WorkflowActor.scala` (`MessageBecomesDeadLetter` → `removeActorRef(dest)`) | again from every peer that still had an in-flight message to the stopped worker | | `unknown identifier: ...` | `PekkoActorRefMappingService.scala:122` | on every send to a worker that was already removed (e.g. resent messages, late stats queries) | So an N-worker region emits ≥N "might have crashed" WARNs on the *normal* completion path — before any real problem exists. This trains operators (and CI log readers) to ignore WARNs, and it buries genuine crash evidence. It is the largest remaining contributor to teardown log noise now that the EndWorker retry storm is fixed (#6916). Note the logger names here are Amber-style (`<actorId>] [<Class>`), which never match package-prefix logback filters like `org.apache=WARN`, so this cannot be silenced by logback configuration — the call sites themselves need to distinguish "removed deliberately during teardown" from "vanished unexpectedly" (e.g. a `removeActorRef(id, expected = true)` variant logged at debug/info). ### How to reproduce? Run any workflow to completion and grep the coordinator log for `might have crashed` — one WARN per worker per region despite a fully successful run. `LoopIntegrationSpec` on CI shows this repeatedly (once per loop iteration per region). ### Version/Branch main (observed at 429be110a7; related to the log-noise follow-ups in #6796 and the fix in #6916). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
