njnu-seafish opened a new issue, #18129: URL: https://github.com/apache/dolphinscheduler/issues/18129
### Search before asking - [x] I had searched in the [DSIP](https://github.com/apache/dolphinscheduler/issues/14102) and found no similar DSIP. ### Motivation Currently, all workflow execution logs are mixed within the Master service logs. This requires operations staff to log in to the server to access them, making it impossible for regular users to view logs related to their specific workflows. ### Design Detail The core of this design relies on Logback's SiftingAppender, which dynamically creates separate log files for each distinct workflow instance. This ensures that logs for different workflows are isolated from one another, facilitating easier debugging and UI presentation. Here is the detailed breakdown of the mechanism: 1. Dynamic Routing with SiftingAppender The WORKFLOWLOGFILE is defined as a ch.qos.logback.classic.sift.SiftingAppender. Unlike standard appenders that write to a single file, this component acts as a router. It evaluates every log event and decides which specific file to write to based on runtime data. 2. Context-Based Discrimination The separation logic is driven by the <Discriminator> block: Key: workflowInstanceLogFullPath Mechanism: The system extracts this key from the MDC (Mapped Diagnostic Context). In DolphinScheduler, the WorkflowLogDiscriminator retrieves the unique path for the current workflow instance from the thread's context. Result: Each unique workflowInstanceLogFullPath triggers the creation of a distinct log file. 3. Filtering Logic Before routing, a specific filter is applied: Filter Class: org.apache.dolphinscheduler.server.master.log.WorkflowLogFilter Function: This ensures that only logs relevant to a workflow instance are processed by this appender. System-level logs (which lack workflow context) are ignored here, preventing them from polluting the instance-specific files. 4. File Storage Strategy The actual file writing is handled by the nested FileAppender inside the <sift> block: Base Directory: Defined by ${log.base}/workflows. File Naming: The final path is constructed dynamically using ${workflowInstanceLogFullPath}. Append Mode: Set to true, ensuring that new logs are added to the existing file rather than overwriting it, which is crucial for long-running workflows. 5. Log Content & Format The logs written to these separated files use a specific pattern: Pattern: %d{yyyy-MM-dd HH:mm:ss.SSS} %-5level [%thread] %logger{10}:[%line] - %message%n Note: Unlike the MASTERLOGFILE, the workflow-specific logs here do not explicitly repeat the [WI-...] prefix in the pattern, as the file name itself identifies the workflow instance. ### Compatibility, Deprecation, and Migration Plan _No response_ ### Test Plan _No response_ ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
