dill21yu commented on issue #17787:
URL:
https://github.com/apache/dolphinscheduler/issues/17787#issuecomment-3809318143
Currently, DolphinScheduler uses FileAppender for task logs without size
limits, which can lead to large log files that consume excessive disk space.
Is this a feasible solution?
Replace FileAppender with RollingFileAppender and implement size-based
rolling policy for task logs, similar to existing MASTERLOGFILE and
WORKERLOGFILE configurations.
1. Configuration Changes:
dolphinscheduler-master/src/main/resources/logback-spring.xml
dolphinscheduler-worker/src/main/resources/logback-spring.xml
dolphinscheduler-standalone-server/src/main/resources/logback-spring.xml
Replace the TASKLOGFILE appender:
`<appender name="FILE-${taskInstanceLogFullPath}"
class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${taskInstanceLogFullPath}</file>
<rollingPolicy
class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<fileNamePattern>${taskInstanceLogFullPath}.%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>
<maxFileSize>${task.log.max.file.size}</maxFileSize>
<maxHistory>${task.log.max.history}</maxHistory>
<totalSizeCap>${task.log.total.size.cap}</totalSizeCap>
<cleanHistoryOnStart>true</cleanHistoryOnStart>
</rollingPolicy>
<encoder>
<pattern>%date{yyyy-MM-dd HH:mm:ss.SSS} %-5level [%thread] -
%message%n</pattern>
<charset>UTF-8</charset>
</encoder>
<append>true</append>
</appender>`
Add configurable properties:
`<property name="task.log.max.file.size" value="200MB"/>
<property name="task.log.max.history" value="3"/>
<property name="task.log.total.size.cap" value="600MB"/>`
2. Log Query Logic Updates
Modify logClientDelegate.getPartLogString() to handle multiple rolled files
Implement file discovery logic to find all rolled log files
Add cross-file line number calculation and content aggregation
Update log download functionality to aggregate all rolled files
And I am willing to submit a PR! @ruanwenjun @SbloodyS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
