geosmart edited a comment on issue #3618: URL: https://github.com/apache/incubator-dolphinscheduler/issues/3618#issuecomment-681893722
just like airflow, wirte task run log to disk produce many small file, but airflow produce plugin to read log from remote (like es), so you can use logstash to collect task logs,and clean the log after collected. --- ds can add some pr to read log from remote log server this is the airflow Writing Logs we can reference * Writing Logs Locally * Writing Logs to Amazon S3 * Writing Logs to Azure Blob Storage * Writing Logs to Google Cloud Storage * Writing Logs to Elasticsearch * Writing Logs to Elasticsearch over TLS ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
