AFAIK, airflow doesn't provide log in this way. Multiple tasks would run in different processes and potentially in parallel, thus writing to the same file at run time would produce log file with mix log lines from different tasks. Also I believe airflow now does not seperate stdour and stderr, they all go to same place. Not sure if there's a good point in code to consolidate the logs from different tasks. Maybe you can have a separate script/service to do the log consolidate job since the log structure and format are known.
Cheers, Kevin Y On Mon, Apr 30, 2018 at 12:16 PM, [email protected] <[email protected] > wrote: > Hi Mailing List, > > Is there a way to consolidate airflow task logs by stdout and stderr? > Currently, the structure is something like > /logs/taskname/task_run_date/log1.txt > which is the log for a particular taskname at a particular run date. What > I would like is two large log files for all tasks, something like > /logs/errors.txt and /logs/outputs.txt Which would contain all the stderr > and stdout messages for all runs of all tasks regardless of run date. I > essentially want two very large log files. For example, if I have task A > and task B, instead of having two directories and then subdirectories for A > and B, I would just like two files one with errors from A and B and one for > outputs from A and B. Does airflow provide this information? > > Thanks! >
