Design of Oozie Logging System
------------------------------
Key: OOZIE-562
URL: https://issues.apache.org/jira/browse/OOZIE-562
Project: Oozie
Issue Type: Question
Reporter: Kiran Nagasubramanian
When large log files are there in the log folder, even if we try to retrieve
the log content(of size of the order of kBs) for a small job, it takes quite
some significant amount of time, as equally as it takes to retrieve log content
for very large jobs(of the order of GBs). This happens because, the list of
files to be scanned for log retrieval is the same for all jobs that run
approximately at around the same time.
Chances that this might materialize in production systems is really high.
Since, hundreds of jobs would be logging to the same file for an hr and this
file size would be really huge. Is it possible to have the logs for the jobs
separately so that scanning large log files of other jobs can be avoided? Would
this be really worth the effort?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
