On Thu, Feb 11, 2016 at 10:58 AM, sebb <[email protected]> wrote: > On 11 February 2016 at 01:53, Sam Ruby <[email protected]> wrote: >> On Wed, Feb 10, 2016 at 8:23 PM, sebb <[email protected]> wrote: >>> Most of the public json data files have dates in them that show when >>> the raw data was last updated. >>> The files themselves are only replaced when the data in them changes, >>> so the If-Modified-Since check works properly. >>> >>> Now some data sources change quite infrequently, so it can be hard to >>> determine of the file really is up to date or the cron job has not >>> been run recently. >>> >>> There are public log files, however these would need to be analysed to >>> determine if the job completed OK, and even an empty file could be >>> associated with a failed job if the error was not caught and logged - >>> i.e. it only went to stderr] >> >> The cron jobs direct stderr to the log files (2>&1): >> >> https://github.com/apache/infrastructure-puppet/blob/deployment/modules/whimsy_server/manifests/cronjobs.pp > > OK. > > But the issue still arises - the contents of the log file have to be > analysed in order to determine if the output was created OK. > However if the job added a marker that would be trivial.
I'm clearly not understanding what problem you are trying to solve. Yes, you have to look at the contents to determine if the job ran successfully. Adding a marker to the contents doesn't eliminate the need to look at the contents. Example problem #1: job didn't run at all. Symptom to look for: no updated mtime. Example problem #2: job ran and blew up. Symptom to look for: unexpected output produced. But whatever, if you want to add a marker, go for it. >>> Possible solutions might be: >>> - scripts add a standard entry to each log file on successful completion. >>> This could include the start and end times of the job. >>> Quite simple to implement, but a bit awkward to use. >>> >>> - scripts update a shared json file on successful completion. >>> More care needed to implement (locking needed), but much easier to use. >>> >>> Thoughts? >> >> The log files themselves have an mtime: https://whimsy-test.apache.org/logs/ > > Yes, but that only shows when cron last ran the job, not when it last > completed OK. > > Also there log file names don't agree with the json files (that could > easily be fixed). > >> status/monitors/public_json.rb could issue a warning (or higher?) if >> the log has not been updated recently. > > Is the monitor itself monitored? Again, I may not be understanding the question. It certainly is: https://www.pingmybox.com/dashboard?location=470 >> Depending on the level chosen, this could trigger an alert. > > That is a good idea. :-) - Sam Ruby
