On 11 February 2016 at 17:34, Sam Ruby <[email protected]> wrote: > On Thu, Feb 11, 2016 at 10:58 AM, sebb <[email protected]> wrote: >> On 11 February 2016 at 01:53, Sam Ruby <[email protected]> wrote: >>> On Wed, Feb 10, 2016 at 8:23 PM, sebb <[email protected]> wrote: >>>> Most of the public json data files have dates in them that show when >>>> the raw data was last updated. >>>> The files themselves are only replaced when the data in them changes, >>>> so the If-Modified-Since check works properly. >>>> >>>> Now some data sources change quite infrequently, so it can be hard to >>>> determine of the file really is up to date or the cron job has not >>>> been run recently. >>>> >>>> There are public log files, however these would need to be analysed to >>>> determine if the job completed OK, and even an empty file could be >>>> associated with a failed job if the error was not caught and logged - >>>> i.e. it only went to stderr] >>> >>> The cron jobs direct stderr to the log files (2>&1): >>> >>> https://github.com/apache/infrastructure-puppet/blob/deployment/modules/whimsy_server/manifests/cronjobs.pp >> >> OK. >> >> But the issue still arises - the contents of the log file have to be >> analysed in order to determine if the output was created OK. >> However if the job added a marker that would be trivial. > > I'm clearly not understanding what problem you are trying to solve.
So that automated scripts can provide info on when the data was last updated. > Yes, you have to look at the contents to determine if the job ran > successfully. Adding a marker to the contents doesn't eliminate the > need to look at the contents. No, but it's a lot easier to check for a specific text marker. > Example problem #1: job didn't run at all. Symptom to look for: no > updated mtime. > > Example problem #2: job ran and blew up. Symptom to look for: > unexpected output produced. Detecting the unexpected is probably fairly easy for a human who has some idea what the script does, but not necessarily easy to define to an automated script. Or even for a human who is not au fait with the app. > But whatever, if you want to add a marker, go for it. Would you be OK with a combined JSON status file instead? >>>> Possible solutions might be: >>>> - scripts add a standard entry to each log file on successful completion. >>>> This could include the start and end times of the job. >>>> Quite simple to implement, but a bit awkward to use. >>>> >>>> - scripts update a shared json file on successful completion. >>>> More care needed to implement (locking needed), but much easier to use. >>>> >>>> Thoughts? >>> >>> The log files themselves have an mtime: https://whimsy-test.apache.org/logs/ >> >> Yes, but that only shows when cron last ran the job, not when it last >> completed OK. >> >> Also there log file names don't agree with the json files (that could >> easily be fixed). >> >>> status/monitors/public_json.rb could issue a warning (or higher?) if >>> the log has not been updated recently. >> >> Is the monitor itself monitored? > > Again, I may not be understanding the question. It certainly is: > > https://www.pingmybox.com/dashboard?location=470 I was forgetting that PMB invokes a cgi, and that invokes the status scripts. As opposed to the status scripts running as a cron job, the output of which is examined. >>> Depending on the level chosen, this could trigger an alert. >> >> That is a good idea. > > :-) As it stands, if one or more cron jobs stop running, I don't think PMB will notice unless the last run happened to fail. > > - Sam Ruby
