On Sun, Jun 11, 2017 at 12:53 PM, John D. Ament <johndam...@apache.org> wrote: > I decided to sleep on this before running into a response.
My apologies if I caused any offense. > On Sat, Jun 10, 2017 at 11:22 AM Sam Ruby <ru...@intertwingly.net> wrote: > >> On Sat, Jun 10, 2017 at 11:10 AM, John D. Ament <johndam...@apache.org> >> wrote: >> > Unless someone objects, I'm going to introduce a shell script that >> executes >> > the contents of the cron jobs. I'm then going to raise a PR to infra to >> > update this section >> > >> https://github.com/apache/infrastructure-puppet/blob/deployment/modules/whimsy_server/manifests/init.pp#L70 >> > to >> > include a call to said script. >> >> Not an objection, but thinking out loud. >> >> Some of those cronjobs are lightning fast. Others may take a while >> and consume network or CPU resources. >> >> For the fast ones, I find it mildly amusing that they use whimsy/asf >> library functions and capture the result in JSON, and you are parsing >> the JSON instead of using the same library functions. What this means >> is that you get stale data when you have ready access to fresher data. > > Do you have a concrete list of these? I often find arbitrary statements > like this hard to follow, mostly because of how unfamiliar I am with > something, in this case Whimsy. Again, my apologies... I misread the code. You are using the library functions. >> For the slow ones, I don't think it is appropriate to run then after every >> push. >> > Again, having a list of these would be useful. i'm not sure I can judge > without having all of the information. The one that I am most familiar with is the site scan. Sebb can comment on others. >> What I would suggest instead is a CGI script that lists the cron jobs >> that are runnable under the apache web server user id, and gives you a >> button that you can push that will run that specific job on request. >> What this would enable is for anybody (or perhaps just ASF members?) >> to rerun the script at will. >> > No, specifically I disagree with this kind of approach. If we're going to > follow an automatic deploy pattern, the application needs to be self > healing. If there's manual intervention something's wrong. Yesterday we > got stuck because: If the sum total of running all of the cron jobs is multiple minutes of elapsed time, running this list of programs every time in order to reduce the time window where data is out of sync seems like overkill. It may also result in the one item that you are looking for to run to actually be run later than expected. I'm also concerned a bit about these scripts possibly running at the same time as the cron jobs. I am also aware that at times there are problems that only show up in production, and we may find ourselves wanting to run scripts because, for example, LDAP has been updated. > - SVN repo list is cached in the app (but not in cron) > - The cron jobs happened to run in such an order that the svn update > happened after public_podlings.json was generated > > What I'm actually now thinking is that we should implement two bug fixes: > > - in ASF::SVN if the repo doesn't exist, reload the repository.yml file and > try again. +1 > - in some scripts, make sure that any repos you need are up to date (rather > than rely on the svn update cron) Absolutely. Operative words: some scripts. The question often comes down to whether it is worthwhile to have less stale information later or have quicker (but sometimes more stale) data responses. Some tools, like the board agenda tool, opt to show possibly stale data quickly and immediately get fresher data and update if necessary. > Thoughts? - Sam Ruby >> I'd start by parsing the whimsy_server/manifests/cronjobs.pp. >> >> > John >> > >> > On Sat, Jun 10, 2017 at 8:58 AM John D. Ament <johndam...@apache.org> >> wrote: >> > >> >> I just pushed up a change that requires a change to the podlings.json >> >> file. However, that file is only generated in cron. I was wondering >> if it >> >> made sense that on deploy, to run a script that would regenerate the >> static >> >> files? >> >> >> >> John >> >> - Sam Ruby >>
