Thanks for the info James. I guess caching introduces a whole new dimension.
Cheers Neill. On 25/09/14 14:41, James HK wrote: > Hi, > >> 1) What does "as of 1.9.2" mean exactly? Is this a change of behaviour >> from older versions? > By the time of the writing (which included SMW 1.9.2 therefore "as of" > and all previous versions) SMW does not support a "real" auto update > mechanism for queries and with "real" auto update we think of "smart > queries" where an algorithm is used to find a data diff which would > trigger a refresh of only those related subjects and its queries > within the store (and hopefully invalidates the MediaWiki html/parser > cache we well otherwise it prevents the page from displaying the > update results). > > [0] refers to the query update not the property/value update of a page. > >> 2) Does a page edit/save where a property value is changed trigger an >> update/refresh or not? > It does not directly trigger an UpdateJob but of course it triggers an > synchronizes update of the Store. Two exceptions are made for the > change of a property type (those on a `Property:` page) and when > creating a redirect (as explained in [1]). > > [0] http://semantic-mediawiki.org/wiki/Automatic_query_update > [1] http://semantic-mediawiki.org/wiki/Help:UpdateJob > > Cheers > > On 9/25/14, Neill Mitchell <ne...@nlkmitchell.com> wrote: >> Hi. >> >> Following [0] led me to this: >> http://semantic-mediawiki.org/wiki/Automatic_query_update which states: >> >> Currently Semantic MediaWiki (as of 1.9.2) does not support a "real" >> auto update (aka "smart" queries) that would refresh the results of >> pages that include the query results. >> >> I want to achieve clarity on this (and it is the entire point of my >> github ticket). So, my questions are: >> >> 1) What does "as of 1.9.2" mean exactly? Is this a change of behaviour >> from older versions? >> 2) Does a page edit/save where a property value is changed trigger an >> update/refresh or not? >> >> Clarification of these will probably answer Bernhard's query as well. >> >> Many thanks. >> >> >> On 25/09/14 13:56, James HK wrote: >>> Hi, >>> >>> A short summary of the current discussion. I further recommend to read >>> [0] which tries to explain as to when the `UpdateJob` is triggered. >>> >>> Krabina Bernhard: >>> >>> Using SMW 1.8.0.5 / MW 1.22 with "Suddenly, without anybody knowing >>> why, ~ 5000 have been done, but > 7000 are still sitting there ", "our >>> problem is not only that we cannot get the hanging jobs to run, but >>> that even new jobs being created (by changing a template or CSV import >>> with data transfer) will not run.... " >>> >>> Yaron Koren >>> >>> Explains that "each job is only attempted a certain number of times >>> before MediaWiki basically just gives up and ignores it" and by >>> executing "UPDATE job SET job_attempts = 0" allows to reset the >>> "job_attempts" field. >>> >>> James Montalvo / Daren Welsh >>> >>> Upgrading to MW 1.23 and SMW 2.0 from 1.22/1.8.0.5 seeing "stuck" jobs >>> for "job_cmd of refreshLinks in job ... other four have job_cmd of >>> SMW\UpdateJob", "Those "stuck" jobs just keep repeating over and >>> over.", "runJobs evidently will eventually stop (or time-out) but only >>> after repeating certain jobs hundreds of times." >>> >>> Phil Legault >>> >>> Explained that "same issue a while ago after I restored a database." >>> but "ran another backup from my production database after I cleared >>> the runjobs, and this worked out fine after". >>> >>> Neill Mitchell >>> >>> I can't think of any reason why the current thread and [1] are related >>> based on the information given in [0]. >>> >>> [0] http://semantic-mediawiki.org/wiki/Help:UpdateJob >>> [1] >>> https://github.com/SemanticMediaWiki/SemanticMediaWiki/issues/518#issuecomment-54269008 >>> >>> Related discussions >>> >>> - https://github.com/SemanticMediaWiki/SemanticMediaWiki/issues/212 >>> - https://github.com/SemanticMediaWiki/SemanticMediaWiki/pull/307 >>> - https://github.com/SemanticMediaWiki/SemanticMediaWiki/issues/330 >>> - https://github.com/SemanticMediaWiki/SemanticMediaWiki/issues/297 >>> - >>> http://wikimedia.7.x6.nabble.com/Re-Job-queue-affecting-semantic-queries-in-MW-1-22-tp5029433.html >>> - http://wikimedia.7.x6.nabble.com/SMW-UpdateJob-error-tp5036300.html >>> >>> Cheers >>> >>> On 9/25/14, Neill Mitchell <ne...@nlkmitchell.com> wrote: >>>> Hi Bernhard. >>>> >>>> Perhaps you are seeing the same issue as me. I have this logged on >>>> github: >>>> https://github.com/SemanticMediaWiki/SemanticMediaWiki/issues/518#issuecomment-54269008 >>>> >>>> Please let me know if you think your issue may be the same as mine. >>>> >>>> Best regards >>>> Neill. >>>> >>>> On 25/09/14 12:02, Krabina Bernhard wrote: >>>>> our problem is not only that we cannot get the hanging jobs to run, but >>>>> that even new jobs being created (by changing a template or CSV import >>>>> with data transfer) will not run.... >>>>> >>>>> regards, >>>>> Bernhard >>>>> >>>>> ----- Ursprüngliche Mail ----- >>>>>> I had the same issue a while ago after I restored a database. >>>>>> I figured it was because when I ran the backup there where jobs in the >>>>>> queue >>>>>> and after the restore they kept running over and over and etc. >>>>>> >>>>>> I ran another backup from my production database after I cleared the >>>>>> runjobs, >>>>>> and this worked out fine after. >>>>>> >>>>>> >>>>>> >>>>>> -----Original Message----- >>>>>> From: James Montalvo [mailto:jamesmontal...@gmail.com] >>>>>> Sent: Wednesday, September 24, 2014 5:55 PM >>>>>> To: Daren Welsh >>>>>> Cc: Semantic MediaWiki developers; Yaron Koren; Semantic MediaWiki >>>>>> users >>>>>> Subject: Re: [Semediawiki-user] [SMW-devel] MediaWiki Job queue problem >>>>>> >>>>>> Daren and I work together, so we have the same issue. One thing to add: >>>>>> Run >>>>>> jobs never finishes. Those "stuck" jobs just keep repeating over and >>>>>> over. >>>>>> >>>>>> On Wed, Sep 24, 2014 at 4:51 PM, Daren Welsh <darenwe...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> I have executed runJobs several times and the job_attempts remains at >>>>>>> 1 for those five jobs. We were thinking of doing a database backup >>>>>>> today, then delete those five jobs from the table, then run the SMW >>>>>>> "repair and upgrade" via the admin special page. >>>>>>> >>>>>>> Even if this clears the job queue, we'd like to understand what caused >>>>>>> this in the first place. I realize that's a very open-ended question >>>>>>> :) >>>>>>> >>>>>>> Daren >>>>>>> >>>>>>> >>>>>>> On Wed, Sep 24, 2014 at 4:30 PM, James HK >>>>>>> <jamesin.hongkon...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>>> We currently have five jobs that are "stuck". All of them have 1 >>>>>>>>> for job_attempts. >>>>>>>>> >>>>>>>>> One has job_cmd of refreshLinks in job namespace 10 and it is for a >>>>>>>>> template page. >>>>>>>>> The other four have job_cmd of SMW\UpdateJob in job namespace 0 and >>>>>>>>> are >>>>>>>> for >>>>>>>>> "standard" pages. These pages do not seem to be related based on >>>>>>>> category >>>>>>>>> or template. >>>>>>>> Just to make sure that I interpret the meaning of "stuck" correctly, >>>>>>>> after finishing `runJobs` those four jobs (five with the >>>>>>>> `refreshLinks` jobs) are still visible in the job table with an >>>>>>>> "job_attempts" of 1. When running `runJobs` again the same four >>>>>>>> `SMW\UpdateJob` (same as in the same title and same Id) jobs are >>>>>>>> executed and increment the "job_attempts" to 2? >>>>>>>> >>>>>>>> If you empty the job table and execute `runJobs` does the same five >>>>>>>> jobs appear again after the run with "job_attempts" = 1? >>>>>>>> >>>>>>>> Cheers >>>>>>>> >>>>>>>> On 9/25/14, Daren Welsh <darenwe...@gmail.com> wrote: >>>>>>>>> We currently have five jobs that are "stuck". All of them have 1 >>>>>>>>> for job_attempts. >>>>>>>>> >>>>>>>>> One has job_cmd of refreshLinks in job namespace 10 and it is for a >>>>>>>>> template page. >>>>>>>>> The other four have job_cmd of SMW\UpdateJob in job namespace 0 and >>>>>>>>> are >>>>>>>> for >>>>>>>>> "standard" pages. These pages do not seem to be related based on >>>>>>>> category >>>>>>>>> or template. >>>>>>>>> >>>>>>>>> On Wed, Sep 24, 2014 at 3:37 PM, James HK >>>>>>>>> <jamesin.hongkon...@gmail.com >>>>>>>>> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>>> runJobs.php will literally run forever. After the non-offending >>>>>>>>>>> jobs are cleared it's easy to see which are the offenders. Thus >>>>>>>>>>> far I think >>>>>>>> all >>>>>>>>>>> offenders have been of type SMW::UpdateJob. >>>>>>>>>> I don't think the problem is with the `SMW\UpdateJob` because it >>>>>>>>>> does a simple "shallow update" of the store while the management >>>>>>>>>> of job status (including how many attempts, id's etc.) are done by >>>>>>>>>> the MW JobQueue (which has first change in 1.22 and then again in >>>>>>>>>> 1.23). >>>>>>>>>> >>>>>>>>>> It does beg the question whether all `SMW\UpdateJob`'s are "stuck" >>>>>>>>>> or only certain jobs belonging to a group of pages or single page? >>>>>>>>>> >>>>>>>>>>> runJobs.php, but for some reason they keep attempting to run >>>>>>>>>>> over and >>>>>>>>>> over. >>>>>>>>>> >>>>>>>>>> How do you know that the same job is run over and over again >>>>>>>>>> because based and above discussion ("job_attempts") a job with too >>>>>>>>>> many attempts is retired after some time. >>>>>>>>>> >>>>>>>>>> If the same job is run over and over again, what is displayed for >>>>>>>>>> the "job_attempts" counter? >>>>>>>>>> >>>>>>>>>> [0] went into SMW 2.0 to counteract any possible job duplicates >>>>>>>>>> for the same `root title`. >>>>>>>>>> >>>>>>>>>> [0] >>>>>>>>>> https://github.com/SemanticMediaWiki/SemanticMediaWiki/pull/307 >>>>>>>>>> >>>>>>>>>> Cheers >>>>>>>>>> >>>>>>>>>> On 9/25/14, James Montalvo <jamesmontal...@gmail.com> wrote: >>>>>>>>>>> I'm not sure if this is related, but on my wiki I'm occasionally >>>>>>>>>>> getting "stuck" jobs. I've only noticed this since upgrading to >>>>>>>>>>> MW 1.23 and >>>>>>>> SMW >>>>>>>>>> 2.0 >>>>>>>>>>> from 1.22/1.8.0.5. >>>>>>>>>>> >>>>>>>>>>> What I mean by "stuck" is that the jobs don't get executed when >>>>>>>>>>> I do runJobs.php, but for some reason they keep attempting to >>>>>>>>>>> run over and >>>>>>>>>> over. >>>>>>>>>>> runJobs.php will literally run forever. After the non-offending >>>>>>>>>>> jobs are cleared it's easy to see which are the offenders. Thus >>>>>>>>>>> far I think >>>>>>>> all >>>>>>>>>>> offenders have been of type SMW::UpdateJob. >>>>>>>>>>> >>>>>>>>>>> Is there some way to debug runJobs.php so I can provide better >>>>>>>>>>> info? >>>>>>>>>>> >>>>>>>>>>> --James >>>>>>>>>>> On Sep 24, 2014 10:55 AM, "Yaron Koren" <ya...@wikiworks.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> I certainly hope so too - or that there's some other standard >>>>>>>>>>>> way to get previously-attempted jobs to be run again. I only >>>>>>>>>>>> know that I tried that SQL trick once, and it worked. Perhaps >>>>>>>>>>>> this is another reason why >>>>>>>> the >>>>>>>>>>>> question should have instead been sent to the mediawiki-l >>>>>>>>>>>> mailing list. >>>>>>>>>>>> :) >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Sep 24, 2014 at 11:35 AM, James HK < >>>>>>>>>> jamesin.hongkon...@gmail.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>>> column is greater than 0 for all the rows in the table; I >>>>>>>>>>>>>> think >>>>>>>> if >>>>>>>>>>>>>> you >>>>>>>>>>>>> just >>>>>>>>>>>>>> go into the database and call something like "UPDATE job >>>>>>>>>>>>>> SET >>>>>>>>>>>>> job_attempts = >>>>>>>>>>>>>> 0", they will get run again. >>>>>>>>>>>>> In case this solves the issue, I sincerely hope there is a >>>>>>>> different >>>>>>>>>>>>> way (a more standard way) to reset the "job_attempts" field >>>>>>>>>>>>> other than by using a SQL statement to manipulate the job >>>>>>>>>>>>> table. >>>>>>>>>>>>> >>>>>>>>>>>>> Cheers >>>>>>>>>>>>> >>>>>>>>>>>>> On 9/25/14, Yaron Koren <ya...@wikiworks.com> wrote: >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I believe the issue is the "job_attempts" field in the "job" >>>>>>>>>>>>>> table. >>>>>>>>>> I >>>>>>>>>>>>>> believe each job is only attempted a certain number of >>>>>>>>>>>>>> times before MediaWiki basically just gives up and ignores >>>>>>>>>>>>>> it. My guess is >>>>>>>> that >>>>>>>>>>>>>> that >>>>>>>>>>>>>> column is greater than 0 for all the rows in the table; I >>>>>>>>>>>>>> think >>>>>>>> if >>>>>>>>>>>>>> you >>>>>>>>>>>>> just >>>>>>>>>>>>>> go into the database and call something like "UPDATE job >>>>>>>>>>>>>> SET >>>>>>>>>>>>> job_attempts = >>>>>>>>>>>>>> 0", they will get run again. >>>>>>>>>>>>>> >>>>>>>>>>>>>> -Yaron >>>>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> WikiWorks · MediaWiki Consulting · http://wikiworks.com >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> --------- >>>>>>>>>>>> Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer >>>>>>>>>>>> Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI >>>>>>>>>>>> DSS Reports Are you Audit-Ready for PCI DSS 3.0 Compliance? >>>>>>>>>>>> Download White paper Comply to PCI DSS 3.0 Requirement 10 and >>>>>>>>>>>> 11.5 with EventLog Analyzer >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg >>>>>>>> .clktrk >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Semediawiki-user mailing list >>>>>>>>>>>> semediawiki-u...@lists.sourceforge.net >>>>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/semediawiki-user >>>>>>>>>>>> >>>>>>>> --------------------------------------------------------------------- >>>>>>>> --------- >>>>>>>>>> Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer >>>>>>>>>> Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS >>>>>>>> Reports >>>>>>>>>> Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White >>>>>>>>>> paper Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog >>>>>>>>>> Analyzer >>>>>>>>>> >>>>>>>>>> >>>>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg >>>>>>>> .clktrk >>>>>>>>>> _______________________________________________ >>>>>>>>>> Semediawiki-devel mailing list >>>>>>>>>> Semediawiki-devel@lists.sourceforge.net >>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/semediawiki-devel >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> __________________ >>>>>>>>> http://mixcloud.com/darenwelsh >>>>>>>>> http://www.beatportfolio.com >>>>>>>>> >>>>>>> -- >>>>>>> __________________ >>>>>>> http://mixcloud.com/darenwelsh >>>>>>> http://www.beatportfolio.com >>>>>>> >>>>>> ------------------------------------------------------------------------------ >>>>>> Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer Achieve >>>>>> PCI >>>>>> DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports Are you >>>>>> Audit-Ready for PCI DSS 3.0 Compliance? Download White paper Comply to >>>>>> PCI >>>>>> DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer >>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk >>>>>> _______________________________________________ >>>>>> Semediawiki-user mailing list >>>>>> semediawiki-u...@lists.sourceforge.net >>>>>> https://lists.sourceforge.net/lists/listinfo/semediawiki-user >>>>>> ------------------------------------------------------------------------------ >>>>>> Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer >>>>>> Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS >>>>>> Reports >>>>>> Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper >>>>>> Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer >>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk >>>>>> _______________________________________________ >>>>>> Semediawiki-user mailing list >>>>>> semediawiki-u...@lists.sourceforge.net >>>>>> https://lists.sourceforge.net/lists/listinfo/semediawiki-user >>>>>> >>>>> ------------------------------------------------------------------------------ >>>>> Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer >>>>> Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports >>>>> Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper >>>>> Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer >>>>> http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk >>>>> _______________________________________________ >>>>> Semediawiki-devel mailing list >>>>> Semediawiki-devel@lists.sourceforge.net >>>>> https://lists.sourceforge.net/lists/listinfo/semediawiki-devel >>>> ------------------------------------------------------------------------------ >>>> Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer >>>> Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports >>>> Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper >>>> Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer >>>> http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk >>>> _______________________________________________ >>>> Semediawiki-devel mailing list >>>> Semediawiki-devel@lists.sourceforge.net >>>> https://lists.sourceforge.net/lists/listinfo/semediawiki-devel >>>> >> >> ------------------------------------------------------------------------------ >> Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer >> Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports >> Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper >> Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer >> http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk >> _______________________________________________ >> Semediawiki-devel mailing list >> Semediawiki-devel@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/semediawiki-devel >> ------------------------------------------------------------------------------ Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk _______________________________________________ Semediawiki-devel mailing list Semediawiki-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/semediawiki-devel
