[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 Mark A. Hershberger m...@everybody.org changed: What|Removed |Added Target Milestone|1.21.x release |--- --- Comment #29 from Mark A. Hershberger m...@everybody.org --- Removing target milestone that was in the past. If you want this in a specific release, have a good reason AND you are willing to find resources to fix this bug, feel free to change it to something appropriate. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #28 from Chris Koerner nob...@gmail.com --- After changing $wgCliPhp to be blank my job queue now runs and clears out jobs without issue. $wgPhpCli = ; As discovered in this thread: https://www.mediawiki.org/w/index.php?title=Project:Support_deskoffset=20140227154556lqt_mustshow=40130#Mediawiki_1.22.2B_Causes_two_of_my_servers_to_hang_indefinitely._37734 -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 Andre Klapper aklap...@wikimedia.org changed: What|Removed |Added See Also||https://bugzilla.wikimedia. ||org/show_bug.cgi?id=60719 -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #26 from Chris Koerner nob...@gmail.com --- Aaron, can you explain a little more about the claim TTL? I don't recall setting that anywhere. I'm trying to pinpoint the cause as much as I can without touching my prod environment. We didn't have this issue in test (berating me for not having more sophisticated QA is justified!) and in order to make changes to prod I have many hoops I must jump through now. One interesting note is that trying to specify the --memory-limit when running runJobs.php throws an error. I'm beginning to think this might be related to available ram. (Server has 2gb, php.ini has 128mb) php runJobs.php --memory-limit 1024 PHP Fatal error: Allowed memory size of 262144 bytes exhausted (tried to allocate 122880 bytes) in /var/www/html/w/includes/AutoLoader.php on line 290 -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #27 from Aaron Schulz aschulz4...@gmail.com --- If the only jobs that linger have job_attempts set to something other than zero, then this is just a problem of failed jobs. You'd want to set wgDebugLogGroups as above to possibly get more insights on why jobs are failing sometimes. Jobs can fail for any number of reasons, mostly specific to the code the job classes run. That in itself doesn't indicate any problem with the job queue itself. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #24 from Chris Koerner nob...@gmail.com --- Another note (and someone tell me to shut up if this isn't proper etiquette) it appears that running php refreshLinks.php will queue up a number of jobs. Running runJobs.php afterward will kick off many jobs (in a queue of 2000, 1500 or so) but does not complete all jobs as a result of refreshLinks.php or any of the other jobs queued. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #25 from Aaron Schulz aschulz4...@gmail.com --- What is $wgJobTypeConf set to? All the jobs you posted had at least 1 attempt. They won't run again until the claim TTL is reached. I don't know what you set that too. By default, jobs the fail are never retried and get deleted after a week. You can try using: $wgJobTypeConf['default']['claimTTL'] = 3600; // 1 hour ...this will let the jobs be retried (after 1 hour of failure). You can also set: $wgDebugLogGroups['runJobs'] = some path ...this will log all jobs run, and may show some failures (fatal errors will not show here though). -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #22 from Chris Koerner nob...@gmail.com --- Nothing appears to run. http://i.imgur.com/eypighU.jpg -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #23 from Chris Koerner nob...@gmail.com --- I should note that certain actions on the site, such as modifying a template or running refreshLinks.php appears to not only add new jobs to the queue (as it should) but I can also run runJobs.php and a number of jobs will process. I don't see a pattern or commonality in what is run however. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #21 from Aaron Schulz aschulz4...@gmail.com --- What does php showJobs.php --group show? What about php showJobs.php --list? Does php runJobs.php --type refreshLinks actually run anything? -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 Chris Koerner nob...@gmail.com changed: What|Removed |Added CC||nob...@gmail.com --- Comment #17 from Chris Koerner nob...@gmail.com --- I had a similar thing happen. We were running 1.16.3 (yeah!) and upgraded to 1.23. Made a copy of the 1.16.3 database, pointed new 1.23wmf11 installation to new database. Ran update.php, everything is hunky-dory. Notice the next day that job queue is backed up. A few old jobs existed from the day of the upgrade (in 1.16.3). Tried clearing job_token and a few would run. Tried clearing out old pre 1.23 jobs, still not running. showJobs.php spits out 0, while api query and db shows jobs in the queue. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #18 from Aaron Schulz aschulz4...@gmail.com --- (In reply to comment #17) I had a similar thing happen. We were running 1.16.3 (yeah!) and upgraded to 1.23. Made a copy of the 1.16.3 database, pointed new 1.23wmf11 installation to new database. Ran update.php, everything is hunky-dory. Notice the next day that job queue is backed up. A few old jobs existed from the day of the upgrade (in 1.16.3). Tried clearing job_token and a few would run. Tried clearing out old pre 1.23 jobs, still not running. showJobs.php spits out 0, while api query and db shows jobs in the queue. What types of jobs? What to some of the rows look like? -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #20 from Chris Koerner nob...@gmail.com --- Note, this is after I removed some of the older jobs in an attempt to 'kick start' the queue. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #19 from Chris Koerner nob...@gmail.com --- Here's the job table from one of our wikis (total separate installations, but configured identically) http://pastebin.com/i4Qatgpa -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #16 from Nemo federicol...@tiscali.it --- WikiApiary still had hundreds of lingering jobs since October, till they dropped them from the DB yesterday because it was impossible to run them. We're still waiting for a general solution to the migration problems. http://lists.thingelstad.com/pipermail/wikiapiary-l/2014-February/000104.html http://lists.thingelstad.com/pipermail/wikiapiary-l/attachments/20140202/fd838fb7/attachment-0002.png http://lists.thingelstad.com/pipermail/wikiapiary-l/attachments/20140202/fd838fb7/attachment-0003.png -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 Nemo federicol...@tiscali.it changed: What|Removed |Added Status|NEW |RESOLVED Resolution|--- |WONTFIX Target Milestone|1.21.x release |--- --- Comment #14 from Nemo federicol...@tiscali.it --- hexmode said this wasn't deemed worth fixing for 1.21 release. I don't know the reasons, but better a partial update than nothing. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 Mark A. Hershberger m...@everybody.org changed: What|Removed |Added Status|RESOLVED|REOPENED Resolution|WONTFIX |--- Target Milestone|--- |1.21.x release --- Comment #15 from Mark A. Hershberger m...@everybody.org --- (In reply to comment #14) hexmode said this wasn't deemed worth fixing for 1.21 release. I don't know the reasons, but better a partial update than nothing. What I meant is that this isn't going to stop 1.21.0 from being released. It is still a valid bug that should be fixed at some point. If we can get it fixed in a 1.21 point release that would be great. I don't know enough about the problem to fix it, though. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 Nemo federicol...@tiscali.it changed: What|Removed |Added Target Milestone|1.21.0 release |1.21.x release -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #5 from Andre Klapper aklap...@wikimedia.org --- Aaron: Could you comment on this please, as the 1.21 tarball release is imminent? -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 Mark A. Hershberger m...@everybody.org changed: What|Removed |Added CC||m...@everybody.org --- Comment #6 from Mark A. Hershberger m...@everybody.org --- (In reply to comment #3) Do we seriously have no way to fix this? Should we just tell people not to upgrade if they have something in the job queue? I'd like to include something in the installation information telling users to clear their job queue before upgrading, but I don't know a lot about this. What would be appropriate? I would like to get this fixed ASAP for a point release. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 Tyler Romeo tylerro...@gmail.com changed: What|Removed |Added CC||tylerro...@gmail.com --- Comment #7 from Tyler Romeo tylerro...@gmail.com --- (In reply to comment #6) (In reply to comment #3) Do we seriously have no way to fix this? Should we just tell people not to upgrade if they have something in the job queue? I'd like to include something in the installation information telling users to clear their job queue before upgrading, but I don't know a lot about this. What would be appropriate? I would like to get this fixed ASAP for a point release. Running maintenance/runJobs.php should clear the job queue. But depending on how long upgrading takes, one or two jobs might still be lost. Maybe put the wiki into read-only mode once the job queue is cleared? -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #8 from Mark A. Hershberger m...@everybody.org --- Could maintenance/runJobs.php be run with the wiki in read-only mode? -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #9 from Tyler Romeo tylerro...@gmail.com --- (In reply to comment #8) Could maintenance/runJobs.php be run with the wiki in read-only mode? Nope, I meant making it read-only after clearing the job queue. Still not a complete solution, but I can't think of anything else. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #10 from Aaron Schulz aschulz4...@gmail.com --- (In reply to comment #0) See http://lists.wikimedia.org/pipermail/mediawiki-l/2013-April/040970.html I remember that WMF had a similar problem, is there a solution apart from dropping the old jobs? Did we? The only problem is that they have a harder time getting picked if there always a bunch of other new jobs. In master and REL1_21 I've tried adding a bunch of jobs and setting the token to 0 for all of them, and runJobs.php works just fine. They weren't lost for me. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #11 from Aaron Schulz aschulz4...@gmail.com --- (In reply to comment #10) In master and REL1_21 I've tried adding a bunch of jobs and setting the token to 0 for all of them, and runJobs.php works just fine. They weren't lost for me. I mean job_random of course, not the token. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #12 from Nemo federicol...@tiscali.it --- (In reply to comment #11) I mean job_random of course, not the token. Ah. :) Well, isn't setting job_random exactly what the user in comment 0 didn't do and should have done? -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #13 from Aaron Schulz aschulz4...@gmail.com --- (In reply to comment #12) (In reply to comment #11) I mean job_random of course, not the token. Ah. :) Well, isn't setting job_random exactly what the user in comment 0 didn't do and should have done? The complaint was that 0 valued ones didn't work, so I set all mine to that value and they still worked. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #3 from Nemo federicol...@tiscali.it --- Do we seriously have no way to fix this? Should we just tell people not to upgrade if they have something in the job queue? -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #4 from Andre Klapper aklap...@wikimedia.org --- Aaron: Could you comment on this, please? -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 Andre Klapper aklap...@wikimedia.org changed: What|Removed |Added Priority|Unprioritized |High CC||g...@wikimedia.org -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 Bartosz Dziewoński matma@gmail.com changed: What|Removed |Added CC||isabel.ganc...@gmail.com --- Comment #1 from Bartosz Dziewoński matma@gmail.com --- *** Bug 46971 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 Bartosz Dziewoński matma@gmail.com changed: What|Removed |Added CC||matma@gmail.com --- Comment #2 from Bartosz Dziewoński matma@gmail.com --- Quoting from the duped bug: (Quoting bug 46971 comment #0) After I upgraded from 1.20.3, in my database (MySQL) some pre-upgrade jobs have job_random set to 0 and do not seem to be picked up -not even when I try to run them by providing their type as an option: php runJobs.php --type=replaceText. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
