Re: [Wikitech-l] deploying the most recent MediaWiki code: which branch?
On Thu, Feb 20, 2014 at 2:37 PM, Ryan Lane wrote: > Note that unless you're willing to keep up to date with WMF's relatively > fast pace of branching, you're going to miss security updates. No matter > what, if you use git you're going to get security updates slower, since > they are released into the tarballs first, then merged into master, then > branches (is this accurate?). Sometimes the current WMF branch won't even > get the security updates since they are already merged locally onto > Wikimedia's deployment server. > I've been releasing tarballs, then pushing the fixes into the release branches and master in gerrit. It all happens within a couple of hours, but the tarballs have a slightly narrower timeframe. I rarely push to wmfXX branches, since those already have the patches applied on the cluster, and the next branch cut from master will contain the fix from master. We're potentially moving to pushing them into gerrit and having jenkins build the tarballs, so this process might be flipped in the near future. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] deploying the most recent MediaWiki code: which branch?
On 2014-02-20 2:50 PM, Bartosz Dziewoński wrote: > On Thu, 20 Feb 2014 23:21:37 +0100, Greg Grossmeier > wrote: > >> >>> It's worth noting that WMF branches also include temporary hacks to >>> keep current JS/CSS and cached HTML output compatible (for at least 30 >>> days), while release branches never contain them (and thus require >>> HTML caches to be purged during the upgrade process). >> >> Isn't that: Feel free to base it off of either. There shouldn't be any WMF-specific things in those wmfXX branches. If there is, it is a commit called something like "Commit of various WMF live hacks". That one commit can be safely reverted. >> >> eg: >> https://git.wikimedia.org/commit/mediawiki%2Fcore.git/a868d086b68f05e7f93727dfb992c3f59c0525cf >> > > Nope, I mean different hacks (that people generally don't bother > ops/deployers with), like the ones being removed by > https://gerrit.wikimedia.org/r/#/c/61075/ or > https://gerrit.wikimedia.org/r/#/c/72151/ or > https://gerrit.wikimedia.org/r/#/c/102492/ or > https://gerrit.wikimedia.org/r/#/c/82102/ . They are required because > (to simplify) generated page HTML (which is cached for up to 30 days > on our cluster) includes links to "autoupdating" JS and CSS code. Thus > any JS/CSS changes need to be compatible with older generated HTML. I've thought for awhile that there are better ways to deal with the cache than this. My thought was a sort of snapshot feature for ResourceLoader: https://www.mediawiki.org/wiki/User:Dantman/Code_Ideas#rl-snapshot - We start outputting &version= on the load.php requests for skin resources. - WMF runs a script that dumps the current state of just about all modules into a snapshot folder. - The MediaWiki upgrade is performed. - Some configuration makes reference to the location of the snapshot. - When load.php sees an old &version= it serves resources from the snapshot instead of MediaWiki, this way even if the styles, scripts, etc... have been modified, the module has been renamed, or even entirely deleted the module is still served to old pages. This way we don't have to create or revert any hacks at all, we are unrestricted in the types of CSS/JS and HTML changes we make, and WMF doesn't get any special treatment since now any wiki large enough to have issues resetting its entire cache can take advantage of it. ~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://danielfriesen.name/] ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] deploying the most recent MediaWiki code: which branch?
> Note that unless you're willing to keep up to date with WMF's relatively > fast pace of branching, you're going to miss security updates. No matter > what, if you use git you're going to get security updates slower, since > they are released into the tarballs first, then merged into master, then > branches (is this accurate?). Sometimes the current WMF branch won't even > get the security updates since they are already merged locally onto > Wikimedia's deployment server. That's a good point, with one small clarification/rewording: Someone who's following wmfXX branches will get the security fixes the next branch after the tarball is released. That's usually with in the working week (we tend to release tarballs on Mon/Tues, with new branches on Thursday). So, yes, if you're pacing behind on the wmfXX branches, you'll want to take note of security releases and backport patches as appropriate (all security bugs have single patches attached to the Bugzilla report, and those are made public after the tarball is released). Greg -- | Greg GrossmeierGPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @gregA18D 1138 8E47 FAC8 1C7D | ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] deploying the most recent MediaWiki code: which branch?
On Thu, 20 Feb 2014 23:21:37 +0100, Greg Grossmeier wrote: It's worth noting that WMF branches also include temporary hacks to keep current JS/CSS and cached HTML output compatible (for at least 30 days), while release branches never contain them (and thus require HTML caches to be purged during the upgrade process). Isn't that: Feel free to base it off of either. There shouldn't be any WMF-specific things in those wmfXX branches. If there is, it is a commit called something like "Commit of various WMF live hacks". That one commit can be safely reverted. eg: https://git.wikimedia.org/commit/mediawiki%2Fcore.git/a868d086b68f05e7f93727dfb992c3f59c0525cf Nope, I mean different hacks (that people generally don't bother ops/deployers with), like the ones being removed by https://gerrit.wikimedia.org/r/#/c/61075/ or https://gerrit.wikimedia.org/r/#/c/72151/ or https://gerrit.wikimedia.org/r/#/c/102492/ or https://gerrit.wikimedia.org/r/#/c/82102/ . They are required because (to simplify) generated page HTML (which is cached for up to 30 days on our cluster) includes links to "autoupdating" JS and CSS code. Thus any JS/CSS changes need to be compatible with older generated HTML. -- Matma Rex ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] deploying the most recent MediaWiki code: which branch?
On Thu, Feb 20, 2014 at 1:20 PM, Sumana Harihareswara wrote: > Reuben Smith of wikiHow asked: > "We're having a hard time figuring out whether we should be basing our > wikiHow code off Mediawiki's external releases (such as the latest 1.22.2), > or off the branches that WMF uses for their internal infrastructure (latest > looks to be wmf/1.23wmf14). > > Do you have any thoughts of guidance on that? We're leaning towards moving > to using the WMF internal branches, since we use MySQL as well, but I > wanted to hear from different people about the drawbacks." > > > Greg Grossmeier responded: > > "The quick answer is: > Feel free to base it off of either. There shouldn't be any WMF-specific > things in those wmfXX branches. If there is, it is a commit called > something like "Commit of various WMF live hacks". That one commit can > be safely reverted. > > The wmfXX branches are made every week on Thursday morning (Pacific) > before we deploy. As we get closer to the next release (1.23) the > MediaWiki Release Managers (our outside contractors, Mark and Markus, > not myself) will pick a wmfXX to call a Release Candidate. > > Going with a 1.22.x would give you stability at the loss of getting > fixes faster and it means a bigger upgrade task when 1.23 is out. > > Summary: If you want to keep closer to WMF, pick a wmfXX branch (this > assumes you'll follow at some pace behind WMF). If you don't want to be > that bleeding edge, stick with 1.22.x. > > Note that unless you're willing to keep up to date with WMF's relatively fast pace of branching, you're going to miss security updates. No matter what, if you use git you're going to get security updates slower, since they are released into the tarballs first, then merged into master, then branches (is this accurate?). Sometimes the current WMF branch won't even get the security updates since they are already merged locally onto Wikimedia's deployment server. That said, due to the poor state of extension management in MediaWiki, the only reasonable way to manage MediaWiki is to use the WMF branches since they handle dependencies for the most popular extensions. I was hoping that composer would make managing extensions easier, but I've been tracking it in SMW and it's actually making things more difficult. - Ryan ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] deploying the most recent MediaWiki code: which branch?
> It's worth noting that WMF branches also include temporary hacks to > keep current JS/CSS and cached HTML output compatible (for at least 30 > days), while release branches never contain them (and thus require > HTML caches to be purged during the upgrade process). Isn't that: >> Feel free to base it off of either. There shouldn't be any WMF-specific >> things in those wmfXX branches. If there is, it is a commit called >> something like "Commit of various WMF live hacks". That one commit can >> be safely reverted. eg: https://git.wikimedia.org/commit/mediawiki%2Fcore.git/a868d086b68f05e7f93727dfb992c3f59c0525cf Greg -- | Greg GrossmeierGPG: B2FA 27B1 F7EB D327 6B8E | | identi.ca: @gregA18D 1138 8E47 FAC8 1C7D | ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] deploying the most recent MediaWiki code: which branch?
On Thu, Feb 20, 2014 at 9:53 PM, Isarra Yos wrote: > It would require pretty consistent maintenance of their own, but it could be > worth it. IOW, you need to hire a Greg Grossmeier :) -Jeremy ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] deploying the most recent MediaWiki code: which branch?
On 20/02/14 21:32, Bartosz Dziewoński wrote: It's worth noting that WMF branches also include temporary hacks to keep current JS/CSS and cached HTML output compatible (for at least 30 days), while release branches never contain them (and thus require HTML caches to be purged during the upgrade process). This usually only affects changes to the Vector skin which is not the default there (I'm not sure if it's even available), but nevertheless wikiHow might prefer one of these depending on how their caching layer is set up. If wikiHow's setup is sufficiently different, it might be worth considering making their own branches from the master - test and fix things from what was the current master, deploy it, move the test up to the new master, and repeat. This would do the same as with the wmf branches to avoid a lot of the usual difficulty involved in upgrading, but could perhaps also be tailored to be more specific to the systems wikiHow uses. It would require pretty consistent maintenance of their own, but it could be worth it. -I ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] deploying the most recent MediaWiki code: which branch?
It's worth noting that WMF branches also include temporary hacks to keep current JS/CSS and cached HTML output compatible (for at least 30 days), while release branches never contain them (and thus require HTML caches to be purged during the upgrade process). This usually only affects changes to the Vector skin which is not the default there (I'm not sure if it's even available), but nevertheless wikiHow might prefer one of these depending on how their caching layer is set up. -- Matma Rex ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] deploying the most recent MediaWiki code: which branch?
I got permission from Reuben Smith of wikihow and WMF release manager Greg Grossmeier to re-post this exchange. Sumana Harihareswara Engineering Community Manager Wikimedia Foundation Reuben Smith of wikiHow asked: "We're having a hard time figuring out whether we should be basing our wikiHow code off Mediawiki's external releases (such as the latest 1.22.2), or off the branches that WMF uses for their internal infrastructure (latest looks to be wmf/1.23wmf14). Do you have any thoughts of guidance on that? We're leaning towards moving to using the WMF internal branches, since we use MySQL as well, but I wanted to hear from different people about the drawbacks." Greg Grossmeier responded: "The quick answer is: Feel free to base it off of either. There shouldn't be any WMF-specific things in those wmfXX branches. If there is, it is a commit called something like "Commit of various WMF live hacks". That one commit can be safely reverted. The wmfXX branches are made every week on Thursday morning (Pacific) before we deploy. As we get closer to the next release (1.23) the MediaWiki Release Managers (our outside contractors, Mark and Markus, not myself) will pick a wmfXX to call a Release Candidate. Going with a 1.22.x would give you stability at the loss of getting fixes faster and it means a bigger upgrade task when 1.23 is out. Summary: If you want to keep closer to WMF, pick a wmfXX branch (this assumes you'll follow at some pace behind WMF). If you don't want to be that bleeding edge, stick with 1.22.x. Hope that helps, Greg PS: To learn more about our deploy cycle, see: https://wikitech.wikimedia.org/wiki/Deployments/One_week PPS: To see where we are at any given point, wrt MediaWiki, see: https://www.mediawiki.org/wiki/MediaWiki_1.23/Roadmap#Schedule_for_the_deployments"; ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
