Re: [Wikitech-l] Nightly shallow clones of mediawiki/core
On Tue, Mar 12, 2013 at 10:48 PM, Chad innocentkil...@gmail.com wrote: Version numbers in Git don't reflect any sort of reality in terms of features or things to look forward Sometimes I mention Semantic Versioning[1] to people that write code for a living and get surprised that they did not hear about it. Željko -- [1] http://semver.org/ ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Nightly shallow clones of mediawiki/core
On Wed, Mar 13, 2013 at 11:02 AM, Željko Filipin zfili...@wikimedia.orgwrote: On Tue, Mar 12, 2013 at 10:48 PM, Chad innocentkil...@gmail.com wrote: Version numbers in Git don't reflect any sort of reality in terms of features or things to look forward Sometimes I mention Semantic Versioning[1] to people that write code for a living and get surprised that they did not hear about it. [1] http://semver.org/ Maybe that's because it's a buzzwordy new name for a very old idea? Although as described there, it tends to break if you have multiple different APIs that aren't necessarily in sync, or if the API is only a minor portion of the product. And marketing may complain if your competitor is at version 10 and you're at 2 (despite that being 2.98.917). ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Nightly shallow clones of mediawiki/core
On Monday, March 11, 2013 at 6:51 PM, Rob Lanphier wrote: On Sun, Mar 10, 2013 at 3:32 PM, Kevin Israel pleasest...@live.com (mailto:pleasest...@live.com) wrote: On 03/10/2013 06:03 PM, Bartosz Dziewoński wrote: A shallow clone certainly shouldn't be as large as a normal one. Something's borked. --depth 0 is what's broken. --depth 1 works fine. $ git clone --depth 1 [...] Receiving objects: 100% (2815/2815), 17.87 MiB | 1.16 MiB/s, done. Yup, I'm seeing more or less the same thing. Importantly: $ du -sh .git 19M .git I was able to do the clone in 50 seconds over HTTPS. Most of that time was spent in data transfer (which would be the same for a snapshot). Ori, have you tried this with --depth 1? Rob Rob, thanks for checking. I tried it yesterday and again just now, and in both cases it took around 15 minutes: vagrant@precise32:~$ time git clone --depth 1 https://gerrit.wikimedia.org/r/p/mediawiki/core.git Cloning into 'core'... remote: Counting objects: 46297, done remote: Finding sources: 100% (46297/46297) remote: Getting sizes: 100% (25843/25843) remote: Compressing objects: 76% (19864/25833) remote: Total 46297 (delta 33063), reused 26399 (delta 20010) Receiving objects: 100% (46297/46297), 102.66 MiB | 194 KiB/s, done. Resolving deltas: 100% (37898/37898), done. real 15m14.500s user 0m27.562s sys 0m13.421s The output of 'git config --list' is blank; this is vanilla git. 'Compressing objects' took the longest. For comparison: vagrant@precise32:~$ time wget -q https://github.com/wikimedia/mediawiki-core/archive/master.zip unzip -x -q master.zip real 1m15.592s user 0m0.184s sys 0m3.480s -- Ori Livneh ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Nightly shallow clones of mediawiki/core
Hi Ori, I'm at the office now, but (heh) that may be a tougher test than my home connection. More below On Tue, Mar 12, 2013 at 12:00 AM, Ori Livneh o...@wikimedia.org wrote: Rob, thanks for checking. I tried it yesterday and again just now, and in both cases it took around 15 minutes: vagrant@precise32:~$ time git clone --depth 1 https://gerrit.wikimedia.org/r/p/mediawiki/core.git Cloning into 'core'... remote: Counting objects: 46297, done remote: Finding sources: 100% (46297/46297) remote: Getting sizes: 100% (25843/25843) remote: Compressing objects: 76% (19864/25833) remote: Total 46297 (delta 33063), reused 26399 (delta 20010) Receiving objects: 100% (46297/46297), 102.66 MiB | 194 KiB/s, done. Resolving deltas: 100% (37898/37898), done. real 15m14.500s user 0m27.562s sys 0m13.421s The output of 'git config --list' is blank; this is vanilla git. 'Compressing objects' took the longest. For comparison: vagrant@precise32:~$ time wget -q https://github.com/wikimedia/mediawiki-core/archive/master.zip unzip -x -q master.zip real 1m15.592s user 0m0.184s sys 0m3.480s I'm on Ubuntu 12.10 (Quantal), so that might have something to do with it. $ git --version git version 1.7.10.4 $ time git clone --depth 1 https://gerrit.wikimedia.org/r/p/mediawiki/core.git core-shallow Cloning into 'core-shallow'... remote: Counting objects: 3456, done remote: Finding sources: 100% (3456/3456) remote: Getting sizes: 100% (3074/3074) remote: Compressing objects: 63% (1958/3069) remote: Total 3456 (delta 690), reused 1496 (delta 379) Receiving objects: 100% (3456/3456), 18.71 MiB | 1.49 MiB/s, done. Resolving deltas: 100% (816/816), done. real 0m34.507s user 0m4.252s sys 0m0.940s I wasn't able to find anything in the Git release notes to indicate anything different on the client side, though I didn't look too hard. However, it appears as though JGit server-side support for shallow clones is relatively recent: https://bugs.eclipse.org/bugs/show_bug.cgi?id=394543 ...characterized here as coming along bit by bit: http://dev.eclipse.org/mhonarc/lists/jgit-dev/msg01976.html ...which would imply that maybe the JGit server side stuff hasn't been tested with a lot of client versions. Rob ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Nightly shallow clones of mediawiki/core
On Tuesday, March 12, 2013 at 8:51 AM, Rob Lanphier wrote: ...which would imply that maybe the JGit server side stuff hasn't been tested with a lot of client versions. Rob, thanks again for investigating. I filed a bug in Bugzilla so we can track this: https://bugzilla.wikimedia.org/show_bug.cgi?id=46041 Could you capture the output of the following command: GIT_CURL_VERBOSE=1 GIT_TRACE=true git clone --verbose --depth 1 https://gerrit.wikimedia.org/r/p/mediawiki/core.git ~/core-git-clone.log And upload it as an attachment? I already uploaded mine. This way we can compare. -- Ori Livneh ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Nightly shallow clones of mediawiki/core
On Tuesday, March 12, 2013 at 1:36 PM, Ori Livneh wrote: I filed a bug in Bugzilla so we can track this: https://bugzilla.wikimedia.org/show_bug.cgi?id=46041 Platonides figured it out. Incredibly enough, in the half-sub-sub-version delta from 1.7.9.5 to 1.7.10, the behavior of git-clone was fixed so that it only fetches a single branch rather than all. The workaround for git 1.7.10 is: mkdir core; cd core git init git fetch --depth=1 https://gerrit.wikimedia.org/r/p/mediawiki/core.git master:refs/remotes/origin/master -- Ori Livneh ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Nightly shallow clones of mediawiki/core
On Tue, Mar 12, 2013 at 5:31 PM, Ori Livneh o...@wikimedia.org wrote: On Tuesday, March 12, 2013 at 1:36 PM, Ori Livneh wrote: I filed a bug in Bugzilla so we can track this: https://bugzilla.wikimedia.org/show_bug.cgi?id=46041 Platonides figured it out. Incredibly enough, in the half-sub-sub-version delta from 1.7.9.5 to 1.7.10, the behavior of git-clone was fixed so that it only fetches a single branch rather than all. The workaround for git 1.7.10 is: mkdir core; cd core git init git fetch --depth=1 https://gerrit.wikimedia.org/r/p/mediawiki/core.git master:refs/remotes/origin/master This totally doesn't surprise me at all. Version numbers in Git don't reflect any sort of reality in terms of features or things to look forward to--if memory serves, a series of really bad performance regressions (and fixes) were introduced in the course of just 1.6.x.y. -Chad ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Nightly shallow clones of mediawiki/core
On Sun, Mar 10, 2013 at 2:39 PM, Chad innocentkil...@gmail.com wrote: I'm open to any other ideas that could help core. I think it would be very useful to provide regular snapshots of core with a shallow '.git' included. Git-cloning from GitHub is faster than Gerrit but is still quite slow, and their snapshots, too, exclude '.git'. --- Ori Livneh o...@wikimedia.org ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Nightly shallow clones of mediawiki/core
On 11/03/13 00:17, Yuri Astrakhan wrote: Answered Inline. Also, I apologize as I think my email was slightly off-topic to Ori's question. On Sun, Mar 10, 2013 at 6:57 PM, Matthew Flaschen mflasc...@wikimedia.orgwrote: PHPMyAdmin also has major security issues. It isn't allowed on Wikimedia Labs and probably shouldn't be used here. Why does SQLite need to be installed exactly? Matthew, we are talking about a developer's virtual machine that has no network connection to anything except the developer's machine itself, and used purely for development. MySql could be accessed through the network if I have the local tools, but if we are talking about the lowest barrier of entry, the novice could follow a few steps to get VM and edit code with anything including notepad, while the VM would have most of the tools to examine and experiment with MediaWiki. BTW, samba would also be a massive security hole - I set mine up to share / with no password root access (no network, no issues) While popular, I don't think PHPMyAdmin is the best tool for managing the tables, either. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Nightly shallow clones of mediawiki/core
On 03/11/2013 07:09 PM, Ori Livneh wrote: I think it would be very useful to provide regular snapshots of core with a shallow '.git' included. Git-cloning from GitHub is faster than Gerrit but is still quite slow, and their snapshots, too, exclude '.git'. You can get a nightly snapshot of mediawiki from: https://toolserver.org/~krinkle/mwSnapshots/#!/mediawiki-core/master There is not .git, but maybe it could be added. -- http://hexmode.com/ Many that live deserve death. And some that die deserve life. Can you give it to them? Then do not be too eager to deal out death in judgment. For even the very wise cannot see all ends. -- J.R.R. Tolkien, The Fellowship of the Ring ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Nightly shallow clones of mediawiki/core
On Mon, Mar 11, 2013 at 8:31 PM, Mark A. Hershberger m...@everybody.orgwrote: You can get a nightly snapshot of mediawiki from: https://toolserver.org/~krinkle/mwSnapshots/#!/mediawiki-core/master There is not .git, but maybe it could be added. This is spectacular. Simple and easy. *--* *Tyler Romeo* Stevens Institute of Technology, Class of 2015 Major in Computer Science www.whizkidztech.com | tylerro...@gmail.com ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Nightly shallow clones of mediawiki/core
Am 12.03.2013 02:51, schrieb Rob Lanphier: I was able to do the clone in 50 seconds over HTTPS Ori, have you tried this with --depth 1? Rob I added - already some weeks ago - this comment to our main download _talk_ page https://www.mediawiki.org/wiki/Talk:Download#Discussion:_Whether_or_not_we_should_add_how_to_.22shallow.22_clone_the_repository_24332 I was unsure, whether it makes sense to add in section Download#Development releases https://www.mediawiki.org/wiki/Download#Development_releases the following: By using the depth parameter as shown in the following, you can download a shallow clone which saves bytes, bandwidth and time, [ but can lead to problems if you are developer ] git clone --depth 1 https://gerrit.wikimedia.org/r/p/mediawiki/core.git If my Url is correct, then someone of you can simply add the command to section https://www.mediawiki.org/wiki/Download#Development_releases of the _article_ page. At least, I wanted to suggest this, because it is the correct place. Thomas ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] Nightly shallow clones of mediawiki/core
Hello, I'm in the process of re-working mediawiki-vagrant, which is a set of scripts for provisioning a virtual machine for MediaWiki development. I'm struggling to identify the best way of fetching mediawiki/core. An ideal solution would have the following attributes: - Fast. - Includes .git metadata, to facilitate contribution of patches. - Viable on slow network connections. - Does not require a Gerrit account (to help newcomers get started quickly) What I tried: - A shallow (--depth=0) git-clone over HTTPS took around half an hour and required transferring 272MB, with 200MB taken up by .git/objects/pack. - The nightlies on integration.mediawiki.org are small (18MB) and easy to retrieve, but the most recent one is from December, and they don't contain any .git metadata. - The snapshots Krinkle maintains on the toolserver are both small and up-to-date, but they too do not contain any .git metadata. - The snapshot link on http://www.mediawiki.org/wiki/Download (https://gerrit.wikimedia.org/r/gitweb?p=mediawiki/core.git;a=snapshot;h=refs/heads/REL1_20;sf=tgz) just didn't work. It hangs for a while and then spits out HTML. - Getting a snapshot from GitHub would probably work, but I am loathe to depend on it. Does anyone have any suggestions? -- Ori Livneh ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Nightly shallow clones of mediawiki/core
Hi Ori, I use your vagrant VM all the time. Thanks!!! If we have a good vagrant setup, getting new devs on-board might be that much easier. I keep all the notes of the things I needed to do to your VM at http://www.mediawiki.org/wiki/User:Yurik/Installing_Linux_virtual_box_under_Windows_7 Most of it is not related to Windows, and I would love to have some (most) of it automated: * reconfigure network to be hostonly instead of bridged - major security issue * install phpmyadmin sqlite * samba (very useful on the windows box, possibly others) * configure git tools (script?) * DEBUG == that's a big one, setting it up for easy nobrainer debugging would be a great help. Including CLI -- i'm having hard time debugging unit tests. An initial shell script * auto-generated the SSH key and place it into the right dir * git submodule update --init Thanks! On Sun, Mar 10, 2013 at 5:15 PM, Ori Livneh o...@wikimedia.org wrote: Hello, I'm in the process of re-working mediawiki-vagrant, which is a set of scripts for provisioning a virtual machine for MediaWiki development. I'm struggling to identify the best way of fetching mediawiki/core. An ideal solution would have the following attributes: - Fast. - Includes .git metadata, to facilitate contribution of patches. - Viable on slow network connections. - Does not require a Gerrit account (to help newcomers get started quickly) What I tried: - A shallow (--depth=0) git-clone over HTTPS took around half an hour and required transferring 272MB, with 200MB taken up by .git/objects/pack. - The nightlies on integration.mediawiki.org are small (18MB) and easy to retrieve, but the most recent one is from December, and they don't contain any .git metadata. - The snapshots Krinkle maintains on the toolserver are both small and up-to-date, but they too do not contain any .git metadata. - The snapshot link on http://www.mediawiki.org/wiki/Download ( https://gerrit.wikimedia.org/r/gitweb?p=mediawiki/core.git;a=snapshot;h=refs/heads/REL1_20;sf=tgz) just didn't work. It hangs for a while and then spits out HTML. - Getting a snapshot from GitHub would probably work, but I am loathe to depend on it. Does anyone have any suggestions? -- Ori Livneh ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Nightly shallow clones of mediawiki/core
Hi, I've been thinking about this for the last week or so because it's becoming incredibly clear to me that core isn't scaling. It's already taking up over 4GB on the Gerrit box, and this is the primary reason core operations are slow. A couple of things I can think of to help the situation. - We can repack core on manganese. This should provide a bit of relief, but won't help long term. Core would have to be read-only for about an hour or two. -We can rewrite history (git-filter-branch) to remove some mistakes that exploded the repo size. Binaries later removed, things accidentally checked into ./extensions, etc. This could potentially greatly reduce object sizes and allow for tighter repacks. Major issue with history rewriting is everyone will have to reclone (all sha1s would change). I've not tested my theory yet. I'm open to any other ideas that could help core. -Chad On Mar 10, 2013 5:15 PM, Ori Livneh o...@wikimedia.org wrote: Hello, I'm in the process of re-working mediawiki-vagrant, which is a set of scripts for provisioning a virtual machine for MediaWiki development. I'm struggling to identify the best way of fetching mediawiki/core. An ideal solution would have the following attributes: - Fast. - Includes .git metadata, to facilitate contribution of patches. - Viable on slow network connections. - Does not require a Gerrit account (to help newcomers get started quickly) What I tried: - A shallow (--depth=0) git-clone over HTTPS took around half an hour and required transferring 272MB, with 200MB taken up by .git/objects/pack. - The nightlies on integration.mediawiki.org are small (18MB) and easy to retrieve, but the most recent one is from December, and they don't contain any .git metadata. - The snapshots Krinkle maintains on the toolserver are both small and up-to-date, but they too do not contain any .git metadata. - The snapshot link on http://www.mediawiki.org/wiki/Download ( https://gerrit.wikimedia.org/r/gitweb?p=mediawiki/core.git;a=snapshot;h=refs/heads/REL1_20;sf=tgz) just didn't work. It hangs for a while and then spits out HTML. - Getting a snapshot from GitHub would probably work, but I am loathe to depend on it. Does anyone have any suggestions? -- Ori Livneh ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Nightly shallow clones of mediawiki/core
The git-archive command is supposed to fulfill this functionality, but it seems it does not work anonymously, i.e., over HTTPS. I think it only works with ssh:// and with git://. Otherwise, GitHub's snapshots seem to be the only solution. *--* *Tyler Romeo* Stevens Institute of Technology, Class of 2015 Major in Computer Science www.whizkidztech.com | tylerro...@gmail.com ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Nightly shallow clones of mediawiki/core
A shallow clone certainly shouldn't be as large as a normal one. Something's borked. On Sun, 10 Mar 2013 22:39:44 +0100, Chad innocentkil...@gmail.com wrote: - We can repack core on manganese. This should provide a bit of relief, but won't help long term. Core would have to be read-only for about an hour or two. This should be just done already, and repeated from time to time. The repo is growing in size much quicker than it should (full clone now is over 250 MB, in September it was under 180 MB IIRC). -We can rewrite history (git-filter-branch) to remove some mistakes that exploded the repo size. Binaries later removed, things accidentally checked into ./extensions, etc. This could potentially greatly reduce object sizes and allow for tighter repacks. Major issue with history rewriting is everyone will have to reclone (all sha1s would change). I've not tested my theory yet. Ugh, the nuclear option. Let's try repacking first, can we? :) This is probably going to make gerrit a little unhappy when it's database thing gets inconsistent with the repo. -- Matma Rex ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Nightly shallow clones of mediawiki/core
On 11/03/13 08:15, Ori Livneh wrote: - Getting a snapshot from GitHub would probably work, but I am loathe to depend on it. We now depend on GitHub for ExtensionDistributor. I don't think it's such a bad thing. External services can be a nasty trap when they have the only copy of your data and it's difficult to get it out, but for a git mirror, it's no big deal. We can always make our own non-jgit mirror of our Git repository if GitHub stops existing. -- Tim Starling ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Nightly shallow clones of mediawiki/core
On 03/10/2013 06:03 PM, Bartosz Dziewoński wrote: A shallow clone certainly shouldn't be as large as a normal one. Something's borked. --depth 0 is what's broken. --depth 1 works fine. $ git clone --depth 1 https://gerrit.wikimedia.org/r/p/mediawiki/core.git core-shallow Cloning into 'core-shallow'... remote: Counting objects: 2815, done remote: Finding sources: 100% (2815/2815) remote: Getting sizes: 100% (2665/2665) remote: Compressing objects: 61% (1631/2660) remote: Total 2815 (delta 317), reused 1182 (delta 147) Receiving objects: 100% (2815/2815), 17.87 MiB | 1.16 MiB/s, done. Resolving deltas: 100% (342/342), done. -- Wikipedia user PleaseStand http://en.wikipedia.org/wiki/User:PleaseStand ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Nightly shallow clones of mediawiki/core
On 10/03/13 22:39, Chad wrote: Hi, I've been thinking about this for the last week or so because it's becoming incredibly clear to me that core isn't scaling. It's already taking up over 4GB on the Gerrit box, and this is the primary reason core operations are slow. 4GB?? My not specially wellpacked clone takes 191M (and that includes all the changes references), plus other 74 MB for the checkout. Core would have to be read-only for about an hour or two. Having core read-only occasionally should not be a problem with jenkins collaboration. Instead of merging make it say this change has been approved and is waiting for the epoch for merging, and have it actually merge everything when it reopens. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Nightly shallow clones of mediawiki/core
On 03/10/2013 05:33 PM, Yuri Astrakhan wrote: * install phpmyadmin sqlite PHPMyAdmin also has major security issues. It isn't allowed on Wikimedia Labs and probably shouldn't be used here. Why does SQLite need to be installed exactly? * DEBUG == that's a big one, setting it up for easy nobrainer debugging would be a great help. Including CLI -- i'm having hard time debugging unit tests. PHP Unit tests, or QUnit? Matt Flaschen ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Nightly shallow clones of mediawiki/core
On 03/10/2013 06:43 PM, Platonides wrote: On 10/03/13 22:39, Chad wrote: Hi, I've been thinking about this for the last week or so because it's becoming incredibly clear to me that core isn't scaling. It's already taking up over 4GB on the Gerrit box, and this is the primary reason core operations are slow. 4GB?? My not specially wellpacked clone takes 191M (and that includes all the changes references), plus other 74 MB for the checkout. Yeah, and one of my git dirs is 812 MB, including a clone of core *and* many extension submodules. I don't know what that 4 GB includes. Either it's hardly packed at all, or it includes a lot more than a regular clone (all non-merged changes to core?). Matt Flaschen ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Nightly shallow clones of mediawiki/core
On 03/10/2013 06:03 PM, Bartosz Dziewoński wrote: -We can rewrite history (git-filter-branch) to remove some mistakes that exploded the repo size. Binaries later removed, things accidentally checked into ./extensions, etc. This could potentially greatly reduce object sizes and allow for tighter repacks. Major issue with history rewriting is everyone will have to reclone (all sha1s would change). I've not tested my theory yet. Ugh, the nuclear option. Let's try repacking first, can we? :) This is probably going to make gerrit a little unhappy when it's database thing gets inconsistent with the repo. Yeah, I agree we should be extremely reluctant to resort to break git history on a project as important as MW core. It would cause a lot of disruption. Matt Flaschen ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Nightly shallow clones of mediawiki/core
Answered Inline. Also, I apologize as I think my email was slightly off-topic to Ori's question. On Sun, Mar 10, 2013 at 6:57 PM, Matthew Flaschen mflasc...@wikimedia.orgwrote: PHPMyAdmin also has major security issues. It isn't allowed on Wikimedia Labs and probably shouldn't be used here. Why does SQLite need to be installed exactly? Matthew, we are talking about a developer's virtual machine that has no network connection to anything except the developer's machine itself, and used purely for development. MySql could be accessed through the network if I have the local tools, but if we are talking about the lowest barrier of entry, the novice could follow a few steps to get VM and edit code with anything including notepad, while the VM would have most of the tools to examine and experiment with MediaWiki. BTW, samba would also be a massive security hole - I set mine up to share / with no password root access (no network, no issues) SQLite is needed because the dev server test with SQLite, and I personally got burnt by their difference (unit tests passed in mysql but not in sqlite due to a bug in the testing framework). This unit testhttps://gerrit.wikimedia.org/r/#/c/48098/ still fails. * DEBUG == that's a big one, setting it up for easy nobrainer debugging would be a great help. Including CLI -- i'm having hard time debugging unit tests. PHP Unit tests, or QUnit? I think both would be good (I only used phpunit so far). ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Nightly shallow clones of mediawiki/core
On 03/10/2013 07:17 PM, Yuri Astrakhan wrote: Matthew, we are talking about a developer's virtual machine that has no network connection to anything except the developer's machine itself, and used purely for development. As you said yourself, it's currently bridged. Doesn't that mean it is in fact accessible to the whole LAN, until that's changed? PHP Unit tests, or QUnit? I think both would be good (I only used phpunit so far). QUnit can be debugged easily at http://127.0.0.1:8080/wiki/Special:JavaScriptTest . Matt Flaschen ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Nightly shallow clones of mediawiki/core
(anonymous) wrote: [...] A couple of things I can think of to help the situation. - We can repack core on manganese. This should provide a bit of relief, but won't help long term. Core would have to be read-only for about an hour or two. [...] Is this read-only to prevent the hiccup caused by the overeager garbage collection some months ago? Tim ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Nightly shallow clones of mediawiki/core
On Sun, Mar 10, 2013 at 7:56 PM, Matthew Flaschen mflasc...@wikimedia.orgwrote: As you said yourself, it's currently bridged. Doesn't that mean it is in fact accessible to the whole LAN, until that's changed? Of course - that's why I put the hostonly as the first requirement, and was the first thing I changed locally. QUnit can be debugged easily at http://127.0.0.1:8080/wiki/Special:JavaScriptTest . Thanks, good to know! Still, it's the phpunit that I have had issues with - the debugger would not reliably attach, it wouldn't break anywhere except on entry point, etc. But I think this should be a separate discussion. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
