Re: Spring cleaning: Reducing Number Footprint of HG Repos
On 27/03/14 00:53, Taras Glek wrote: *User Repos* TLDR: I would like to make user repos read-only by April 30th. We should archive them by May 31st. I think that if you truly intend to go ahead with this, the news will need way, way wider circulation than mozilla.dev.platform. I have some useful software stored in a user repo, and I only happen to read this group. It will also need much more lead time than a month. I'm also somewhat surprised that this has been proposed without any previous attempt to measure the impact of doing it. Or has such work been done, but the results not published? How often are all these repos pulled from or pushed to? Could we achieve many of the gains by getting people to clean up after themselves, rather than eliminating the capability entirely? I don't think you're suggesting this, but just to be clear: I'm against storing our repo of record for anything of long-term importance on any system other than our own. And yes, I know about B2G. Gerv ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Spring cleaning: Reducing Number Footprint of HG Repos
Also, if you are using a COW filesystem, initial clones should be nearly free and you'd only pay the extra copy cost for changesets added afterwards. This could help dramatically with mozilla-central clones. Out of curiosity, is there open source software for a shared Git object store? git. git also has a wide array of interesting backends(eg swift) to choose from, etc. It's slightly less painful to host than hg. Yet I still don't see a compelling reason to roll our own poor imitation of github/bitbucket. re busted self-serve deletes in another email: https://bugzilla.mozilla.org/show_bug.cgi?id=983085 Taras ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Graceful Platform Degradation
Jet Villegas: I've asked Cameron McCormack to look into how Firefox and other browsers should behave when under mild to severe stress. As all browser engines have to manage how to run under low memory, feeble network, pegged CPU, weak GPU, low battery, small/slow screens, etc., I think web authors should know what to expect when their content is run on high-end machines and wristwatches. The hypothesis is that web authors will be frustrated by multiple browsers modifying their content for performance reasons without consistent/documented fallback scenarios. Cameron is initially focused on Web Rendering but we may want to specify what happens in other parts of the Platform as well, including the conditions that will trigger the graceful degradation of content. Some features that may have value as normative specs for all browsers: reduced or no antialiasing reduced color depth image downsampling image resizing/culling animation throttling (frame-rate reduction or frame dropping) reduced font/text feature usage audio/video bit-rate throttling content purging (how?) definition of browser stress levels (device profiles?) I haven’t thought about the topic in much depth yet, so here are a bunch of questions for everyone. First I think we need to answer the question of what such functionality would gain us. Two aspects come to mind: 1. Someone has a slow machine and visits a complex page that causes responsiveness to be poor. 2. An author wants to write a page that performs well on a range of devices with different performance characteristics. For different browsers running on the same machine, some given content might perform poorly or well, depending on how features are implemented. In (1) we might want to reduce quality/correctness to make the experience better for the user. If the user finds that content doesn’t work performantly in Firefox, we might want to trade off quality/ correctness so that they don’t want to switch to another browser where certain Web features are more performant. (Alternatively we might just want to concentrate on closing the performance gap in that case. :-)) But if we do back off on quality/correctness, is this something that should happen in a predictable way? If we detect that a page is having performance problems (and exactly what that means might be something to dive into, but assume for a moment we’re just looking at whether we’re able to paint in time for the next frame) and that it is due, say, too many box-shadows and SVG filters being used, should we have a defined order of e.g. reducing SVG filter resolution, turning filters off, then turning box-shadows off? How coarse or fine grained should the change to these features be? How standardised should this process be? Jet’s comment about authors being frustrated that things degrade differently on different browsers sounds right, or even more likely, that they won’t test on different browsers resulting in pages being unusable in the untested browsers. One other worry I would have is that the user might place different value on quality/correctness and performance than we might. If they are someone with an old computer who is used to things being slow, and we make it so that pages degrade their use of complex features so that performance becomes acceptable, might they prefer the original slow experience? PC games allow the user to turn certain features (mostly graphics related ones) on and off so that they can find their own level of acceptable performance/quality. This doesn’t seem like the right approach for viewing Web content. For (2), we might want to have a mechanism for the author to declare whether a given feature is “important”, or to give a priority order to features that the browser is allowed to reduce or turn off when it feels it needs to. One suggestion that was made at a recent SVG WG meeting where I brought this topic up was to have a fixed, small number of performance levels (high, medium, low) that the browser would decide it is in based on how well the page is running. The page could then declare (with media queries?) what to do in a given performance level, with the idea that you’d turn off your box-shadows in low performance mode for example. How many of these problems would go away anyway if we had APZC, e10s, etc.? There are a lot of open questions here so I’m interested in hearing others’ thoughts. -- Cameron McCormack ≝ http://mcc.id.au/ ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Spring cleaning: Reducing Number Footprint of HG Repos
Want to move to github? (0) sudo apt-get install python-setuptools (1) sudo easy_install hg-git (2) add |hggit =| under [extensions] in your .hgrc file (3) Go to GitHub.com and create your new repo. (4) cd hg_repo (5) hg bookmark -r default master (6) hg push git+ssh://g...@github.com/you/name of your repo you created in step 3 On Wednesday, March 26, 2014, Taras Glek tg...@mozilla.com wrote: *User Repos* TLDR: I would like to make user repos read-only by April 30th. We should archive them by May 31st. Time spent operating user repositories could be spent reducing our end-to-end continuous integration cycles. These do not seem like mission-critical repos, seems like developers would be better off hosting these on bitbucket or github. Using a 3rd-party host has obvious benefits for collaboration self-service that our existing system will never meet. We are happy to help move specific hg repos to bitbucket. Once you have migrated your repository, please comment in https://bugzilla.mozilla.org/show_bug.cgi?id=988628so we can free some disk space. *Non-User Repos* There are too many non-user repos. I'm not convinced we should host ash, oak, other project branches internally. I think we should focus on mission-critical repos only. There should be less than a dozen of those. I would like to stop hosting non-mission-critical repositories by end of Q2. This is a soft target. I don't have a concrete plan here. I'd like to start experimenting with moving project branches elsewhere and see where that takes us. *What my hg repo needs X/Y that 3rd-party services do not provide?* If you have a good reason to use a feature not supported by github/bitbucket, we should continue hosting your repo at Mozilla. *Why Not Move Everything to Github/Bitbucket/etc?* Mozilla prefers to keep repositories public by-default. This does not fit Github's business model which is built around private repos. Github's free service does not provide any availability guarantee. There is also a problem of github not supporting hg. I'm not completely sure why we can't move everything to bitbucket. Some of it is to do with anecdotal evidence of robustness problems. Some of it is lack of hooks (sans post-receive POSTs).Additionally, as with Github there is no availability guarantee. Hosting arbitrary Moz-related hg repositories does not make strategic sense. We should do the absolute minimum(eg http://bke.ro/?p=380) required to keep Firefox shipping smoothly and focus our efforts on making Firefox better. Taras ps. Footprint stats: *Largest User Repos Out Of ~130GB* 1.1Gdmt.alexandre_gmail.com 1.1Gjblandy_mozilla.com 1.1Gjparsons_mozilla.com 1.2Gbugzilla_standard8.plus.com 1.2Gmbrubeck_mozilla.com 1.2Gmrbkap_mozilla.com 1.3Gdcamp_campd.org 1.3Gjst_mozilla.com 1.4Gblassey_mozilla.com 1.4Ggszorc_mozilla.com 1.4Giacobcatalin_gmail.com 1.5Gcpearce_mozilla.com 1.5Ghurley_mozilla.com 1.6Gbsmedberg_mozilla.com 1.6Gdglastonbury_mozilla.com 1.6Gdtc-moz_scieneer.com 1.6Gjlund_mozilla.com 1.6Gsarentz_mozilla.com 1.6Gsbruno_mozilla.com 1.7Gmshal_mozilla.com 1.9Gmhammond_skippinet.com.au 2.1Glwagner_mozilla.com 2.4Garmenzg_mozilla.com 2.4Gdougt_mozilla.com 2.5Gbschouten_mozilla.com 2.7Ghwine_mozilla.com 2.8Geakhgari_mozilla.com 2.8Gmozilla_kewis.ch -- // mobile ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Spring cleaning: Reducing Number Footprint of HG Repos
On Wed, Mar 26, 2014 at 11:58:48PM -0700, Doug Turner wrote: Want to move to github? (0) sudo apt-get install python-setuptools (1) sudo easy_install hg-git (2) add |hggit =| under [extensions] in your .hgrc file (3) Go to GitHub.com and create your new repo. (4) cd hg_repo (5) hg bookmark -r default master (6) hg push git+ssh://g...@github.com/you/name of your repo you created in step 3 I don't know the state of github backend, but it used to be recommended to start from a fork than to push something fresh, especially when it's as massive as mozilla-central. Mike ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Graceful Platform Degradation
This sounds like a worthy and interesting idea, but also a very difficult one. PC games allow the user to turn certain features (mostly graphics related ones) on and off so that they can find their own level of acceptable performance/quality. This doesn't seem like the right approach for viewing Web content. Yeah, games are a much easier case. The content is known ahead of time (so the degradation can be carefully tested), and typically graphics dominates the hardware requirements. In a browser, the former is untrue, and the latter is often untrue -- degradation of audiovisual elements seems tractable, but what if it's JS execution that's causing the slowness? Perhaps there could be a way to annotate the HTML/JS/CSS code to indicate which parts are less important. I.e. let the page author dictate what is less important. That would facilitate testing -- a web developer with a powerful machine could turn on the browser's stress mode and get a good sense of what would change. Whether developers would bother with it, though, I don't know. Nick ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Spring cleaning: Reducing Number Footprint of HG Repos
For talos development we allow pointing at a user specific repo instead of the master one. This has greatly reduced the time to bring up new tests. This could easily be hosted elsewhere, but we chose to restrict it to user repos for a security measure. You have to have cleared some form of basic authentication with user repos and now if someone wants to see how their talos modifications run on talos they can do that without checking them in. A change like this will require us to either remove this functionality, make it less secure, or create busy work whenever someone new wants to point to a custom talos repository. -Joel ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Spring cleaning: Reducing Number Footprint of HG Repos
On 14-03-26 07:53 PM, Taras Glek wrote: *User Repos* TLDR: I would like to make user repos read-only by April 30th. We should archive them by May 31st. Time spent operating user repositories could be spent reducing our end-to-end continuous integration cycles. These do not seem like mission-critical repos, seems like developers would be better off hosting these on bitbucket or github. Using a 3rd-party host has obvious benefits for collaboration self-service that our existing system will never meet. We are happy to help move specific hg repos to bitbucket. Once you have migrated your repository, please comment in https://bugzilla.mozilla.org/show_bug.cgi?id=988628so we can free some disk space. *Non-User Repos* There are too many non-user repos. I'm not convinced we should host ash, oak, other project branches internally. I think we should focus on mission-critical repos only. There should be less than a dozen of those. I would like to stop hosting non-mission-critical repositories by end of Q2. First of all, I applaud this and it's important to get it done. However, we need to review what is used within the releng system and the security implications of using non-mozilla hosting for repos. Our infra also allows on the try server to test talos repositories under hg.m.o/users/blah. We should also get security sign-off for a different type of hosting of those repos. We're putting an etherpad together with repos important to releng systems. cheers, Armen ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Spring cleaning: Reducing Number Footprint of HG Repos
On 3/27/2014 1:11 AM, Mike Hommey wrote: On Wed, Mar 26, 2014 at 05:40:36PM -0700, Gregory Szorc wrote: On 3/26/14, 4:53 PM, Taras Glek wrote: *User Repos* TLDR: I would like to make user repos read-only by April 30th. We should archive them by May 31st. Time spent operating user repositories could be spent reducing our end-to-end continuous integration cycles. These do not seem like mission-critical repos, seems like developers would be better off hosting these on bitbucket or github. Using a 3rd-party host has obvious benefits for collaboration self-service that our existing system will never meet. How much time do we spend operating user repositories? I follow the repos bugzilla components and most of the requests I see have little if anything to do with user repositories. And I reckon that's because user repositories are self-service. Note that while user repositories are self-service on the creation side, there is no obvious way to self-service a user repo removal. I'm not in Taras's list, but after looking, I figured I had an old m-c copy with old patches on top of it. Prior to the hg migration to local disk there was (well technically still is): ssh hg.mozilla.org edit repo which allowed you to delete it. We even had/have this info on MDN. The bug exists today that the deletion does not propogate out to the local-storage webheads. ~Justin Wood (Callek) ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Spring cleaning: Reducing Number Footprint of HG Repos
On 3/26/2014 9:15 PM, Taras Glek wrote: Bobby Holley mailto:bobbyhol...@gmail.com Wednesday, March 26, 2014 17:27 I don't understand what the overhead is. We don't run CI on user repos. It's effectively just ssh:// + disk space, right? That seems totally negligible. Human overhead in keeping infra running could be spent making our infra better elsewhere. Also, project branches are pretty useful for teams working together on large projects that aren't ready to land in m-c. We only use them when we need them, so why would we shut them down? I'm not suggesting killing it. My suggestion is that project branch experience would likely be better when not hosted by mozilla. It would still trigger our c-i systems. Except when you consider the disposable project branches get Level 2 commit privs needed, and that to commit to our repos you need to have signed the committer agreement, which grants some legal recompense if malice is done. These project branches run on non try based machines which have elevated rights vs what try does, and can do much much more harm if there is malice here. I for one would not be happy from a sec standpoint if we allowed bitbucket-hosted repos to execute arbitrary code this way. ~Justin Wood (Callek) ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Spring cleaning: Reducing Number Footprint of HG Repos
On 3/27/14, 12:53 AM, Taras Glek wrote: *User Repos* TLDR: I would like to make user repos read-only by April 30th. We should archive them by May 31st. Time spent operating user repositories could be spent reducing our end-to-end continuous integration cycles. These do not seem like mission-critical repos, seems like developers would be better off hosting these on bitbucket or github. Using a 3rd-party host has obvious benefits for collaboration self-service that our existing system will never meet. We are happy to help move specific hg repos to bitbucket. Once you have migrated your repository, please comment in https://bugzilla.mozilla.org/show_bug.cgi?id=988628so we can free some disk space. I think it's utterly sad making that we're giving up on hosting, instead of fixing it. I have several things in my user repos that only run on our hg server, mostly because all other repo hoster don't send correct mimetypes for raw files. In particular this affects dashboards I created to share aggregated bugzilla data etc. I'm also sad that we're removing the ability for contributors to share their mozilla-central clones, at least in large parts of the world. Pushing a full clone to some random server just isn't working for large parts of teh world. And all that while the opportunity for us to help you on the data consumption is just broken. Sad. Note, strategically, I think that mozilla needs to support developing o the web, and the github editor isn't it. It'll be web-based IDEs, which require good tooling and hosting to be on the same infrastructure. Axel *Non-User Repos* There are too many non-user repos. I'm not convinced we should host ash, oak, other project branches internally. I think we should focus on mission-critical repos only. There should be less than a dozen of those. I would like to stop hosting non-mission-critical repositories by end of Q2. This is a soft target. I don't have a concrete plan here. I'd like to start experimenting with moving project branches elsewhere and see where that takes us. *What my hg repo needs X/Y that 3rd-party services do not provide?* If you have a good reason to use a feature not supported by github/bitbucket, we should continue hosting your repo at Mozilla. *Why Not Move Everything to Github/Bitbucket/etc?* Mozilla prefers to keep repositories public by-default. This does not fit Github's business model which is built around private repos. Github's free service does not provide any availability guarantee. There is also a problem of github not supporting hg. I'm not completely sure why we can't move everything to bitbucket. Some of it is to do with anecdotal evidence of robustness problems. Some of it is lack of hooks (sans post-receive POSTs).Additionally, as with Github there is no availability guarantee. Hosting arbitrary Moz-related hg repositories does not make strategic sense. We should do the absolute minimum(eg http://bke.ro/?p=380) required to keep Firefox shipping smoothly and focus our efforts on making Firefox better. Taras ps. Footprint stats: *Largest User Repos Out Of ~130GB* 1.1Gdmt.alexandre_gmail.com 1.1Gjblandy_mozilla.com 1.1Gjparsons_mozilla.com 1.2Gbugzilla_standard8.plus.com 1.2Gmbrubeck_mozilla.com 1.2Gmrbkap_mozilla.com 1.3Gdcamp_campd.org 1.3Gjst_mozilla.com 1.4Gblassey_mozilla.com 1.4Ggszorc_mozilla.com 1.4Giacobcatalin_gmail.com 1.5Gcpearce_mozilla.com 1.5Ghurley_mozilla.com 1.6Gbsmedberg_mozilla.com 1.6Gdglastonbury_mozilla.com 1.6Gdtc-moz_scieneer.com 1.6Gjlund_mozilla.com 1.6Gsarentz_mozilla.com 1.6Gsbruno_mozilla.com 1.7Gmshal_mozilla.com 1.9Gmhammond_skippinet.com.au 2.1Glwagner_mozilla.com 2.4Garmenzg_mozilla.com 2.4Gdougt_mozilla.com 2.5Gbschouten_mozilla.com 2.7Ghwine_mozilla.com 2.8Geakhgari_mozilla.com 2.8Gmozilla_kewis.ch 2.9Grcampbell_mozilla.com 3.1Gbhearsum_mozilla.com 3.1Grjesup_wgate.com 3.2Gagal_mozilla.com 3.3Gaxel_mozilla.com 3.3Gprepr-ffxbld 4.2Gjford_mozilla.com 4.3Gmgervasini_mozilla.com 4.6Glsblakk_mozilla.com 5.0Gbsmith_mozilla.com 5.5Gnthomas_mozilla.com 5.8Gcoop_mozilla.com 6.5Gjhopkins_mozilla.com 7.7Graliiev_mozilla.com 9.2Gcatlee_mozilla.com 13Gstage-ffxbld *Space Usage by Non-user repos ~100GB* 24K integration/gaia-1_4 28K addon-sdk 28K projects/collusion 32K integration/gaia-1_1_0 32K projects/emscripten 32K projects/Moz2D 32K releases/mozilla-b2g18_v1_1_0 144Kprojects/addon-sdk-jetperf-tests 268Kipccode 452Ktestpilot-l10n 500Kreleases/firefox-hotfixes 700Kprojects/python-nss 896Kschema-validation 1.2Mprojects/mccoy 1.4Mpyxpcom 2.4Mplatform-model 2.4M
Re: Spring cleaning: Reducing Number Footprint of HG Repos
On 3/27/2014 2:58 AM, Doug Turner wrote: Want to move to github? (0) sudo apt-get install python-setuptools (1) sudo easy_install hg-git (2) add |hggit =| under [extensions] in your .hgrc file (3) Go to GitHub.com and create your new repo. (4) cd hg_repo (5) hg bookmark -r default master (6) hg push git+ssh://g...@github.com/you/name of your repo you created in step 3 hg-git can't run without a very very custom and difficult-to-setup hg on windows. Specifically because hg uses py2exe which strips out EVERY unused python library. And even doing hg in a virtualenv is hard because you get a MUCH slower hg due to no compiled code. I have never further tested hg-git on windows after I encountered the two issues above. ~Justin Wood (Callek) ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Spring cleaning: Reducing Number Footprint of HG Repos
On 3/27/2014 1:58 AM, Doug Turner wrote: Want to move to github? (0) sudo apt-get install python-setuptools (1) sudo easy_install hg-git (2) add |hggit =| under [extensions] in your .hgrc file (3) Go to GitHub.com and create your new repo. (4) cd hg_repo (5) hg bookmark -r default master (6) hg push git+ssh://g...@github.com/you/name of your repo you created It's worth noting that hg-git is having some performance issues with github right now. A basic clone of a 1MB repository takes well over a minute before it starts doing anything. -- Joshua Cranmer Thunderbird and DXR developer Source code archæologist ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Spring cleaning: Reducing Number Footprint of HG Repos
On 14-03-26 08:27 PM, Bobby Holley wrote: I don't understand what the overhead is. We don't run CI on user repos. It's effectively just ssh:// + disk space, right? That seems totally negligible. FTR from an operations standpoint, it is never just. Never. If it was *just* we wouldn't even be having this conversation. Trust me. regards, Armen ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Spring cleaning: Reducing Number Footprint of HG Repos
On 27/03/2014 13:43, Justin Wood (Callek) wrote: On 3/27/2014 2:58 AM, Doug Turner wrote: Want to move to github? (0) sudo apt-get install python-setuptools (1) sudo easy_install hg-git (2) add |hggit =| under [extensions] in your .hgrc file (3) Go to GitHub.com and create your new repo. (4) cd hg_repo (5) hg bookmark -r default master (6) hg push git+ssh://g...@github.com/you/name of your repo you created in step 3 hg-git can't run without a very very custom and difficult-to-setup hg on windows. Specifically because hg uses py2exe which strips out EVERY unused python library. And even doing hg in a virtualenv is hard because you get a MUCH slower hg due to no compiled code. I have never further tested hg-git on windows after I encountered the two issues above. ~Justin Wood (Callek) IME tortoisehg ships a much happier-making hg (than mozilla-build) that has a bunch of python libs you want. I've never used hg-git, however, so I don't know if it has enough of what you need. ~ Gijs ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Spring cleaning: Reducing Number Footprint of HG Repos
On 27/03/14 14:17, Armen Zambrano G. wrote: On 14-03-26 08:27 PM, Bobby Holley wrote: I don't understand what the overhead is. We don't run CI on user repos. It's effectively just ssh:// + disk space, right? That seems totally negligible. FTR from an operations standpoint, it is never just. Never. If it was *just* we wouldn't even be having this conversation. Trust me. To be fair there are also considerable costs associated with outsourcing VCS hosting, mostly associated with integrating the external hosting with other systems that need to work with the repository. For example W3C's web-platform-tests testsuite is being hosted on GitHub and as a result we have spent a non-trivial amount of effort on integration with a system for ensuring contributers agree to a CLA, a code review tool, synchronization of HEAD with a web server and various other things. This might be less effort than doing all the hosting at the W3C (although the reason we did it was purely that GitHub is familiar to potential contributers), but of course it will all have to be thrown away if we want to move providers in the future. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Spring cleaning: Reducing Number Footprint of HG Repos
What are mission critical repos since you just put everything in the same list? If we start removing project branches to be put on outsourced VCS we remove any sheriff support for that project branch since, as been pointed out many times, we dont have access to the server side commit hooks and can't close the tree. This may (I want to use *want* but don't have the data to prove it) impact engineering productivity. We have this situation with Gaia which has its canonical repo on Github. Sheriffs can land checkin-needed but can't close the tree. The way the B2G people do it is to remove everyone from the repo and then re-add (or thats how they used to do it) which then spams you with you are now getting notifications for repository X which is annoying. There is the other thing we need to worry about is the constant DDoS of Github[1]. We have seen that when there is a massive one it will take down their site for hours impacting engineering productivity again since people can't pull or push. I couldn't find similar reports on bitbucket but it can happen to any third party we may use. David [1] https://github.com/blog/1796-denial-of-service-attacks On 26/03/2014 23:53, Taras Glek wrote: *User Repos* TLDR: I would like to make user repos read-only by April 30th. We should archive them by May 31st. Time spent operating user repositories could be spent reducing our end-to-end continuous integration cycles. These do not seem like mission-critical repos, seems like developers would be better off hosting these on bitbucket or github. Using a 3rd-party host has obvious benefits for collaboration self-service that our existing system will never meet. We are happy to help move specific hg repos to bitbucket. Once you have migrated your repository, please comment in https://bugzilla.mozilla.org/show_bug.cgi?id=988628so we can free some disk space. *Non-User Repos* There are too many non-user repos. I'm not convinced we should host ash, oak, other project branches internally. I think we should focus on mission-critical repos only. There should be less than a dozen of those. I would like to stop hosting non-mission-critical repositories by end of Q2. This is a soft target. I don't have a concrete plan here. I'd like to start experimenting with moving project branches elsewhere and see where that takes us. *What my hg repo needs X/Y that 3rd-party services do not provide?* If you have a good reason to use a feature not supported by github/bitbucket, we should continue hosting your repo at Mozilla. *Why Not Move Everything to Github/Bitbucket/etc?* Mozilla prefers to keep repositories public by-default. This does not fit Github's business model which is built around private repos. Github's free service does not provide any availability guarantee. There is also a problem of github not supporting hg. I'm not completely sure why we can't move everything to bitbucket. Some of it is to do with anecdotal evidence of robustness problems. Some of it is lack of hooks (sans post-receive POSTs).Additionally, as with Github there is no availability guarantee. Hosting arbitrary Moz-related hg repositories does not make strategic sense. We should do the absolute minimum(eg http://bke.ro/?p=380) required to keep Firefox shipping smoothly and focus our efforts on making Firefox better. Taras ps. Footprint stats: *Largest User Repos Out Of ~130GB* 1.1Gdmt.alexandre_gmail.com 1.1Gjblandy_mozilla.com 1.1Gjparsons_mozilla.com 1.2Gbugzilla_standard8.plus.com 1.2Gmbrubeck_mozilla.com 1.2Gmrbkap_mozilla.com 1.3Gdcamp_campd.org 1.3Gjst_mozilla.com 1.4Gblassey_mozilla.com 1.4Ggszorc_mozilla.com 1.4Giacobcatalin_gmail.com 1.5Gcpearce_mozilla.com 1.5Ghurley_mozilla.com 1.6Gbsmedberg_mozilla.com 1.6Gdglastonbury_mozilla.com 1.6Gdtc-moz_scieneer.com 1.6Gjlund_mozilla.com 1.6Gsarentz_mozilla.com 1.6Gsbruno_mozilla.com 1.7Gmshal_mozilla.com 1.9Gmhammond_skippinet.com.au 2.1Glwagner_mozilla.com 2.4Garmenzg_mozilla.com 2.4Gdougt_mozilla.com 2.5Gbschouten_mozilla.com 2.7Ghwine_mozilla.com 2.8Geakhgari_mozilla.com 2.8Gmozilla_kewis.ch 2.9Grcampbell_mozilla.com 3.1Gbhearsum_mozilla.com 3.1Grjesup_wgate.com 3.2Gagal_mozilla.com 3.3Gaxel_mozilla.com 3.3Gprepr-ffxbld 4.2Gjford_mozilla.com 4.3Gmgervasini_mozilla.com 4.6Glsblakk_mozilla.com 5.0Gbsmith_mozilla.com 5.5Gnthomas_mozilla.com 5.8Gcoop_mozilla.com 6.5Gjhopkins_mozilla.com 7.7Graliiev_mozilla.com 9.2Gcatlee_mozilla.com 13Gstage-ffxbld *Space Usage by Non-user repos ~100GB* 24K integration/gaia-1_4 28K addon-sdk 28K projects/collusion 32K integration/gaia-1_1_0
Re: Spring cleaning: Reducing Number Footprint of HG Repos
Taras Glek schrieb: *User Repos* TLDR: I would like to make user repos read-only by April 30th. When that happens, I will stop running any custom crash reports and dashboards that the stability program depends on, at least until further notice. I do not want to run a non-Mozilla-hosted repo for Mozilla work stuff. KaiRo ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Spring cleaning: Reducing Number Footprint of HG Repos
On 3/27/14, 6:37 AM, Justin Wood (Callek) wrote: On 3/26/2014 9:15 PM, Taras Glek wrote: Bobby Holley mailto:bobbyhol...@gmail.com Wednesday, March 26, 2014 17:27 I don't understand what the overhead is. We don't run CI on user repos. It's effectively just ssh:// + disk space, right? That seems totally negligible. Human overhead in keeping infra running could be spent making our infra better elsewhere. Also, project branches are pretty useful for teams working together on large projects that aren't ready to land in m-c. We only use them when we need them, so why would we shut them down? I'm not suggesting killing it. My suggestion is that project branch experience would likely be better when not hosted by mozilla. It would still trigger our c-i systems. Except when you consider the disposable project branches get Level 2 commit privs needed, and that to commit to our repos you need to have signed the committer agreement, which grants some legal recompense if malice is done. These project branches run on non try based machines which have elevated rights vs what try does, and can do much much more harm if there is malice here. I for one would not be happy from a sec standpoint if we allowed bitbucket-hosted repos to execute arbitrary code this way. The security concern should be on the scheduling front, not where the code is hosted. If a repo push incurs automation activity, we have established trust that anyone who can push to that repo can be trusted. If we don't have this automatic scheduling on push, no trust is established and there is no security concern. If a user is able to schedule automation manually (say by calling a web API), we trust the user isn't doing something nefarious. Since the scheduling API requires authentication, there shouldn't be a new security concern here. Even if there is an increased security concern over MITM or silent repo modification by 3rd party, these concerns can be mitigated through proper security settings (our Mercurial clients in automation aren't currently validating x509 fingerprints) and moving our automation jobs to execute in containers, which I believe is already in the works. That leaves us pretty much with kernel vulnerabilities (that can escape from containers), which we should be protecting ourselves against anyway. This problem is little different than what insert cloud hosting service provider here deals with. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Spring cleaning: Reducing Number Footprint of HG Repos
*User Repos* TLDR: I would like to make user repos read-only by April 30th. We should archive them by May 31st. As mentioned, too fast. Time spent operating user repositories could be spent reducing our end-to-end continuous integration cycles. If we're spending any significant time or money running these, let's solve that instead - I really don't think much time or money *should* be needed to run low-priority repos with non-mission-critical availability requirements. These do not seem like mission-critical repos, seems like developers would be better off hosting these on bitbucket or github. Using a 3rd-party host has obvious benefits for collaboration self-service that our existing system will never meet. Some issues I raised privately before this post went public, but I don't see addressed here: * Security implications Any dev who works on security bugs (and most do at one point or another, or might) who puts a patch queue on an external host is proxying to that host all security assurances. This makes that external hosting a tempting target for people who want to find 0-days. I'd like to say this is an excessive amount of paranoia, but given both the lucrative market for 0-days and NSA's interest in 0-days (and ability to compel or buy silence from companies or employees at companies), I no longer think this is excessive. :-( I'm less worried about silent changes to the repos to slip stuff in (though it's possible) than someone silently cataloging possible 0-day targets in repos associated with devs, especially ones marked as referring to bugs that aren't visible. * cleanup Per previous comments, it wasn't aware I could get rid of a user repo in any easy way (and it may actually be busted right now). Likely 50% of what's in user repos (or more) is dusty stuff that people could simply delete. I have one large and one medium repo I need to keep and some patch queues (most of which are deletable now). Anything else can go. But there's no trivial way for me to see what I have and delete them. A simple 1/month nag mail listing your private repos and their sizes would help. * side note: my repo names are tied to the email address in my key. It's dead. I'd change my key to the new permanent email address, but I worry I might lose all my user repos. * Backup/data-integrity/availability Already mentioned was availability guarantees or lack thereof. We'd need to back up these external repos (and find them somehow). Taras commented to me that we use expensive storage solutions for user repos (similar to primary repos). IMHO that's not needed: User repos needs lower SLA gear I'd imagine - redundancy, but could probably just live in a RAID-1+1 array with consumer drives with very high reliability (two RAID-1 arrays in a RAID-1 configuration) - you'd need a 3-drive simultaneous failure to have to fall back on backups. Hell, a single RAID-1 is probably good enough, so long as it's backed up frequently. Taras mentioned that this is time not spent doing other things; my response: I imagine you can buy a RAID drive and just drop it in in place of $$$-expensive-drive. But yes this requires some thought/planning/etc time for them; while moving to random-VCS-storage requires some time by N devs - net result may be more time than if we keep it inhouse. Plus time by IT to set up remote backup and negotiate something with random-VCS to let that happen. If we're dropping backup and just relying on the service, there are some additional concerns. -- Randell Jesup, Mozilla Corp remove news for personal email ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Spring cleaning: Reducing Number Footprint of HG Repos
On 03/27/2014 10:10 AM, Joshua Cranmer wrote: It's worth noting that hg-git is having some performance issues with github right now. A basic clone of a 1MB repository takes well over a minute before it starts doing anything. When I was converting my repositories last night I found that although the push to github from hg(-git) was hanging, it in fact had completed all of its work already. After a second or two you could control-C, re-push, and it would say there was nothing to do. If you checked on github, the commits would in fact be all there, and they would be there before the second push attempt or hitting control-C. Obviously, if you are pushing something huge like a clone of mozilla-central, you may need to legitimately wait a long time. But for clones of mozilla-central it's probably most advisable and polite to fork the gecko-dev repo and either do a light-weight import of any branches using git fast-import or the fancy tooling used to produce gecko-dev in the first place. A very cursory exploration shows http://repo.or.cz/w/fast-export.git provides fast-export from hg for fast-import to git, but it's probably best to read the blog-posts for the gecko-dev conversion instead. Andrew ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Spring cleaning: Reducing Number Footprint of HG Repos
On 3/27/2014 12:11 PM, Andrew Sutherland wrote: On 03/27/2014 10:10 AM, Joshua Cranmer wrote: It's worth noting that hg-git is having some performance issues with github right now. A basic clone of a 1MB repository takes well over a minute before it starts doing anything. When I was converting my repositories last night I found that although the push to github from hg(-git) was hanging, it in fact had completed all of its work already. After a second or two you could control-C, re-push, and it would say there was nothing to do. If you checked on github, the commits would in fact be all there, and they would be there before the second push attempt or hitting control-C. I saw that too, but the clone/pull is a different error, reported here: https://bitbucket.org/durin42/hg-git/issue/90/stuck-clone-over-git-ssh-to-bitbucketorg. Note that the hang happens here well before the work is done. -- Joshua Cranmer Thunderbird and DXR developer Source code archæologist ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Graceful Platform Degradation
For different browsers running on the same machine, some given content might perform poorly or well, depending on how features are implemented. In (1) we might want to reduce quality/correctness to make the experience better for the user. If the user finds that content doesn’t work performantly in Firefox, we might want to trade off quality/ correctness so that they don’t want to switch to another browser where certain Web features are more performant. (Alternatively we might just want to concentrate on closing the performance gap in that case. :-)) For TenFourFox, I've often toyed with implementing switches for box-shadow, blur, etc., so that people on the very low end of the spec (we still support G3 Macintoshes) can turn these rather expensive features off. I'd rather do that in a web-standard way than a one-off local change, though. Since none of our supported systems are WebGL-capable, I'm mostly interested in graceful degradation of expensive CSS, such as reducing animation smoothness or interpolation, or browser-generated effects like blur etc. However, I personally doubt that a web author will implement this, reasoning that it should be at the taste of the user. Maybe suggestions for an automated means would be nice, but a determined user will want to strip things out in just the way they would like, and I think they should have the opportunity to do so. There's nothing that says this has to be one or the other. Cameron Kaiser ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
