Re: [Rubygems-developers] Gemcutter
On Aug 26, 2009, at 21:08 , Nick Quaranto wrote: Just a note here: I haven't added support for the legacy indexes for in Gemcutter. Since you need at least RubyGems 1.3.3 to use the gem plugins, I figured supporting the legacy ones shouldn't be a priority as the project started. Currently I'm only generating and serving up the modern indexes (and these are undergoing some serious refactoring and rework internally as well). Agreed. Cut and run. ___ Rubygems-developers mailing list http://rubyforge.org/projects/rubygems Rubygems-developers@rubyforge.org http://rubyforge.org/mailman/listinfo/rubygems-developers
[Rubygems-developers] Gemcutter
For the past few months I've been working on Gemcutter (http://gemcutter.org), which aims to deliver a fresh take on gem hosting for the Ruby community. The purpose of the project is three-fold: * Create more transparent and accessible project pages * Enable the community to improve and enhance the site * Provide a better API for dealing with gems Gemcutter is now up and running on Heroku, and the gem serving is done entirely through S3. Patches are beginning to roll in from contributors, and a redesign is in progress at http://staging.gemcutter.org thanks to some designer time donated from Thoughtbot. I would like to emphasize that I've worked hard to make this an open, honest effort towards improving a core part of the Ruby world that's easily accessible to every Ruby developer. The overall goal of the project is to become the default gem host for RubyGems. However, this can't happen without the support of the RubyGems and RubyForge team. Here's what I propose: 1) Redirect gems.rubyforge.org to gemcutter.org for gem serving. (all gems are currently mirrored from RubyForge and are ready for consumption) 2) Make gemcutter.org the default gem host for the next major release of RubyGems. 3) Keep RubyForge online since many projects depend on its bug trackers, mailing lists, and more. Please note I'm not suggesting that all of these happen *this instant*, I just want to open up discussion about making this happen and what would be involved. Your thoughts and comments would be appreciated. -Nick Quaranto ___ Rubygems-developers mailing list http://rubyforge.org/projects/rubygems Rubygems-developers@rubyforge.org http://rubyforge.org/mailman/listinfo/rubygems-developers
Re: [Rubygems-developers] Gemcutter
Nick, On Aug 26, 2009, at 1:42 PM, Nick Quaranto wrote: 1) Redirect gems.rubyforge.org to gemcutter.org for gem serving. (all gems are currently mirrored from RubyForge and are ready for consumption) RubyForge currently has a pretty good mirror system for supporting Gem downloads. I expect Tom could get your some figures on how much bandwidth they use. If Gemcutter was the default RG source, how would you deal with this? Are you planning on eating S3 costs yourself? Are you thinking about using something like CloudFront, since S3 itself isn't strictly a CDN? ~ j. ___ Rubygems-developers mailing list http://rubyforge.org/projects/rubygems Rubygems-developers@rubyforge.org http://rubyforge.org/mailman/listinfo/rubygems-developers
Re: [Rubygems-developers] Gemcutter
On Wed, Aug 26, 2009 at 1:42 PM, Nick Quaranton...@quaran.to wrote: Please note I'm not suggesting that all of these happen *this instant*, I just want to open up discussion about making this happen and what would be involved. Your thoughts and comments would be appreciated. To be honest, this seems Not Ready For Prime Time: * Pagination/sorting is wrong and broken * Project pages contain obviously broken/incorrect links (version and homepage links) * Project pages seem to have much less information than I have on RubyForge. No release notes, individual file links (.gem, .tar, etc), etc... (broken and less != transparent and accessible) * I uploaded a gem to RubyForge yesterday just before midnight, it is still not mirrored sixteen hours later (and it was available almost immediately on Rubyforge, which shows their mirror process is working very well) That's just what I found in one minute of poking around, it doesn't give me a good feeling for the overall quality and stability of the effort. I'm also concerned about long-term support. No offense to you, but Tom Copeland is a machine, he is always there to support Rubyforge, year after year... In general, if it Ain't Broken, Don't Fix It - and think especially hard if the replacement isn't a no-brainer out-of-the-park homerun... Is there a reason we can't get many of these benefits from RubyForge.org? For example, your API seems like a good idea, but why can't it be a wrapper or enhancement to the current RubyGems/RubyForge ecosystem? A Gem plugin, even? Finally, I don't see why there needs to be a change to the canonical source, since Rubyforge isn't even really one anymore. That ship has already sailed, many people don't even publish gems to RubyForge, they just use Github or their own servers. If anything there needs to be better support for multiple sources (There are good ideas here not yet implemented). The multiple-naming problem is a bigger but related issue as Eric discussed (http://blog.segment7.net/articles/2009/02/04/a-rubygems-github-proposal)... -- Chad ___ Rubygems-developers mailing list http://rubyforge.org/projects/rubygems Rubygems-developers@rubyforge.org http://rubyforge.org/mailman/listinfo/rubygems-developers
Re: [Rubygems-developers] Gemcutter
On Thu, Aug 27, 2009 at 1:24 AM, Ryan Davisryand-r...@zenspider.com wrote: On Aug 26, 2009, at 14:57 , John Barnette wrote: Nick, On Aug 26, 2009, at 1:42 PM, Nick Quaranto wrote: 1) Redirect gems.rubyforge.org to gemcutter.org for gem serving. (all gems are currently mirrored from RubyForge and are ready for consumption) RubyForge currently has a pretty good mirror system for supporting Gem downloads. I expect Tom could get your some figures on how much bandwidth they use. If Gemcutter was the default RG source, how would you deal with this? Are you planning on eating S3 costs yourself? Are you thinking about using something like CloudFront, since S3 itself isn't strictly a CDN? (possibly) even better, I'm pretty sure we could get Tom convinced that gemcutter is a good idea and we should migrate rubyforge to it. The thing is that RubyForge is more than just gems. Take as example the installers for Ruby for Windows: http://rubyforge.org/frs/?group_id=167release_id=38052 I believe that GemCutter API and interface has potential, but as was previously mentioned, the support and cost of backing it up raises some concerns. Taking in consideration there are lot of gems with more than 10K of downloads, the incurring charge of running that form S3 and adding CloudFront CDN on top will need to be payed my someone. -- Luis Lavena AREA 17 - Perfection in design is achieved not when there is nothing more to add, but rather when there is nothing more to take away. Antoine de Saint-Exupéry ___ Rubygems-developers mailing list http://rubyforge.org/projects/rubygems Rubygems-developers@rubyforge.org http://rubyforge.org/mailman/listinfo/rubygems-developers
Re: [Rubygems-developers] Gemcutter
Chad, I'll admit, the website as it is right now sucks, this is not a finished project yet. There's a full fledged redesign in progress that is being pushed out to: http://staging.gemcutter.org Check out the preview post here too: http://robots.thoughtbot.com/post/165832471/gemcutter-org-redesign-preview I'd also like to pull in all of the READMEs for each gem and show them at the bottom of the gem page and get them indexed for searching. The API is already being used by the gem plugins provided by the gemcutter gem...that's how `gem push` works for instance. -Nick On Wed, Aug 26, 2009 at 7:55 PM, Chad Woolley thewoolley...@gmail.comwrote: On Wed, Aug 26, 2009 at 1:42 PM, Nick Quaranton...@quaran.to wrote: Please note I'm not suggesting that all of these happen *this instant*, I just want to open up discussion about making this happen and what would be involved. Your thoughts and comments would be appreciated. To be honest, this seems Not Ready For Prime Time: * Pagination/sorting is wrong and broken * Project pages contain obviously broken/incorrect links (version and homepage links) * Project pages seem to have much less information than I have on RubyForge. No release notes, individual file links (.gem, .tar, etc), etc... (broken and less != transparent and accessible) * I uploaded a gem to RubyForge yesterday just before midnight, it is still not mirrored sixteen hours later (and it was available almost immediately on Rubyforge, which shows their mirror process is working very well) That's just what I found in one minute of poking around, it doesn't give me a good feeling for the overall quality and stability of the effort. I'm also concerned about long-term support. No offense to you, but Tom Copeland is a machine, he is always there to support Rubyforge, year after year... In general, if it Ain't Broken, Don't Fix It - and think especially hard if the replacement isn't a no-brainer out-of-the-park homerun... Is there a reason we can't get many of these benefits from RubyForge.org? For example, your API seems like a good idea, but why can't it be a wrapper or enhancement to the current RubyGems/RubyForge ecosystem? A Gem plugin, even? Finally, I don't see why there needs to be a change to the canonical source, since Rubyforge isn't even really one anymore. That ship has already sailed, many people don't even publish gems to RubyForge, they just use Github or their own servers. If anything there needs to be better support for multiple sources (There are good ideas here not yet implemented). The multiple-naming problem is a bigger but related issue as Eric discussed (http://blog.segment7.net/articles/2009/02/04/a-rubygems-github-proposal).http://blog.segment7.net/articles/2009/02/04/a-rubygems-github-proposal%29. .. -- Chad ___ Rubygems-developers mailing list http://rubyforge.org/projects/rubygems Rubygems-developers@rubyforge.orghttp://rubyforge.org/projects/rubygems%0arubygems-develop...@rubyforge.org http://rubyforge.org/mailman/listinfo/rubygems-developers ___ Rubygems-developers mailing list http://rubyforge.org/projects/rubygems Rubygems-developers@rubyforge.org http://rubyforge.org/mailman/listinfo/rubygems-developers
Re: [Rubygems-developers] Gemcutter
On Thu, Aug 27, 2009 at 12:21 AM, Nick Quaranton...@quaran.to wrote: I'll take a look into CloudFront, I just haven't yet since the need hasn't come up. My main concern is keeping the gap between publishing a gem and being able to install it low. CloudFront is a transparent CDN, you only need to subscribe to the service and enable it for the S3 bucket you're using. Then, change the URL to instead of using S3 bucket directly uses the CNAME of the CloudFront enabled bucket. old url: http://my-bucket.s3.amazonaws.com/path/to/my/gem.gem new url: http://assets.gemcutter.org/path/to/my/gem.gem Or something similar. I would love to get an estimate on bandwidth since the costs are easy to calculate for S3. So far with over 2,000 gem downloads and uploading all of the gems from RubyForge over the past 2 months it's been less than $5. Taking in consideration RubyForge has lot of mirrors, collecting real life numbers for it would be really great. -- Luis Lavena AREA 17 - Perfection in design is achieved not when there is nothing more to add, but rather when there is nothing more to take away. Antoine de Saint-Exupéry ___ Rubygems-developers mailing list http://rubyforge.org/projects/rubygems Rubygems-developers@rubyforge.org http://rubyforge.org/mailman/listinfo/rubygems-developers
Re: [Rubygems-developers] Gemcutter
On Aug 26, 2009, at 17:16 , Luis Lavena wrote: (possibly) even better, I'm pretty sure we could get Tom convinced that gemcutter is a good idea and we should migrate rubyforge to it. The thing is that RubyForge is more than just gems. I wasn't suggesting that gemcutter take over rubyforge, but that rubyforge migrate to using gemcutter for gem services. With a little bit of work, it wouldn't be hard to do. Taking in consideration there are lot of gems with more than 10K of downloads, the incurring charge of running that form S3 and adding CloudFront CDN on top will need to be payed my someone. There is no incurred charge if S3 is not involved. That is one benefit of having it brought under the wing of ruby central / rubyforge. ___ Rubygems-developers mailing list http://rubyforge.org/projects/rubygems Rubygems-developers@rubyforge.org http://rubyforge.org/mailman/listinfo/rubygems-developers
Re: [Rubygems-developers] Gemcutter
On Aug 26, 2009, at 5:57 PM, John Barnette wrote: Nick, On Aug 26, 2009, at 1:42 PM, Nick Quaranto wrote: 1) Redirect gems.rubyforge.org to gemcutter.org for gem serving. (all gems are currently mirrored from RubyForge and are ready for consumption) RubyForge currently has a pretty good mirror system for supporting Gem downloads. I expect Tom could get your some figures on how much bandwidth they use. I just ran webalizer on the Apache log from the Bytemark mirror and it did 14M gems of gem traffic on Monday. Figure Bytemark serves about a quarter of the traffic, that's about 60M/day. File downloads eat a little more bandwidth - that Bytemark mirror did 49M yesterday of files... it's serving about 60% of the traffic, so files account for 80M/day, maybe. And of course there are spikes here and there. Yours, Tom ___ Rubygems-developers mailing list http://rubyforge.org/projects/rubygems Rubygems-developers@rubyforge.org http://rubyforge.org/mailman/listinfo/rubygems-developers
Re: [Rubygems-developers] Gemcutter
On Aug 26, 2009, at 6:21 PM, Nick Quaranto wrote: As for the costs, I'd like to approach a non-profit organization such as RubyCentral to help deal with them. I originally sent an email out to them first about supporting the project a few weeks ago, but I haven't heard anything back. I'm interested to hear their opinions on the site as well. It would definitely be good to get Chad or Rich or David to weigh in since they're on the board of Ruby Central, and Ruby Central rides herd on RubyForge. Yours, tom ___ Rubygems-developers mailing list http://rubyforge.org/projects/rubygems Rubygems-developers@rubyforge.org http://rubyforge.org/mailman/listinfo/rubygems-developers
Re: [Rubygems-developers] Gemcutter
On Aug 26, 2009, at 9:22 PM, Tom Copeland wrote: On Aug 26, 2009, at 5:57 PM, John Barnette wrote: Nick, On Aug 26, 2009, at 1:42 PM, Nick Quaranto wrote: 1) Redirect gems.rubyforge.org to gemcutter.org for gem serving. (all gems are currently mirrored from RubyForge and are ready for consumption) RubyForge currently has a pretty good mirror system for supporting Gem downloads. I expect Tom could get your some figures on how much bandwidth they use. I just ran webalizer on the Apache log from the Bytemark mirror and it did 14M gems of gem traffic on Monday. Figure Bytemark serves about a quarter of the traffic, that's about 60M/day. File downloads eat a little more bandwidth - that Bytemark mirror did 49M yesterday of files... it's serving about 60% of the traffic, so files account for 80M/day, maybe. And of course there are spikes here and there. Oh, also, space-wise there are 6GB of gems and 29G of files. Yours, tom ___ Rubygems-developers mailing list http://rubyforge.org/projects/rubygems Rubygems-developers@rubyforge.org http://rubyforge.org/mailman/listinfo/rubygems-developers
Re: [Rubygems-developers] Gemcutter
So I'm trying out: http://calculator.s3.amazonaws.com/calc5.html 14mb/day, is less than 1GB a month. For S3 that's less than $1/month just for data transfer out, and a little more with CloudFront. It doesn't seem like this would be a problem unless if we were in the range of terabytes/month. As for hosting files on Gemcutter, I'm open to that if that's what is necessary for the transition. 80 mb/day is still a trivial amount for S3 or CloudFront to host, cost-wise. If I interpreted your findings wrong Tom, let me know. -Nick On Wed, Aug 26, 2009 at 9:22 PM, Tom Copeland t...@infoether.com wrote: On Aug 26, 2009, at 5:57 PM, John Barnette wrote: Nick, On Aug 26, 2009, at 1:42 PM, Nick Quaranto wrote: 1) Redirect gems.rubyforge.org to gemcutter.org for gem serving. (all gems are currently mirrored from RubyForge and are ready for consumption) RubyForge currently has a pretty good mirror system for supporting Gem downloads. I expect Tom could get your some figures on how much bandwidth they use. I just ran webalizer on the Apache log from the Bytemark mirror and it did 14M gems of gem traffic on Monday. Figure Bytemark serves about a quarter of the traffic, that's about 60M/day. File downloads eat a little more bandwidth - that Bytemark mirror did 49M yesterday of files... it's serving about 60% of the traffic, so files account for 80M/day, maybe. And of course there are spikes here and there. Yours, Tom ___ Rubygems-developers mailing list http://rubyforge.org/projects/rubygems Rubygems-developers@rubyforge.org http://rubyforge.org/mailman/listinfo/rubygems-developers ___ Rubygems-developers mailing list http://rubyforge.org/projects/rubygems Rubygems-developers@rubyforge.org http://rubyforge.org/mailman/listinfo/rubygems-developers
Re: [Rubygems-developers] Gemcutter
Just a note here: I haven't added support for the legacy indexes for in Gemcutter. Since you need at least RubyGems 1.3.3 to use the gem plugins, I figured supporting the legacy ones shouldn't be a priority as the project started. Currently I'm only generating and serving up the modern indexes (and these are undergoing some serious refactoring and rework internally as well). It would be interesting to watch the traffic of the legacy indexes vs the modern indexes to see if it's even worth it to support those going forward. At the very least, we could drop the legacy indexes into S3 and not update them. -Nick On Wed, Aug 26, 2009 at 11:59 PM, Tom Copeland t...@infoether.com wrote: On Aug 26, 2009, at 11:12 PM, Tom Copeland wrote: On Aug 26, 2009, at 10:27 PM, Nick Quaranto wrote: So I'm trying out: http://calculator.s3.amazonaws.com/calc5.html 14mb/day, is less than 1GB a month. For S3 that's less than $1/month just for data transfer out, and a little more with CloudFront. It doesn't seem like this would be a problem unless if we were in the range of terabytes/month. As for hosting files on Gemcutter, I'm open to that if that's what is necessary for the transition. 80 mb/day is still a trivial amount for S3 or CloudFront to host, cost-wise. Oh, one more thing - serving up the gem indexes (yaml, Marshal.4.8, Marshal.4.8.Z, etc) also accounts for 100 MB or so a day. The yaml file is about 60MB of that. Anyhow, it's still a pretty small amount S3-wise... Yours, tom ___ Rubygems-developers mailing list http://rubyforge.org/projects/rubygems Rubygems-developers@rubyforge.org http://rubyforge.org/mailman/listinfo/rubygems-developers ___ Rubygems-developers mailing list http://rubyforge.org/projects/rubygems Rubygems-developers@rubyforge.org http://rubyforge.org/mailman/listinfo/rubygems-developers
Re: [Rubygems-developers] Gemcutter
On Aug 27, 2009, at 12:08 AM, Nick Quaranto wrote: Just a note here: I haven't added support for the legacy indexes for in Gemcutter. Since you need at least RubyGems 1.3.3 to use the gem plugins, I figured supporting the legacy ones shouldn't be a priority as the project started. Currently I'm only generating and serving up the modern indexes (and these are undergoing some serious refactoring and rework internally as well). It would be interesting to watch the traffic of the legacy indexes vs the modern indexes to see if it's even worth it to support those going forward. At the very least, we could drop the legacy indexes into S3 and not update them. Here's the full stats for gems.rubyforge.org (not the mirrors, just the main domain) for Wednesday: http://infoether.com/~tom/gems.rubyforge.org.logs/usage_200908.html 6K hits on the old-timey yaml index. We only rebuild it once a day... I should put it up on S3 and do a rewrite rule for just that one file... Yours, Tom ___ Rubygems-developers mailing list http://rubyforge.org/projects/rubygems Rubygems-developers@rubyforge.org http://rubyforge.org/mailman/listinfo/rubygems-developers
