Re: [Numpy-discussion] Move scipy.org docs to Github?
Hi, On Wed, Mar 15, 2017 at 4:20 PM, Pauli Virtanen wrote: > Wed, 15 Mar 2017 14:56:52 -0700, Nathaniel Smith kirjoitti: > [clip] >> As long as we can fit under the 1 gig size limit then GH pages seems >> like the best option so far... it's reliable, widely understood, and all >> of the limits besides the 1 gig size are soft limits where they say >> they'll work with us to figure things out. > > The Scipy html docs weigh 60M apiece and numpy is 35M, so it can be done > if a limited number of releases are kept, and the rest and the auxiliary > files are put as release downloads. > > Otherwise there's probably no problem as you can stick a CDN in front if > it's too heavy otherwise. > > Sounds sensible? Certainly it's the lowest-effort approach, and would > simplify management of S3/etc origin site access permissions. Sounds very sensible to me, Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Move scipy.org docs to Github?
Wed, 15 Mar 2017 14:56:52 -0700, Nathaniel Smith kirjoitti: [clip] > As long as we can fit under the 1 gig size limit then GH pages seems > like the best option so far... it's reliable, widely understood, and all > of the limits besides the 1 gig size are soft limits where they say > they'll work with us to figure things out. The Scipy html docs weigh 60M apiece and numpy is 35M, so it can be done if a limited number of releases are kept, and the rest and the auxiliary files are put as release downloads. Otherwise there's probably no problem as you can stick a CDN in front if it's too heavy otherwise. Sounds sensible? Certainly it's the lowest-effort approach, and would simplify management of S3/etc origin site access permissions. Pauli ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Move scipy.org docs to Github?
On Mar 15, 2017 05:47, "Ralf Gommers" wrote: On Wed, Mar 15, 2017 at 3:21 PM, Matthew Brett wrote: > Hi, > > The scipy.org site is down at the moment, and has been for more than 36 > hours: > > https://github.com/numpy/numpy/issues/8779#issue-213781439 > > This has happened before: > > https://github.com/scipy/scipy.org/issues/187#issue-186426408 > > I think it was down for about 24 hours that time. > > From the number of people opening issues or commenting on the > scipy.org website this time, it seems to be causing quite a bit of > disruption. > > It seems to me that we would have a much better chances of avoiding > significant down-time, if we switched to hosting the docs on github > pages. > > What do y'all think? > Once the site is back up we should look at migrating to a better (hosted) infrastructure. I suspect that Github Pages won't work, we'll exceed or be close to exceeding both the 1 GB site size limit and the 100 GB/month bandwidth limit [1]. Rough bandwidth estimate (using page size from http://scipy.github.io/ devdocs/ and Alexa stats): 2 million visits per month, 2.5 page views per visit, 5 kb/page = 25 GB/month (html). Add to that pdf docs, which are ~20 MB in size: if only a small fraction of visitors download those, we'll be at >100 GB. Ralf [1] https://help.github.com/articles/what-is-github-pages/#usage-limits Would it be possible to include the PDF and zipped HTML as sources in GitHub releases and (if desired) link to those on the documentation page? That way only the individual HTML pages would need to be stored in GitHub pages, reducing both the size and bandwidth. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Move scipy.org docs to Github?
On Wed, Mar 15, 2017 at 6:16 AM, Bryan Van de ven wrote: > NumPy is a NumFocus fiscally sponsored project, perhaps they can help with > the costs of different/better hosting. Enthought already provides hosting and operations support (thanks!) – the problem is that it doesn't make sense to have a full-time ops person just for numpy, but if we're taking a tiny slice of someone's time then occasional outages are going to happen. The advantage of something like github pages is that it's big enough that it *does* have dedicated ops support. As long as we can fit under the 1 gig size limit then GH pages seems like the best option so far... it's reliable, widely understood, and all of the limits besides the 1 gig size are soft limits where they say they'll work with us to figure things out. -n -- Nathaniel J. Smith -- https://vorpus.org ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Move scipy.org docs to Github?
On Mar 15, 2017 12:28 PM, "Pauli Virtanen" wrote: Wed, 15 Mar 2017 12:11:09 -0400, Marten van Kerkwijk kirjoitti: > Astropy uses readthedocs quite happily (auto-updates on merges to master > too). AFAIK, scipy cannot be built on readthedocs. Another issue is that switching to rtd would (I think?) force us into their URL structure and break all incoming links, which would be really bad. -n ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Move scipy.org docs to Github?
Wed, 15 Mar 2017 12:11:09 -0400, Marten van Kerkwijk kirjoitti: > Astropy uses readthedocs quite happily (auto-updates on merges to master > too). AFAIK, scipy cannot be built on readthedocs. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Move scipy.org docs to Github?
Astropy uses readthedocs quite happily (auto-updates on merges to master too). -- Marten ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Move scipy.org docs to Github?
What about readthedocs? I haven't seen any explicit limit in traffic. On 15 March 2017 at 16:33, Matthew Brett wrote: > Hi, > > On Wed, Mar 15, 2017 at 2:47 AM, Ralf Gommers wrote: >> >> >> On Wed, Mar 15, 2017 at 3:21 PM, Matthew Brett >> wrote: >>> >>> Hi, >>> >>> The scipy.org site is down at the moment, and has been for more than 36 >>> hours: >>> >>> https://github.com/numpy/numpy/issues/8779#issue-213781439 >>> >>> This has happened before: >>> >>> https://github.com/scipy/scipy.org/issues/187#issue-186426408 >>> >>> I think it was down for about 24 hours that time. >>> >>> From the number of people opening issues or commenting on the >>> scipy.org website this time, it seems to be causing quite a bit of >>> disruption. >>> >>> It seems to me that we would have a much better chances of avoiding >>> significant down-time, if we switched to hosting the docs on github >>> pages. >>> >>> What do y'all think? >> >> >> Once the site is back up we should look at migrating to a better (hosted) >> infrastructure. I suspect that Github Pages won't work, we'll exceed or be >> close to exceeding both the 1 GB site size limit and the 100 GB/month >> bandwidth limit [1]. >> >> Rough bandwidth estimate (using page size from >> http://scipy.github.io/devdocs/ and Alexa stats): 2 million visits per >> month, 2.5 page views per visit, 5 kb/page = 25 GB/month (html). Add to that >> pdf docs, which are ~20 MB in size: if only a small fraction of visitors >> download those, we'll be at >100 GB. > > Maybe we could host the PDF docs somewhere else? I wonder if Github > would consider allowing us to go a bit over if necessary? > > Cheers, > > Matthew > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Move scipy.org docs to Github?
Hi, On Wed, Mar 15, 2017 at 2:47 AM, Ralf Gommers wrote: > > > On Wed, Mar 15, 2017 at 3:21 PM, Matthew Brett > wrote: >> >> Hi, >> >> The scipy.org site is down at the moment, and has been for more than 36 >> hours: >> >> https://github.com/numpy/numpy/issues/8779#issue-213781439 >> >> This has happened before: >> >> https://github.com/scipy/scipy.org/issues/187#issue-186426408 >> >> I think it was down for about 24 hours that time. >> >> From the number of people opening issues or commenting on the >> scipy.org website this time, it seems to be causing quite a bit of >> disruption. >> >> It seems to me that we would have a much better chances of avoiding >> significant down-time, if we switched to hosting the docs on github >> pages. >> >> What do y'all think? > > > Once the site is back up we should look at migrating to a better (hosted) > infrastructure. I suspect that Github Pages won't work, we'll exceed or be > close to exceeding both the 1 GB site size limit and the 100 GB/month > bandwidth limit [1]. > > Rough bandwidth estimate (using page size from > http://scipy.github.io/devdocs/ and Alexa stats): 2 million visits per > month, 2.5 page views per visit, 5 kb/page = 25 GB/month (html). Add to that > pdf docs, which are ~20 MB in size: if only a small fraction of visitors > download those, we'll be at >100 GB. Maybe we could host the PDF docs somewhere else? I wonder if Github would consider allowing us to go a bit over if necessary? Cheers, Matthew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Move scipy.org docs to Github?
How is it that scipy.org's bandwidth usage is that much greater than matplotlib.org's? We are quite image-heavy, but we haven't hit any bandwidth limits that I am aware of. Ben Root On Wed, Mar 15, 2017 at 9:16 AM, Bryan Van de ven wrote: > NumPy is a NumFocus fiscally sponsored project, perhaps they can help with > the costs of different/better hosting. > > Bryan > > > > On Mar 15, 2017, at 07:24, Nathaniel Smith wrote: > > > > On Mar 15, 2017 02:47, "Ralf Gommers" wrote: > > > > > > On Wed, Mar 15, 2017 at 3:21 PM, Matthew Brett > wrote: > > Hi, > > > > The scipy.org site is down at the moment, and has been for more than 36 > hours: > > > > https://github.com/numpy/numpy/issues/8779#issue-213781439 > > > > This has happened before: > > > > https://github.com/scipy/scipy.org/issues/187#issue-186426408 > > > > I think it was down for about 24 hours that time. > > > > From the number of people opening issues or commenting on the > > scipy.org website this time, it seems to be causing quite a bit of > > disruption. > > > > It seems to me that we would have a much better chances of avoiding > > significant down-time, if we switched to hosting the docs on github > > pages. > > > > What do y'all think? > > > > Once the site is back up we should look at migrating to a better > (hosted) infrastructure. I suspect that Github Pages won't work, we'll > exceed or be close to exceeding both the 1 GB site size limit and the 100 > GB/month bandwidth limit [1]. > > > > Rough bandwidth estimate (using page size from http://scipy.github.io/ > devdocs/ and Alexa stats): 2 million visits per month, 2.5 page views per > visit, 5 kb/page = 25 GB/month (html). Add to that pdf docs, which are ~20 > MB in size: if only a small fraction of visitors download those, we'll be > at >100 GB. > > > > No matter where we go, we can likely reduce the endpoint bandwidth > requirements substantially by putting something like cloudflare's free tier > in front. That doesn't help for the actual disk size though of course... > > > > -n > > ___ > > NumPy-Discussion mailing list > > NumPy-Discussion@scipy.org > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Move scipy.org docs to Github?
NumPy is a NumFocus fiscally sponsored project, perhaps they can help with the costs of different/better hosting. Bryan > On Mar 15, 2017, at 07:24, Nathaniel Smith wrote: > > On Mar 15, 2017 02:47, "Ralf Gommers" wrote: > > > On Wed, Mar 15, 2017 at 3:21 PM, Matthew Brett > wrote: > Hi, > > The scipy.org site is down at the moment, and has been for more than 36 hours: > > https://github.com/numpy/numpy/issues/8779#issue-213781439 > > This has happened before: > > https://github.com/scipy/scipy.org/issues/187#issue-186426408 > > I think it was down for about 24 hours that time. > > From the number of people opening issues or commenting on the > scipy.org website this time, it seems to be causing quite a bit of > disruption. > > It seems to me that we would have a much better chances of avoiding > significant down-time, if we switched to hosting the docs on github > pages. > > What do y'all think? > > Once the site is back up we should look at migrating to a better (hosted) > infrastructure. I suspect that Github Pages won't work, we'll exceed or be > close to exceeding both the 1 GB site size limit and the 100 GB/month > bandwidth limit [1]. > > Rough bandwidth estimate (using page size from > http://scipy.github.io/devdocs/ and Alexa stats): 2 million visits per month, > 2.5 page views per visit, 5 kb/page = 25 GB/month (html). Add to that pdf > docs, which are ~20 MB in size: if only a small fraction of visitors download > those, we'll be at >100 GB. > > No matter where we go, we can likely reduce the endpoint bandwidth > requirements substantially by putting something like cloudflare's free tier > in front. That doesn't help for the actual disk size though of course... > > -n > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Move scipy.org docs to Github?
Doesn't help all the CI builds that are failing because they utilize intersphinx to link to the official docs, unfortunately. Bryan > On Mar 15, 2017, at 05:59, Ilhan Polat wrote: > > By the way, this is not bad at all in the absence of the actual documentation > http://devdocs.io/numpy~1.12/ > > On Wed, Mar 15, 2017 at 11:55 AM, Ilhan Polat wrote: > In the meantime maybe it's a good idea to keep one of the issues open so that > people can see that this is an open issue? As we close them they disappear > from the issues tab on Github > > On Wed, Mar 15, 2017 at 10:47 AM, Ralf Gommers wrote: > > > On Wed, Mar 15, 2017 at 3:21 PM, Matthew Brett > wrote: > Hi, > > The scipy.org site is down at the moment, and has been for more than 36 hours: > > https://github.com/numpy/numpy/issues/8779#issue-213781439 > > This has happened before: > > https://github.com/scipy/scipy.org/issues/187#issue-186426408 > > I think it was down for about 24 hours that time. > > From the number of people opening issues or commenting on the > scipy.org website this time, it seems to be causing quite a bit of > disruption. > > It seems to me that we would have a much better chances of avoiding > significant down-time, if we switched to hosting the docs on github > pages. > > What do y'all think? > > Once the site is back up we should look at migrating to a better (hosted) > infrastructure. I suspect that Github Pages won't work, we'll exceed or be > close to exceeding both the 1 GB site size limit and the 100 GB/month > bandwidth limit [1]. > > Rough bandwidth estimate (using page size from > http://scipy.github.io/devdocs/ and Alexa stats): 2 million visits per month, > 2.5 page views per visit, 5 kb/page = 25 GB/month (html). Add to that pdf > docs, which are ~20 MB in size: if only a small fraction of visitors download > those, we'll be at >100 GB. > > Ralf > > [1] https://help.github.com/articles/what-is-github-pages/#usage-limits > > > > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Move scipy.org docs to Github?
On Mar 15, 2017 02:47, "Ralf Gommers" wrote: On Wed, Mar 15, 2017 at 3:21 PM, Matthew Brett wrote: > Hi, > > The scipy.org site is down at the moment, and has been for more than 36 > hours: > > https://github.com/numpy/numpy/issues/8779#issue-213781439 > > This has happened before: > > https://github.com/scipy/scipy.org/issues/187#issue-186426408 > > I think it was down for about 24 hours that time. > > From the number of people opening issues or commenting on the > scipy.org website this time, it seems to be causing quite a bit of > disruption. > > It seems to me that we would have a much better chances of avoiding > significant down-time, if we switched to hosting the docs on github > pages. > > What do y'all think? > Once the site is back up we should look at migrating to a better (hosted) infrastructure. I suspect that Github Pages won't work, we'll exceed or be close to exceeding both the 1 GB site size limit and the 100 GB/month bandwidth limit [1]. Rough bandwidth estimate (using page size from http://scipy.github.io/ devdocs/ and Alexa stats): 2 million visits per month, 2.5 page views per visit, 5 kb/page = 25 GB/month (html). Add to that pdf docs, which are ~20 MB in size: if only a small fraction of visitors download those, we'll be at >100 GB. No matter where we go, we can likely reduce the endpoint bandwidth requirements substantially by putting something like cloudflare's free tier in front. That doesn't help for the actual disk size though of course... -n ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Move scipy.org docs to Github?
By the way, this is not bad at all in the absence of the actual documentation http://devdocs.io/numpy~1.12/ On Wed, Mar 15, 2017 at 11:55 AM, Ilhan Polat wrote: > In the meantime maybe it's a good idea to keep one of the issues open so > that people can see that this is an open issue? As we close them they > disappear from the issues tab on Github > > On Wed, Mar 15, 2017 at 10:47 AM, Ralf Gommers > wrote: > >> >> >> On Wed, Mar 15, 2017 at 3:21 PM, Matthew Brett >> wrote: >> >>> Hi, >>> >>> The scipy.org site is down at the moment, and has been for more than 36 >>> hours: >>> >>> https://github.com/numpy/numpy/issues/8779#issue-213781439 >>> >>> This has happened before: >>> >>> https://github.com/scipy/scipy.org/issues/187#issue-186426408 >>> >>> I think it was down for about 24 hours that time. >>> >>> From the number of people opening issues or commenting on the >>> scipy.org website this time, it seems to be causing quite a bit of >>> disruption. >>> >>> It seems to me that we would have a much better chances of avoiding >>> significant down-time, if we switched to hosting the docs on github >>> pages. >>> >>> What do y'all think? >>> >> >> Once the site is back up we should look at migrating to a better (hosted) >> infrastructure. I suspect that Github Pages won't work, we'll exceed or be >> close to exceeding both the 1 GB site size limit and the 100 GB/month >> bandwidth limit [1]. >> >> Rough bandwidth estimate (using page size from >> http://scipy.github.io/devdocs/ and Alexa stats): 2 million visits per >> month, 2.5 page views per visit, 5 kb/page = 25 GB/month (html). Add to >> that pdf docs, which are ~20 MB in size: if only a small fraction of >> visitors download those, we'll be at >100 GB. >> >> Ralf >> >> [1] https://help.github.com/articles/what-is-github-pages/#usage-limits >> >> >> >> >> ___ >> NumPy-Discussion mailing list >> NumPy-Discussion@scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Move scipy.org docs to Github?
In the meantime maybe it's a good idea to keep one of the issues open so that people can see that this is an open issue? As we close them they disappear from the issues tab on Github On Wed, Mar 15, 2017 at 10:47 AM, Ralf Gommers wrote: > > > On Wed, Mar 15, 2017 at 3:21 PM, Matthew Brett > wrote: > >> Hi, >> >> The scipy.org site is down at the moment, and has been for more than 36 >> hours: >> >> https://github.com/numpy/numpy/issues/8779#issue-213781439 >> >> This has happened before: >> >> https://github.com/scipy/scipy.org/issues/187#issue-186426408 >> >> I think it was down for about 24 hours that time. >> >> From the number of people opening issues or commenting on the >> scipy.org website this time, it seems to be causing quite a bit of >> disruption. >> >> It seems to me that we would have a much better chances of avoiding >> significant down-time, if we switched to hosting the docs on github >> pages. >> >> What do y'all think? >> > > Once the site is back up we should look at migrating to a better (hosted) > infrastructure. I suspect that Github Pages won't work, we'll exceed or be > close to exceeding both the 1 GB site size limit and the 100 GB/month > bandwidth limit [1]. > > Rough bandwidth estimate (using page size from http://scipy.github.io/ > devdocs/ and Alexa stats): 2 million visits per month, 2.5 page views per > visit, 5 kb/page = 25 GB/month (html). Add to that pdf docs, which are ~20 > MB in size: if only a small fraction of visitors download those, we'll be > at >100 GB. > > Ralf > > [1] https://help.github.com/articles/what-is-github-pages/#usage-limits > > > > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Move scipy.org docs to Github?
On Wed, Mar 15, 2017 at 3:21 PM, Matthew Brett wrote: > Hi, > > The scipy.org site is down at the moment, and has been for more than 36 > hours: > > https://github.com/numpy/numpy/issues/8779#issue-213781439 > > This has happened before: > > https://github.com/scipy/scipy.org/issues/187#issue-186426408 > > I think it was down for about 24 hours that time. > > From the number of people opening issues or commenting on the > scipy.org website this time, it seems to be causing quite a bit of > disruption. > > It seems to me that we would have a much better chances of avoiding > significant down-time, if we switched to hosting the docs on github > pages. > > What do y'all think? > Once the site is back up we should look at migrating to a better (hosted) infrastructure. I suspect that Github Pages won't work, we'll exceed or be close to exceeding both the 1 GB site size limit and the 100 GB/month bandwidth limit [1]. Rough bandwidth estimate (using page size from http://scipy.github.io/devdocs/ and Alexa stats): 2 million visits per month, 2.5 page views per visit, 5 kb/page = 25 GB/month (html). Add to that pdf docs, which are ~20 MB in size: if only a small fraction of visitors download those, we'll be at >100 GB. Ralf [1] https://help.github.com/articles/what-is-github-pages/#usage-limits ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
