Are weekly releases an option? It was brought up as concern that we might lose pip as a pretty common distribution channel where people consume nightly builds. I don't feel like that concern has been properly addressed so far.
-Marco Lausen, Leonard <lau...@amazon.com.invalid> schrieb am Mi., 4. Dez. 2019, 04:09: > As a simple POC to test distribution, you can try installing MXNet based on > these 3 URLs: > > pip install --no-cache-dir > > https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl > pip install --no-cache-dir > > https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl > pip install --no-cache-dir https://d19zq12jzu4w95.cloudfront.net/ > mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl > <https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl> > > where --no-cache-dir prevents caching the downloaded file, for the purpose > of > testing. (cu101 chosen based on large size) > > The first URL uses standard S3 bucket in US. The second uses S3 Accelerate > based > on CloudFront CDN. And the third uses CloudFront CDN. I'm adding the third > URL, > as S3 Accelerate may or may not use all new CloudFront endpoints yet. > > Regarding voting: Uploading to Pypi is currently impossible, which is a > reality > (so there is no option to continue as we do currently). Pypi folks > indicated > they will unblock our uploads to Pypi once we stop uploading nightly > releases > and taking up 20% of their ressources [1]. > > If there are any shortcomings or problems identified with uploading to S3, > we > can work to address them. But for now, status quo is broken and this seems > the > only solution addressing Pypi's problem. > > I don't mind if you state that you object to lazy consensus and start a > vote. If > your "maybe [...] start a proper vote" was supposed to be an objection to > lazy > consensus, please state so clearly (I'm not sure if "maybe" qualifies as > objection). Though I think it only makes sense with at least 2 options to > vote > on. Status quo is not a meaningful option, as it is already broken. > > Best regards > Leonard > > [1]: https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706 > > On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote: > > Excellent! Could we maybe come up with a POC and a quick writeup and then > > start a proper vote after everyone verified that it covers their > use-cases? > > > > -Marco > > > > Sheng Zha <zhash...@apache.org> schrieb am Di., 3. Dez. 2019, 19:24: > > > > > Yes, there is. We can also make it easier to access by using a > > > geo-location based DNS server so that China users are directed to that > > > local mirror. The rest of the world is already covered by the global > > > cloudfront. > > > > > > -sz > > > > > > On 2019/12/03 18:22:22, Marco de Abreu <marco.g.ab...@gmail.com> > wrote: > > > > Isn't there an s3 endpoint in Beijing? > > > > > > > > It seems like this topic still warrants some discussion and thus I'd > > > > > > prefer > > > > if we don't move forward with lazy consensus. > > > > > > > > -Marco > > > > > > > > Tao Lv <mutou...@gmail.com> schrieb am Di., 3. Dez. 2019, 14:31: > > > > > > > > > * For pypi, we can use mirrors. > > > > > > > > > > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mutou...@gmail.com> wrote: > > > > > > > > > > > As we have many users in China, I'm considering the > accessibility of > > > > > > S3. > > > > > > For pip, we can mirrors. > > > > > > > > > > > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard > > > > > > <lau...@amazon.com.invalid > > > > > > > > > > > > wrote: > > > > > > > > > > > > > I would like to remind everyone that lazy consensus is assumed > if no > > > > > > > objections > > > > > > > are raised before 2019-12-05 at 05:42 UTC. There has been some > > > > > > > > > > discussion > > > > > > > about > > > > > > > the proposal, but to my understanding no objections were > raised. > > > > > > > > > > > > > > If the proposal is accepted, MXNet releases would be installed > via > > > > > > > > > > > > > > pip install mxnet > > > > > > > > > > > > > > And release candidates via > > > > > > > > > > > > > > pip install --pre mxnet > > > > > > > > > > > > > > (or with the respective cuda version specifier appended etc.) > > > > > > > > > > > > > > To obtain releases built automatically from the master branch, > users > > > > > > > would need > > > > > > > to specify something like "-f > > > > > > > http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html"; option to > pip. > > > > > > > > > > > > > > Best regards > > > > > > > Leonard > > > > > > > > > > > > > > On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote: > > > > > > > > Hi MXNet Community, > > > > > > > > > > > > > > > > since more than 2 months our binary Python nightly releases > > > > > > published > > > > > > > on Pypi > > > > > > > > are broken. The problem is that our binaries exceed Pypi's > size > > > > > > limit. > > > > > > > > Decreasing the binary size by adding compression breaks > > > > > > third-party > > > > > > > libraries > > > > > > > > loading libmxnet.so > > > > > > > > > > > > > > https://github.com/apache/incubator-mxnet/issues/16193 > > > > > > > > > > > > > > > > Sheng requested Pypi to increase their size limit: > > > > > > > > https://github.com/pypa/pypi-support/issues/50 > > > > > > > > > > > > > > > > Currently "the biggest cost for PyPI from [the many MXNet > binaries > > > > > > > > > > with > > > > > > > > nightly > > > > > > > > release to Pypi] is the bandwidth consumed when several > hundred > > > > > > > > > > mirrors > > > > > > > > attempt > > > > > > > > to mirror each release immediately after it's published". So > Pypi > > > > > > is > > > > > not > > > > > > > > inclined to allow us to upload even larger binaries on a > nightly > > > > > > > > > > > > > > schedule. > > > > > > > > Their compromise is to allow it on a weekly cadence. > > > > > > > > > > > > > > > > However, I would like the community to revisit the necessity > of > > > > > > > > > > > > > > releasing pre- > > > > > > > > release binaries to Pypi on a nightly (or weekly) cadence. > > > > > > Instead, we > > > > > > > can > > > > > > > > release nightly binaries ONLY to a public S3 bucket and > instruct > > > > > > users > > > > > > > to > > > > > > > > install from there. On our side, we only need to prepare a > html > > > > > > > > > > > > > > document that > > > > > > > > contains links to all released nightly binaries. > > > > > > > > Finally users will install the nightly releases via > > > > > > > > > > > > > > > > pip install --pre mxnet-cu101 -f > > > > > > > > > > > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/ > > > > > > > > nightly.html > > > > > > > > > > > > > > > > Instead of > > > > > > > > > > > > > > > > pip install --pre mxnet-cu101 > > > > > > > > > > > > > > > > Of course proper releases and release candidates should > still be > > > > > > made > > > > > > > > available > > > > > > > > via Pypi. Thus releases would be installed via > > > > > > > > > > > > > > > > pip install mxnet-cu101 > > > > > > > > > > > > > > > > And release candidates via > > > > > > > > > > > > > > > > pip install --pre mxnet-cu101 > > > > > > > > > > > > > > > > This will substantially reduce the costs of the Pypi project > and > > > > > > in > > > > > fact > > > > > > > > matches > > > > > > > > the installation experience provided by PyTorch. I don't > think the > > > > > > > > > > > > > > benefit of > > > > > > > > not including "-f > > > > > > > > > > > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"; > > > > > > > > matches the costs we currently externalize to the Pypi team. > > > > > > > > > > > > > > > > This suggestion seems uncontroversial to me. Thus I would > like to > > > > > > > > > > start > > > > > > > lazy > > > > > > > > consensus. If there are no objections, I will assume lazy > > > > > > consensus on > > > > > > > > stopping > > > > > > > > nightly releases to Pypi in 72hrs. > > > > > > > > > > > > > > > > Best regards > > > > > > > > Leonard >
