Hi Leonard, Is there any reason why we shouldn't take both options? Ie we do weekly build on PyPi and provide the s3 option. I would be inclined to make sure we provide as many avenues as possible to reduce friction for developers. The d2l.ai book by Alex Smola is attracting a community that so far has been relying on the nightly builds in advance of stable.
Sincerely, Alex Chung ________________________________________ From: Lausen, Leonard <lau...@amazon.com.INVALID> Sent: Sunday, December 1, 2019 9:42 PM To: d...@mxnet.apache.org Subject: Stopping nightly releases to Pypi Hi MXNet Community, since more than 2 months our binary Python nightly releases published on Pypi are broken. The problem is that our binaries exceed Pypi's size limit. Decreasing the binary size by adding compression breaks third-party libraries loading libmxnet.so https://github.com/apache/incubator-mxnet/issues/16193 Sheng requested Pypi to increase their size limit: https://github.com/pypa/pypi-support/issues/50 Currently "the biggest cost for PyPI from [the many MXNet binaries with nightly release to Pypi] is the bandwidth consumed when several hundred mirrors attempt to mirror each release immediately after it's published". So Pypi is not inclined to allow us to upload even larger binaries on a nightly schedule. Their compromise is to allow it on a weekly cadence. However, I would like the community to revisit the necessity of releasing pre- release binaries to Pypi on a nightly (or weekly) cadence. Instead, we can release nightly binaries ONLY to a public S3 bucket and instruct users to install from there. On our side, we only need to prepare a html document that contains links to all released nightly binaries. Finally users will install the nightly releases via pip install --pre mxnet-cu101 -f http://mxnet.s3.amazonaws.com/mxnet-cu101/ nightly.html Instead of pip install --pre mxnet-cu101 Of course proper releases and release candidates should still be made available via Pypi. Thus releases would be installed via pip install mxnet-cu101 And release candidates via pip install --pre mxnet-cu101 This will substantially reduce the costs of the Pypi project and in fact matches the installation experience provided by PyTorch. I don't think the benefit of not including "-f http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"; matches the costs we currently externalize to the Pypi team. This suggestion seems uncontroversial to me. Thus I would like to start lazy consensus. If there are no objections, I will assume lazy consensus on stopping nightly releases to Pypi in 72hrs. Best regards Leonard
