Agree, we should add the selection in the installation page for nightly build.
https://mxnet.apache.org/get_started?version=master&platform=linux&language=python&environ=pip&processor=cpu# > -----Original Message----- > From: Haibin Lin <haibin.lin....@gmail.com> > Sent: Tuesday, December 17, 2019 2:40 PM > To: dev@mxnet.incubator.apache.org > Cc: d...@mxnet.apache.org > Subject: Re: Stopping nightly releases to Pypi > > Shall we update the website installation page with nightly build information > as well (after we figure out the CD details)? > > Best, > Haibin > > On Tue, Dec 10, 2019 at 10:15 PM Lausen, Leonard > <lau...@amazon.com.invalid> > wrote: > > > Not yet. As a community, we first need to add the nightly build > > hosting feature to the community run CD and then we can add the page > > so that the exact date doesn't need to be specified. > > > > I'm not sure what steps are required for this. Do we need to host the > > artifacts on Apache's infrastructure? Or can we host the nightly CD > > artifacts as part of the AWS sponsored community-maintained CD (S3 > > bucket associated to the account)? > > > > In the meantime, the "proprietary" AWS build solution could be > > extended to publish an html page per artifact type (mxnet, > > mxnet-cu100, ...) containing a link to all recent builds. > > > > Best regards > > Leonard > > > > On Tue, 2019-12-10 at 22:03 -0800, Lin Yuan wrote: > > > Is there a way to install the latest nightly package without having > > > to specify exact date? > > > > > > Thanks, > > > > > > Lin > > > > > > On Sun, Dec 8, 2019 at 6:13 PM Lausen, Leonard > > > <lau...@amazon.com.invalid > > > > > > wrote: > > > > > > > From Shanghai, the closest endpoint (automatically chosen > > > > endpoint) is > > in > > > > Tokyo > > > > and download speed for mxnet-mkl was on average 1.7 MB/s with a > > maximum of > > > > 5 > > > > MB/s during my test. > > > > > > > > On Sun, 2019-12-08 at 01:30 +0000, Sheng Zha wrote: > > > > > > Heres a set of links for today’s builds > > > > > > > > > > > > (Plain mxnet, no mkl no cuda) > > > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12- > 07/dist/m > > xnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > (mxnet-mkl) > > > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12- > 07/dist/m > > xnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > (mxnet-cuXXX) > > > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12- > 07/dist/m > > xnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12- > 07/dist/m > > xnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12- > 07/dist/m > > xnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12- > 07/dist/m > > xnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > (mxnet-cuXXXmkl) > > > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12- > 07/dist/m > > xnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12- > 07/dist/m > > xnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12- > 07/dist/m > > xnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12- > 07/dist/m > > xnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > These links are not utilizing the s3 accelerate feature (i.e. > > > > > not > > backed > > > > by > > > > > cloudfront edges). Please use repo.mxnet.io instead. The updated > > links > > > > are: > > > > > (Plain mxnet, no mkl no cuda) > > > > > > > > > > > https://repo.mxnet.io/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py > > 3-none-manylinux1_x86_64.whl > > > > > (mxnet-mkl) > > > > > > > > > > > https://repo.mxnet.io/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py > > 2.py3-none-manylinux1_x86_64.whl > > > > > (mxnet-cuXXX) > > > > > > > > > > > https://repo.mxnet.io/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-p > > y2.py3-none-manylinux1_x86_64.whl > > > > > > https://repo.mxnet.io/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-p > > y2.py3-none-manylinux1_x86_64.whl > > > > > > https://repo.mxnet.io/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207- > > py2.py3-none-manylinux1_x86_64.whl > > > > > > https://repo.mxnet.io/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207- > > py2.py3-none-manylinux1_x86_64.whl > > > > > (mxnet-cuXXXmkl) > > > > > > > > > > > https://repo.mxnet.io/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b2019120 > > 7-py2.py3-none-manylinux1_x86_64.whl > > > > > > https://repo.mxnet.io/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b2019120 > > 7-py2.py3-none-manylinux1_x86_64.whl > > > > > > https://repo.mxnet.io/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b201912 > > 07-py2.py3-none-manylinux1_x86_64.whl > > > > > > https://repo.mxnet.io/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b201912 > > 07-py2.py3-none-manylinux1_x86_64.whl > > > > > When updating the installation doc we should use repo.mxnet.io > > domain > > > > name > > > > > too. > > > > > > > > > > Best, > > > > > -sz > > > > > > > > > > On 2019/12/07 17:39:40, "Skalicky, Sam" > > > > > <sska...@amazon.com.INVALID> > > > > wrote: > > > > > > Hi MXNet Community, > > > > > > > > > > > > We have been working on getting nightly builds fixed and made > > available > > > > > > again. We’ve made another system using AWS CodeBuild & S3 to > > > > > > work > > > > around the > > > > > > problems with Jenkins CI, PyPI, etc. It is currently building > > > > > > all > > the > > > > > > flavors and publishing to an S3 bucket here: > > > > > > > > > > > > https://us-west-2.console.aws.amazon.com/s3/buckets/apache- > mxnet/dist/ > > ?region=us-west-2 > > > > > > There are folders for each set of nightly builds, try out the > > wheels > > > > > > starting today 2019-12-07. Builds start at 1:30am PT (9:30am > > > > > > GMT) > > and > > > > arrive > > > > > > in the bucket 30min-2hours later. Inside each folder are the > > > > > > wheels > > > > for each > > > > > > flavor of MXNet. Currently we’re only building for linux, > > > > > > builds > > for > > > > > > windows/Mac will come later. > > > > > > > > > > > > If you want to download the wheels easily you can use a URL in > > > > > > the > > > > form of: > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/ > > > > <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3- > none- > > > > manylinux1_x86_64.whl > > > > > > Heres a set of links for today’s builds > > > > > > > > > > > > (Plain mxnet, no mkl no cuda) > > > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12- > 07/dist/m > > xnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > (mxnet-mkl) > > > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12- > 07/dist/m > > xnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > (mxnet-cuXXX) > > > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12- > 07/dist/m > > xnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12- > 07/dist/m > > xnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12- > 07/dist/m > > xnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12- > 07/dist/m > > xnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > (mxnet-cuXXXmkl) > > > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12- > 07/dist/m > > xnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12- > 07/dist/m > > xnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12- > 07/dist/m > > xnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12- > 07/dist/m > > xnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > You can easily install these pip wheels in your system either > > > > > > by > > > > downloading > > > > > > them to your machine first and then installing by doing: > > > > > > > > > > > > pip install /path/to/downloaded/wheel.whl > > > > > > > > > > > > Or you can install directly by just giving the link to pip > > > > > > like > > this: > > > > > > > > > > > > pip install > > > > > > > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12- > 07/dist/m > > xnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > > > Credit goes to everyone involved (in no particular order) > > > > > > Rakesh Vasudevan Zach Kimberg Manu Seth Sheng Zha Jun Wu > Pedro > > > > > > Larroy Chaitanya Bapat > > > > > > > > > > > > Thanks! > > > > > > Sam > > > > > > > > > > > > > > > > > > On Dec 5, 2019, at 1:16 AM, Lausen, Leonard > > <lau...@amazon.com.INVALID > > > > <mailt > > > > > > o:lau...@amazon.com.INVALID>> wrote: > > > > > > > > > > > > We don't loose pip by hosting on S3. We just don't host > > > > > > nightly > > > > releases on > > > > > > Pypi > > > > > > servers and mirror them to several hundred mirrors immediately > > after > > > > each > > > > > > build > > > > > > is published which is very expensive for the Pypi project.. > > > > > > People > > can > > > > still > > > > > > install the nightly builds with pip by specifying the -f option. > > > > > > > > > > > > Uploading weekly releases to Pypi will reduce the cost for > > > > > > Pypi by > > > > ~75% [1]. > > > > > > It > > > > > > may be acceptable to Pypi, but does it make sense for us? I'm > > > > > > not > > > > convinced > > > > > > weekly release on Pypi is a good idea. Consider one release is > > buggy, > > > > users > > > > > > will > > > > > > need to wait for 7 days for a fix. It doesn't provide good > > > > > > user > > > > experience. > > > > > > If someone has a stronger conviction about the value of weekly > > > > releases on > > > > > > Pypi, > > > > > > that person shall please go ahead and propose it in a separate > > > > discussion > > > > > > thread. > > > > > > > > > > > > Currently we don't have generally working nightly builds to > > > > > > Pypi > > and > > > > as a > > > > > > matter > > > > > > of fact we know that we can't have them due to Pypi's policy > > > > > > and > > our > > > > > > apparent > > > > > > need for large binaries. Given this fact and that no objection > > > > > > was > > > > raised by > > > > > > 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on > > stopping > > > > > > upload > > > > > > attempts of nightly builds to Pypi. > > > > > > > > > > > > With consensus established, we can change the CI job to stop > > trying to > > > > > > upload > > > > > > the nightly builds and then request Pypi to increase the limit. > > Then > > > > we have > > > > > > one > > > > > > less blocker for the 1.6 release. > > > > > > > > > > > > Best regards > > > > > > Leonard > > > > > > > > > > > > [1]: Lower cost due to less releases, but higher cost due to > > > > > > 500MB > > -> > > > > 800MB > > > > > > limit increase. Assuming that the limit increase translates > > > > > > into > > > > actually > > > > > > larger > > > > > > binaries. > > > > > > > > > > > > > > > > > > On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote: > > > > > > Are weekly releases an option? It was brought up as concern > > > > > > that we > > > > might > > > > > > lose pip as a pretty common distribution channel where people > > consume > > > > > > nightly builds. I don't feel like that concern has been > > > > > > properly > > > > addressed > > > > > > so far. > > > > > > > > > > > > -Marco > > > > > > > > > > > > Lausen, Leonard <lau...@amazon.com.invalid<mailto: > > > > lau...@amazon.com.invalid> > > > > > > > schrieb am Mi., 4. Dez. 2019, > > > > > > 04:09: > > > > > > > > > > > > As a simple POC to test distribution, you can try installing > > > > > > MXNet > > > > based on > > > > > > these 3 URLs: > > > > > > > > > > > > pip install --no-cache-dir > > > > > > > > > > > > > > > > > > https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3- > non > > e-manylinux1_x86_64.whl > > > > > > pip install --no-cache-dir > > > > > > > > > > > > > > > > > > https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101- > 1.5.1.post0- > > py2.py3-none-manylinux1_x86_64.whl > > > > > > pip install --no-cache-dir > > > > > > https://d19zq12jzu4w95.cloudfront.net/ > > > > > > mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl > > > > > > < > > > > > > > > > > > > https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0- > py2.py3- > > none-manylinux1_x86_64.whl > > > > > > > > > > > > where --no-cache-dir prevents caching the downloaded file, for > > > > > > the > > > > purpose > > > > > > of > > > > > > testing. (cu101 chosen based on large size) > > > > > > > > > > > > The first URL uses standard S3 bucket in US. The second uses > > > > > > S3 > > > > Accelerate > > > > > > based > > > > > > on CloudFront CDN. And the third uses CloudFront CDN. I'm > > > > > > adding > > the > > > > third > > > > > > URL, > > > > > > as S3 Accelerate may or may not use all new CloudFront > > > > > > endpoints > > yet. > > > > > > > > > > > > Regarding voting: Uploading to Pypi is currently impossible, > > > > > > which > > is a > > > > > > reality > > > > > > (so there is no option to continue as we do currently). Pypi > > > > > > folks indicated they will unblock our uploads to Pypi once we > > > > > > stop uploading > > nightly > > > > > > releases > > > > > > and taking up 20% of their ressources [1]. > > > > > > > > > > > > If there are any shortcomings or problems identified with > > uploading to > > > > S3, > > > > > > we > > > > > > can work to address them. But for now, status quo is broken > > > > > > and > > this > > > > seems > > > > > > the > > > > > > only solution addressing Pypi's problem. > > > > > > > > > > > > I don't mind if you state that you object to lazy consensus > > > > > > and > > start a > > > > > > vote. If > > > > > > your "maybe [...] start a proper vote" was supposed to be an > > objection > > > > to > > > > > > lazy > > > > > > consensus, please state so clearly (I'm not sure if "maybe" > > qualifies > > > > as > > > > > > objection). Though I think it only makes sense with at least 2 > > options > > > > to > > > > > > vote > > > > > > on. Status quo is not a meaningful option, as it is already broken. > > > > > > > > > > > > Best regards > > > > > > Leonard > > > > > > > > > > > > [1]: > > > > https://github.com/pypa/pypi-support/issues/50#issuecomment- > 560479 > > > > 706 > > > > > > On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote: > > > > > > Excellent! Could we maybe come up with a POC and a quick > > > > > > writeup > > and > > > > then > > > > > > start a proper vote after everyone verified that it covers > > > > > > their use-cases? > > > > > > -Marco > > > > > > > > > > > > Sheng Zha <zhash...@apache.org> schrieb am Di., 3. Dez. 2019, > > 19:24: > > > > > > > > > > > > Yes, there is. We can also make it easier to access by using a > > > > > > geo-location based DNS server so that China users are directed > > > > > > to > > that > > > > > > local mirror. The rest of the world is already covered by the > > global > > > > > > cloudfront. > > > > > > > > > > > > -sz > > > > > > > > > > > > On 2019/12/03 18:22:22, Marco de Abreu > > > > > > <marco.g.ab...@gmail.com> > > > > > > wrote: > > > > > > Isn't there an s3 endpoint in Beijing? > > > > > > > > > > > > It seems like this topic still warrants some discussion and > > > > > > thus > > I'd > > > > > > > > > > > > prefer > > > > > > if we don't move forward with lazy consensus. > > > > > > > > > > > > -Marco > > > > > > > > > > > > Tao Lv <mutou...@gmail.com> schrieb am Di., 3. Dez. 2019, 14:31: > > > > > > > > > > > > * For pypi, we can use mirrors. > > > > > > > > > > > > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mutou...@gmail.com> > wrote: > > > > > > > > > > > > As we have many users in China, I'm considering the > > > > > > accessibility of S3. > > > > > > For pip, we can mirrors. > > > > > > > > > > > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard > > > > > > > > > > > > <lau...@amazon.com.invalid > > > > > > wrote: > > > > > > > > > > > > I would like to remind everyone that lazy consensus is assumed > > > > > > if no objections are raised before 2019-12-05 at 05:42 UTC. > > > > > > There has been some > > > > > > > > > > > > discussion > > > > > > about > > > > > > the proposal, but to my understanding no objections were > > > > > > raised. > > > > > > If the proposal is accepted, MXNet releases would be installed > > > > > > via > > > > > > pip install mxnet > > > > > > > > > > > > And release candidates via > > > > > > > > > > > > pip install --pre mxnet > > > > > > > > > > > > (or with the respective cuda version specifier appended etc.) > > > > > > > > > > > > To obtain releases built automatically from the master branch, > > > > > > users would need to specify something like "-f > > > > > > http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html"; option to > > > > > > pip. > > > > > > Best regards > > > > > > Leonard > > > > > > > > > > > > On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote: > > > > > > Hi MXNet Community, > > > > > > > > > > > > since more than 2 months our binary Python nightly releases > > > > > > > > > > > > published > > > > > > on Pypi > > > > > > are broken. The problem is that our binaries exceed Pypi's > > > > > > size limit. > > > > > > Decreasing the binary size by adding compression breaks > > > > > > > > > > > > third-party > > > > > > libraries > > > > > > loading libmxnet.so > > > > > > > > > > > > https://github.com/apache/incubator-mxnet/issues/16193 > > > > > > Sheng requested Pypi to increase their size limit: > > > > > > https://github.com/pypa/pypi-support/issues/50 > > > > > > > > > > > > Currently "the biggest cost for PyPI from [the many MXNet > > > > > > binaries with nightly release to Pypi] is the bandwidth > > > > > > consumed when several hundred mirrors attempt to mirror each > > > > > > release immediately after it's published". So Pypi is not > > > > > > inclined to allow us to upload even larger binaries on a > > > > > > nightly schedule. > > > > > > Their compromise is to allow it on a weekly cadence. > > > > > > > > > > > > However, I would like the community to revisit the necessity > > > > > > of releasing pre- release binaries to Pypi on a nightly (or > > > > > > weekly) cadence. > > > > > > > > > > > > Instead, we > > > > > > can > > > > > > release nightly binaries ONLY to a public S3 bucket and > > > > > > instruct users to install from there. On our side, we only > > > > > > need to prepare a html document that contains links to all > > > > > > released nightly binaries. > > > > > > Finally users will install the nightly releases via > > > > > > > > > > > > pip install --pre mxnet-cu101 -f > > > > > > > > > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/ > > > > > > nightly.html > > > > > > > > > > > > Instead of > > > > > > > > > > > > pip install --pre mxnet-cu101 > > > > > > > > > > > > Of course proper releases and release candidates should still > > > > > > be made available via Pypi. Thus releases would be installed > > > > > > via > > > > > > > > > > > > pip install mxnet-cu101 > > > > > > > > > > > > And release candidates via > > > > > > > > > > > > pip install --pre mxnet-cu101 > > > > > > > > > > > > This will substantially reduce the costs of the Pypi project > > > > > > and in fact matches the installation experience provided by > > > > > > PyTorch. I don't think the benefit of not including "-f > > > > > > > > > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"; > > > > > > matches the costs we currently externalize to the Pypi team. > > > > > > > > > > > > This suggestion seems uncontroversial to me. Thus I would like > > > > > > to start lazy consensus. If there are no objections, I will > > > > > > assume lazy > > > > > > > > > > > > consensus on > > > > > > stopping > > > > > > nightly releases to Pypi in 72hrs. > > > > > > > > > > > > Best regards > > > > > > Leonard > > > > > > > > > > > > > >
