Not yet. As a community, we first need to add the nightly build hosting feature to the community run CD and then we can add the page so that the exact date doesn't need to be specified.
I'm not sure what steps are required for this. Do we need to host the artifacts on Apache's infrastructure? Or can we host the nightly CD artifacts as part of the AWS sponsored community-maintained CD (S3 bucket associated to the account)? In the meantime, the "proprietary" AWS build solution could be extended to publish an html page per artifact type (mxnet, mxnet-cu100, ...) containing a link to all recent builds. Best regards Leonard On Tue, 2019-12-10 at 22:03 -0800, Lin Yuan wrote: > Is there a way to install the latest nightly package without having to > specify exact date? > > Thanks, > > Lin > > On Sun, Dec 8, 2019 at 6:13 PM Lausen, Leonard <lau...@amazon.com.invalid> > wrote: > > > From Shanghai, the closest endpoint (automatically chosen endpoint) is in > > Tokyo > > and download speed for mxnet-mkl was on average 1.7 MB/s with a maximum of > > 5 > > MB/s during my test. > > > > On Sun, 2019-12-08 at 01:30 +0000, Sheng Zha wrote: > > > > Heres a set of links for today’s builds > > > > > > > > (Plain mxnet, no mkl no cuda) > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > (mxnet-mkl) > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > (mxnet-cuXXX) > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > (mxnet-cuXXXmkl) > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > These links are not utilizing the s3 accelerate feature (i.e. not backed > > by > > > cloudfront edges). Please use repo.mxnet.io instead. The updated links > > are: > > > (Plain mxnet, no mkl no cuda) > > > > > https://repo.mxnet.io/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > (mxnet-mkl) > > > > > https://repo.mxnet.io/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > (mxnet-cuXXX) > > > > > https://repo.mxnet.io/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > https://repo.mxnet.io/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > https://repo.mxnet.io/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > https://repo.mxnet.io/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > (mxnet-cuXXXmkl) > > > > > https://repo.mxnet.io/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > https://repo.mxnet.io/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > https://repo.mxnet.io/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > https://repo.mxnet.io/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > When updating the installation doc we should use repo.mxnet.io domain > > name > > > too. > > > > > > Best, > > > -sz > > > > > > On 2019/12/07 17:39:40, "Skalicky, Sam" <sska...@amazon.com.INVALID> > > wrote: > > > > Hi MXNet Community, > > > > > > > > We have been working on getting nightly builds fixed and made available > > > > again. We’ve made another system using AWS CodeBuild & S3 to work > > around the > > > > problems with Jenkins CI, PyPI, etc. It is currently building all the > > > > flavors and publishing to an S3 bucket here: > > > > > > https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2 > > > > There are folders for each set of nightly builds, try out the wheels > > > > starting today 2019-12-07. Builds start at 1:30am PT (9:30am GMT) and > > arrive > > > > in the bucket 30min-2hours later. Inside each folder are the wheels > > for each > > > > flavor of MXNet. Currently we’re only building for linux, builds for > > > > windows/Mac will come later. > > > > > > > > If you want to download the wheels easily you can use a URL in the > > form of: > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/ > > <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none- > > manylinux1_x86_64.whl > > > > Heres a set of links for today’s builds > > > > > > > > (Plain mxnet, no mkl no cuda) > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > (mxnet-mkl) > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > (mxnet-cuXXX) > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > (mxnet-cuXXXmkl) > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > You can easily install these pip wheels in your system either by > > downloading > > > > them to your machine first and then installing by doing: > > > > > > > > pip install /path/to/downloaded/wheel.whl > > > > > > > > Or you can install directly by just giving the link to pip like this: > > > > > > > > pip install > > > > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > > > Credit goes to everyone involved (in no particular order) > > > > Rakesh Vasudevan > > > > Zach Kimberg > > > > Manu Seth > > > > Sheng Zha > > > > Jun Wu > > > > Pedro Larroy > > > > Chaitanya Bapat > > > > > > > > Thanks! > > > > Sam > > > > > > > > > > > > On Dec 5, 2019, at 1:16 AM, Lausen, Leonard <lau...@amazon.com.INVALID > > <mailt > > > > o:lau...@amazon.com.INVALID>> wrote: > > > > > > > > We don't loose pip by hosting on S3. We just don't host nightly > > releases on > > > > Pypi > > > > servers and mirror them to several hundred mirrors immediately after > > each > > > > build > > > > is published which is very expensive for the Pypi project.. People can > > still > > > > install the nightly builds with pip by specifying the -f option. > > > > > > > > Uploading weekly releases to Pypi will reduce the cost for Pypi by > > ~75% [1]. > > > > It > > > > may be acceptable to Pypi, but does it make sense for us? I'm not > > convinced > > > > weekly release on Pypi is a good idea. Consider one release is buggy, > > users > > > > will > > > > need to wait for 7 days for a fix. It doesn't provide good user > > experience. > > > > If someone has a stronger conviction about the value of weekly > > releases on > > > > Pypi, > > > > that person shall please go ahead and propose it in a separate > > discussion > > > > thread. > > > > > > > > Currently we don't have generally working nightly builds to Pypi and > > as a > > > > matter > > > > of fact we know that we can't have them due to Pypi's policy and our > > > > apparent > > > > need for large binaries. Given this fact and that no objection was > > raised by > > > > 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on stopping > > > > upload > > > > attempts of nightly builds to Pypi. > > > > > > > > With consensus established, we can change the CI job to stop trying to > > > > upload > > > > the nightly builds and then request Pypi to increase the limit. Then > > we have > > > > one > > > > less blocker for the 1.6 release. > > > > > > > > Best regards > > > > Leonard > > > > > > > > [1]: Lower cost due to less releases, but higher cost due to 500MB -> > > 800MB > > > > limit increase. Assuming that the limit increase translates into > > actually > > > > larger > > > > binaries. > > > > > > > > > > > > On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote: > > > > Are weekly releases an option? It was brought up as concern that we > > might > > > > lose pip as a pretty common distribution channel where people consume > > > > nightly builds. I don't feel like that concern has been properly > > addressed > > > > so far. > > > > > > > > -Marco > > > > > > > > Lausen, Leonard <lau...@amazon.com.invalid<mailto: > > lau...@amazon.com.invalid> > > > > > schrieb am Mi., 4. Dez. 2019, > > > > 04:09: > > > > > > > > As a simple POC to test distribution, you can try installing MXNet > > based on > > > > these 3 URLs: > > > > > > > > pip install --no-cache-dir > > > > > > > > > > https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl > > > > pip install --no-cache-dir > > > > > > > > > > https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl > > > > pip install --no-cache-dir https://d19zq12jzu4w95.cloudfront.net/ > > > > mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl > > > > < > > > > > > https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl > > > > > > > > where --no-cache-dir prevents caching the downloaded file, for the > > purpose > > > > of > > > > testing. (cu101 chosen based on large size) > > > > > > > > The first URL uses standard S3 bucket in US. The second uses S3 > > Accelerate > > > > based > > > > on CloudFront CDN. And the third uses CloudFront CDN. I'm adding the > > third > > > > URL, > > > > as S3 Accelerate may or may not use all new CloudFront endpoints yet. > > > > > > > > Regarding voting: Uploading to Pypi is currently impossible, which is a > > > > reality > > > > (so there is no option to continue as we do currently). Pypi folks > > > > indicated > > > > they will unblock our uploads to Pypi once we stop uploading nightly > > > > releases > > > > and taking up 20% of their ressources [1]. > > > > > > > > If there are any shortcomings or problems identified with uploading to > > S3, > > > > we > > > > can work to address them. But for now, status quo is broken and this > > seems > > > > the > > > > only solution addressing Pypi's problem. > > > > > > > > I don't mind if you state that you object to lazy consensus and start a > > > > vote. If > > > > your "maybe [...] start a proper vote" was supposed to be an objection > > to > > > > lazy > > > > consensus, please state so clearly (I'm not sure if "maybe" qualifies > > as > > > > objection). Though I think it only makes sense with at least 2 options > > to > > > > vote > > > > on. Status quo is not a meaningful option, as it is already broken. > > > > > > > > Best regards > > > > Leonard > > > > > > > > [1]: > > https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706 > > > > On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote: > > > > Excellent! Could we maybe come up with a POC and a quick writeup and > > then > > > > start a proper vote after everyone verified that it covers their > > > > use-cases? > > > > -Marco > > > > > > > > Sheng Zha <zhash...@apache.org> schrieb am Di., 3. Dez. 2019, 19:24: > > > > > > > > Yes, there is. We can also make it easier to access by using a > > > > geo-location based DNS server so that China users are directed to that > > > > local mirror. The rest of the world is already covered by the global > > > > cloudfront. > > > > > > > > -sz > > > > > > > > On 2019/12/03 18:22:22, Marco de Abreu <marco.g.ab...@gmail.com> > > > > wrote: > > > > Isn't there an s3 endpoint in Beijing? > > > > > > > > It seems like this topic still warrants some discussion and thus I'd > > > > > > > > prefer > > > > if we don't move forward with lazy consensus. > > > > > > > > -Marco > > > > > > > > Tao Lv <mutou...@gmail.com> schrieb am Di., 3. Dez. 2019, 14:31: > > > > > > > > * For pypi, we can use mirrors. > > > > > > > > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mutou...@gmail.com> wrote: > > > > > > > > As we have many users in China, I'm considering the > > > > accessibility of > > > > S3. > > > > For pip, we can mirrors. > > > > > > > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard > > > > > > > > <lau...@amazon.com.invalid > > > > wrote: > > > > > > > > I would like to remind everyone that lazy consensus is assumed > > > > if no > > > > objections > > > > are raised before 2019-12-05 at 05:42 UTC. There has been some > > > > > > > > discussion > > > > about > > > > the proposal, but to my understanding no objections were > > > > raised. > > > > If the proposal is accepted, MXNet releases would be installed > > > > via > > > > pip install mxnet > > > > > > > > And release candidates via > > > > > > > > pip install --pre mxnet > > > > > > > > (or with the respective cuda version specifier appended etc.) > > > > > > > > To obtain releases built automatically from the master branch, > > > > users > > > > would need > > > > to specify something like "-f > > > > http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html"; option to > > > > pip. > > > > Best regards > > > > Leonard > > > > > > > > On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote: > > > > Hi MXNet Community, > > > > > > > > since more than 2 months our binary Python nightly releases > > > > > > > > published > > > > on Pypi > > > > are broken. The problem is that our binaries exceed Pypi's > > > > size > > > > limit. > > > > Decreasing the binary size by adding compression breaks > > > > > > > > third-party > > > > libraries > > > > loading libmxnet.so > > > > > > > > https://github.com/apache/incubator-mxnet/issues/16193 > > > > Sheng requested Pypi to increase their size limit: > > > > https://github.com/pypa/pypi-support/issues/50 > > > > > > > > Currently "the biggest cost for PyPI from [the many MXNet > > > > binaries > > > > with > > > > nightly > > > > release to Pypi] is the bandwidth consumed when several > > > > hundred > > > > mirrors > > > > attempt > > > > to mirror each release immediately after it's published". So > > > > Pypi > > > > is > > > > not > > > > inclined to allow us to upload even larger binaries on a > > > > nightly > > > > schedule. > > > > Their compromise is to allow it on a weekly cadence. > > > > > > > > However, I would like the community to revisit the necessity > > > > of > > > > releasing pre- > > > > release binaries to Pypi on a nightly (or weekly) cadence. > > > > > > > > Instead, we > > > > can > > > > release nightly binaries ONLY to a public S3 bucket and > > > > instruct > > > > users > > > > to > > > > install from there. On our side, we only need to prepare a > > > > html > > > > document that > > > > contains links to all released nightly binaries. > > > > Finally users will install the nightly releases via > > > > > > > > pip install --pre mxnet-cu101 -f > > > > > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/ > > > > nightly.html > > > > > > > > Instead of > > > > > > > > pip install --pre mxnet-cu101 > > > > > > > > Of course proper releases and release candidates should > > > > still be > > > > made > > > > available > > > > via Pypi. Thus releases would be installed via > > > > > > > > pip install mxnet-cu101 > > > > > > > > And release candidates via > > > > > > > > pip install --pre mxnet-cu101 > > > > > > > > This will substantially reduce the costs of the Pypi project > > > > and > > > > in > > > > fact > > > > matches > > > > the installation experience provided by PyTorch. I don't > > > > think the > > > > benefit of > > > > not including "-f > > > > > > > > http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"; > > > > matches the costs we currently externalize to the Pypi team. > > > > > > > > This suggestion seems uncontroversial to me. Thus I would > > > > like to > > > > start > > > > lazy > > > > consensus. If there are no objections, I will assume lazy > > > > > > > > consensus on > > > > stopping > > > > nightly releases to Pypi in 72hrs. > > > > > > > > Best regards > > > > Leonard > > > > > > > >
