Hi Sam, Can you simply change the check on L21 in SortTable from `if (x > y)` to `if (x < y)`? Seems then we can have the newest build at the top of table.
Thanks, -tao On Tue, Jan 14, 2020 at 7:07 AM Lin Yuan <apefor...@gmail.com> wrote: > Awesome work! It's really convenient to have this page. > > Two cents: > (1) create a link on mxnet page to this one > (2) reorder the nightly as Tao suggested. Newest first. > > On Mon, Jan 13, 2020 at 10:25 AM Skalicky, Sam <sska...@amazon.com.invalid > > > wrote: > > > Hi All, > > > > The html page source is available at the link (view source, its all in a > > single html file), if someone wants to make modifications I’ll be happy > to > > help integrate those changes and get the latest version published in the > S3 > > bucket. Whenever the final location of the nightly builds is identified > we > > can move/modify the script appropriately. > > > > Sam > > > > > On Jan 12, 2020, at 5:41 PM, Tao Lv <mutou...@gmail.com> wrote: > > > > > > Thank you for the effort, Sam. One minor suggestion: can we sort and > put > > > the latest build at the top of the table? > > > > > > -tao > > > > > > On Mon, Jan 13, 2020 at 7:03 AM Marco de Abreu < > marco.g.ab...@gmail.com> > > > wrote: > > > > > >> Hi Sam, > > >> > > >> that's a great idea, thanks! Can you please adjust the script so it > uses > > >> the artifacts that will be published once Shengs PR gets merged? > > >> > > >> Best regards, > > >> Marco > > >> > > >> Skalicky, Sam <sska...@amazon.com.invalid> schrieb am So., 12. Jan. > > 2020, > > >> 23:23: > > >> > > >>> Hi dev, > > >>> > > >>> I made an html page that generates the links to the nightly builds > > >>> available in the public S3 bucket so you don’t have to log into AWS > to > > >> see > > >>> them. > > >>> > > >>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/index.html > > >>> > > >>> Keep in mind we only have builds from January 2020 and December 2019 > so > > >>> far. > > >>> > > >>> Sam > > >>> > > >>> On Jan 10, 2020, at 3:05 AM, Sheng Zha <zhash...@apache.org<mailto: > > >>> zhash...@apache.org>> wrote: > > >>> > > >>> Size of a change doesn't necessarily reflect the time one spends on > the > > >>> navigating the code base and finding the solution. Also, I tend to > > >> believe > > >>> that everyone genuinely wants what's best for the project, just from > > >>> different perspectives. > > >>> > > >>> Let's focus on improving the CD solution so that security concerns > can > > be > > >>> addressed too. > > >>> > > >>> -sz > > >>> > > >>> On 2020/01/09 21:57:30, Chris Olivier <cjolivie...@apache.org > <mailto: > > >>> cjolivie...@apache.org>> wrote: > > >>> If this tiny fix is representative of the bulk of the reasoning > behind > > >> all > > >>> the the CD churn recently, then this seems to be of some concern. > > >>> > > >>> -Chris > > >>> > > >>> On Thu, Jan 9, 2020 at 6:32 AM Marco de Abreu < > marco.g.ab...@gmail.com > > >>> <mailto:marco.g.ab...@gmail.com>> > > >>> wrote: > > >>> > > >>> Great, thanks a lot sheng! > > >>> > > >>> -Marco > > >>> > > >>> Sheng Zha <zhash...@apache.org<mailto:zhash...@apache.org>> schrieb > am > > >>> Do., 9. Jan. 2020, 14:28: > > >>> > > >>> I'm fixing the CD pipeline in > > >>> https://github.com/apache/incubator-mxnet/pull/17259/files and will > > >>> update the s3 publish path so that it's friendly for automatically > > >>> generating such page. > > >>> > > >>> -sz > > >>> > > >>> On 2020/01/06 18:19:52, "Lausen, Leonard" <lau...@amazon.com.INVALID > > >>> <mailto:lau...@amazon.com.INVALID>> > > >>> wrote: > > >>> Consider a user finds a bug in a nightly version but we can't narrow > > >>> down the > > >>> version of mxnet used as the name is constant over time. Or users > wan't > > >>> to > > >>> revert back to the previous nightly version installed but don't know > > >>> which date > > >>> it was from due to constant name. > > >>> > > >>> Instead I suggest we introduce an autogenerated page like > > >>> https://download.pytorch.org/whl/nightly/cu101/torch_nightly.html > > >>> > > >>> Then "pip install -f URLTOPAGE mxnet" will install the latest > available > > >>> version. > > >>> Maybe the team maintaining the S3 bucket can reconsider creating such > > >>> page for > > >>> the intermediate time until the CD based nighlty build is operating. > > >>> > > >>> On Mon, 2020-01-06 at 10:01 -0800, Lin Yuan wrote: > > >>> +1 for a nightly pip with fixed name. > > >>> > > >>> We need this to track mxnet integration with other packages such as > > >>> Horovod. > > >>> > > >>> Sam, when do you think we can have this nightly build with a fixed > > >>> name? > > >>> > > >>> Thanks, > > >>> > > >>> Lin > > >>> > > >>> On Sun, Jan 5, 2020 at 7:48 PM Skalicky, Sam > > >>> <sska...@amazon.com.invalid<mailto:sska...@amazon.com.invalid>> > > >>> wrote: > > >>> > > >>> Hi Tao, > > >>> > > >>> We dont have this yet, but we did think about putting the latest > > >>> wheels in > > >>> a specific place in the s3 bucket so they are always updated. > > >>> Initially we > > >>> decided not to do this since the main MXNet CD should have been > > >>> fixed. But > > >>> since its still not fixed yet, we might try and go ahead and do > > >>> this. > > >>> > > >>> Sam > > >>> > > >>> On Jan 5, 2020, at 6:02 PM, Lv, Tao A <tao.a...@intel.com<mailto: > > >>> tao.a...@intel.com><mailto: > > >>> tao.a...@intel.com<mailto:tao.a...@intel.com>>> wrote: > > >>> > > >>> Hi, > > >>> > > >>> How to install the latest available build of a flavor without > > >>> specifying > > >>> the build date? Something like `pip install mxnet --pre`. > > >>> > > >>> Thanks, > > >>> -tao > > >>> > > >>> -----Original Message----- > > >>> From: Skalicky, Sam <sska...@amazon.com.INVALID<mailto: > > >>> sska...@amazon.com.INVALID><mailto: > > >>> sska...@amazon.com.INVALID<mailto:sska...@amazon.com.INVALID>>> > > >>> Sent: Monday, January 6, 2020 2:09 AM > > >>> To: dev@mxnet.incubator.apache.org<mailto: > > dev@mxnet.incubator.apache.org > > >>>> <mailto: > > >>> dev@mxnet.incubator.apache.org<mailto:dev@mxnet.incubator.apache.org > >> > > >>> Subject: Re: Stopping nightly releases to Pypi > > >>> > > >>> Hi Haibin, > > >>> > > >>> You typed the correct URLs, the cu100 build has been failing since > > >>> December 30th but other builds have succeeded. The wheels are being > > >>> delivered into a public bucket that anyone with an AWS account can > > >>> access > > >>> and go poke around, here’s the URL for web access: > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >> > > > https://s3.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/2020-01-01/dist/?region=us-west-2&tab=overview > > >>> > > >>> You will have to log into your AWS account to access it however > > >>> (which > > >>> means you’ll need an AWS account). > > >>> > > >>> It looks like only the following flavors are available for > > >>> 2020-01-01: > > >>> mxnet > > >>> mxnet-cu92 > > >>> mxnet-cu92mkl > > >>> mxnet-mkl > > >>> > > >>> Sam > > >>> > > >>> On Jan 4, 2020, at 9:06 PM, Haibin Lin <haibin.lin....@gmail.com > > >>> <mailto: > > >>> haibin.lin....@gmail.com><mailto:haibin.lin....@gmail.com>> wrote: > > >>> > > >>> I was trying the nightly builds, but none of them is available: > > >>> > > >>> pip3 install > > >>> > > >>> > > >>> > > >>> > > >>> > > >> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl > > >>> --user > > >>> < > > >> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl--user > > >>> > > >>> < > > >>> > > >>> > > >>> > > >>> > > >> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-01/dist/mxnet_cu100-1.6.0b20200101-py2.py3-none-manylinux1_x86_64.whl--user > > >>> > > >>> pip3 install > > >>> > > >>> > > >>> > > >>> > > >>> > > >> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl > > >>> --user > > >>> < > > >> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl--user > > >>> > > >>> < > > >>> > > >>> > > >>> > > >>> > > >> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-02/dist/mxnet_cu100-1.6.0b20200102-py2.py3-none-manylinux1_x86_64.whl--user > > >>> > > >>> pip3 install > > >>> > > >>> > > >>> > > >>> > > >>> > > >> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl > > >>> --user > > >>> < > > >> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl--user > > >>> > > >>> < > > >>> > > >>> > > >>> > > >>> > > >> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl--user > > >>> > > >>> pip3 install > > >>> > > >>> > > >>> > > >>> > > >>> > > >> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl > > >>> --user > > >>> < > > >> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl--user > > >>> > > >>> < > > >>> > > >>> > > >>> > > >>> > > >> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-04/dist/mxnet_cu100-1.6.0b20200104-py2.py3-none-manylinux1_x86_64.whl--user > > >>> > > >>> > > >>> ERROR: Could not install requirement mxnet-cu100==1.6.0b20200103 > > >>> from > > >>> > > >>> > > >>> > > >>> > > >> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl > > >>> because of HTTP error 404 Client Error: Not Found for url: > > >>> > > >>> > > >>> > > >>> > > >>> > > >> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl > > >>> for URL > > >>> > > >>> > > >>> > > >>> > > >>> > > >> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2020-01-03/dist/mxnet_cu100-1.6.0b20200103-py2.py3-none-manylinux1_x86_64.whl > > >>> > > >>> Please let me know if I typed wrong URLs. > > >>> > > >>> 1. The discoverability of available nightly builds needs > > >>> improvement. If > > >>> someone can help write a script to list all links that exist, that > > >>> would be > > >>> very helpful. > > >>> 2. If any nightly build is not built successfully, how do the > > >>> community > > >>> know the reason of the failure, and potentially offer helps? > > >>> Currently I > > >>> don't have much visibility of the nightly build status. > > >>> > > >>> Best, > > >>> Haibin > > >>> > > >>> > > >>> On Fri, Jan 3, 2020 at 5:47 PM Pedro Larroy < > > >>> pedro.larroy.li...@gmail.com > > >>> <mailto:pedro.larroy.li...@gmail.com>> > > >>> wrote: > > >>> > > >>> Just to clarify, the current CI is quite an overhead to maintain > > >>> for > > >>> several reasons, this complexity is overkill for CD. Jenkins also > > >>> has > > >>> constant plugin upgrades, security vulnerabilities, has to be > > >>> restarted > > >>> from time to time as it stops working... and to make binary builds > > >>> from an > > >>> environment which runs unsafe code, I don't think is good practice. > > >>> So for > > >>> that, having a separate Jenkins, CodeBuild, Drone or using a > > >>> separate > > >>> Jenkins node is the right solution. Agree with you that is just a > > >>> scheduler, but somebody is making efforts to keep it running. If > > >>> you > > >>> have > > >>> the appetite and resources to duplicate it for CD please go ahead. > > >>> > > >>> On Fri, Jan 3, 2020 at 3:25 PM Marco de Abreu < > > >>> marco.g.ab...@gmail.com > > >>> <mailto:marco.g.ab...@gmail.com>> > > >>> wrote: > > >>> > > >>> Regarding your point of finding somebody to maintain the solution: > > >>> At > > >>> Apache we usually retire things if there's no maintainer, since > > >>> that > > >>> indicates that the feature/system is not of enough interest to > > >>> warrant > > >>> maintenance - otherwise, someone would step up. > > >>> > > >>> While assistance in the form of a fix is always appreciated, the > > >>> fix > > >>> still > > >>> has to conform with the way this project and Apache operates. Next > > >>> time I'd > > >>> recommend to contribute time on improving the existing community > > >>> solution > > >>> instead of developing an internal system. > > >>> > > >>> -Marco > > >>> > > >>> Marco de Abreu <marco.g.ab...@gmail.com<mailto: > > >>> marco.g.ab...@gmail.com>> > > >>> schrieb am Sa., 4. Jan. 2020, > > >>> 00:21: > > >>> > > >>> Sam, while I understand that this solution was developed out of > > >>> necessity, > > >>> my question why a new system has been developed instead of fixing > > >>> the > > >>> existing one or adapting the solution. CodeBuild is a scheduler in > > >>> the same > > >>> fashion as Jenkins is. It runs code. So you can adapt it to Jenkins > > >>> without > > >>> much hassle. > > >>> > > >>> I'm not volunteering for this - why should I? The role of a PMC > > >>> member is > > >>> to steer the direction of the project. Just because a manager > > >>> points > > >>> towards a certain direction, if doesn't mean that they're going to > > >>> do it. > > >>> > > >>> Apparently there was enough time at some point to develop a new > > >>> solution > > >>> from scratch. It might have been a solution for your internal team > > >>> and > > >>> that's fine, but upgrading it "temporarily" to be the advertised > > >>> way > > >>> on the > > >>> official website is something different. > > >>> > > >>> I won't argue about how the veto can be enforced. I think it's in > > >>> the best > > >>> interest of the project if we try working on a solution instead of > > >>> spending > > >>> time on trying to figure out the power of the PMC. > > >>> > > >>> Pedro, that's certainly a step towards the right direction. But > > >>> committers > > >>> would also need access to the control plane of the system - to > > >>> trigger, > > >>> stop and audit builds. We could go down that road, but i think the > > >>> fewer > > >>> systems, the better - also for the sake of maintainability. > > >>> > > >>> Best regards, > > >>> Marco > > >>> > > >>> > > >>> > > >>> Pedro Larroy <pedro.larroy.li...@gmail.com<mailto: > > >>> pedro.larroy.li...@gmail.com>> schrieb am Fr., 3. Jan. > > >>> 2020, > > >>> 20:55: > > >>> > > >>> I'm not involved in such efforts, but one possibility is to have > > >>> the > > >>> yaml > > >>> files that describe the pipelines for CD in the Apache > > >>> repositories, > > >>> would > > >>> that be acceptable from the Apache POV? In the end they should be > > >>> very thin > > >>> and calling the scripts that are part of the CD packages. > > >>> > > >>> On Fri, Jan 3, 2020 at 6:56 AM Marco de Abreu < > > >>> marco.g.ab...@gmail.com > > >>> <mailto:marco.g.ab...@gmail.com>> > > >>> wrote: > > >>> > > >>> Agree, but the question how a non Amazonian is able to maintain and > > >>> access > > >>> the system is still open. As it stands right now, the community has > > >>> taken a > > >>> step back and loses some control if we continue down that road. > > >>> > > >>> I personally am disapproving of that approach since committers are > > >>> no > > >>> longer in control of that process. So far it seems like my > > >>> questions > > >>> were > > >>> skipped and further actions have been taken. As openness and the > > >>> community > > >>> having control are part of our graduation criteria, I'm putting in > > >>> my veto > > >>> with a grace period until 15th of January. Please bring the system > > >>> into a > > >>> state that aligns with Apache values or revert the changes. > > >>> > > >>> -Marco > > >>> > > >>> Pedro Larroy <pedro.larroy.li...@gmail.com<mailto: > > >>> pedro.larroy.li...@gmail.com>> schrieb am Fr., 3. Jan. > > >>> 2020, > > >>> 03:33: > > >>> > > >>> CD should be separate from CI for security reasons in any case. > > >>> > > >>> > > >>> On Sat, Dec 7, 2019 at 10:04 AM Marco de Abreu < > > >>> marco.g.ab...@gmail.com > > >>> <mailto:marco.g.ab...@gmail.com>> > > >>> wrote: > > >>> > > >>> Could you elaborate how a non-Amazonian is able to access, maintain > > >>> and > > >>> review the CodeBuild pipeline? How come we've diverted from the > > >>> community > > >>> agreed-on standard where the public Jenkins serves for the purpose > > >>> of > > >>> testing and releasing MXNet? I'd be curious about the issues you're > > >>> encountering with Jenkins CI that led to a non-standard solution. > > >>> > > >>> -Marco > > >>> > > >>> > > >>> Skalicky, Sam <sska...@amazon.com.invalid<mailto: > > >>> sska...@amazon.com.invalid>> schrieb am Sa., 7. > > >>> Dez. > > >>> 2019, > > >>> 18:39: > > >>> > > >>> Hi MXNet Community, > > >>> > > >>> We have been working on getting nightly builds fixed and made > > >>> available > > >>> again. We’ve made another system using AWS CodeBuild & S3 to work > > >>> around > > >>> the problems with Jenkins CI, PyPI, etc. It is currently building > > >>> all the > > >>> flavors and publishing to an S3 bucket here: > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >> > > > https://us-west-2.console.aws.amazon.com/s3/buckets/apache-mxnet/dist/?region=us-west-2 > > >>> > > >>> There are folders for each set of nightly builds, try out the > > >>> wheels > > >>> starting today 2019-12-07. Builds start at 1:30am PT (9:30am > > >>> GMT) > > >>> and > > >>> arrive in the bucket 30min-2hours later. Inside each folder are the > > >>> wheels > > >>> for each flavor of MXNet. Currently we’re only building for linux, > > >>> builds > > >>> for windows/Mac will come later. > > >>> > > >>> If you want to download the wheels easily you can use a URL in the > > >>> form > > >>> of: > > >>> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/ > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> <YYYY-MM-DD>/dist/<mxnet_build>-1.6.0b<YYYYMMDD>-py2.py3-none- > > >>> manylinux1_x86_64.whl > > >>> > > >>> Heres a set of links for today’s builds > > >>> > > >>> (Plain mxnet, no mkl no cuda) > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > >>> (mxnet-mkl > > >>> < > > >> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl > > >>> > > >>> < > > >>> > > >>> > > >>> > > >>> > > >> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl > > >>> > > >>> < > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl > > >>> > > >>> ) > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > >>> (mxnet-cuXXX > > >>> < > > >> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX > > >>> > > >>> < > > >>> > > >>> > > >>> > > >>> > > >> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX > > >>> > > >>> < > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX > > >>> > > >>> ) > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > >>> (mxnet-cuXXXmkl > > >>> < > > >> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl > > >>> > > >>> < > > >>> > > >>> > > >>> > > >>> > > >> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl > > >>> > > >>> < > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl > > >>> > > >>> ) > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > >>> > > >>> You can easily install these pip wheels in your system either by > > >>> downloading them to your machine first and then installing by > > >>> doing: > > >>> > > >>> pip install /path/to/downloaded/wheel.whl > > >>> > > >>> Or you can install directly by just giving the link to pip like > > >>> this: > > >>> > > >>> pip install > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >> > > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl > > >>> > > >>> Credit goes to everyone involved (in no particular order) Rakesh > > >>> Vasudevan > > >>> Zach Kimberg Manu Seth Sheng Zha Jun Wu Pedro Larroy Chaitanya > > >>> Bapat > > >>> > > >>> Thanks! > > >>> Sam > > >>> > > >>> > > >>> On Dec 5, 2019, at 1:16 AM, Lausen, Leonard > > >>> <lau...@amazon.com.INVALID > > >>> <mailto:lau...@amazon.com.INVALID> <mailto: > > >>> lau...@amazon.com.INVALID > > >>> > > >>> wrote: > > >>> > > >>> We don't loose pip by hosting on S3. We just don't host nightly > > >>> releases > > >>> on Pypi servers and mirror them to several hundred mirrors > > >>> immediately > > >>> after each build is published which is very expensive for the Pypi > > >>> project.. > > >>> People > > >>> can > > >>> still > > >>> install the nightly builds with pip by specifying the -f option. > > >>> > > >>> Uploading weekly releases to Pypi will reduce the cost for Pypi by > > >>> ~75% > > >>> [1]. It may be acceptable to Pypi, but does it make sense for us? > > >>> I'm not > > >>> convinced weekly release on Pypi is a good idea. Consider one > > >>> release is > > >>> buggy, users will need to wait for 7 days for a fix. It doesn't > > >>> provide > > >>> good user experience. > > >>> If someone has a stronger conviction about the value of weekly > > >>> releases on > > >>> Pypi, that person shall please go ahead and propose it in a > > >>> separate > > >>> discussion thread. > > >>> > > >>> Currently we don't have generally working nightly builds to Pypi > > >>> and > > >>> as a > > >>> matter of fact we know that we can't have them due to Pypi's policy > > >>> and our > > >>> apparent need for large binaries. Given this fact and that no > > >>> objection was > > >>> raised by > > >>> 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on > > >>> stopping > > >>> upload attempts of nightly builds to Pypi. > > >>> > > >>> With consensus established, we can change the CI job to stop trying > > >>> to > > >>> upload the nightly builds and then request Pypi to increase the > > >>> limit. > > >>> Then > > >>> we > > >>> have one > > >>> less blocker for the 1.6 release. > > >>> > > >>> Best regards > > >>> Leonard > > >>> > > >>> [1]: Lower cost due to less releases, but higher cost due to 500MB > > >>> -> > > >>> 800MB limit increase. Assuming that the limit increase translates > > >>> into > > >>> actually larger binaries. > > >>> > > >>> > > >>> On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote: > > >>> Are weekly releases an option? It was brought up as concern that we > > >>> might > > >>> lose pip as a pretty common distribution channel where people > > >>> consume > > >>> nightly builds. I don't feel like that concern has been properly > > >>> addressed > > >>> so far. > > >>> > > >>> -Marco > > >>> > > >>> Lausen, Leonard <lau...@amazon.com.invalid<mailto: > > >>> lau...@amazon.com.invalid><mailto: > > >>> lau...@amazon.com.invalid<mailto:lau...@amazon.com.invalid>>> > > >>> schrieb am > > >>> Mi., 4. Dez. 2019, > > >>> 04:09: > > >>> > > >>> As a simple POC to test distribution, you can try installing MXNet > > >>> based > > >>> on these 3 URLs: > > >>> > > >>> pip install --no-cache-dir > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >> > > > https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl > > >>> pip install --no-cache-dir > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >> > > > https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl > > >>> pip install --no-cache-dir > > >>> https://d19zq12jzu4w95.cloudfront.net/ > > >>> mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl > > >>> < > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >> > > > https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl > > >>> > > >>> < > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >> > > > https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl > > >>> > > >>> > > >>> where --no-cache-dir prevents caching the downloaded file, for the > > >>> purpose > > >>> of testing. (cu101 chosen based on large size) > > >>> > > >>> The first URL uses standard S3 bucket in US. The second uses > > >>> S3 > > >>> Accelerate > > >>> based > > >>> on CloudFront CDN. And the third uses CloudFront CDN. I'm adding > > >>> the > > >>> third > > >>> URL, as S3 Accelerate may or may not use all new CloudFront > > >>> endpoints yet. > > >>> > > >>> Regarding voting: Uploading to Pypi is currently impossible, which > > >>> is a > > >>> reality (so there is no option to continue as we do currently). > > >>> Pypi > > >>> folks > > >>> indicated they will unblock our uploads to Pypi once we stop > > >>> uploading > > >>> nightly releases and taking up 20% of their ressources [1]. > > >>> > > >>> If there are any shortcomings or problems identified with uploading > > >>> to S3, > > >>> we can work to address them. But for now, status quo is broken and > > >>> this > > >>> seems the only solution addressing Pypi's problem. > > >>> > > >>> I don't mind if you state that you object to lazy consensus and > > >>> start a > > >>> vote. If your "maybe [...] start a proper vote" was supposed to be > > >>> an > > >>> objection to lazy consensus, please state so clearly (I'm not sure > > >>> if > > >>> "maybe" > > >>> qualifies > > >>> as > > >>> objection). Though I think it only makes sense with at least 2 > > >>> options to > > >>> vote on. Status quo is not a meaningful option, as it is already > > >>> broken. > > >>> > > >>> Best regards > > >>> Leonard > > >>> > > >>> [1]: > > >>> > > >>> > > >>> > https://github.com/pypa/pypi-support/issues/50#issuecomment-560479706 > > >>> > > >>> On Tue, 2019-12-03 at 19:28 +0100, Marco de Abreu wrote: > > >>> Excellent! Could we maybe come up with a POC and a quick writeup > > >>> and > > >>> then > > >>> start a proper vote after everyone verified that it covers their > > >>> use-cases? > > >>> -Marco > > >>> > > >>> Sheng Zha <zhash...@apache.org<mailto:zhash...@apache.org>> > > >>> schrieb > > >>> am > > >>> Di., 3. Dez. 2019, > > >>> 19:24: > > >>> > > >>> Yes, there is. We can also make it easier to access by using a > > >>> geo-location based DNS server so that China users are directed to > > >>> that > > >>> local mirror. The rest of the world is already covered by the > > >>> global > > >>> cloudfront. > > >>> > > >>> -sz > > >>> > > >>> On 2019/12/03 18:22:22, Marco de Abreu < marco.g.ab...@gmail.com > > >>> <mailto: > > >>> marco.g.ab...@gmail.com> > > >>> > > >>> wrote: > > >>> Isn't there an s3 endpoint in Beijing? > > >>> > > >>> It seems like this topic still warrants some discussion and thus > > >>> I'd > > >>> > > >>> prefer > > >>> if we don't move forward with lazy consensus. > > >>> > > >>> -Marco > > >>> > > >>> Tao Lv <mutou...@gmail.com<mailto:mutou...@gmail.com>> schrieb am > > >>> Di., 3. > > >>> Dez. 2019, > > >>> 14:31: > > >>> > > >>> * For pypi, we can use mirrors. > > >>> > > >>> On Tue, Dec 3, 2019 at 9:28 PM Tao Lv <mutou...@gmail.com<mailto: > > >>> mutou...@gmail.com>> > > >>> wrote: > > >>> > > >>> As we have many users in China, I'm considering the accessibility > > >>> of > > >>> S3. > > >>> For pip, we can mirrors. > > >>> > > >>> On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard > > >>> > > >>> <lau...@amazon.com.invalid<mailto:lau...@amazon.com.invalid> > > >>> wrote: > > >>> > > >>> I would like to remind everyone that lazy consensus is assumed if > > >>> no > > >>> objections are raised before 2019-12-05 at 05:42 UTC. There has > > >>> been > > >>> some > > >>> > > >>> discussion > > >>> about > > >>> the proposal, but to my understanding no objections were raised. > > >>> If the proposal is accepted, MXNet releases would be installed via > > >>> pip > > >>> install mxnet > > >>> > > >>> And release candidates via > > >>> > > >>> pip install --pre mxnet > > >>> > > >>> (or with the respective cuda version specifier appended etc.) > > >>> > > >>> To obtain releases built automatically from the master branch, > > >>> users > > >>> would > > >>> need to specify something like "-f > > >>> http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html"; option to pip. > > >>> Best regards > > >>> Leonard > > >>> > > >>> On Mon, 2019-12-02 at 05:42 +0000, Lausen, Leonard wrote: > > >>> Hi MXNet Community, > > >>> > > >>> since more than 2 months our binary Python nightly releases > > >>> > > >>> published > > >>> on Pypi > > >>> are broken. The problem is that our binaries exceed Pypi's size > > >>> limit. > > >>> Decreasing the binary size by adding compression breaks > > >>> > > >>> third-party > > >>> libraries > > >>> loading libmxnet.so > > >>> > > >>> https://github.com/apache/incubator-mxnet/issues/16193 > > >>> Sheng requested Pypi to increase their size limit: > > >>> https://github.com/pypa/pypi-support/issues/50 > > >>> > > >>> Currently "the biggest cost for PyPI from [the many MXNet binaries > > >>> with > > >>> nightly release to Pypi] is the bandwidth consumed when several > > >>> hundred > > >>> mirrors attempt to mirror each release immediately after it's > > >>> published". > > >>> So Pypi is not inclined to allow us to upload even larger binaries > > >>> on a > > >>> nightly schedule. > > >>> Their compromise is to allow it on a weekly cadence. > > >>> > > >>> However, I would like the community to revisit the necessity of > > >>> releasing > > >>> pre- release binaries to Pypi on a nightly (or weekly) cadence. > > >>> > > >>> Instead, we > > >>> can > > >>> release nightly binaries ONLY to a public S3 bucket and instruct > > >>> users to > > >>> install from there. On our side, we only need to prepare a html > > >>> document > > >>> that contains links to all released nightly binaries. > > >>> Finally users will install the nightly releases via > > >>> > > >>> pip install --pre mxnet-cu101 -f > > >>> > > >>> http://mxnet.s3.amazonaws.com/mxnet-cu101/ > > >>> nightly.html > > >>> > > >>> Instead of > > >>> > > >>> pip install --pre mxnet-cu101 > > >>> > > >>> Of course proper releases and release candidates should still be > > >>> made > > >>> available via Pypi. Thus releases would be installed via > > >>> > > >>> pip install mxnet-cu101 > > >>> > > >>> And release candidates via > > >>> > > >>> pip install --pre mxnet-cu101 > > >>> > > >>> This will substantially reduce the costs of the Pypi project and in > > >>> fact > > >>> matches the installation experience provided by PyTorch. I don't > > >>> think the > > >>> benefit of not including "-f > > >>> > > >>> http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html"; > > >>> matches the costs we currently externalize to the Pypi team. > > >>> > > >>> This suggestion seems uncontroversial to me. Thus I would like to > > >>> start > > >>> lazy consensus. If there are no objections, I will assume lazy > > >>> > > >>> consensus on > > >>> stopping > > >>> nightly releases to Pypi in 72hrs. > > >>> > > >>> Best regards > > >>> Leonard > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >> > > > > >
