Hi Tao, What is the current timeline for 1.5.1 release? Since it is a patch release to include only critical bug fix, would it make sense to have a short release time? I propose to have code freeze as early as next week. Please let me know if there is any other comments.
Best, Lin On Thu, Aug 29, 2019 at 3:23 PM Lin Yuan <apefor...@gmail.com> wrote: > Hi Tao, > > 5) is not a bug. It's just a large tensor support requirement. The PR was > to fix a memory alignment issue introduced in master but not in 1.5.1 > (since you did not cherry pick that PR). So, I have crossed out 5) in the > doc and I don't think we need to mention it in release note. > > Lin > > On Thu, Aug 29, 2019 at 8:12 AM Tao Lv <ta...@apache.org> wrote: > >> @Aaron, >> Thank you for looking into these two issues. I have removed the #15609 >> from >> the scope of 1.5.1. Please let me know if you have any update about >> #15608. >> >> @Lai, >> I'm fine with the decision. License issue about MKL-DNN, cub and pybind is >> moved to next release. >> >> @Sam, >> I also removed the sidebar issue [3] from the scope of 1.5.1. Besides, I >> notice one of your cherry picks is stopped by the CI. Please take a look >> at >> it. Thanks. >> >> *Nice progress since the last update:* >> 1. Per the discussion, we decided to remove #15609, the license issue >> about >> MKL-DNN, cub and pybind, and the sidebar issue [3] from the scope of 1.5.1 >> patch release; >> 2. 3 fixes [4] [5] [6] were merged into the v1.5.x branch. >> >> *Opens (suggested owners are highlighted):* >> 1. @Aaron is working on #15608 to see if we can have it in v1.5.x; >> 2. Two cherry pick PRs [7] [8] cannot pass the CI. I have pinged the >> authors to take a look at the CI failures. >> 3. @Kellen proposed 5 fixes [9] for TensorRT but till now only 3 are >> picked >> to v1.5.x. Please help to confirm if the other 2 are still needed. >> 4. Sorry that I missed the proposal for fixing the nightly build [10] in >> previous update. @Lai, can you help to confirm if it's still valid? >> 5. @Lin please help to make a conclusion for the GPU OOM issue caused by >> topk regression [11]. If it cannot be addressed on v1.5.x branch, I will >> remove it from the scope of this release and mark it as a known issue in >> the release note. >> >> Please find the details in >> >> https://cwiki.apache.org/confluence/display/MXNET/1.5.1+Release+Plan+and+Status >> . >> >> Thanks, >> -tao >> >> [1] https://github.com/apache/incubator-mxnet/pull/15609 >> [2] https://github.com/apache/incubator-mxnet/pull/15608 >> [3] https://github.com/apache/incubator-mxnet/issues/15200 >> [4] https://github.com/apache/incubator-mxnet/pull/16029 >> [5] https://github.com/apache/incubator-mxnet/pull/16026 >> [6] https://github.com/apache/incubator-mxnet/pull/16028 >> [7] https://github.com/apache/incubator-mxnet/pull/15803 >> [8] https://github.com/apache/incubator-mxnet/pull/16027 >> [9] >> >> https://github.com/apache/incubator-mxnet/issues/15613#issuecomment-520688668 >> [10] >> >> https://github.com/apache/incubator-mxnet/issues/15613#issuecomment-516937546 >> [11] https://github.com/apache/incubator-mxnet/issues/15703 >> >> >> >> On Thu, Aug 29, 2019 at 1:06 AM Skalicky, Sam <sska...@amazon.com.invalid >> > >> wrote: >> >> > Hi Tao, >> > >> > I just talked with Aaron, lets leave the sidebar issue for later. >> > >> > I created PRs in the v1.5.x branch to cherry pick the fixes into the >> 1.5.1 >> > release: >> > https://github.com/apache/incubator-mxnet/pull/16027 >> > https://github.com/apache/incubator-mxnet/pull/16028 >> > >> > Thanks for your work on this release! >> > Sam >> > >> > On Aug 28, 2019, at 9:35 AM, Lai Wei <roywei...@gmail.com<mailto: >> > roywei...@gmail.com>> wrote: >> > >> > Hi, >> > >> > Regrading the license issue[1], we still have item 3, 4, 5 left. >> > I think it's better to remove them from 1.5.1 release scope and target >> for >> > 1.6.0 as it need more time and requires changes that should not go into >> > patch release. >> > >> > >> > [1] https://github.com/apache/incubator-mxnet/issues/15542 >> > >> > Best Regards >> > >> > Lai >> > >> > >> > On Wed, Aug 28, 2019 at 9:20 AM Aaron Markham < >> aaron.s.mark...@gmail.com >> > <mailto:aaron.s.mark...@gmail.com>> >> > wrote: >> > >> > 5 no. Install page defaults to master so you don't need to pick it. >> > 6 probably, but there might be other PRs needed. I'd check out the >> branch >> > and attempt the install across platforms to be sure. >> > >> > On Wed, Aug 28, 2019, 08:55 Tao Lv <ta...@apache.org<mailto: >> > ta...@apache.org>> wrote: >> > >> > Hi Aaron, >> > >> > They were proposed to be ported to v1.5.x at the beginning of the >> > discussion but I didn't see any action for that. So I'm wondering if >> > they're still needed. I asked for that in the last update on 8/20 but >> > didn't get a response. >> > >> > If they're still needed, I hope someone who is more familiar with Julia >> > frontend can help to cherry pick the commits to the v1.5.x branch. >> > >> > thanks, >> > -tao >> > >> > On Wed, Aug 28, 2019 at 11:43 PM Aaron Markham < >> > aaron.s.mark...@gmail.com<mailto:aaron.s.mark...@gmail.com>> >> > wrote: >> > >> > I don't see any request for action on the Julia PRs: 5 or 6. >> > We didn't put the change in right away because we wanted it to not >> > break >> > anything. But the changes are needed to make Julia setup more seamless. >> > >> > What "update" is needed? >> > >> > >> > On Wed, Aug 28, 2019, 08:36 Tao Lv <ta...@apache.org<mailto: >> > ta...@apache.org>> wrote: >> > >> > @Pedro, seems the issue is still open on the master branch. Do you >> > still >> > think we can have your fix on the 1.5.x branch? >> > >> > Progress since last update: >> > 1. We received several more proposals in the github thread [1]. I >> > humbly >> > ask the reporters to pick the fixes to the v1.5.x. I will keep >> > tracking >> > the >> > progress and the healthy status of the release branch. >> > 2. Thanks to @Lai, the licence issue of julia cat image was fixed on >> > the >> > master branch and I opened a PR to pick it to v1.5.x [2]. >> > 3. The GPU OOM issue was fixed on the master branch by @Lin [3] . But >> > there >> > is a problem with porting the fix to v1.5.x branch [4]. >> > >> > Opens: >> > 1. https://github.com/apache/incubator-mxnet/pull/15803 still can >> > not >> > pass >> > the CI; >> > 2. Call for a update from julia folks about the back porting for [5] >> > and >> > [6] >> > 3. License issue of cub and pybind is still open. @Lai opened a PR >> > [7] >> > to >> > update cub submodule but seems it need more effort than just commit >> > id >> > update. I suspect that we cannot finish this work in 1.5.1 patch >> > release. >> > 4. Still no progress for the sidebar issue on web page [8]. >> > 5. Call for a conclusion about fixing the GPU OOM issue in 1.5.1 >> > >> > Besides, I would like to ask if there is any preference for the >> > release >> > timeline of 1.5.1 patch release? Please share so I can propose the >> > time >> > for >> > code freeze. >> > >> > Thanks, >> > -tao >> > >> > [1] https://github.com/apache/incubator-mxnet/issues/15613. >> > [2] https://github.com/apache/incubator-mxnet/pull/16026 >> > [3] https://github.com/apache/incubator-mxnet/pull/15948 >> > [4] https://github.com/apache/incubator-mxnet/pull/15999 >> > [5] https://github.com/apache/incubator-mxnet/pull/15609 >> > [6] https://github.com/apache/incubator-mxnet/pull/15608 >> > [7] https://github.com/apache/incubator-mxnet/pull/15963 >> > [8] https://github.com/apache/incubator-mxnet/issues/15200 >> > >> > On Wed, Aug 28, 2019 at 5:50 AM Pedro Larroy < >> > pedro.larroy.li...@gmail.com<mailto:pedro.larroy.li...@gmail.com> >> > >> > wrote: >> > >> > Ok. I was just asking if we want this fix in 1.5.1 since it >> > addresses >> > crashes using multiprocessing. The problem with cherry picking is >> > that >> > the >> > patch contains the dynamic load change which shouldn't impact >> > anything >> > else >> > but is not supposed to go in a release branch. >> > >> > On Tue, Aug 27, 2019 at 1:19 PM Lin Yuan <apefor...@gmail.com<mailto: >> > apefor...@gmail.com>> >> > wrote: >> > >> > https://github.com/apache/incubator-mxnet/pull/15762 contains >> > some >> > unrelated changes which is being reverted. Please do not cherry >> > pick >> > it >> > yet. >> > >> > On Mon, Aug 26, 2019 at 4:25 PM Pedro Larroy < >> > pedro.larroy.li...@gmail.com<mailto:pedro.larroy.li...@gmail.com> >> > >> > wrote: >> > >> > There's a fix that I did which seems to still produce crashes >> > in >> > 1.5 >> > for >> > some users, which I got notice today and is fixed in master. >> > >> > Might be useful to put in 1.5.1: >> > https://github.com/apache/incubator-mxnet/pull/15762 ? >> > >> > Pedro. >> > >> > On Tue, Aug 20, 2019 at 7:49 AM Tao Lv <ta...@apache.org<mailto: >> > ta...@apache.org>> >> > wrote: >> > >> > Hi dev, >> > >> > Here is an update for the 1.5.1 patch release. >> > >> > 1. Thanks for the effort from whole community, we have cherry >> > picked >> > a >> > bunch of fixes to v1.5.x branch. So far, the branch looks >> > healthy: >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/NightlyTestsForBinaries/activity/ >> > 2. https://github.com/apache/incubator-mxnet/pull/15803 >> > cannot >> > pass >> > the >> > CI; >> > 3. I hope julia folks can take a look at the back porting for >> > https://github.com/apache/incubator-mxnet/pull/15609 and >> > https://github.com/apache/incubator-mxnet/pull/15608 - do we >> > still >> > need >> > them? >> > 4. License issue of cub and pybind is still not fixed. We >> > also >> > has >> > a >> > license issue of a cat image in julia examples. >> > https://github.com/apache/incubator-mxnet/issues/15542 >> > 5. Still no progress for the sidebar issue: >> > https://github.com/apache/incubator-mxnet/issues/15200 >> > 6. There is a GPU OOM issue in 1.5.0 release and already root >> > caused >> > by >> > Lin: >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> https://github.com/apache/incubator-mxnet/issues/15703#issuecomment-522780492 >> > . >> > We need decide whether we want to get it fixed in the 1.5.1 >> > patch >> > release. >> > >> > Please find details in >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> https://cwiki.apache.org/confluence/display/MXNET/1.5.1+Release+Plan+and+Status >> > . >> > >> > Thanks, >> > -tao >> > >> > On Mon, Aug 12, 2019 at 9:57 PM Zhao, Patric < >> > patric.z...@intel.com> >> > wrote: >> > >> > Thanks for the explanation, Marco & Tao. Sounds great! >> > >> > -----Original Message----- >> > From: Tao Lv <ta...@apache.org> >> > Sent: Monday, August 12, 2019 9:54 PM >> > To: dev@mxnet.incubator.apache.org >> > Subject: Re: [Discussion] MXNet 1.5.1 release >> > >> > Regarding the open issue, is there default code >> > owner/maintainer? >> > If >> > so, he/she will be the right people to look into the >> > issue. >> > >> > https://github.com/apache/incubator-mxnet/blob/master/CODEOWNERS >> > >> > >> > I have no idea. But the CODEOWNERS is used to receive >> > change >> > notificaitons, >> > not actually indicates the maintainer of a piece of code. >> > >> > Do we have regularly build, run, functionality and >> > performance >> > testing >> > for >> > this release? >> > >> > >> > As Marco mentioned, build, run and functionality of >> > v1.5.x >> > branch >> > are >> > tracked >> > automatically by the CI for each cherry pick pull request >> > and >> > the >> > nightly tests >> > here: >> > http://jenkins.mxnet-ci.amazon- >> > >> > ml.com/blue/organizations/jenkins/NightlyTestsForBinaries/activity >> > . >> > I see it's healthy so far. >> > >> > For performance, Shufan will track CPU performance with >> > his >> > test >> > suite >> > and >> > send out the report once the branch is frozen. I'm not >> > sure >> > if >> > there >> > are >> > any >> > other performance tests. >> > >> > On Mon, Aug 12, 2019 at 9:36 PM Marco de Abreu >> > <marco.g.ab...@gmail.com> >> > wrote: >> > >> > Hi Patric, >> > >> > CI should automatically pick up the branch and validate >> > it >> > as >> > usual. >> > >> > Best regards, >> > Marco >> > >> > Zhao, Patric <patric.z...@intel.com> schrieb am Mo., >> > 12. >> > Aug. >> > 2019, >> > 15:22: >> > >> > It's great works, Tao 😊 >> > >> > Regarding the open issue, is there default code >> > owner/maintainer? >> > If >> > so, he/she will be the right people to look into the >> > issue. >> > https://github.com/apache/incubator- >> > mxnet/blob/master/CODEOWNERS >> > >> > Do we have regularly build, run, functionality and >> > performance >> > testing >> > for >> > this release? >> > >> > Thanks, >> > >> > --Patric >> > >> > -----Original Message----- >> > From: Tao Lv <ta...@apache.org> >> > Sent: Monday, August 12, 2019 8:59 PM >> > To: dev@mxnet.incubator.apache.org >> > Subject: Re: [Discussion] MXNet 1.5.1 release >> > >> > Update: >> > >> > We're cherry picking fixes from the master to the >> > v1.5.x >> > branch. >> > Some >> > of >> > them are already merged. Please find details on the >> > cwiki >> > page: >> > >> > >> > https://cwiki.apache.org/confluence/display/MXNET/1.5.1+Release+Pl >> > an+a >> > nd+Status >> > >> > >> > There are still 3 opens: >> > 1. Nightly test failure on CI ( >> > >> > https://github.com/apache/incubator-mxnet/issues/15374 >> > ): >> > The >> > issue >> > is >> > still >> > open. I'm wondering if it has been fixed or not. If >> > not, >> > is >> > there >> > anyone >> > working on it? >> > 2. Broken Sidebar on website API for master and >> > 1.5.0 ( >> > >> > https://github.com/apache/incubator-mxnet/issues/15200 >> > ): >> > I >> > don't >> > see >> > any >> > progress on this issue? Do we still want to include >> > it >> > into >> > 1.5.1 >> > patch >> > release? >> > 3. License issues need to be fixed before 1.6 >> > release ( >> > >> > https://github.com/apache/incubator-mxnet/issues/15542 >> > ): >> > Currently >> > the license issue for code and images is partially >> > fixed >> > on >> > the >> > master >> > branch and >> > will be picked to v1.5.x soon. MKLML license issue >> > is >> > pushed >> > out >> > to 1.6 release. But license issue for cub and >> > pybind >> > is >> > still >> > open. >> > >> > Let me know if you any suggestion. Thanks for your >> > support! >> > >> > -tao >> > >> > >> > On Wed, Aug 7, 2019 at 11:03 PM Tao Lv < >> > ta...@apache.org >> > >> > wrote: >> > >> > >> > Update: >> > >> > Thanks to wkcn's report, Issue #15774 [1] and the >> > fix >> > #15751 >> > [2] >> > are added to the scope of 1.5.1 patch release. >> > For issue #15703 [3], I'm still waiting from the >> > response >> > from >> > the reporter. >> > Issue #15431 [4] was closed as false positive >> > report. >> > I also included several MKL-DNN backend issues >> > reported >> > by >> > mxnet >> > users >> > and downstream projects. They are already fixed >> > on >> > the >> > master >> > branch. >> > >> > Please kindly check the full list of issues need >> > be >> > included >> > in >> > the >> > 1.5.1 patch release: >> > >> > >> > >> > https://cwiki.apache.org/confluence/display/MXNET/1.5.1+Release+Pl >> > an+a >> > nd+Status >> > >> > For issues which are already fixed on the master >> > branch, >> > we >> > will >> > start >> > to cherry pick the fix commit to the v1.5.x >> > branch. >> > For >> > issues >> > which are still open, we will start to track the >> > fix >> > process. >> > >> > Thanks for your great support. Let me know if you >> > have >> > any >> > questions or concerns. >> > >> > -tao >> > >> > [1] >> > https://github.com/apache/incubator-mxnet/issues/15774 >> > [2] >> > https://github.com/apache/incubator-mxnet/pull/15751 >> > [3] >> > https://github.com/apache/incubator-mxnet/issues/15703 >> > [4] >> > https://github.com/apache/incubator-mxnet/issues/15431 >> > >> > >> > On Tue, Aug 6, 2019 at 2:04 PM Tao Lv < >> > ta...@apache.org> >> > wrote: >> > >> > >> > Per Sam's proposal [1], Issue #15737 [2] and the >> > fix >> > [3] >> > are >> > added >> > to >> > the scope of 1.5.1 patch release. >> > >> > A friendly reminder: the issue proposing will be >> > closed >> > before >> > 11pm >> > 8/7 CST (8am 8/7 PST). After that, we will start >> > to >> > cherry >> > pick >> > fixes >> > to the v1.5.x branch. >> > >> > >> > [1] >> > https://github.com/apache/incubator- >> > mxnet/issues/15613#issuecomment-5 >> > 18430120 [2] >> > >> > https://github.com/apache/incubator-mxnet/issues/15737 >> > [3] >> > https://github.com/apache/incubator-mxnet/pull/15692 >> > >> > On Thu, Aug 1, 2019 at 4:24 PM Tao Lv < >> > ta...@apache.org >> > >> > wrote: >> > >> > Hi Sandeep/Lai, >> > >> > Thank you for the prompt response! >> > >> > >> > https://github.com/apache/incubator-mxnet/issues/15200 >> > is >> > added >> > to >> > the list to track the sidebar issue. >> > >> > On Thu, Aug 1, 2019 at 7:54 AM sandeep >> > krishnamurthy >> > < >> > sandeep.krishn...@gmail.com> wrote: >> > >> > Thank you Tao and Shufan. >> > Sidebar missing bug in API documentation is >> > inconvenience >> > for >> > the >> > user. >> > It >> > would great if we can fix it with 1.5.1 >> > >> > On Wed, Jul 31, 2019, 10:14 AM Lai Wei < >> > roywei...@gmail.com >> > >> > wrote: >> > >> > Hi Tao, >> > >> > Thank you so much for driving it. Currently >> > nightly >> > test >> > on >> > tutorials are >> > failing and it need to be fixed. [3] I have >> > updated >> > the >> > issue[1] and cwiki.[2] >> > >> > [1] >> > https://github.com/apache/incubator-mxnet/issues/15613 >> > [2] >> > >> > >> > >> > >> > >> > https://cwiki.apache.org/confluence/display/MXNET/1.5.1+Release+Pl >> > a >> > n+and+Status >> > [3] >> > https://github.com/apache/incubator-mxnet/issues/15374 >> > >> > Best Regards >> > >> > Lai >> > >> > >> > On Wed, Jul 31, 2019 at 8:04 AM Tao Lv < >> > ta...@apache.org> >> > wrote: >> > >> > Hi community, >> > >> > >> > >> > Thanks for the initiative from Sam >> > (samskalicky@github >> > ), >> > we already >> > have a >> > discussion thread [1] on github about the >> > defects >> > and >> > bugs exposed >> > in the >> > 1.5.0 release. >> > >> > Shufan (juliusshufan@github) and I >> > (TaoLv@github) >> > would >> > like >> > to >> > manage >> > the >> > release of 1.5.1. This will be our first >> > debut >> > on >> > the >> > release >> > process, >> > your >> > comments are always valuable. >> > >> > >> > >> > Per the SemVer 2.0 [2], MXNet 1.5.1 will >> > be >> > a >> > patch >> > release which >> > contains >> > backwards-compatible fixes only. >> > >> > I have created a page on cwiki [3] to >> > track >> > the >> > release >> > process >> > and >> > moved >> > the issues and PRs mentioned in the github >> > discussion >> > thread >> > to >> > the >> > page. >> > >> > >> > >> > Here I would like to ask the community to: >> > >> > (1) Raise any other defect or regression >> > you >> > identified >> > in the >> > 1.5.0 release. Please file a github issue >> > for >> > it >> > and >> > note >> > the issue >> > number in >> > this thread; >> > >> > (2) Please comment with one sentence for >> > why >> > you >> > think >> > the issue is critical and must have in the >> > 1.5.1 >> > release; >> > >> > (3) If the issue is already fixed on >> > master >> > branch >> > or >> > already have >> > a PR >> > WIP, please also note the fix commit id or >> > PR >> > number; >> > >> > (4) If the issue is still open and there >> > is >> > no >> > PR >> > WIP, >> > please >> > indicate >> > whether you'd be willing to help it out; >> > >> > (5) Feel free to comment if any other >> > suggestion >> > for >> > the >> > release. >> > >> > >> > >> > I suggest to keep this thread open for one >> > week >> > to >> > collect enough information and proposals >> > before >> > we >> > decide >> > the timeline for the >> > release. >> > So >> > your timely response will be highly >> > appreciated! >> > >> > >> > >> > PS: Sorry to say that even as a committer, >> > this >> > is >> > the >> > first time >> > for me >> > to >> > manage a release. So it would be great if >> > an >> > experienced >> > committer >> > can >> > help >> > to guide the process. >> > >> > >> > >> > -tao >> > >> > >> > >> > [1] >> > >> > https://github.com/apache/incubator-mxnet/issues/15613 >> > >> > [2] https://semver.org/ >> > >> > [3] >> > >> > >> > >> > >> > >> > >> > https://cwiki.apache.org/confluence/display/MXNET/1.5.1+Release+Pl >> > a >> > n+and+Status >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> >
