Hi Tao, I just talked with Aaron, lets leave the sidebar issue for later.
I created PRs in the v1.5.x branch to cherry pick the fixes into the 1.5.1 release: https://github.com/apache/incubator-mxnet/pull/16027 https://github.com/apache/incubator-mxnet/pull/16028 Thanks for your work on this release! Sam On Aug 28, 2019, at 9:35 AM, Lai Wei <[email protected]<mailto:[email protected]>> wrote: Hi, Regrading the license issue[1], we still have item 3, 4, 5 left. I think it's better to remove them from 1.5.1 release scope and target for 1.6.0 as it need more time and requires changes that should not go into patch release. [1] https://github.com/apache/incubator-mxnet/issues/15542 Best Regards Lai On Wed, Aug 28, 2019 at 9:20 AM Aaron Markham <[email protected]<mailto:[email protected]>> wrote: 5 no. Install page defaults to master so you don't need to pick it. 6 probably, but there might be other PRs needed. I'd check out the branch and attempt the install across platforms to be sure. On Wed, Aug 28, 2019, 08:55 Tao Lv <[email protected]<mailto:[email protected]>> wrote: Hi Aaron, They were proposed to be ported to v1.5.x at the beginning of the discussion but I didn't see any action for that. So I'm wondering if they're still needed. I asked for that in the last update on 8/20 but didn't get a response. If they're still needed, I hope someone who is more familiar with Julia frontend can help to cherry pick the commits to the v1.5.x branch. thanks, -tao On Wed, Aug 28, 2019 at 11:43 PM Aaron Markham < [email protected]<mailto:[email protected]>> wrote: I don't see any request for action on the Julia PRs: 5 or 6. We didn't put the change in right away because we wanted it to not break anything. But the changes are needed to make Julia setup more seamless. What "update" is needed? On Wed, Aug 28, 2019, 08:36 Tao Lv <[email protected]<mailto:[email protected]>> wrote: @Pedro, seems the issue is still open on the master branch. Do you still think we can have your fix on the 1.5.x branch? Progress since last update: 1. We received several more proposals in the github thread [1]. I humbly ask the reporters to pick the fixes to the v1.5.x. I will keep tracking the progress and the healthy status of the release branch. 2. Thanks to @Lai, the licence issue of julia cat image was fixed on the master branch and I opened a PR to pick it to v1.5.x [2]. 3. The GPU OOM issue was fixed on the master branch by @Lin [3] . But there is a problem with porting the fix to v1.5.x branch [4]. Opens: 1. https://github.com/apache/incubator-mxnet/pull/15803 still can not pass the CI; 2. Call for a update from julia folks about the back porting for [5] and [6] 3. License issue of cub and pybind is still open. @Lai opened a PR [7] to update cub submodule but seems it need more effort than just commit id update. I suspect that we cannot finish this work in 1.5.1 patch release. 4. Still no progress for the sidebar issue on web page [8]. 5. Call for a conclusion about fixing the GPU OOM issue in 1.5.1 Besides, I would like to ask if there is any preference for the release timeline of 1.5.1 patch release? Please share so I can propose the time for code freeze. Thanks, -tao [1] https://github.com/apache/incubator-mxnet/issues/15613. [2] https://github.com/apache/incubator-mxnet/pull/16026 [3] https://github.com/apache/incubator-mxnet/pull/15948 [4] https://github.com/apache/incubator-mxnet/pull/15999 [5] https://github.com/apache/incubator-mxnet/pull/15609 [6] https://github.com/apache/incubator-mxnet/pull/15608 [7] https://github.com/apache/incubator-mxnet/pull/15963 [8] https://github.com/apache/incubator-mxnet/issues/15200 On Wed, Aug 28, 2019 at 5:50 AM Pedro Larroy < [email protected]<mailto:[email protected]> wrote: Ok. I was just asking if we want this fix in 1.5.1 since it addresses crashes using multiprocessing. The problem with cherry picking is that the patch contains the dynamic load change which shouldn't impact anything else but is not supposed to go in a release branch. On Tue, Aug 27, 2019 at 1:19 PM Lin Yuan <[email protected]<mailto:[email protected]>> wrote: https://github.com/apache/incubator-mxnet/pull/15762 contains some unrelated changes which is being reverted. Please do not cherry pick it yet. On Mon, Aug 26, 2019 at 4:25 PM Pedro Larroy < [email protected]<mailto:[email protected]> wrote: There's a fix that I did which seems to still produce crashes in 1.5 for some users, which I got notice today and is fixed in master. Might be useful to put in 1.5.1: https://github.com/apache/incubator-mxnet/pull/15762 ? Pedro. On Tue, Aug 20, 2019 at 7:49 AM Tao Lv <[email protected]<mailto:[email protected]>> wrote: Hi dev, Here is an update for the 1.5.1 patch release. 1. Thanks for the effort from whole community, we have cherry picked a bunch of fixes to v1.5.x branch. So far, the branch looks healthy: http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/NightlyTestsForBinaries/activity/ 2. https://github.com/apache/incubator-mxnet/pull/15803 cannot pass the CI; 3. I hope julia folks can take a look at the back porting for https://github.com/apache/incubator-mxnet/pull/15609 and https://github.com/apache/incubator-mxnet/pull/15608 - do we still need them? 4. License issue of cub and pybind is still not fixed. We also has a license issue of a cat image in julia examples. https://github.com/apache/incubator-mxnet/issues/15542 5. Still no progress for the sidebar issue: https://github.com/apache/incubator-mxnet/issues/15200 6. There is a GPU OOM issue in 1.5.0 release and already root caused by Lin: https://github.com/apache/incubator-mxnet/issues/15703#issuecomment-522780492 . We need decide whether we want to get it fixed in the 1.5.1 patch release. Please find details in https://cwiki.apache.org/confluence/display/MXNET/1.5.1+Release+Plan+and+Status . Thanks, -tao On Mon, Aug 12, 2019 at 9:57 PM Zhao, Patric < [email protected]> wrote: Thanks for the explanation, Marco & Tao. Sounds great! -----Original Message----- From: Tao Lv <[email protected]> Sent: Monday, August 12, 2019 9:54 PM To: [email protected] Subject: Re: [Discussion] MXNet 1.5.1 release Regarding the open issue, is there default code owner/maintainer? If so, he/she will be the right people to look into the issue. https://github.com/apache/incubator-mxnet/blob/master/CODEOWNERS I have no idea. But the CODEOWNERS is used to receive change notificaitons, not actually indicates the maintainer of a piece of code. Do we have regularly build, run, functionality and performance testing for this release? As Marco mentioned, build, run and functionality of v1.5.x branch are tracked automatically by the CI for each cherry pick pull request and the nightly tests here: http://jenkins.mxnet-ci.amazon- ml.com/blue/organizations/jenkins/NightlyTestsForBinaries/activity . I see it's healthy so far. For performance, Shufan will track CPU performance with his test suite and send out the report once the branch is frozen. I'm not sure if there are any other performance tests. On Mon, Aug 12, 2019 at 9:36 PM Marco de Abreu <[email protected]> wrote: Hi Patric, CI should automatically pick up the branch and validate it as usual. Best regards, Marco Zhao, Patric <[email protected]> schrieb am Mo., 12. Aug. 2019, 15:22: It's great works, Tao 😊 Regarding the open issue, is there default code owner/maintainer? If so, he/she will be the right people to look into the issue. https://github.com/apache/incubator- mxnet/blob/master/CODEOWNERS Do we have regularly build, run, functionality and performance testing for this release? Thanks, --Patric -----Original Message----- From: Tao Lv <[email protected]> Sent: Monday, August 12, 2019 8:59 PM To: [email protected] Subject: Re: [Discussion] MXNet 1.5.1 release Update: We're cherry picking fixes from the master to the v1.5.x branch. Some of them are already merged. Please find details on the cwiki page: https://cwiki.apache.org/confluence/display/MXNET/1.5.1+Release+Pl an+a nd+Status There are still 3 opens: 1. Nightly test failure on CI ( https://github.com/apache/incubator-mxnet/issues/15374 ): The issue is still open. I'm wondering if it has been fixed or not. If not, is there anyone working on it? 2. Broken Sidebar on website API for master and 1.5.0 ( https://github.com/apache/incubator-mxnet/issues/15200 ): I don't see any progress on this issue? Do we still want to include it into 1.5.1 patch release? 3. License issues need to be fixed before 1.6 release ( https://github.com/apache/incubator-mxnet/issues/15542 ): Currently the license issue for code and images is partially fixed on the master branch and will be picked to v1.5.x soon. MKLML license issue is pushed out to 1.6 release. But license issue for cub and pybind is still open. Let me know if you any suggestion. Thanks for your support! -tao On Wed, Aug 7, 2019 at 11:03 PM Tao Lv < [email protected] wrote: Update: Thanks to wkcn's report, Issue #15774 [1] and the fix #15751 [2] are added to the scope of 1.5.1 patch release. For issue #15703 [3], I'm still waiting from the response from the reporter. Issue #15431 [4] was closed as false positive report. I also included several MKL-DNN backend issues reported by mxnet users and downstream projects. They are already fixed on the master branch. Please kindly check the full list of issues need be included in the 1.5.1 patch release: https://cwiki.apache.org/confluence/display/MXNET/1.5.1+Release+Pl an+a nd+Status For issues which are already fixed on the master branch, we will start to cherry pick the fix commit to the v1.5.x branch. For issues which are still open, we will start to track the fix process. Thanks for your great support. Let me know if you have any questions or concerns. -tao [1] https://github.com/apache/incubator-mxnet/issues/15774 [2] https://github.com/apache/incubator-mxnet/pull/15751 [3] https://github.com/apache/incubator-mxnet/issues/15703 [4] https://github.com/apache/incubator-mxnet/issues/15431 On Tue, Aug 6, 2019 at 2:04 PM Tao Lv < [email protected]> wrote: Per Sam's proposal [1], Issue #15737 [2] and the fix [3] are added to the scope of 1.5.1 patch release. A friendly reminder: the issue proposing will be closed before 11pm 8/7 CST (8am 8/7 PST). After that, we will start to cherry pick fixes to the v1.5.x branch. [1] https://github.com/apache/incubator- mxnet/issues/15613#issuecomment-5 18430120 [2] https://github.com/apache/incubator-mxnet/issues/15737 [3] https://github.com/apache/incubator-mxnet/pull/15692 On Thu, Aug 1, 2019 at 4:24 PM Tao Lv < [email protected] wrote: Hi Sandeep/Lai, Thank you for the prompt response! https://github.com/apache/incubator-mxnet/issues/15200 is added to the list to track the sidebar issue. On Thu, Aug 1, 2019 at 7:54 AM sandeep krishnamurthy < [email protected]> wrote: Thank you Tao and Shufan. Sidebar missing bug in API documentation is inconvenience for the user. It would great if we can fix it with 1.5.1 On Wed, Jul 31, 2019, 10:14 AM Lai Wei < [email protected] wrote: Hi Tao, Thank you so much for driving it. Currently nightly test on tutorials are failing and it need to be fixed. [3] I have updated the issue[1] and cwiki.[2] [1] https://github.com/apache/incubator-mxnet/issues/15613 [2] https://cwiki.apache.org/confluence/display/MXNET/1.5.1+Release+Pl a n+and+Status [3] https://github.com/apache/incubator-mxnet/issues/15374 Best Regards Lai On Wed, Jul 31, 2019 at 8:04 AM Tao Lv < [email protected]> wrote: Hi community, Thanks for the initiative from Sam (samskalicky@github ), we already have a discussion thread [1] on github about the defects and bugs exposed in the 1.5.0 release. Shufan (juliusshufan@github) and I (TaoLv@github) would like to manage the release of 1.5.1. This will be our first debut on the release process, your comments are always valuable. Per the SemVer 2.0 [2], MXNet 1.5.1 will be a patch release which contains backwards-compatible fixes only. I have created a page on cwiki [3] to track the release process and moved the issues and PRs mentioned in the github discussion thread to the page. Here I would like to ask the community to: (1) Raise any other defect or regression you identified in the 1.5.0 release. Please file a github issue for it and note the issue number in this thread; (2) Please comment with one sentence for why you think the issue is critical and must have in the 1.5.1 release; (3) If the issue is already fixed on master branch or already have a PR WIP, please also note the fix commit id or PR number; (4) If the issue is still open and there is no PR WIP, please indicate whether you'd be willing to help it out; (5) Feel free to comment if any other suggestion for the release. I suggest to keep this thread open for one week to collect enough information and proposals before we decide the timeline for the release. So your timely response will be highly appreciated! PS: Sorry to say that even as a committer, this is the first time for me to manage a release. So it would be great if an experienced committer can help to guide the process. -tao [1] https://github.com/apache/incubator-mxnet/issues/15613 [2] https://semver.org/ [3] https://cwiki.apache.org/confluence/display/MXNET/1.5.1+Release+Pl a n+and+Status
