Re: [apache/incubator-mxnet] [RFC] Double dependency for ONNX (#18824)

[Sheng Zha]

Can we drop onnx-tensorrt in favor of native tensorrt integration? It seems 
perfectly ok to run that outside of MXNet as it's more of an ONNX feature.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/18824#issuecomment-666989953

Re: assimilation of mshadow into the MXNet codebase

[Sheng Zha]

Hi,

Here's the second update. At the moment we are only missing ICLAs from 20
(out of 70) contributors, accounting for 31 (out of 913) commits left.

3 @zhenlinluo
3 @jpauwels
3 @hjk41
3 @DrustZ
2 @zhangchen-qinyinghua
2 @yinghu5
2 @reyoung
1 @xinyu-intel
1 @xingmingjie
1 @qiaohaijun
1 @loveisp
1 @lebeg
1 @kaleidoscopical
1 @jason-xuan
1 @happynear
1 @glingyan
1 @asitstands
1 @antoine-wdg-rmz
1 @alextnewman
1 @Harmonicahappy

Regarding whether the contributors are employed by a company that requires
CCLA for contribution, I have no way of verifying the contributors'
employment status at the time of contribution, and not enough bandwidth to
verify the individual company policies on such contribution. As such, I
will solely rely on the ICLAs.

Given the current status, I think the rest of the 31 commits is manageable
for me even if I end up having to revert and rework all of them. Let me
know if you have any concern on starting the IP clearance process.
Otherwise I think we can start it on general@incubator soon.

Cheers,
Sheng

On Sun, Jul 26, 2020 at 8:59 PM Justin Mclean 
wrote:

> Hi,
>
> In the case of Intel and other companies, it may be that their employee
> contracts do not allow employees to contribute to OS projects. It more
> likely that the contributor doesn’t own copyright of the code but their
> employer does. A CCLA give a clear indication that the contributors are
> intact allowed to contrive code and own the copyright of their
> contributions. We have CCLAs from Intel on file from other contributions so
> it would seem that Intel requires this.
>
> Thanks,
> Justin

Re: Distribution of release candidates

[Sheng Zha]

Hi Justin,

Please excuse the delay. I replied on general@ [1]. I suggest that we
continue the discussion in that thread and not fork the discussion.

Regards,
Sheng

[1]
https://lists.apache.org/thread.html/r149bb0721c40f145d3a6a9c847f0172cffa5ec78f84d1ee775b4103c%40%3Cgeneral.incubator.apache.org%3E

On Sun, Jul 26, 2020 at 7:43 PM Justin Mclean  wrote:

> Hi,
>
> I asked on the incubator vote but didn't get a reply.
>
> Can the PPMC please explain why release candidates are being released in
> this way, in particular by companies who employee MXnet PPMC members.
>
> What steps will the PPMC will take to stop this from happening in the
> future?
>
> Thanks,
> Justin
>
>

Re: assimilation of mshadow into the MXNet codebase

[Sheng Zha]

Justin,

Are you OK with proceeding?

Regards,
Sheng

On Sun, Jul 26, 2020 at 8:30 PM Tianqi Chen 
wrote:

> As long as we have CLA covering for the majority of the code(which I
> believe so), I think we should be good.
> Just like the case of Apache only requires iCLA from committers.
>
> The rationale is that normal contributions are already in the form of ALv2,
> in the case of a(unlikely) dispute, the community can quickly rewrite the
> code(since that is non-majority).
>
> TQ
>
> On Sun, Jul 26, 2020 at 5:49 PM Sheng Zha  wrote:
>
> > Hi Justin,
> >
> > Thanks, that's a good point. I think we have already received CCLA from
> > Intel. I will take that into account when providing the next update.
> >
> > Regards,
> > Sheng
> >
> > On Sun, Jul 26, 2020 at 5:39 PM Justin Mclean 
> > wrote:
> >
> > > Hi,
> > >
> > > > Several peoples in below list are from Intel and I have added them
> into
> > > CC.
> > >
> > > Has Intel signed a CCLA? And if so does it list people who are allowed
> to
> > > contribute to this project? Are there any others on that list who
> > > employer’s may need to also sign CCLAs if we don’t have them?
> > >
> > > Thanks,
> > > Justin
> >
>

Re: assimilation of mshadow into the MXNet codebase

[Sheng Zha]

Hi Justin,

Thanks, that's a good point. I think we have already received CCLA from
Intel. I will take that into account when providing the next update.

Regards,
Sheng

On Sun, Jul 26, 2020 at 5:39 PM Justin Mclean 
wrote:

> Hi,
>
> > Several peoples in below list are from Intel and I have added them into
> CC.
>
> Has Intel signed a CCLA? And if so does it list people who are allowed to
> contribute to this project? Are there any others on that list who
> employer’s may need to also sign CCLAs if we don’t have them?
>
> Thanks,
> Justin

Re: assimilation of mshadow into the MXNet codebase

[Sheng Zha]

Hi,

Here's an update on this issue. We are still missing the ICLAs from 32 (out
of 70) mshadow contributors, accounting for a total of 62 (out of 913)
commits. (@ap-hynninen passed away a few years ago and is not included). I
reached out to them through email and other channels to collect ICLA for
mshadow. I will wait for a day or two before updating on the progress
again, and we can decide then whether we are good to start the IP clearance.

The complete list of mshadow contributors' GitHub logins that are missing
ICLA is here ("#commits @github-login"):

8 @Lorrainexun
6 @tornadomeet
5 @asmushetzel
3 @zhenlinluo
3 @stefanhenneking
3 @jpauwels
3 @hjk41
3 @DrustZ
2 @zhangchen-qinyinghua
2 @yinghu5
2 @reyoung
2 @forwchen
1 @yupbank
1 @yllan
1 @xinyu-intel
1 @xingmingjie
1 @xianyi
1 @tdomhan
1 @siemanko
1 @qiaohaijun
1 @maxint
1 @loveisp
1 @lebeg
1 @kdavis-mozilla
1 @kaleidoscopical
1 @jason-xuan
1 @happynear
1 @glingyan
1 @asitstands
1 @antoine-wdg-rmz
1 @alextnewman
1 @Harmonicahappy

Best,
Sheng

On Thu, Jul 23, 2020 at 12:28 AM Sheng Zha  wrote:

> Hi,
>
> No, I don’t think we used ICLAs for mshadow before.
>
> Out of the 42 people who made more than 1 commit or more than 10 lines of
> code change to mshadow, 26 signed ICLA with Apache (and additionally one
> member is unfortunately deceased...). Would this be a better criteria as
> “the major ones”? I wasn’t part of the initial code donation or the initial
> PPMC group, so apologies if the questions were silly.
>
> I think the rest of the commits are manageable so that I could do a revert
> and rework for those commits if/when necessary.
>
> Regards,
> Sheng
>
> > On Jul 22, 2020, at 11:50 PM, Justin Mclean 
> wrote:
> >
> > Hi,
> >
> >> Thanks for clarifying. All contributors who made more than 10 commits
> to msahdow before are committers of MXNet, so their ICLAs should already be
> on file: tqchen, bingxu, eric.xie, sxjscience, mli, yajiedesign [1]. If you
> think this is OK, one of the mentors or I can start the notification.
> >
> >
> > What about the other 60 contributors? More than 10 commits is not a line
> I would feel comfortable with. You need to be able to account for the IP
> provenance of every line of code, just like in your initial code donation.
> It would probably be best to make a list all contributors and if they have
> an ICLA or not. Did the mshadow project use ICLAs? If so that may also help.
> >
> > Thanks,
> > Justin
>

Re: [apache/incubator-mxnet] [RFC] MXNet 2.0 JVM Language development (#17783)

[Sheng Zha]

@saudet this looks awesome! An 18% improvement in throughput is quite 
significant for switching the way of integration for a frontend binding. I 
think we should definitely start with this offering. @lanking520 @gigasquid 
what do you think?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/17783#issuecomment-663064169

Re: assimilation of mshadow into the MXNet codebase

[Sheng Zha]

Hi,

No, I don’t think we used ICLAs for mshadow before.

Out of the 42 people who made more than 1 commit or more than 10 lines of code 
change to mshadow, 26 signed ICLA with Apache (and additionally one member is 
unfortunately deceased...). Would this be a better criteria as “the major 
ones”? I wasn’t part of the initial code donation or the initial PPMC group, so 
apologies if the questions were silly.

I think the rest of the commits are manageable so that I could do a revert and 
rework for those commits if/when necessary.

Regards,
Sheng

> On Jul 22, 2020, at 11:50 PM, Justin Mclean  wrote:
> 
> Hi,
> 
>> Thanks for clarifying. All contributors who made more than 10 commits to 
>> msahdow before are committers of MXNet, so their ICLAs should already be on 
>> file: tqchen, bingxu, eric.xie, sxjscience, mli, yajiedesign [1]. If you 
>> think this is OK, one of the mentors or I can start the notification.
> 
> 
> What about the other 60 contributors? More than 10 commits is not a line I 
> would feel comfortable with. You need to be able to account for the IP 
> provenance of every line of code, just like in your initial code donation. It 
> would probably be best to make a list all contributors and if they have an 
> ICLA or not. Did the mshadow project use ICLAs? If so that may also help.
> 
> Thanks,
> Justin

Re: assimilation of mshadow into the MXNet codebase

[Sheng Zha]

Thanks for clarifying. All contributors who made more than 10 commits to
msahdow before are committers of MXNet, so their ICLAs should already be on
file: tqchen, bingxu, eric.xie, sxjscience, mli, yajiedesign [1]. If you
think this is OK, one of the mentors or I can start the notification.

Regards,
Sheng

[1] https://github.com/dmlc/mshadow/graphs/contributors

On Wed, Jul 22, 2020 at 10:37 PM Justin Mclean 
wrote:

> Hi,
>
> > Thank you, Justin. Though I’m still uncertain about what the definition
> of IP clearance process is.
>
> The bit you quoted there is for an initial code base, it the second part
> of that document you need to look at.
>
> In short as well as the SGA you need to get signed ICLA from all of the
> contributors to the code base. It might be OK to just get the major ones
> depending on the type of contributions. You then need to notify the
> incubator of the IP clearance and see if they have any questions about it.
>
> Here’s an example:
>
> https://lists.apache.org/thread.html/r750880f7295c1a8c31c99e7a40f3466c177bd714254d0c98a506dede%40%3Cgeneral.incubator.apache.org%3E
>
> Thanks,
> Justin

Re: assimilation of mshadow into the MXNet codebase

[Sheng Zha]

Thank you, Justin. Though I’m still uncertain about what the definition of IP 
clearance process is, I found the following paragraphs that seem relevant. 
Sounds like we need three votes from our mentors here for this acceptance. If 
that’s the case, I can start a vote on it.

Regards,
Sheng

> The Incubator PMC must approve the clearance. This indicates that the project 
> is happy to receive the code donated. When a new podling is created, this is 
> done by the identification of existing codebases in the proposal. Otherwise, 
> the IPMC delegates this decision to the PPMC.
> As usual, three binding votes are required. So, Mentors need to be involved 
> in IP clearance for podlings. If too few binding VOTEs are posted on list, 
> the VOTE will need to be posted to the general list for ratification.


> On Jul 22, 2020, at 6:31 PM, Justin Mclean  wrote:
> 
> Hi,
> 
> See also:
> https://incubator.apache.org/guides/ip_clearance.html
> 
> Thanks,
> Justin

Re: assimilation of mshadow into the MXNet codebase

[Sheng Zha]

Hi Justin,

Yes and yes. I filed the software grant and received confirmation from 
secretary@.

I’m not sure if I should be updating the page, and if so, how.

Regards,
Sheng

> On Jul 22, 2020, at 1:59 AM, Justin Mclean  wrote:
> 
> Hi,
> 
> Has the IP clearance process been followed? I don't see it listed on this 
> page [1]
> 
> Does the current release being voted on contain this code?
> 
> Thanks,
> Justin
> 
> 1. https://incubator.apache.org/ip-clearance/

Re: [apache/incubator-mxnet] [RFC] Apache MXNet 2.0 Roadmap (#16167)

[Sheng Zha]

@fhieber we are planning to release the first public beta on this somewhere in 
August. At the moment we are finalizing some API changes and also validating 
them in GluonNLP. We will publish a transition doc as part of the public beta.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/16167#issuecomment-662620865

Re: [DISCUSS] Examples Repo for 2.0

[Sheng Zha]

I created the repository at https://github.com/apache/incubator-mxnet-examples. 
Next, we will need to set up CI and move some examples there. I'm currently 
occupied and may not get to it before next month. If someone volunteers to help 
on this I'd appreciate it.

Best,
Sheng

On 2020/07/17 06:03:19, Chaitanya Bapat  wrote: 
> Testing nightly is the perfect middle. Daily/per commit would be too much
> and Never would be too less.
> 
> +1 for the proposal!
> 
> On Thu, 16 Jul 2020 at 02:08, Kshitij Kalambarkar <
> kshitijkalambar...@gmail.com> wrote:
> 
> > +1 Great Idea! It is quite useful to have working examples to refer to.
> >
> > On Thu, Jul 16, 2020 at 12:17 PM Marco de Abreu 
> > wrote:
> >
> > > +1 good idea!
> > >
> > > On Thu, Jul 16, 2020, 5:39 AM Skalicky, Sam 
> > > wrote:
> > >
> > > > +1 For regular testing, enhanced doc/tutorial
> > > >
> > > > > On Jul 15, 2020, at 7:40 PM, Sheng Zha  wrote:
> > > > >
> > > > > CAUTION: This email originated from outside of the organization. Do
> > > not
> > > > click links or open attachments unless you can confirm the sender and
> > > know
> > > > the content is safe.
> > > > >
> > > > >
> > > > >
> > > > > Hi,
> > > > >
> > > > > Over the years, MXNet accumulated many examples in the example folder
> > > > [1]. However, due to not testing them in the CI and in releases, many
> > of
> > > > them are currently broken. I'd like to propose that we create a new
> > > > examples repo for MXNet 2.0 similar to how PyTorch hosts them [2].
> > > Further
> > > > more, the new examples repo should be evaluated periodically (e.g.
> > > weekly)
> > > > against the master branch to ensure that they are not broken.
> > > > >
> > > > > Thoughts and comments are welcome.
> > > > >
> > > > > Sheng
> > > > >
> > > > > [1] https://github.com/apache/incubator-mxnet/tree/master/example
> > > > > [2] https://github.com/pytorch/examples
> > > >
> > >
> >
> 
> 
> -- 
> *Chaitanya Prakash Bapat*
> *+1 (973) 953-6299*
> 
> [image: https://www.linkedin.com//in/chaibapat25]
> <https://github.com/ChaiBapchya>[image: https://www.facebook.com/chaibapat]
> <https://www.facebook.com/chaibapchya>[image:
> https://twitter.com/ChaiBapchya] <https://twitter.com/ChaiBapchya>[image:
> https://www.linkedin.com//in/chaibapat25]
> <https://www.linkedin.com//in/chaibapchya/>
> 

Re: [VOTE] Release Apache MXNet (incubating) version 1.7.0.rc1

[Sheng Zha]

+1. I checked:

[x] Are release files in correct location? Yes
[x] Do release files have the word incubating in their name? Yes
[x] Are the digital signature and hashes correct? Yes
[x] Does DISCLAIMER file exist? Yes, DISCLAIMER-WIP
[x] Do LICENSE and NOTICE files exists? Yes
[x] Is the LICENSE and NOTICE text correct?
Yes, though the license still reads "Copyright [] [name of copyright 
owner]", which needs correction.

[x] Is the NOTICE year correct? Yes
[x] Un-included software dependencies are not mentioned in LICENSE or NOTICE?
No. mshadow is now contributed to MXNet via software grant and should be 
removed from NOTICE.

[x] License information is not mentioned in NOTICE? Confirmed

Is there any 3rd party code contained inside the release? If so:
[x] Does the software have a compatible license? Yes. Minor issue:
Dual license in cmake/Modules/FindJeMalloc.cmake.

[x] Are all software licenses mentioned in LICENSE? Yes
[x] Is the full text of the licenses (or pointers to it) in LICENSE? Yes

Is any of this code Apache licensed? Do they have NOTICE files? If so:
[x] Have relevant parts of those NOTICE files been added to this NOTICE file?
No. TVM NOTICE file hasn't been included.

[x] Do all source files have ASF headers?
Yes, except those in 3rdparty folder and those mentioned in license.
[x] Do the contents of the release match with what's tagged in version control? 
Yes
[x] Are there any unexpected binary files in the release? No
[x] Can you compile from source? Are the instruction clear? Yes, Makefile is 
present and is straightforward.
Is the issue minor? Yes
Could it possibly be fixed in the next release? Yes

I vote with:
[x] +1 release the software


On 2020/07/20 17:25:50, "Skalicky, Sam"  wrote: 
> +1
> 
> Tested:
> - Make flow building from source, verified all example/extensions/* work 
> correctly
> - staticbuild flow cpu & cu102 variants producing the pip wheels, tested with 
> custom extension library
> 
> Sam
> 
> On 7/20/20, 4:07 AM, "Chen, Ciyong"  wrote:
> 
> CAUTION: This email originated from outside of the organization. Do not 
> click links or open attachments unless you can confirm the sender and know 
> the content is safe.
> 
> 
> 
> Thanks Aston, Patric for the vote.
> 
> Hi Community,
> 
> I would like to call for action to test/validate/vote for the release 
> candidate (1.7.0.rc1).
> As we've not reached the quorum, I would like to extend the voting 
> process to July 22, 23:59:59 PST.
> Please prepare your time and provide feedback if you've tried with the 
> pre-released code base, thanks!
> 
> Best Regards,
> Ciyong
> 
> -Original Message-
> From: Zhao, Patric 
> Sent: Monday, July 20, 2020 11:36 AM
> To: dev@mxnet.incubator.apache.org
> Cc: d...@mxnet.apache.org; Bob Paulin ; Henri Yandell 
> ; Jason Dai ; Markus Weimer 
> ; Michael Wall 
> Subject: RE: [VOTE] Release Apache MXNet (incubating) version 1.7.0.rc1
> 
> +1
> 
> Passed the performance benchmarking for CPU tests and no regression is 
> found.
> 
> 
> > -Original Message-
> > From: Aston Zhang 
> > Sent: Sunday, July 19, 2020 1:45 PM
> > To: dev@mxnet.incubator.apache.org
> > Cc: d...@mxnet.apache.org; Bob Paulin ; Henri Yandell
> > ; Jason Dai ; Markus Weimer
> > ; Michael Wall 
> > Subject: Re: [VOTE] Release Apache MXNet (incubating) version
> > 1.7.0.rc1
> >
> > +1
> > Passed d2l-en v0.14.1:
> > https://github.com/d2l-ai/d2l-en/releases/tag/v0.14.1
> >
> > On Thu, Jul 16, 2020 at 2:34 AM Chen, Ciyong  
> wrote:
> >
> > > Dear MXNet community,
> > >
> > > This is the vote to release Apache MXNet (incubating) version 1.7.0.
> > > Voting will start 16th July 23:59:59 PST and close on 19th July
> > > 23:59:59 PST.
> > >
> > > Link to release notes:
> > > https://cwiki.apache.org/confluence/display/MXNET/1.7.0+Release+note
> > > s
> > >
> > > Link to release candidate:
> > > https://github.com/apache/incubator-mxnet/releases/tag/1.7.0.rc1
> > >
> > > Link to source and signatures on apache dist server:
> > > https://dist.apache.org/repos/dist/dev/incubator/mxnet/1.7.0.rc1
> > >
> > > Please remember to TEST first before voting accordingly:
> > > +1 = approve
> > > +0 = no opinion
> > > -1 = disapprove (provide reason)
> > >
> > > Here's the changes comparing to 1.7.0.rc0:
> > >
> > >   *   Revert "Fix memory leaks in Gluon (#18328) (#18358) (#18692)
> > >   *   revise activations (#18700)
> > >   *   Fix the monitor_callback invalid issue during calibration with
> > > variable input shapes (#18632) (#18703)
> > >
> > >
> > > Best regards,
> > > Ciyong Chen
> > >
> 
> 

Re: [apache/incubator-mxnet] [RFC] Use TVMOp with GPU & Build without libcuda.so in CI (#18716)

[Sheng Zha]

Instead of linking tvm to mxnet, can we use TVM to generate source code and 
test as custom c++ operator?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/18716#issuecomment-659126976

[DISCUSS] Examples Repo for 2.0

[Sheng Zha]

Hi,

Over the years, MXNet accumulated many examples in the example folder [1]. 
However, due to not testing them in the CI and in releases, many of them are 
currently broken. I'd like to propose that we create a new examples repo for 
MXNet 2.0 similar to how PyTorch hosts them [2]. Further more, the new examples 
repo should be evaluated periodically (e.g. weekly) against the master branch 
to ensure that they are not broken.

Thoughts and comments are welcome.

Sheng

[1] https://github.com/apache/incubator-mxnet/tree/master/example
[2] https://github.com/pytorch/examples

Re: [VOTE] Release Apache MXNet (incubating) version 1.7.0.rc0

[Sheng Zha]

Hi Marco,

Since the license issues apply to binary distribution, we should still be able 
to make official source releases.

Regards,
Sheng

> On Jul 12, 2020, at 1:10 AM, Marco de Abreu  wrote:
> 
> Are we in the position to make a release given that we have open license
> issues with the ipmc and Apache board? I want to avoid giving the
> impression that we are ignoring their requests - my current understanding
> is that we are non compliant.
> 
> -Marco
> 
>> On Sat, Jul 11, 2020, 9:46 AM Tong He  wrote:
>> 
>> My +1 on the R binding.
>> 
>> Tested with
>> 
>> - Build from source
>> - Install the R package and check it passed all tests.
>> 
>>> On 2020/07/10 18:31:27, Patrick Mu  wrote:
>>> Hi Ciyong,
>>> 
>>> I just discovered an issue with the 1.7, which causes the Yolo training
>> with latest Gluon CV Yolo to fail.
>>> 
>>> The PR that causes the failure is
>> https://github.com/apache/incubator-mxnet/pull/18358, which modifies
>> basic blocks of Gluon to fix a memory leak issue.
>>> 
>>> Talked with Leonard, the author of the PR, and he said he found the root
>> cause, but patching that PR would modifies those Gluon basic blocks
>> further, which might be risky towards existing models and various customer
>> models.
>>> 
>>> So my 2-cents is reverting this PR in 1.7, and try patching the PR in
>> 1.x and 2.0, meaning that the 1.7 won't have memory usage optimized by that
>> feature.
>>> 
>>> I'd like to hear what you think about this issue.
>>> 
>>> Thanks,
>>> Ziyi
>>> 
>>> 
>>> On 2020/07/10 06:18:02, "Chen, Ciyong"  wrote:
 Hi Community,
 
 I would like to call for action to test/validate/vote for the release
>> candidate (1.7.0.rc0)
 As there's not any voting result during the scheduled time window, I
>> would like to extend the time windows to July 13, 23:59:59 PST.
 Please prepare your time and provide feedback if you've tried with the
>> pre-release code bases, thanks!
 
 Best regards,
 Ciyong
 
 -Original Message-
 From: Chen, Ciyong 
 Sent: Monday, July 6, 2020 10:48 PM
 To: d...@mxnet.apache.org
 Cc: Bob Paulin ; Henri Yandell ;
>> Jason Dai ; Markus Weimer ;
>> Michael Wall 
 Subject: RE: [VOTE] Release Apache MXNet (incubating) version 1.7.0.rc0
 
 For the language bindings and windows platform, may I have your
>> support to help verify these features? Thanks!
 
 @lanking520 to help verify the Scala/Java @gigasquid to help verify
>> the Clojure
 @hetong007 to help verify the R
 @yajiedesign to help verify the windows platform
 
 Best regards,
 Ciyong Chen
 
 -Original Message-
 From: Chen, Ciyong 
 Sent: Monday, July 6, 2020 10:39 PM
 To: d...@mxnet.apache.org
 Cc: Bob Paulin ; Henri Yandell ;
>> Jason Dai ; Markus Weimer ;
>> Michael Wall 
 Subject: [VOTE] Release Apache MXNet (incubating) version 1.7.0.rc0
 
 Dear MXNet community,
 
 This is the vote to release Apache MXNet (incubating) version 1.7.0.
>> Voting will start July 6, 23:59:59 PST and close on July 9, 23:59:59 PST.
 
 Link to release notes:
 https://cwiki.apache.org/confluence/display/MXNET/1.7.0+Release+notes
 
 Link to release candidate:
 https://github.com/apache/incubator-mxnet/releases/tag/1.7.0.rc0
 
 Link to source and signatures on apache dist server:
 https://dist.apache.org/repos/dist/dev/incubator/mxnet/1.7.0.rc0<
>> https://dist.apache.org/repos/dist/dev/incubator/mxnet/1.7.0.rc0/>
 
 Please remember to TEST first before voting accordingly:
 +1 = approve
 +0 = no opinion
 -1 = disapprove (provide reason)
 
 Additional notes:
 
  *   There was an issue and discussion[1] regarding on a few numpy
>> operators failed due to numpy 1.19.0 released on Jun 20, 2020, which exists
>> in all branches (works with numpy <= 1.18.5). As numpy operator is still an
>> experimental feature in 1.7.0 release and mainly targeting in MXNet 2.0
>> release, so I decided to not block the voting and instead let the Community
>> decide whether this is a blocker for the release.
 
 [1] https://github.com/apache/incubator-mxnet/issues/18600
 
 Best regards,
 Ciyong Chen
 
 
>>> 
>> 

Re: Access to MXNet Slack Channel for joshr-mx...@joshr.com

[Sheng Zha]

Invite sent. Welcome!

-sz

On 2020/07/08 13:13:58, Josh Rabinowitz  wrote: 
> Hello,
> 
> Can you please allow access for the email
> 
>joshr-mx...@joshr.com
>(which is my email address, but not the email address this email comes
> from)
> 
> to the MXNet Slack Group?
> 
> Thank you
> 
> Josh
> 

Re: Joining on Slack

[Sheng Zha]

Invite sent. Welcome!

-sz

On 2020/07/05 05:35:54, Veesh Goldman  wrote: 
> Hi, I have interest in getting involved in the MXNet community. Could I get
> an invite for slack?
> 

Re: assimilation of mshadow into the MXNet codebase

[Sheng Zha]

I found the template in the link Marco provided and filed the software
grant to the secretary.

Sheng

On Sun, Jul 5, 2020 at 10:09 AM Michael Wall  wrote:

> Yes, to secretary@.  Do you need a template?
>
> Thanks Sheng
>
> Mike
>
> On Sun, Jul 5, 2020 at 12:59 PM Sheng Zha  wrote:
> >
> > Hi Michael,
> >
> > Thanks for offering help. I can represent the code donors and file the
> software grant. Should the filing go to secretary@?
> >
> > Sheng
> >
> > > On Jul 5, 2020, at 9:50 AM, Michael Wall  wrote:
> > >
> > > Is this being tracked in a ticket anywhere?  What help can I offer?
> > >
> > > Mike
> > >
> > >> On Fri, Jun 12, 2020 at 6:44 PM Marco de Abreu <
> marco.g.ab...@gmail.com> wrote:
> > >>
> > >> Hi Sheng,
> > >>
> > >> since this is a "large one off code contribution", the policy [1]
> states
> > >> that they should be brought in through a software grant.
> > >>
> > >> Best regards,
> > >> Marco
> > >>
> > >> [1]: https://www.apache.org/foundation/how-it-works/legal.html
> > >>
> > >>> On Fri, Jun 12, 2020 at 11:41 PM Sheng Zha 
> wrote:
> > >>>
> > >>> To mentors,
> > >>>
> > >>> Do we the PPMC need to fill out IP clearance for this code donation?
> > >>>
> > >>> -sz
> > >>>
> > >>> On 2019/04/24 21:19:49, Sheng Zha  wrote:
> > >>>> The community has agreed to donate mshadow to the mxnet code base. I
> > >>> will start the migration and build logic changes soon.
> > >>>>
> > >>>> -sz
> > >>>>
> > >>>> On 2019/04/07 21:47:39, Sheng Zha  wrote:
> > >>>>> I agree it would make development easier to donate mshadow to mxnet
> > >>> code base, since mshadow is only used in MXNet. I support donating
> the
> > >>> mshadow code to mxnet and I started an RFC for this in mshadow [1].
> > >>>>>
> > >>>>> [1] https://github.com/dmlc/mshadow/issues/373
> > >>>>>
> > >>>>> -sz
> > >>>>>
> > >>>>> On 2019/04/06 04:38:19, Tianqi Chen 
> wrote:
> > >>>>>> Technically, mshadow is sufficient for MXNet. Adopting other
> > >>> libraries (
> > >>>>>> eigen or xtensor) will unnecessarily increase the codebase
> complexity
> > >>>>>> without any additional gains.
> > >>>>>>
> > >>>>>> Given that mshadow is only used by mxnet. I do support donating it
> > >>> into
> > >>>>>> mxnet codebase.
> > >>>>>> To respect the original mshadow community. I would recommend
> > >>> starting a
> > >>>>>> community RFC In the mshadow github issue for a week, before we
> > >>> start the
> > >>>>>> migrating process.
> > >>>>>> Also, I would recommend a rebase merge just like the case of
> > >>> MXNet.jl code
> > >>>>>> base to preserve the contribution history.
> > >>>>>>
> > >>>>>> Tianqi
> > >>>>>>
> > >>>>>>
> > >>>>>> On Fri, Apr 5, 2019 at 9:25 PM Alfredo Luque
> > >>>>>>  wrote:
> > >>>>>>
> > >>>>>>> Do you have a link to both of these proposals?
> > >>>>>>>
> > >>>>>>> On Fri, Apr 5, 2019 at 20:14 Anirudh Acharya <
> > >>> anirudhk...@gmail.com>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> Hi Pedro,
> > >>>>>>>>
> > >>>>>>>> mshadow is mostly used for tensor arithmetic. There have been
> > >>> discussions
> > >>>>>>>> about including it within mxnet. I think it is a good idea.
> > >>>>>>>>
> > >>>>>>>> As a more long term solution using libraries like eigen to
> > >>> perform linear
> > >>>>>>>> algebra operations was also suggested by anirudh2290@. I think
> > >>> xtensor(
> > >>>>>>>> https://github.com/QuantStack/xtensor ) can also be a candidate
> > >>> here.
> > >>>>>>>>
> > >>>>>>>> -
> > >>>>>>>> Anirudh
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On Fri, Apr 5, 2019 at 7:03 PM Pedro Larroy <
> > >>>>>>> pedro.larroy.li...@gmail.com>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Hi
> > >>>>>>>>>
> > >>>>>>>>> Some developers have noticed that working in mshadow is
> > >>> cumbersome as
> > >>>>>>>>> it's a 3rdparty subrepo.
> > >>>>>>>>>
> > >>>>>>>>> Since mshadow is a bunch of headers which don't have much of
> > >>>>>>>>> independent tests / library functionality, me and other
> > >>> developers
> > >>>>>>>>> believe that it would be good to assimilate this code in the
> > >>>>>>>>> repository for ease of contribution and changes without having
> > >>> to go
> > >>>>>>>>> trough contortions to test PRs that modify mshadow.
> > >>>>>>>>>
> > >>>>>>>>> Would anybody oppose this change?
> > >>>>>>>>>
> > >>>>>>>>> Thanks and have a nice weekend.
> > >>>>>>>>>
> > >>>>>>>>> Pedro.
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
>

Re: assimilation of mshadow into the MXNet codebase

[Sheng Zha]

Hi Michael,

Thanks for offering help. I can represent the code donors and file the software 
grant. Should the filing go to secretary@?

Sheng

> On Jul 5, 2020, at 9:50 AM, Michael Wall  wrote:
> 
> Is this being tracked in a ticket anywhere?  What help can I offer?
> 
> Mike
> 
>> On Fri, Jun 12, 2020 at 6:44 PM Marco de Abreu  
>> wrote:
>> 
>> Hi Sheng,
>> 
>> since this is a "large one off code contribution", the policy [1] states
>> that they should be brought in through a software grant.
>> 
>> Best regards,
>> Marco
>> 
>> [1]: https://www.apache.org/foundation/how-it-works/legal.html
>> 
>>> On Fri, Jun 12, 2020 at 11:41 PM Sheng Zha  wrote:
>>> 
>>> To mentors,
>>> 
>>> Do we the PPMC need to fill out IP clearance for this code donation?
>>> 
>>> -sz
>>> 
>>> On 2019/04/24 21:19:49, Sheng Zha  wrote:
>>>> The community has agreed to donate mshadow to the mxnet code base. I
>>> will start the migration and build logic changes soon.
>>>> 
>>>> -sz
>>>> 
>>>> On 2019/04/07 21:47:39, Sheng Zha  wrote:
>>>>> I agree it would make development easier to donate mshadow to mxnet
>>> code base, since mshadow is only used in MXNet. I support donating the
>>> mshadow code to mxnet and I started an RFC for this in mshadow [1].
>>>>> 
>>>>> [1] https://github.com/dmlc/mshadow/issues/373
>>>>> 
>>>>> -sz
>>>>> 
>>>>> On 2019/04/06 04:38:19, Tianqi Chen  wrote:
>>>>>> Technically, mshadow is sufficient for MXNet. Adopting other
>>> libraries (
>>>>>> eigen or xtensor) will unnecessarily increase the codebase complexity
>>>>>> without any additional gains.
>>>>>> 
>>>>>> Given that mshadow is only used by mxnet. I do support donating it
>>> into
>>>>>> mxnet codebase.
>>>>>> To respect the original mshadow community. I would recommend
>>> starting a
>>>>>> community RFC In the mshadow github issue for a week, before we
>>> start the
>>>>>> migrating process.
>>>>>> Also, I would recommend a rebase merge just like the case of
>>> MXNet.jl code
>>>>>> base to preserve the contribution history.
>>>>>> 
>>>>>> Tianqi
>>>>>> 
>>>>>> 
>>>>>> On Fri, Apr 5, 2019 at 9:25 PM Alfredo Luque
>>>>>>  wrote:
>>>>>> 
>>>>>>> Do you have a link to both of these proposals?
>>>>>>> 
>>>>>>> On Fri, Apr 5, 2019 at 20:14 Anirudh Acharya <
>>> anirudhk...@gmail.com>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Hi Pedro,
>>>>>>>> 
>>>>>>>> mshadow is mostly used for tensor arithmetic. There have been
>>> discussions
>>>>>>>> about including it within mxnet. I think it is a good idea.
>>>>>>>> 
>>>>>>>> As a more long term solution using libraries like eigen to
>>> perform linear
>>>>>>>> algebra operations was also suggested by anirudh2290@. I think
>>> xtensor(
>>>>>>>> https://github.com/QuantStack/xtensor ) can also be a candidate
>>> here.
>>>>>>>> 
>>>>>>>> -
>>>>>>>> Anirudh
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Fri, Apr 5, 2019 at 7:03 PM Pedro Larroy <
>>>>>>> pedro.larroy.li...@gmail.com>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hi
>>>>>>>>> 
>>>>>>>>> Some developers have noticed that working in mshadow is
>>> cumbersome as
>>>>>>>>> it's a 3rdparty subrepo.
>>>>>>>>> 
>>>>>>>>> Since mshadow is a bunch of headers which don't have much of
>>>>>>>>> independent tests / library functionality, me and other
>>> developers
>>>>>>>>> believe that it would be good to assimilate this code in the
>>>>>>>>> repository for ease of contribution and changes without having
>>> to go
>>>>>>>>> trough contortions to test PRs that modify mshadow.
>>>>>>>>> 
>>>>>>>>> Would anybody oppose this change?
>>>>>>>>> 
>>>>>>>>> Thanks and have a nice weekend.
>>>>>>>>> 
>>>>>>>>> Pedro.
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 

Re: Ownership of discuss.mxnet.io

[Sheng Zha]

Hi Michael,

Thank you for offering to help. Yes, that’s the correct understanding.

For brand management, we still haven’t started the review yet. On the other 
hand, if the hosting is on Apache infra and PPMC manages the site, then the 
site will no longer be owned by a third party.

Thanks,
Sheng

> On Jul 5, 2020, at 9:25 AM, Michael Wall  wrote:
> 
> Did a ticket ever get opened for this?  I can open one, but want to
> verify my understanding:
> 
> "We want to explore the possibility of hosting discuss.mxnet.io and
> discuss.gluon.ai on Apache infrastructure and hook into Apache user
> management."
> 
> Shane also asked if we started a conversation with Brand management.
> How can I help there?
> 
> Mike
> 
>> On Thu, Jun 18, 2020 at 6:03 AM Marco de Abreu  
>> wrote:
>> 
>> I think we can start by opening a ticket with infra referring to this
>> thread.
>> 
>> -Marco
>> 
>> Sheng Zha  schrieb am Do., 18. Juni 2020, 09:45:
>> 
>>> Hi Shane,
>>> 
>>> Thanks for the comment. With my apache hat on, I definitely agree that it
>>> would be better to have such forum on Apache managed infra. I’ve started
>>> with discussion archive to ensure that everything is recorded, and I think
>>> I can quickly try to make the backups of that forum accessible to the PPMC
>>> and Apache infra.
>>> 
>>> I think the best way to achieve the goal of continuity, eventually, would
>>> be to host it on Apache infrastructure and have its hosting funded by ASF.
>>> How do we go about requesting this?
>>> 
>>> As for the trademark review, we started working on reviewing the binary
>>> distribution usage and are currently focusing on the more pressing issue
>>> there. We will get to the domain name usage afterwards.
>>> 
>>> -sz
>>> 
>>>> On Jun 17, 2020, at 6:07 AM, Shane Curcuru  wrote:
>>>> 
>>>> On 2020/06/12 22:39:10, Marco de Abreu 
>>> wrote: ...
>>>>> The mxnet.io domain is not under control of the ASF. If we say that a
>>> user
>>>>> facing platform is managed and endorsed by the PPMC, we should make it
>>>>> available under the ASF domain system to further represent the official
>>>>> affiliation. Also, it removes a weak link in the chain - e.g. the case
>>> when
>>>>> the mxnet.io domain ran out and broke quite a bit of stuff. Since the
>>> ASF
>>>>> offers subdomains for projects, I don't see any reason why not to use
>>> them.
>>>> 
>>>> This is the core issue for me, from the larger Apache perspective.
>>>> Domain names using Apache marks - especially when they have interactive
>>>> traffic about the Apache project itself - work best when the domain name
>>>> itself is owned by the ASF.  That ensures continuity for the future, in
>>>> the case where a party (which includes individual PMC members) stops
>>>> contributing to the project.  When the ASF Infra team has direct admin
>>>> access to a service or domain, we know that the ASF will be able to
>>>> manage it for the future.
>>>> 
>>>> Also, has the PPMC worked with Apache Brand Management on use of
>>>> trademarks in domain names?
>>>> 
>>>> https://apache.org/foundation/marks/domains
>>>> 
>>>> --
>>>> 
>>>> - Shane
>>>> Director & Member
>>>> The Apache Software Foundation
>>> 

Re: [Lazy consensus] Turn on Branch Protection for Release branches and v1.x

[Sheng Zha]

Branch protection has been turned on for v*.x branches in 
https://issues.apache.org/jira/browse/INFRA-20471

-sz

On 2020/06/19 23:01:45, Sheng Zha  wrote: 
> Hi,
> 
> I'd like to propose that we turn on branch protection and CI check 
> enforcement that's consistent with our master branch for all release branches 
> and v1.x. So far we only have branch protection and enforce CI checks on 
> master branch, while allowing force push without CI checks on release 
> branches.
> 
> Feel free to reply if you have questions or concerns, or otherwise I will 
> wait for 96hrs before proceeding with changes.
> 
> -sz
> 

Re: Podling Mxnet Report Reminder - July 2020

[Sheng Zha]

Working on it.

-sz

On 2020/06/25 03:38:02, jmcl...@apache.org wrote: 
> Dear podling,
> 
> This email was sent by an automated system on behalf of the Apache
> Incubator PMC. It is an initial reminder to give you plenty of time to
> prepare your quarterly board report.
> 
> The board meeting is scheduled for Wed, 15 July 2020.
> The report for your podling will form a part of the Incubator PMC
> report. The Incubator PMC requires your report to be submitted 2 weeks
> before the board meeting, to allow sufficient time for review and
> submission (Wed, July 01).
> 
> Please submit your report with sufficient time to allow the Incubator
> PMC, and subsequently board members to review and digest. Again, the
> very latest you should submit your report is 2 weeks prior to the board
> meeting.
> 
> Candidate names should not be made public before people are actually
> elected, so please do not include the names of potential committers or
> PPMC members in your report.
> 
> Thanks,
> 
> The Apache Incubator PMC
> 
> Submitting your Report
> 
> --
> 
> Your report should contain the following:
> 
> *   Your project name
> *   A brief description of your project, which assumes no knowledge of
> the project or necessarily of its field
> *   A list of the three most important issues to address in the move
> towards graduation.
> *   Any issues that the Incubator PMC or ASF Board might wish/need to be
> aware of
> *   How has the community developed since the last report
> *   How has the project developed since the last report.
> *   How does the podling rate their own maturity.
> 
> This should be appended to the Incubator Wiki page at:
> 
> https://cwiki.apache.org/confluence/display/INCUBATOR/July2020
> 
> Note: This is manually populated. You may need to wait a little before
> this page is created from a template.
> 
> Note: The format of the report has changed to use markdown.
> 
> Mentors
> ---
> 
> Mentors should review reports for their project(s) and sign them off on
> the Incubator wiki page. Signing off reports shows that you are
> following the project - projects that are not signed may raise alarms
> for the Incubator PMC.
> 
> Incubator PMC
> 

[Lazy consensus] Turn on Branch Protection for Release branches and v1.x

[Sheng Zha]

Hi,

I'd like to propose that we turn on branch protection and CI check enforcement 
that's consistent with our master branch for all release branches and v1.x. So 
far we only have branch protection and enforce CI checks on master branch, 
while allowing force push without CI checks on release branches.

Feel free to reply if you have questions or concerns, or otherwise I will wait 
for 96hrs before proceeding with changes.

-sz

Re: Ownership of discuss.mxnet.io

[Sheng Zha]

Hi Shane,

Thanks for the comment. With my apache hat on, I definitely agree that it would 
be better to have such forum on Apache managed infra. I’ve started with 
discussion archive to ensure that everything is recorded, and I think I can 
quickly try to make the backups of that forum accessible to the PPMC and Apache 
infra.

I think the best way to achieve the goal of continuity, eventually, would be to 
host it on Apache infrastructure and have its hosting funded by ASF. How do we 
go about requesting this?

As for the trademark review, we started working on reviewing the binary 
distribution usage and are currently focusing on the more pressing issue there. 
We will get to the domain name usage afterwards.

-sz

> On Jun 17, 2020, at 6:07 AM, Shane Curcuru  wrote:
> 
> On 2020/06/12 22:39:10, Marco de Abreu  wrote: ...
>> The mxnet.io domain is not under control of the ASF. If we say that a user
>> facing platform is managed and endorsed by the PPMC, we should make it
>> available under the ASF domain system to further represent the official
>> affiliation. Also, it removes a weak link in the chain - e.g. the case when
>> the mxnet.io domain ran out and broke quite a bit of stuff. Since the ASF
>> offers subdomains for projects, I don't see any reason why not to use them.
> 
> This is the core issue for me, from the larger Apache perspective.
> Domain names using Apache marks - especially when they have interactive
> traffic about the Apache project itself - work best when the domain name
> itself is owned by the ASF.  That ensures continuity for the future, in
> the case where a party (which includes individual PMC members) stops
> contributing to the project.  When the ASF Infra team has direct admin
> access to a service or domain, we know that the ASF will be able to
> manage it for the future.
> 
> Also, has the PPMC worked with Apache Brand Management on use of
> trademarks in domain names?
> 
>  https://apache.org/foundation/marks/domains
> 
> -- 
> 
> - Shane
>  Director & Member
>  The Apache Software Foundation

Re: [CI] Staggered build pipelines enabled

[Sheng Zha]

Hi Marco,

> We've always had the policy of not enforcing protected feature branches and 
> this pipeline is causing human errors.

There are several problems with this statement:
1. This is a release branch as defined in our release process. It's not a 
feature branch.
2. There was no explicit decision or an agreed upon policy of not protecting 
the release branch. To me it doesn't make sense to only protect the development 
branch while allowing force push and revision of history on release branches, 
and this is what needs to be fixed.
3. I made a conscious decision when merging it given the CI sanity check at the 
time, given that there were changes from Chai to CI pipelines which may not 
take effect. While a better approach would have been to test this separately 
first, I see such issue as a normal part of development and I wouldn't be too 
concerned given that we use proper version control.

Regarding the benefit of having the sanity check step, you can easily measure 
its impact by looking at the recent PR build history of the sanity step [1], 
which I believe you do have access to. Out of the recent 563 builds in PR 
validation, 137 builds have broken sanity check, or 24.3%. Given the amount of 
reduction in wasted compute on already failed validation pipelines, to me 
waiting for two docker cache builds in rare occasions is justified.

Since the increase in PR validation time is due to the docker build time, you 
can help by improving the docker cache of the CI. While a couple of people are 
already investing time in improving the status quo in this regard, you are more 
than welcome to step up and participate on improving it.

-sz

[1] 
http://jenkins.mxnet-ci.amazon-ml.com/job/mxnet-validation/job/sanity/view/change-requests/builds

On 2020/06/15 23:19:51, Marco de Abreu  wrote: 
> Hello,
> 
> I'd like to revisit this decision and review whether the expected benefit
> (cost reduction) was achieved and how the overall PR validation duration
> has changed. Could you guys share some information on this matter?
> 
> Just today, we've had two incidents which were caused by this change:
> 
> 1. This PR was merged prematurely because the follow up pipelines didn't
> run (whether it's a timing issue or something else I don't know). We've
> always had the policy of not enforcing protected feature branches and this
> pipeline is causing human errors.
> https://github.com/apache/incubator-mxnet/pull/18560
> 
> 2. The sanity check, which got into the critical path here, starts running
> into timeouts. I'm aware that this is in case of cache misses, but none the
> less does that heavily increase the waiting duration since developers now
> have to wait for two entire cache build cycles instead of just one:
> https://github.com/apache/incubator-mxnet/pull/18568
> 
> I understand that money has to be conserved, but I still stand by my
> opinion that this was the wrong move and development speed was sacrificed.
> 
> If there are no other compelling arguments, I'd prefer if the previous
> state of parallel pipelines could be restored.
> 
> Best regards
> Marco
> 
> sandeep krishnamurthy  schrieb am Di., 28.
> Apr. 2020, 07:55:
> 
> > Thanks a lot Joe for your contributions. Thank you Marco, Chai and Leo for
> > helping this.
> > Especially given that you had seen around 57% build failing in sanity
> > check, this should be very helpful to provide faster feedback for PR
> > authors on sanity issues plus save a lot of unnecessary builds.
> >
> > Best,
> > Sandeep
> >
> > On Mon, 27 Apr 2020, 10:20 pm Joe Evans,  wrote:
> >
> > > Hi dev community,
> > >
> > >
> > > We have made the changes to the mxnet CI system to incorporate the
> > > staggered build pipelines. With this change, when a new PR is created or
> > an
> > > existing PR is updated, the status checks will only show
> > > “ci/jenkins/mxnet-validation/sanity” build job at first. Once this build
> > > completes successfully (avg. run time is about 10min), the remaining CI
> > > build jobs will appear and function as previously.
> > >
> > >
> > > Please let me know if you experience any issues with this change.
> > >
> > >
> > > Thanks!
> > >
> > > Joe
> > >
> > >
> > > References:
> > >
> > >
> > > https://github.com/apache/incubator-mxnet/issues/17802
> > >
> >
> 

Re: Ownership of discuss.mxnet.io

[Sheng Zha]

Hi Marco,

As shared in another reply, I've set up an archive for all future posts on the 
forum. I can't get to the committer authorization item soon and if you want to 
help and need access from the forum, I'm happy to help grant you access.

-sz

On 2020/06/15 23:20:40, Marco de Abreu  wrote: 
> Hello,
> 
> Has there been any update on this matter?
> 
> Best regards
> Marco
> 
> Marco de Abreu  schrieb am Sa., 13. Juni 2020,
> 00:39:
> 
> > Thanks for elaborating!
> >
> > You are right, there's no requirement to host on ASF infrastructure. I was
> > just thinking about what happens when that funding is cut or when there's
> > an issue. What's the strategy to ensure no data loss in case of priority
> > shift or defunding from Amazon? Opposed to the CI which is fully codified
> > and under version control, we as PMC have no means to recover from a
> > disaster in case of Discourse.
> >
> > The mxnet.io domain is not under control of the ASF. If we say that a
> > user facing platform is managed and endorsed by the PPMC, we should make it
> > available under the ASF domain system to further represent the official
> > affiliation. Also, it removes a weak link in the chain - e.g. the case when
> > the mxnet.io domain ran out and broke quite a bit of stuff. Since the ASF
> > offers subdomains for projects, I don't see any reason why not to use them.
> >
> > Could you open a ticket with INFRA to check what authentication options
> > they are offering? I couldn't find any info on the website.
> >
> > -Marco
> >
> > On Sat, Jun 13, 2020 at 12:24 AM Sheng Zha  wrote:
> >
> >> > Could you elaborate about the hosting of the forum? Is there the
> >> > possibility to move it to some ASF-managed webhost (I'm not sure about
> >> the
> >> > complexity of hosting Discourse)?
> >>
> >> The website is currently a subscribed hosted service provided by
> >> discourse, donated by Amazon. It is possible to host it ourselves but it
> >> would incur additional operational cost. In addition, I'm not aware of a
> >> requirement to have it on ASF infrastructure, given that it's only for
> >> answering user questions and we don't conduct official Apache MXNet
> >> businesses there.
> >>
> >> > Also, I'd suggest to make an Apache managed subdomain the main domain
> >> (e.g.
> >> > https://discuss.mxnet.apache.org/ <https://mxnet.apache.org/>). The
> >> > existing domain could be a redirect.
> >>
> >> I don't see a benefit of doing that, nor am I aware of a requirement for
> >> it.
> >>
> >> > Additionally, can we hook it up to the Apache IDP to allow committers
> >> and
> >> > pmc members to be automatically authorized?
> >>
> >> This sounds like a good idea though I'm not sure how to proceed. If
> >> someone knows how to proceed I'm happy to help grant necessary access to 
> >> it.
> >>
> >> -sz
> >>
> >>
> >> On 2020/06/12 22:07:47, Marco de Abreu  wrote:
> >> > Hi Sheng,
> >> >
> >> > thanks for the quick response.
> >> >
> >> > Could you elaborate about the hosting of the forum? Is there the
> >> > possibility to move it to some ASF-managed webhost (I'm not sure about
> >> the
> >> > complexity of hosting Discourse)?
> >> >
> >> > Also, I'd suggest to make an Apache managed subdomain the main domain
> >> (e.g.
> >> > https://discuss.mxnet.apache.org/ <https://mxnet.apache.org/>). The
> >> > existing domain could be a redirect.
> >> >
> >> > Additionally, can we hook it up to the Apache IDP to allow committers
> >> and
> >> > pmc members to be automatically authorized?
> >> >
> >> > Best regards
> >> > Marco
> >> >
> >> > On Fri, Jun 12, 2020 at 11:24 PM Sheng Zha  wrote:
> >> >
> >> > > Hi Marco,
> >> > >
> >> > > I'm an admin of that website along with a couple of other PPMC
> >> members.
> >> > > This forum has been around for some time and has been helpful in
> >> connecting
> >> > > users of mxnet to the developer community. I think it's in the best
> >> > > interest of this community to consider this under PPMC control for
> >> better
> >> > > discoverability under the mxnet domain name. I'm happy to grant you
> >> admin
> &g

Re: Ownership of discuss.mxnet.io

[Sheng Zha]

To address the data loss concern, I'm setting up an archive for all future 
posts on discuss.mxnet.io to the new archive list [1]. It will take me a while 
before I get time to work on the rest.

-sz

[1] https://lists.apache.org/list.html?discuss-arch...@mxnet.apache.org

On 2020/06/12 22:39:10, Marco de Abreu  wrote: 
> Thanks for elaborating!
> 
> You are right, there's no requirement to host on ASF infrastructure. I was
> just thinking about what happens when that funding is cut or when there's
> an issue. What's the strategy to ensure no data loss in case of priority
> shift or defunding from Amazon? Opposed to the CI which is fully codified
> and under version control, we as PMC have no means to recover from a
> disaster in case of Discourse.
> 
> The mxnet.io domain is not under control of the ASF. If we say that a user
> facing platform is managed and endorsed by the PPMC, we should make it
> available under the ASF domain system to further represent the official
> affiliation. Also, it removes a weak link in the chain - e.g. the case when
> the mxnet.io domain ran out and broke quite a bit of stuff. Since the ASF
> offers subdomains for projects, I don't see any reason why not to use them.
> 
> Could you open a ticket with INFRA to check what authentication options
> they are offering? I couldn't find any info on the website.
> 
> -Marco
> 
> On Sat, Jun 13, 2020 at 12:24 AM Sheng Zha  wrote:
> 
> > > Could you elaborate about the hosting of the forum? Is there the
> > > possibility to move it to some ASF-managed webhost (I'm not sure about
> > the
> > > complexity of hosting Discourse)?
> >
> > The website is currently a subscribed hosted service provided by
> > discourse, donated by Amazon. It is possible to host it ourselves but it
> > would incur additional operational cost. In addition, I'm not aware of a
> > requirement to have it on ASF infrastructure, given that it's only for
> > answering user questions and we don't conduct official Apache MXNet
> > businesses there.
> >
> > > Also, I'd suggest to make an Apache managed subdomain the main domain
> > (e.g.
> > > https://discuss.mxnet.apache.org/ <https://mxnet.apache.org/>). The
> > > existing domain could be a redirect.
> >
> > I don't see a benefit of doing that, nor am I aware of a requirement for
> > it.
> >
> > > Additionally, can we hook it up to the Apache IDP to allow committers and
> > > pmc members to be automatically authorized?
> >
> > This sounds like a good idea though I'm not sure how to proceed. If
> > someone knows how to proceed I'm happy to help grant necessary access to it.
> >
> > -sz
> >
> >
> > On 2020/06/12 22:07:47, Marco de Abreu  wrote:
> > > Hi Sheng,
> > >
> > > thanks for the quick response.
> > >
> > > Could you elaborate about the hosting of the forum? Is there the
> > > possibility to move it to some ASF-managed webhost (I'm not sure about
> > the
> > > complexity of hosting Discourse)?
> > >
> > > Also, I'd suggest to make an Apache managed subdomain the main domain
> > (e.g.
> > > https://discuss.mxnet.apache.org/ <https://mxnet.apache.org/>). The
> > > existing domain could be a redirect.
> > >
> > > Additionally, can we hook it up to the Apache IDP to allow committers and
> > > pmc members to be automatically authorized?
> > >
> > > Best regards
> > > Marco
> > >
> > > On Fri, Jun 12, 2020 at 11:24 PM Sheng Zha  wrote:
> > >
> > > > Hi Marco,
> > > >
> > > > I'm an admin of that website along with a couple of other PPMC members.
> > > > This forum has been around for some time and has been helpful in
> > connecting
> > > > users of mxnet to the developer community. I think it's in the best
> > > > interest of this community to consider this under PPMC control for
> > better
> > > > discoverability under the mxnet domain name. I'm happy to grant you
> > admin
> > > > access if you or any other PPMC members need it. If you feel otherwise,
> > > > feel free to propose how you suggest we proceed.
> > > >
> > > > -sz
> > > >
> > > > On 2020/06/12 21:07:28, Marco de Abreu  wrote:
> > > > > Hi everyone,
> > > > >
> > > > > Due to a PR [1] I had a closer look at discuss.mxnet.io which is a
> > > > forum to
> > > > > discuss about MXNet. It's a very helpful platform and provides great
> > > 

Re: Ownership of discuss.mxnet.io

[Sheng Zha]

> Could you elaborate about the hosting of the forum? Is there the
> possibility to move it to some ASF-managed webhost (I'm not sure about the
> complexity of hosting Discourse)?

The website is currently a subscribed hosted service provided by discourse, 
donated by Amazon. It is possible to host it ourselves but it would incur 
additional operational cost. In addition, I'm not aware of a requirement to 
have it on ASF infrastructure, given that it's only for answering user 
questions and we don't conduct official Apache MXNet businesses there.

> Also, I'd suggest to make an Apache managed subdomain the main domain (e.g.
> https://discuss.mxnet.apache.org/ <https://mxnet.apache.org/>). The
> existing domain could be a redirect.

I don't see a benefit of doing that, nor am I aware of a requirement for it.

> Additionally, can we hook it up to the Apache IDP to allow committers and
> pmc members to be automatically authorized?

This sounds like a good idea though I'm not sure how to proceed. If someone 
knows how to proceed I'm happy to help grant necessary access to it.

-sz


On 2020/06/12 22:07:47, Marco de Abreu  wrote: 
> Hi Sheng,
> 
> thanks for the quick response.
> 
> Could you elaborate about the hosting of the forum? Is there the
> possibility to move it to some ASF-managed webhost (I'm not sure about the
> complexity of hosting Discourse)?
> 
> Also, I'd suggest to make an Apache managed subdomain the main domain (e.g.
> https://discuss.mxnet.apache.org/ <https://mxnet.apache.org/>). The
> existing domain could be a redirect.
> 
> Additionally, can we hook it up to the Apache IDP to allow committers and
> pmc members to be automatically authorized?
> 
> Best regards
> Marco
> 
> On Fri, Jun 12, 2020 at 11:24 PM Sheng Zha  wrote:
> 
> > Hi Marco,
> >
> > I'm an admin of that website along with a couple of other PPMC members.
> > This forum has been around for some time and has been helpful in connecting
> > users of mxnet to the developer community. I think it's in the best
> > interest of this community to consider this under PPMC control for better
> > discoverability under the mxnet domain name. I'm happy to grant you admin
> > access if you or any other PPMC members need it. If you feel otherwise,
> > feel free to propose how you suggest we proceed.
> >
> > -sz
> >
> > On 2020/06/12 21:07:28, Marco de Abreu  wrote:
> > > Hi everyone,
> > >
> > > Due to a PR [1] I had a closer look at discuss.mxnet.io which is a
> > forum to
> > > discuss about MXNet. It's a very helpful platform and provides great
> > > assistance for the community.
> > >
> > > When I looked into the terms of services [2] though, I noticed that the
> > ASF
> > > and MXNet ppmc are mentioned as owners.
> > >
> > > From my understanding, that forum and the domain are not managed by the
> > ASF
> > > or the mxnet ppmc. Thus, that statement does not seem correct and is
> > rather
> > > a trademark infringement. My understanding is that the platform is
> > operated
> > > by a third party as a community service towards MXNet. Thus, I think it's
> > > important to properly express who's responsible for that platform.
> > >
> > > If somebody has any knowledge about the owner of that forum, I'd
> > appreciate
> > > if they could give them a heads up for this thread.
> > >
> > > Best regards
> > > Marco
> > >
> > > [1]: https://github.com/apache/incubator-mxnet/pull/18553/files
> > > [2]: https://discuss.mxnet.io/tos
> > >
> >
> 

Re: assimilation of mshadow into the MXNet codebase

[Sheng Zha]

To mentors,

Do we the PPMC need to fill out IP clearance for this code donation?

-sz

On 2019/04/24 21:19:49, Sheng Zha  wrote: 
> The community has agreed to donate mshadow to the mxnet code base. I will 
> start the migration and build logic changes soon.
> 
> -sz
> 
> On 2019/04/07 21:47:39, Sheng Zha  wrote: 
> > I agree it would make development easier to donate mshadow to mxnet code 
> > base, since mshadow is only used in MXNet. I support donating the mshadow 
> > code to mxnet and I started an RFC for this in mshadow [1].
> > 
> > [1] https://github.com/dmlc/mshadow/issues/373
> > 
> > -sz
> > 
> > On 2019/04/06 04:38:19, Tianqi Chen  wrote: 
> > > Technically, mshadow is sufficient for MXNet. Adopting other libraries (
> > > eigen or xtensor) will unnecessarily increase the codebase complexity
> > > without any additional gains.
> > > 
> > > Given that mshadow is only used by mxnet. I do support donating it into
> > > mxnet codebase.
> > > To respect the original mshadow community. I would recommend starting a
> > > community RFC In the mshadow github issue for a week, before we start the
> > > migrating process.
> > > Also, I would recommend a rebase merge just like the case of MXNet.jl code
> > > base to preserve the contribution history.
> > > 
> > > Tianqi
> > > 
> > > 
> > > On Fri, Apr 5, 2019 at 9:25 PM Alfredo Luque
> > >  wrote:
> > > 
> > > > Do you have a link to both of these proposals?
> > > >
> > > > On Fri, Apr 5, 2019 at 20:14 Anirudh Acharya 
> > > > wrote:
> > > >
> > > > > Hi Pedro,
> > > > >
> > > > > mshadow is mostly used for tensor arithmetic. There have been 
> > > > > discussions
> > > > > about including it within mxnet. I think it is a good idea.
> > > > >
> > > > > As a more long term solution using libraries like eigen to perform 
> > > > > linear
> > > > > algebra operations was also suggested by anirudh2290@. I think 
> > > > > xtensor(
> > > > > https://github.com/QuantStack/xtensor ) can also be a candidate here.
> > > > >
> > > > > -
> > > > > Anirudh
> > > > >
> > > > >
> > > > > On Fri, Apr 5, 2019 at 7:03 PM Pedro Larroy <
> > > > pedro.larroy.li...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi
> > > > > >
> > > > > > Some developers have noticed that working in mshadow is cumbersome 
> > > > > > as
> > > > > > it's a 3rdparty subrepo.
> > > > > >
> > > > > > Since mshadow is a bunch of headers which don't have much of
> > > > > > independent tests / library functionality, me and other developers
> > > > > > believe that it would be good to assimilate this code in the
> > > > > > repository for ease of contribution and changes without having to go
> > > > > > trough contortions to test PRs that modify mshadow.
> > > > > >
> > > > > > Would anybody oppose this change?
> > > > > >
> > > > > > Thanks and have a nice weekend.
> > > > > >
> > > > > > Pedro.
> > > > > >
> > > > >
> > > >
> > > 
> > 
> 

Re: Ownership of discuss.mxnet.io

[Sheng Zha]

Hi Marco,

I'm an admin of that website along with a couple of other PPMC members. This 
forum has been around for some time and has been helpful in connecting users of 
mxnet to the developer community. I think it's in the best interest of this 
community to consider this under PPMC control for better discoverability under 
the mxnet domain name. I'm happy to grant you admin access if you or any other 
PPMC members need it. If you feel otherwise, feel free to propose how you 
suggest we proceed.

-sz

On 2020/06/12 21:07:28, Marco de Abreu  wrote: 
> Hi everyone,
> 
> Due to a PR [1] I had a closer look at discuss.mxnet.io which is a forum to
> discuss about MXNet. It's a very helpful platform and provides great
> assistance for the community.
> 
> When I looked into the terms of services [2] though, I noticed that the ASF
> and MXNet ppmc are mentioned as owners.
> 
> From my understanding, that forum and the domain are not managed by the ASF
> or the mxnet ppmc. Thus, that statement does not seem correct and is rather
> a trademark infringement. My understanding is that the platform is operated
> by a third party as a community service towards MXNet. Thus, I think it's
> important to properly express who's responsible for that platform.
> 
> If somebody has any knowledge about the owner of that forum, I'd appreciate
> if they could give them a heads up for this thread.
> 
> Best regards
> Marco
> 
> [1]: https://github.com/apache/incubator-mxnet/pull/18553/files
> [2]: https://discuss.mxnet.io/tos
> 

Re: [apache/incubator-mxnet] [RFC] MXNet 2.0 API Deprecation (#17676)

[Sheng Zha]

@yifeim module API will continue to be supported in 1.x and users are free to 
stay on that version. For 2.x, we will only support numpy/npx API so users who 
adopt those API will have to reimplement the model anyway. the main function 
you listed will all be available in 2.x.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/17676#issuecomment-639131712

Re: Issue with releases / feedback from ASF board

[Sheng Zha]

Hi Justin,

Here's an update on the progress of addressing the license issues:

Done:
- Add disclaimer to repo.mxnet.io and dist.mxnet.io to clarify that the
nightly releases there are not intended for public consumption. (changed.
pending cache refresh)
- Remove non-Apache releases in github release page.

Ongoing:
- Delete problematic Maven releases. dev@ is notified and per discussion
it's pending lazy consensus [1].
- Review with Apache Trademark on the current third-party binary
distributions of mxnet and make necessary correction. Review initiated with
trademarks@.
- Review with Apache Legal on the appropriate license for convenience
binary distribution and display it prominently. Currently waiting for reply
[2]

To do:
- Add source distribution to PyPI package `apache-mxnet`. This is intended
to be the official source release that is compliant with incubator
distribution guidelines [3].
- Non-official third-party Maven binary releases. We need input on the
requirements for observing the proper trademark usage first.

As we proceed, there may be more work to do, and we are tracking the
progress in https://github.com/apache/incubator-mxnet/issues/18397.

Let us know if you have any question.

Regards,
Sheng

[1]
https://lists.apache.org/thread.html/ra4e9572ac74857a80c64a31e8bf292d353e74cfa87bf457f47450303%40%3Cdev.mxnet.apache.org%3E
[2] https://issues.apache.org/jira/browse/LEGAL-515
[3]
https://cwiki.apache.org/confluence/display/INCUBATOR/DistributionGuidelines

On Sun, May 31, 2020 at 2:01 AM Justin Mclean  wrote:

> Hi,
>
> I'm writing my board report in the next couple of days and just wondering
> what progress has been made on this.
>
> Thanks,
> Justin
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

Re: [DISCUSS] Deprecating and Forbidding Terraform and Other Services by HashiCorp in MXNet

[Sheng Zha]

Hi Marco,

At the time I started the thread, the specific term related to China reads as 
follows:

PLEASE NOTE THAT THE SOFWARE MAY NOT BE USED, DEPLOYED, OR INSTALLED IN THE 
PEOPLE'S REPUBLIC OF CHINA. [1]

My concern with the above clause is that it starts the practice of HashiCorp 
targeting a group of the community resides in a specific region. I'm worried 
that they could change terms of use at any moment that could render the normal 
usage illegal and our community members liable.

The situation has changed as HashiCorp indeed revised the specific clause to 
include only Vault enterprise version. From the reply I think you only read the 
updated version. Since the current version of their term of use doesn't affect 
our project anymore, I'm OK to leave it as is.

-sz

[1] https://pbs.twimg.com/media/EZLJJwjUYAM8Xk-?format=jpg=large

On 2020/05/31 09:58:08, Marco de Abreu  wrote: 
> The statement is specifically about HashiCorp Vault enterprise edition -
> speak a single module of their enterprise suite which is about credential
> encryption.
> 
> I didn't read further, but my first guess is that they are not offering it
> in China since the Chinese government - as far as I can recall - is
> restricting the usage of encryption methods to those, which they consider
> weak and exploitable.
> 
> From that point on, it becomes more a political question. I personally am
> quite concerned about a government using their power to undermine security
> of software.
> 
> Since this statement is only about vault enterprise edition and we're not
> using vault at all, I'd leave it as is. Supporting such a horrible practice
> by "boycotting" HashiCorp in general would send the wrong signals  from an
> open source communities point of view.
> 
> -Marco
> 
> Sheng Zha  schrieb am So., 31. Mai 2020, 05:08:
> 
> > Dear community,
> >
> > Yesterday, HashiCorp added to their terms of evaluation the clause that
> > forbids usage of the enterprise version of any HashiCorp software in the
> > People's Republic of China [1]. While this does not affect the usage of the
> > community version, it does signal the potential legal risk to many of our
> > community members in China. In light of this recent development, I'm
> > initiating discussion on deprecating the usage of HashiCorp software and
> > services and forbidding their usage in MXNet infrastructure until situation
> > changes.
> >
> > I believe that the community cannot be traded for the technological
> > convenience. Currently, we have part of our CI and GitHub labeling robot
> > bootstrapping logic that relies on Terraform, a service provided by
> > HashiCorp [2]. Since all its usage that I found are for bootstrapping on
> > AWS, replacing them with CloudFormation is feasible and likely
> > straightforward.
> >
> > Your input is appreciated.
> >
> > Regards,
> > Sheng
> >
> > [1] https://www.hashicorp.com/terms-of-evaluation
> > [2]
> > https://github.com/apache/incubator-mxnet-ci/search?q=terraform_q=terraform
> >
> >
> 

[DISCUSS] Deprecating and Forbidding Terraform and Other Services by HashiCorp in MXNet

[Sheng Zha]

Dear community,

Yesterday, HashiCorp added to their terms of evaluation the clause that forbids 
usage of the enterprise version of any HashiCorp software in the People's 
Republic of China [1]. While this does not affect the usage of the community 
version, it does signal the potential legal risk to many of our community 
members in China. In light of this recent development, I'm initiating 
discussion on deprecating the usage of HashiCorp software and services and 
forbidding their usage in MXNet infrastructure until situation changes.

I believe that the community cannot be traded for the technological 
convenience. Currently, we have part of our CI and GitHub labeling robot 
bootstrapping logic that relies on Terraform, a service provided by HashiCorp 
[2]. Since all its usage that I found are for bootstrapping on AWS, replacing 
them with CloudFormation is feasible and likely straightforward.

Your input is appreciated.

Regards,
Sheng

[1] https://www.hashicorp.com/terms-of-evaluation
[2] 
https://github.com/apache/incubator-mxnet-ci/search?q=terraform_q=terraform

Re: Issue with releases / feedback from ASF board

[Sheng Zha]

Thanks for the reply. Besides the clarification below, we will track and
address here [1] the rest of the comments in this thread.

> > - We will introduce source releases on PyPI and Maven using apache-mxnet
> > name to provide users with the option to have Apache distribution and
allow
> > users to compile CUDA/cuDNN code as an option.
>
> I assume that mean you will two distribution on PyPI? Or is the intent to
remove the other one?

- We PPMC will create apache-mxnet package with Apache releases in the form
of source distributions. This is intended as first-party release to comply
with the draft release policy.
- As individual, and with appropriate names and description as determined
by trademark review, I intend to continue to provide binary releases on
PyPI for the convenience of the community. This is out of necessity as
compiling all the CUDA code in MXNet can take hours even on powerful CPUs.

-sz

[1] https://github.com/apache/incubator-mxnet/issues/18397

On Sat, May 23, 2020 at 5:15 PM Justin Mclean 
wrote:

> Hi,
>
> > - MXNet publishes only nightly pre-release builds to dist.mxnet.io for
> the
> > verification purpose for projects in the ecosystem and for developers who
> > work on the bleeding edge.
>
> IMO Something more needs to be done here, at the very least I would expect
> to se a very large disclaimer that these are not releases for use of the
> general public and for internal development use only and are not Apache
> Releases.  A google search brings up that page and it is incorrectly titled
> "Apache MXNet Python Binary Distribution”. I see other download pages that
> just contain links and no information to what the files are.
>
> It also needs to be clear what a user is installed from this install page
> [1] which I assume is using the above? The GitHub download page [2] is also
> confusing as it contains a mix of Apache and non-Apache releases.
>
> > - PyPI releases are not the act of this PPMC, but are from individuals
> > acting as third-party for the convenience of the community.
>
> Currently this page isn’t in line with Apache trademark policy.
>
> > With my PPMC hat on, we will take the following actions:
> >
> > - We will try to take down problematic Maven releases immediately and
> > discuss in the community how to proceed with future Maven releases.
>
> Thanks for that.
>
> > - We will introduce source releases on PyPI and Maven using apache-mxnet
> > name to provide users with the option to have Apache distribution and
> allow
> > users to compile CUDA/cuDNN code as an option.
>
> I assume that mean you will two distribution on PyPI? Or is the intent to
> remove the other one?
>
> > Finally, we appreciate any lesson other projects can share on licensing
> and
> > distribution that involves CUDA code. Thanks.
>
> I’m not aware of any other project that uses CUDU code.
>
> Thanks,
> Justin
>
> 1. https://mxnet.apache.org/get_started/?
> 2. https://github.com/apache/incubator-mxnet/releases
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>

Re: Issue with releases / feedback from ASF board

[Sheng Zha]

Hi Justin and incubator,

Thank you for the feedback.

1. We fully intend to address the licensing issues in releases and software
distributions as part of our efforts towards graduation.

2. To my knowledge, there was no intention from this PPMC to bypass the
software release policy.

To clarify on 2:

- MXNet publishes only nightly pre-release builds to dist.mxnet.io for the
verification purpose for projects in the ecosystem and for developers who
work on the bleeding edge.

- PyPI releases are not the act of this PPMC, but are from individuals
acting as third-party for the convenience of the community. The binary
releases there are not from dist.mxnet.io, but are produced from the same
or similar build scripts without modification to the source code.

- We made a mistake on Maven releases that we took the same solution as
PyPI without carefully reviewing that it complies with the license
requirement.

With my PPMC hat off, as the individual who made releases to PyPI:

- I do NOT intend to confuse the users of our PyPI releases to be the act
of MXNet PPMC as I didn't use the apache-mxnet as PyPI package name, which
is specified by the draft policy on releases [1]. I will initiate review
with Apache Trademark to make sure it is clear.

- I do NOT intend to confuse the users of our PyPI releases to be under
Apache License 2.0 and it's an artifact from using the same build script. I
will seek input from Apache Legal on LEGAL-515 to clarify what the proper
licensing should be.

With my PPMC hat on, we will take the following actions:

- We will try to take down problematic Maven releases immediately and
discuss in the community how to proceed with future Maven releases.

- We will introduce source releases on PyPI and Maven using apache-mxnet
name to provide users with the option to have Apache distribution and allow
users to compile CUDA/cuDNN code as an option.


Finally, we appreciate any lesson other projects can share on licensing and
distribution that involves CUDA code. Thanks.

Regards,

Sheng

On behalf of Apache MXNet (Incubating) PPMC


On Fri, May 22, 2020 at 8:49 PM Justin Mclean  wrote:

> Hi,
>
> The incubator report had the following feedback:
>  Incubator needs to address the software distribution issues
>  regarding MXNet (not reporting this month). The PPMC is
>  effectively bypassing our software release policies by creating
>  distribution packages that are combined with non-open-source
>  platform libraries and publishing them on dist.mxnet.io, where
>  they are picked up and distributed using the PyPi channel to
>  all Python users. The resulting pages on PyPi mislead users
>  into thinking they are installing an Apache License 2.0 package
>  even though it contains software that we are not allowed to
>  distribute under our license.
>  https://issues.apache.org/jira/browse/LEGAL-515
>  https://dist.mxnet.io/python/cu102
>  https://pypi.org/project/mxnet-cu102/
>
> I can see you have been discussing this here [1] so some progress has been
> made. It would be a good idea to keep the Incubator PMC in the loop on what
> actions the PMC is going to take to resolve this.
>
> Thanks,
> Justin
>
> 1. https://s.apache.org/5102c
>
>

Re: [apache/incubator-mxnet] [RFC] MXNet 2.0 JVM Language development (#17783)

[Sheng Zha]

My understanding is that DJL depends on MXNet, so if you want to bring JNA from 
DJL into MXNet, it will create circular dependency as a 3rdparty module. In 
terms of stability, I was referring to the development of code base rather than 
the performance.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/17783#issuecomment-620815186

Re: [apache/incubator-mxnet] [RFC] MXNet 2.0 JVM Language development (#17783)

[Sheng Zha]

@lanking520 would it create circular dependency? and how stable is the JNA and 
what changes are expected? it would be great if you could share a pointer on 
the JNA code to help clarify these concerns.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/17783#issuecomment-620803313

Re: [apache/incubator-mxnet] [RFC] MXNet 2.0 API Deprecation (#17676)

[Sheng Zha]

mxnet.rnn module should be deprecated and removed too given it's designed for 
interacting with symbol API.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/17676#issuecomment-619624835

Re: [apache/incubator-mxnet] [RFC] MXNet 2.0 API Deprecation (#17676)

[Sheng Zha]

@zhreshold thanks for bringing this up. Currently the test for that is the 
longest running one too, so if there's no objection I hope that we could move 
forward in removing it soon

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/17676#issuecomment-615394771

Re: [apache/incubator-mxnet] [RFC] MXNet 2.0 API Deprecation (#17676)

[Sheng Zha]

caffe usage is very low now and let's deprecate caffe converter too.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/17676#issuecomment-613546835

Re: Podling Mxnet Report Reminder - April 2020

[Sheng Zha]

I posted the draft at 
https://cwiki.apache.org/confluence/display/INCUBATOR/April2020#MXNet. 
Suggestions and updates are welcome. Thanks.

-sz

On 2020/03/25 18:18:04, Sheng Zha  wrote: 
> I'm working on a draft now and will report back with link once ready.
> 
> -sz
> 
> On 2020/03/20 03:18:09, jmcl...@apache.org wrote: 
> > Dear podling,
> > 
> > This email was sent by an automated system on behalf of the Apache
> > Incubator PMC. It is an initial reminder to give you plenty of time to
> > prepare your quarterly board report.
> > 
> > The board meeting is scheduled for Wed, 15 April 2020, 10:30 am PDT.
> > The report for your podling will form a part of the Incubator PMC
> > report. The Incubator PMC requires your report to be submitted 2 weeks
> > before the board meeting, to allow sufficient time for review and
> > submission (Wed, April 01).
> > 
> > Please submit your report with sufficient time to allow the Incubator
> > PMC, and subsequently board members to review and digest. Again, the
> > very latest you should submit your report is 2 weeks prior to the board
> > meeting.
> > 
> > Candidate names should not be made public before people are actually
> > elected, so please do not include the names of potential committers or
> > PPMC members in your report.
> > 
> > Thanks,
> > 
> > The Apache Incubator PMC
> > 
> > Submitting your Report
> > 
> > --
> > 
> > Your report should contain the following:
> > 
> > *   Your project name
> > *   A brief description of your project, which assumes no knowledge of
> > the project or necessarily of its field
> > *   A list of the three most important issues to address in the move
> > towards graduation.
> > *   Any issues that the Incubator PMC or ASF Board might wish/need to be
> > aware of
> > *   How has the community developed since the last report
> > *   How has the project developed since the last report.
> > *   How does the podling rate their own maturity.
> > 
> > This should be appended to the Incubator Wiki page at:
> > 
> > https://cwiki.apache.org/confluence/display/INCUBATOR/April2020
> > 
> > Note: This is manually populated. You may need to wait a little before
> > this page is created from a template.
> > 
> > Note: The format of the report has changed to use markdown.
> > 
> > Mentors
> > ---
> > 
> > Mentors should review reports for their project(s) and sign them off on
> > the Incubator wiki page. Signing off reports shows that you are
> > following the project - projects that are not signed may raise alarms
> > for the Incubator PMC.
> > 
> > Incubator PMC
> > 
> 

Re: [apache/incubator-mxnet] [RFC] Deferred compute in imperative interface to unify imperative and symbolic interface (#16376)

[Sheng Zha]

Closed #16376.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/16376#event-3175578824

Re: [apache/incubator-mxnet] [RFC] Deferred compute in imperative interface to unify imperative and symbolic interface (#16376)

[Sheng Zha]

This feature has been merged in #17530. Thanks for the great work @leezu 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/16376#issuecomment-605558414

Re: Podling Mxnet Report Reminder - April 2020

[Sheng Zha]

I'm working on a draft now and will report back with link once ready.

-sz

On 2020/03/20 03:18:09, jmcl...@apache.org wrote: 
> Dear podling,
> 
> This email was sent by an automated system on behalf of the Apache
> Incubator PMC. It is an initial reminder to give you plenty of time to
> prepare your quarterly board report.
> 
> The board meeting is scheduled for Wed, 15 April 2020, 10:30 am PDT.
> The report for your podling will form a part of the Incubator PMC
> report. The Incubator PMC requires your report to be submitted 2 weeks
> before the board meeting, to allow sufficient time for review and
> submission (Wed, April 01).
> 
> Please submit your report with sufficient time to allow the Incubator
> PMC, and subsequently board members to review and digest. Again, the
> very latest you should submit your report is 2 weeks prior to the board
> meeting.
> 
> Candidate names should not be made public before people are actually
> elected, so please do not include the names of potential committers or
> PPMC members in your report.
> 
> Thanks,
> 
> The Apache Incubator PMC
> 
> Submitting your Report
> 
> --
> 
> Your report should contain the following:
> 
> *   Your project name
> *   A brief description of your project, which assumes no knowledge of
> the project or necessarily of its field
> *   A list of the three most important issues to address in the move
> towards graduation.
> *   Any issues that the Incubator PMC or ASF Board might wish/need to be
> aware of
> *   How has the community developed since the last report
> *   How has the project developed since the last report.
> *   How does the podling rate their own maturity.
> 
> This should be appended to the Incubator Wiki page at:
> 
> https://cwiki.apache.org/confluence/display/INCUBATOR/April2020
> 
> Note: This is manually populated. You may need to wait a little before
> this page is created from a template.
> 
> Note: The format of the report has changed to use markdown.
> 
> Mentors
> ---
> 
> Mentors should review reports for their project(s) and sign them off on
> the Incubator wiki page. Signing off reports shows that you are
> following the project - projects that are not signed may raise alarms
> for the Incubator PMC.
> 
> Incubator PMC
> 

Re: [apache/incubator-mxnet] [RFC] MXNet 2.0 JVM Language development (#17783)

[Sheng Zha]

+1 for option 1 and 2

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/17783#issuecomment-599800518

Re: [apache/incubator-mxnet] [RFC] MXNet 2.0 API Deprecation (#17676)

[Sheng Zha]

@lanking520 @zachgk @terrytangyuan @aaronmarkham could one of you start a 
discussion in a new issue on the JVM ecosystem support in 2.0? This topic seems 
to require extended discussion.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/17676#issuecomment-595911490

Re: [apache/incubator-mxnet] [RFC] MXNet 2.0 API Deprecation (#17676)

[Sheng Zha]

I vote for "upgrade/rewrite Scala API and bring up MXNet 2.0 features" as
it took us a lot of efforts to bring MXNet to Scala originally and there
are already adopters of Scala API in industries.

On Fri, Mar 6, 2020 at 11:02 AM Wang Jiajun 
wrote:

> > We may also drop ONNX in MXNet 2. I'm not aware of anyone working on
> ONNX in MXNet and TVM can be used as a replacement.
>
> +1 for keeping ONNX support. Although it has a lot of small problems, but
> I've converted a lot of pytorch models to mxnet for deploying with the
> following pipeline:
>
> https://docs.aws.amazon.com/dlami/latest/devguide/tutorial-onnx-pytorch-mxnet.html
>
> --
> You are receiving this because you authored the thread.
> Reply to this email directly or view it on GitHub:
>
> https://github.com/apache/incubator-mxnet/issues/17676#issuecomment-595835658



-- 
Yuan Tang
https://terrytangyuan.github.io/about/ 



-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/17676#issuecomment-595840489

Re: [apache/incubator-mxnet] [RFC] New Branches for MXNet 1.x, 1.7.x, and 2.x (#17701)

[Sheng Zha]

Let's continue the discussion on website in #17497 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/17701#issuecomment-595544177

Re: [apache/incubator-mxnet] [RFC] New Branches for MXNet 1.x, 1.7.x, and 2.x (#17701)

[Sheng Zha]

Closed #17701.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/17701#event-3103498526

Re: [apache/incubator-mxnet] [RFC] MXNet 2.0 API Deprecation (#17676)

[Sheng Zha]

@TaoLv the search result shows API in the following categories:
- operator (these will be deprecated and the newest version should be covered 
in https://github.com/apache/incubator-mxnet/issues/17096)
- gluon blocks (e.g. Con**v1**D). they are not legacy ops and will be kept
- io (these will be deprecated and replacement is covered in 2.0 roadmap item 
4.8 data API enhancement)
- model zoo (e.g. ResNet**V1**). they are not legacy API and will be kept

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/17676#issuecomment-595018894

Re: [apache/incubator-mxnet] [RFC] New Branches for MXNet 1.x, 1.7.x, and 2.x (#17701)

[Sheng Zha]

Website versioning is tracked in 
https://github.com/apache/incubator-mxnet/issues/17497

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/17701#issuecomment-594916076

Re: [apache/incubator-mxnet] [RFC] New Branches for MXNet 1.x, 1.7.x, and 2.x (#17701)

[Sheng Zha]

Let's switch the doc generation for the website to 1.x for now. 2.0 doc should 
be generated when version selection of the website is available.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/17701#issuecomment-593653255

Re: [apache/incubator-mxnet] [RFC] New Branches for MXNet 1.x, 1.7.x, and 2.x (#17701)

[Sheng Zha]

Pinging @apache/mxnet-committers again in case there's any more concern. 
Otherwise I will assume lazy consensus by the end of Wednesday of this week.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/17701#issuecomment-593638561

Re: [apache/incubator-mxnet] [RFC] MXNet 2.0 API Deprecation (#17676)

[Sheng Zha]

TensorRT support is currently using ONNX to convert from NNVM: 
https://github.com/apache/incubator-mxnet/blob/746cbc55fd666bb4529e88d247fed8e0907270f9/src/operator/subgraph/tensorrt/tensorrt.cc#L313-L318

Although I would like to see TensorRT support moved away from ONNX with a 
native integration using the Accelerator API compile support: 
https://github.com/apache/incubator-mxnet/pull/17623. But the migration from 
ONNX to AccAPI is still in discussion and the compile support PR is not merged 
yet (shameless plug: please review! :-D)

Sam

On Feb 28, 2020, at 9:06 PM, JackieWu 
mailto:notificati...@github.com>> wrote:

I think we should keep ONNX APIs, since it is able to export many basic models, 
although it is not perfect. Users will train their models in MXNet 2.0, and 
export ONNX model,  then use the ONNX model in their deployment frameworks. 
(http://onnx.ai/supported-tools).

It is useful to attract users to use MXNet 2.0 to train their models with ONNX.

--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/17676#issuecomment-592878029



-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/17676#issuecomment-593574187

Re: [apache/incubator-mxnet] [RFC] New Branches for MXNet 1.x, 1.7.x, and 2.x (#17701)

[Sheng Zha]

Contributors are encouraged to create PR against master branch. Since master 
branch is the development branch for 2.0, backward incompatible changes will be 
allowed, while 1.x branch only allows backward compatible changes.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/17701#issuecomment-592984631

Re: [apache/incubator-mxnet] [RFC] New Branches for MXNet 1.x, 1.7.x, and 2.x (#17701)

[Sheng Zha]

cc @apache/mxnet-committers 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/17701#issuecomment-592944338

[apache/incubator-mxnet] [RFC] New Branches for MXNet 1.x, 1.7.x, and 2.x (#17701)

[Sheng Zha]

Hi,

I created the following branches in the MXNet repo: 1.x, 1.7.x. I propose that 
these branches will function as the following:
- 1.7.x: release branch for 1.7.x versions. 1.7.0 roadmap is specified at 
https://github.com/apache/incubator-mxnet/issues/16864
- 1.x: development branch for MXNet's future 1.x versions (e.g. 1.8.x, 1.9.x, 
...)
- master: development branch for 2.x. 2.0 roadmap is specified at 
https://github.com/apache/incubator-mxnet/issues/16167

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/17701

Re: [apache/incubator-mxnet] [RFC] MXNet 2.0 API Deprecation (#17676)

[Sheng Zha]

cc @apache/mxnet-committers 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/17676#issuecomment-591743914

[apache/incubator-mxnet] [RFC] MXNet 2.0 API Deprecation (#17676)

[Sheng Zha]

As the MXNet community is working on the next major version of MXNet as 
described in #16167, this RFC seeks to clarify the scope of API deprecation, to 
inform the community of the replacement API design, and to ensure informed 
consensus.

Thanks to the long history of MXNet and the accumulated efforts of the 
community, MXNet now supports a wide range of neural network model training and 
deployment use cases. Many of these use cases have seen several generations of 
API design and implementation. Take model training as an example, there have 
been the Symbol Model API, Symbol Module API, and Gluon Hybrid Block API, all 
of which coexist in MXNet. Older generations of API often have a significant 
body of users and thus require time from the community to maintain, even though 
the supported use cases are mostly covered by a newer generation of API. Such 
requirement for maintenance not only consumes time and energy of the MXNet 
community and can distract the community from its longer term goal, but also 
causes pressure on CI, binary distribution.

In this RFC, we list several candidate API to be deprecated and the 
corresponding new generation of API as replacement. Unless otherwise stated, 
these APIs will continue to be supported in the future 1.x releases that happen 
in parallel to the 2.0 development. On the other hand, participating in the RFC 
for the new replacement feature of the feature you are interested in is the 
best way to ensure continued support in 2.0 for that feature. To make it easier 
to navigate, the replacement feature RFCs are linked in each section.

To make the discussion more productive, I recommend the following actions:

* If a feature in MXNet that you are interested in currently depend on any of 
the deprecated API, and you plan to switch from 1.x to 2.0, please participate 
in the RFC for the replacement feature. Please also direct any related 
questions with respect to how the replacement feature covers your use cases to 
the replacement feature RFCs.
* If after discussion in the replacement RFCs, you believe that the replacement 
feature cannot replace the candidate feature for deprecation, please call out 
in this RFC.
* Make sure to include your argument on why it’s the case, and clarify what 
use case cannot be supported in the new feature.
* If you wish to commit time and sponsor the continued maintenance beyond 
what’s specified in the following sections, please state so along with your 
comment.
* You may also seek other community members to sponsor the feature as 
comments in this RFC.
* The group of sponsors needs to collectively clarify what additional 
support the feature will receive from them, and commit to the cost of 
maintenance, development, and operations (such as CI).


Please always keep the discussion civilized and informative. Comments otherwise 
will be folded.

## mxnet.numpy and mxnet.ndarray

Traditionally MXNet provided `mx.nd` API with operators inspired, but often 
incompatible with Numpy. Based on RFC 
[#14253](https://github.com/apache/incubator-mxnet/issues/14253) there has been 
a large and ongoing effort to provide Numpy compatible operators in `mx.np` 
namespace.
This means that MXNet currently provides two incompatible APIs with separate 
backend implementations achieving similar goals, doubling the maintenance 
burden of developers. Note that there are some deep learning operators in 
`mx.nd` that don't have the counterparts in `mx.np`. These operators will be 
migrated to `mx.npx` namespace and will be tracked in #17096.

Given the wide impact of this decision, these people convened on 2/19/2020 and 
reached consensus on recommending **Removal** and parallel maintenance of 1.x 
and 2.x as the option forward: @eric-haibin-lin, @mli, @haojin2, @szhengac, 
@yizhiliu, @sxjscience, @reminisce, @leezu, @zhreshold, @apeforest, @oorqueda, 
@rondogency

| Options | Removal 

| Deprecation   




  | Separate Compatibility API  




  |

Re: MXNet CD pipelines cost savings

[Sheng Zha]

Hi Marco,

I've taken up the task to fix the CD pipeline [1] and it's pending CI checks 
and merge. I also made the adjustments to the namespace layout of the mxnet s3 
bucket and updated the static page which is now accessible from 
https://repo.mxnet.io/dist/index.html.

Meanwhile, the access from the CodeBuild to publish to the mxnet s3 bucket has 
been revoked so its status should no longer be relevant. I will update the 
installation doc to reflect only the artifacts published by Jenkins CD. I hope 
this brings conclusion to this situation.

Let me know if there's any further question.

-sz

[1] https://github.com/apache/incubator-mxnet/pull/17551

On 2020/02/10 23:57:47, Marco de Abreu  wrote: 
> Thanks for bringing this to our attention.
> 
> I'm quite devastated that our two vetoes have been ignored and the
> CodeBuild pipeline is actually supplying our user-facing binaries.
> Suggesting to turn of the Jenkins based CD now adds insult to injury.
> 
> I'd like to hear a plan how to make the project compliant again. I already
> announced that I will remove any mentions of that publishing method (speak,
> all links on our website pointing to the bucket) if the sourcing system is
> not our Jenkins CD. So far I believed that this was actually done, but
> seems like we got played here.
> 
> For the sake of our users, I'm giving one week (until 2/17) to come up with
> a proposal and until the end of the month 2/29 to have CodeBuild turned off
> and Jenkins CD fixed. Considering we have been tricked last time, I want to
> have confirmation that CodeBuild has been turned off and a description how
> we can verify that all artifacts are now coming from Jenkins CD.
> 
> Best regards
> Marco
> 
> Zha, Sheng  schrieb am Mo., 10. Feb. 2020,
> 19:40:
> 
> > +dev@
> >
> > -sz
> >
> > On Feb 10, 2020, at 1:35 PM, Zha, Sheng  wrote:
> >
> >  As already stated in the public threads, I’ve vetoed the CodeBuild
> > solution from becoming the long term solution as it’s not publicly
> > manageable.
> >
> > As communicated before, the team should have put efforts in maintaining
> > and fixing the Jenkins CD pipeline but has neglected to do so. Promoting
> > the CodeBuild solution this way is a step in the wrong direction that has
> > to be stopped.
> >
> > -sz
> >
> > On Feb 10, 2020, at 1:13 PM, Davydenko, Denis  wrote:
> >
> > 
> > Hello guys,
> >
> > I would like to start this discussion so that we can align on handling CD
> > pipelines we currently have. There are two of them: one in Jenkins<
> > http://jenkins.mxnet-ci.amazon-ml.com/job/restricted-mxnet-cd/> and one
> > in CodeBuild. The one in
> > Jenkins is currently functioning but its runs are always failing. The one
> > in CodeBuild is currently functioning and publishing artifacts to S3 bucket<
> > https://tiny.amazon.com/39negmk0/IsenLink>.
> >
> > MXNet Engineering team proposal is to shut down Jenkins based CD
> > completely as it is currently just a waste of resources and use CodeBuild
> > based setup to continue publishing nightly builds to S3 bucket, which
> > provides public access to all binaries stored in it. This doesn’t affect a
> > discussion of whether to publish binaries to S3 or to pypi – once that
> > concludes (if ever) we can switch destination of CodeBuild projects so that
> > they would upload MXNet nightly binaries to pypi instead of S3.
> >
> > This is an effort to get alignment internally, if possible, before
> > bringing this as a proposal for community discussion.
> >
> > --
> > Thanks,
> > Denis
> >
> 

Re: [VOTE] Release Apache MXNet (incubating) version 1.6.0.rc2

[Sheng Zha]

+1 as the disclaimer-WIP is now used which will allow us more time to fix 
license issues.

On 2020/02/04 02:43:42, Chaitanya Bapat  wrote: 
> +1
> Successfully built MXNet 1.6.0rc2 on Linux
> Tested for OpPerf utility
> For CPU -
> https://gist.github.com/ChaiBapchya/d5ecc3e971c5a3c558d672477b4b6b9c
> 
> Works well!
> 
> 
> 
> On Mon, 3 Feb 2020 at 15:43, Lin Yuan  wrote:
> 
> > +1
> >
> > Tested Horovod with mnist example. My compiler flags are below:
> >
> > [✔ CUDA, ✔ CUDNN, ✔ NCCL, ✔ CUDA_RTC, ✖ TENSORRT, ✔ CPU_SSE, ✔ CPU_SSE2, ✔
> > CPU_SSE3, ✔ CPU_SSE4_1, ✔ CPU_SSE4_2, ✖ CPU_SSE4A, ✔ CPU_AVX, ✖ CPU_AVX2, ✔
> > OPENMP, ✖ SSE, ✔ F16C, ✖ JEMALLOC, ✔ BLAS_OPEN, ✖ BLAS_ATLAS, ✖ BLAS_MKL, ✖
> > BLAS_APPLE, ✔ LAPACK, ✖ MKLDNN, ✔ OPENCV, ✖ CAFFE, ✖ PROFILER, ✔
> > DIST_KVSTORE, ✖ CXX14, ✖ INT64_TENSOR_SIZE, ✖ SIGNAL_HANDLER, ✖ DEBUG, ✖
> > TVM_OP]
> >
> > Lin
> >
> > On Sat, Feb 1, 2020 at 9:55 PM Tao Lv  wrote:
> >
> > > +1
> > >
> > > I tested below items:
> > > 1. download artifacts from Apache dist repo;
> > > 2. the signature looks good;
> > > 3. build from source code with MKL-DNN and MKL on centos;
> > > 4. run fp32 and int8 inference of ResNet50 under /example/quantization/.
> > >
> > > thanks,
> > > -tao
> > >
> > > On Sun, Feb 2, 2020 at 11:00 AM Tao Lv  wrote:
> > >
> > > > I see. I was looking at this page:
> > > > https://github.com/apache/incubator-mxnet/releases/tag/1.6.0.rc2
> > > >
> > > > On Sun, Feb 2, 2020 at 4:54 AM Przemysław Trędak 
> > > > wrote:
> > > >
> > > >> Hi Tao,
> > > >>
> > > >> Could you tell me where did you look for it and did not find it? I
> > just
> > > >> checked and both
> > > >> https://dist.apache.org/repos/dist/dev/incubator/mxnet/1.6.0.rc2/ and
> > > >> draft of the release on GitHub have them.
> > > >>
> > > >> Thank you
> > > >> Przemek
> > > >>
> > > >> On 2020/02/01 14:23:11, Tao Lv  wrote:
> > > >> > It seems the src tar and signature are missing from the tag.
> > > >> >
> > > >> > On Fri, Jan 31, 2020 at 11:09 AM Przemysław Trędak <
> > > ptre...@apache.org>
> > > >> > wrote:
> > > >> >
> > > >> > > Dear MXNet community,
> > > >> > >
> > > >> > > This is the vote to release Apache MXNet (incubating) version
> > 1.6.0.
> > > >> > > Voting starts today and will close on Monday 2/3/2020 23:59 PST.
> > > >> > >
> > > >> > > Link to release notes:
> > > >> > >
> > > https://cwiki.apache.org/confluence/display/MXNET/1.6.0+Release+notes
> > > >> > >
> > > >> > > Link to release candidate:
> > > >> > > https://github.com/apache/incubator-mxnet/releases/tag/1.6.0.rc2
> > > >> > >
> > > >> > > Link to source and signatures on apache dist server:
> > > >> > > https://dist.apache.org/repos/dist/dev/incubator/mxnet/1.6.0.rc2/
> > > >> > >
> > > >> > > The differences comparing to previous release candidate 1.6.0.rc1:
> > > >> > >  * Fixes for license issues (#17361, #17375, #17370, #17460)
> > > >> > >  * Bugfix for saving LSTM layer parameter (#17288)
> > > >> > >  * Bugfix for downloading the model from model zoo from multiple
> > > >> processes
> > > >> > > (#17372)
> > > >> > >  * Fixed a symbol.py in AMP for GluonNLP (#17408)
> > > >> > >
> > > >> > >
> > > >> > > Please remember to TEST first before voting accordingly:
> > > >> > > +1 = approve
> > > >> > > +0 = no opinion
> > > >> > > -1 = disapprove (provide reason)
> > > >> > >
> > > >> > >
> > > >> > > Best regards,
> > > >> > > Przemyslaw Tredak
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
> 
> 
> -- 
> *Chaitanya Prakash Bapat*
> *+1 (973) 953-6299*
> 
> [image: https://www.linkedin.com//in/chaibapat25]
> [image: https://www.facebook.com/chaibapat]
> [image:
> https://twitter.com/ChaiBapchya] [image:
> https://www.linkedin.com//in/chaibapat25]
> 
> 

Re: [DISCUSS] Enforce tighter control on API related changes

[Sheng Zha]

> 2)  Regarding issue #17292, it was not broken by 4ed14e2 but an C API
> change in in https://github.com/apache/incubator-mxnet/pull/17128. The
> later commit 4ed14e2 was trying to fix this API change but it did not seem
> to work yet.

None of the existing C API was changed in #17128. #17128 had an unnecessary 
addition of a C API which was removed in 4ed14e2. Neither change should have 
broken third party integration if it's not making assumptions on where to find 
the implementation.

As I don't see any further discussion on this in the issue #17292, let's make 
sure the related details are added there please.

-sz

On 2020/01/14 18:24:13, Lin Yuan  wrote: 
> Hi Sheng,
> 
> Thanks for your reply.
> 
> 1) Adding a "API Change" label is a good way to flag PRs with API change.
> It would be great if we could make this labeling automatic with some hook
> in API related modules, so we don't miss them in PRs.
> 
> 2)  Regarding issue #17292, it was not broken by 4ed14e2 but an C API
> change in in https://github.com/apache/incubator-mxnet/pull/17128. The
> later commit 4ed14e2 was trying to fix this API change but it did not seem
> to work yet.
> 
> Horovod integration does not call any inline function from MXNet, it
> includes an exported header c_api_error.h from mxnet to throw and catch
> mxnet exception. Same header is included in other project, such as BytePS
> https://github.com/bytedance/byteps/blob/master/byteps/mxnet/ops.h#L22.
> 
> I agree with you we need a better design to allow other third party
> libraries to build on top of MXNet. E.g. TensorFlow provides customer
> operators for Horovod to push their allreduce actions to TensorFlow as a
> Custom Operator instead of low-level C API calls. It seems our Custom
> Dynamic Operator
> <https://cwiki.apache.org/confluence/display/MXNET/Dynamic+CustomOp+Support>
> project may enable this feature in MXNet 2.0 and I am looking forward to it
> :)
> 
> Cheers,
> 
> Lin
> 
> 
> 
> 
> On Mon, Jan 13, 2020 at 7:24 PM Sheng Zha  wrote:
> 
> > Hi Lin,
> >
> > Thanks for the suggestions.
> >
> > With respect to your proposal:
> >
> > > (2) Any PR that contains API change should clearly state this in PR
> > title.
> > > Otherwise, committer can reject the PR
> >
> > I agree that PRs with API changes should be made more prominent. Another
> > mechanism that has already been used is to tag the PRs with the "API
> > change" label [1].
> >
> > On the other hand, relying on the community to call out the PRs with API
> > changes may not be reliable. Oftentimes, people didn't realize that a
> > change constitutes an API change. If a committer identifies such a change,
> > a more friendly response would be to just label the PR and call out where
> > the API change happens in a comment.
> >
> > > (1) Any PR involving change of APIs should be approved by at least one of
> > > the committers from a "API Committee" before it can be merged. I will
> > > explain how to form this committee at the end of email
> >
> > I'm not convinced that more hierarchy should be created among committers.
> > All committers are entrusted by the PPMC to use their judgement to the best
> > interest of this project, and additional qualification seems
> > counter-productive.
> >
> > With respect to your linked issue #17292, as @stephenrawls pointed out, it
> > comes from 4ed14e2 where the inline definition of MXAPIHandleException is
> > moved to a .cc file, and I'm the one that actually made this change to
> > unblock the PR. I want to call out that:
> > - This is not an API change in that there's no change in the function
> > signature or visibility in the symbol table of libmxnet.so.
> > - It should not be the responsibility of MXNet to maintain the assumption
> > that downstream projects like horovod make in their building logic.
> >
> > A more pressing issue should have been the way that a third-party
> > communication library like horovod integrates with MXNet. So far the
> > horovod integration seemed brittle and there have been many issues [2]. For
> > this specific issue, to me, it doesn't seem like a good decision on the
> > horovod side to assume any function would be defined inline on the MXNet
> > side.
> >
> > With the development of MXNet 2.0, it's a good time to rethink how horovod
> > integration should work with MXNet. I'm hoping that MXNet 2.0 item 4.11
> > AbstractKVStore interface (See #17115) could help simplify and alleviate
> > the coupling in the current way of integration.
> >
> > -sz
> >
&

Re: [DISCUSS] Enforce tighter control on API related changes

[Sheng Zha]

Hi Lin,

Thanks for the suggestions.

With respect to your proposal:

> (2) Any PR that contains API change should clearly state this in PR title.
> Otherwise, committer can reject the PR

I agree that PRs with API changes should be made more prominent. Another 
mechanism that has already been used is to tag the PRs with the "API change" 
label [1].

On the other hand, relying on the community to call out the PRs with API 
changes may not be reliable. Oftentimes, people didn't realize that a change 
constitutes an API change. If a committer identifies such a change, a more 
friendly response would be to just label the PR and call out where the API 
change happens in a comment.

> (1) Any PR involving change of APIs should be approved by at least one of
> the committers from a "API Committee" before it can be merged. I will
> explain how to form this committee at the end of email

I'm not convinced that more hierarchy should be created among committers. All 
committers are entrusted by the PPMC to use their judgement to the best 
interest of this project, and additional qualification seems counter-productive.

With respect to your linked issue #17292, as @stephenrawls pointed out, it 
comes from 4ed14e2 where the inline definition of MXAPIHandleException is moved 
to a .cc file, and I'm the one that actually made this change to unblock the 
PR. I want to call out that:
- This is not an API change in that there's no change in the function signature 
or visibility in the symbol table of libmxnet.so.
- It should not be the responsibility of MXNet to maintain the assumption that 
downstream projects like horovod make in their building logic.

A more pressing issue should have been the way that a third-party communication 
library like horovod integrates with MXNet. So far the horovod integration 
seemed brittle and there have been many issues [2]. For this specific issue, to 
me, it doesn't seem like a good decision on the horovod side to assume any 
function would be defined inline on the MXNet side.

With the development of MXNet 2.0, it's a good time to rethink how horovod 
integration should work with MXNet. I'm hoping that MXNet 2.0 item 4.11 
AbstractKVStore interface (See #17115) could help simplify and alleviate the 
coupling in the current way of integration.

-sz

[1] 
https://github.com/apache/incubator-mxnet/pulls?utf8=%E2%9C%93=is%3Apr+label%3A%22API+change%22+
[2] 
https://github.com/apache/incubator-mxnet/issues?utf8=%E2%9C%93=is%3Aissue+horovod

On 2020/01/14 00:37:55, Lin Yuan  wrote: 
> Dear Community,
> 
> Recently, there were some changes to C APIs that broke another downstream
> project Horovod: https://github.com/apache/incubator-mxnet/issues/17292.
> Since we do not have integration tests for downstream project, it becomes
> critical for us to update APIs with extreme caution.
> 
> I would like to propose the following mechanism for us to maintain a clean
> and robust APIs: including both C API and Python API at the moment.
> 
> (1) Any PR involving change of APIs should be approved by at least one of
> the committers from a "API Committee" before it can be merged. I will
> explain how to form this committee at the end of email
> 
> (2) Any PR that contains API change should clearly state this in PR title.
> Otherwise, committer can reject the PR
> 
> API Committee:
> - This committee should consist of both seasoned MXNet developers and users.
> - Committee members should have a comprehensive view of MXNet APIs to make
> sure their usage are consistent across stack.
> - Committee members review PRs that involve API change with extra caution.
> - Committee members are required to attend the roadmap discussion for each
> new release.
> - For API breaking changes, committe members should reach consensus before
> the change is made.
> 
> Any other suggestion is welcome here.
> 
> Best,
> 
> Lin
> 

Re: Stopping nightly releases to Pypi

[Sheng Zha]

Size of a change doesn't necessarily reflect the time one spends on the 
navigating the code base and finding the solution. Also, I tend to believe that 
everyone genuinely wants what's best for the project, just from different 
perspectives.

Let's focus on improving the CD solution so that security concerns can be 
addressed too.

-sz

On 2020/01/09 21:57:30, Chris Olivier  wrote: 
> If this tiny fix is representative of the bulk of the reasoning behind all
> the the CD churn recently, then this seems to be of some concern.
> 
> -Chris
> 
> On Thu, Jan 9, 2020 at 6:32 AM Marco de Abreu 
> wrote:
> 
> > Great, thanks a lot sheng!
> >
> > -Marco
> >
> > Sheng Zha  schrieb am Do., 9. Jan. 2020, 14:28:
> >
> > > I'm fixing the CD pipeline in
> > > https://github.com/apache/incubator-mxnet/pull/17259/files and will
> > > update the s3 publish path so that it's friendly for automatically
> > > generating such page.
> > >
> > > -sz
> > >
> > > On 2020/01/06 18:19:52, "Lausen, Leonard" 
> > > wrote:
> > > > Consider a user finds a bug in a nightly version but we can't narrow
> > > down the
> > > > version of mxnet used as the name is constant over time. Or users wan't
> > > to
> > > > revert back to the previous nightly version installed but don't know
> > > which date
> > > > it was from due to constant name.
> > > >
> > > > Instead I suggest we introduce an autogenerated page like
> > > > https://download.pytorch.org/whl/nightly/cu101/torch_nightly.html
> > > >
> > > > Then "pip install -f URLTOPAGE mxnet" will install the latest available
> > > version.
> > > > Maybe the team maintaining the S3 bucket can reconsider creating such
> > > page for
> > > > the intermediate time until the CD based nighlty build is operating.
> > > >
> > > > On Mon, 2020-01-06 at 10:01 -0800, Lin Yuan wrote:
> > > > > +1 for a nightly pip with fixed name.
> > > > >
> > > > > We need this to track mxnet integration with other packages such as
> > > Horovod.
> > > > >
> > > > > Sam, when do you think we can have this nightly build with a fixed
> > > name?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Lin
> > > > >
> > > > > On Sun, Jan 5, 2020 at 7:48 PM Skalicky, Sam
> > > 
> > > > > wrote:
> > > > >
> > > > > > Hi Tao,
> > > > > >
> > > > > > We dont have this yet, but we did think about putting the latest
> > > wheels in
> > > > > > a specific place in the s3 bucket so they are always updated.
> > > Initially we
> > > > > > decided not to do this since the main MXNet CD should have been
> > > fixed. But
> > > > > > since its still not fixed yet, we might try and go ahead and do
> > this.
> > > > > >
> > > > > > Sam
> > > > > >
> > > > > > On Jan 5, 2020, at 6:02 PM, Lv, Tao A  > > > > > tao.a...@intel.com>> wrote:
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > How to install the latest available build of a flavor without
> > > specifying
> > > > > > the build date? Something like `pip install mxnet --pre`.
> > > > > >
> > > > > > Thanks,
> > > > > > -tao
> > > > > >
> > > > > > -Original Message-
> > > > > > From: Skalicky, Sam  > > > > > sska...@amazon.com.INVALID>>
> > > > > > Sent: Monday, January 6, 2020 2:09 AM
> > > > > > To: dev@mxnet.incubator.apache.org > > dev@mxnet.incubator.apache.org>
> > > > > > Subject: Re: Stopping nightly releases to Pypi
> > > > > >
> > > > > > Hi Haibin,
> > > > > >
> > > > > > You typed the correct URLs, the cu100 build has been failing since
> > > > > > December 30th but other builds have succeeded. The wheels are being
> > > > > > delivered into a public bucket that anyone with an AWS account can
> > > access
> > > > > > and go poke around, here’s the URL for web access:
> > > > > >
> > > > > >
> > > > >

Re: Stopping nightly releases to Pypi

[Sheng Zha]

.py3-none-manylinux1_x86_64.whl
> > > (mxnet-mkl
> > > <
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> > > >
> > > <
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-mkl
> > > 
> > > )
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > (mxnet-cuXXX
> > > <
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> > > >
> > > <
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXX
> > > 
> > > )
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > (mxnet-cuXXXmkl
> > > <
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> > > >
> > > <
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl(mxnet-cuXXXmkl
> > > 
> > > )
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu90mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > 
> > > You can easily install these pip wheels in your system either by
> > > downloading them to your machine first and then installing by
> > > doing:
> > > 
> > > pip install /path/to/downloaded/wheel.whl
> > > 
> > > Or you can install directly by just giving the link to pip like
> > > this:
> > > 
> > > pip install
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> > > 
> > > Credit goes to everyone involved (in no particular order) Rakesh Vasudevan
> > > Zach Kimberg Manu Seth Sheng Zha Jun Wu Pedro Larroy Chaitanya Bapat
> > > 
> > > Thanks!

Re: Podling Mxnet Report Reminder - January 2020

[Sheng Zha]

This is listed in the active projects and (I believe) still being worked on.

-sz

On 2019/12/31 12:04:20, Marco de Abreu  wrote: 
> Hey sheng,
> 
> Thanks for driving the report! Shall we maybe mention the reboot of the
> website and also include some statistics?
> 
> Best regards
> Marco
> 
> Sheng Zha  schrieb am Di., 31. Dez. 2019, 13:01:
> 
> > I just posted the draft at
> > https://cwiki.apache.org/confluence/display/INCUBATOR/January2020.
> >
> > -sz
> >
> > On 2019/12/28 06:04:57, Sheng Zha  wrote:
> > > I'm working on a draft and will post it to the link and update here soon.
> > >
> > > -sz
> > >
> > > On 2019/12/28 05:12:03, jmcl...@apache.org wrote:
> > > > Dear podling,
> > > >
> > > > This email was sent by an automated system on behalf of the Apache
> > > > Incubator PMC. It is an initial reminder to give you plenty of time to
> > > > prepare your quarterly board report.
> > > >
> > > > The board meeting is scheduled for Wed, 15 January 2020, 10:30 am PDT.
> > > > The report for your podling will form a part of the Incubator PMC
> > > > report. The Incubator PMC requires your report to be submitted 2 weeks
> > > > before the board meeting, to allow sufficient time for review and
> > > > submission (Wed, January 01).
> > > >
> > > > Please submit your report with sufficient time to allow the Incubator
> > > > PMC, and subsequently board members to review and digest. Again, the
> > > > very latest you should submit your report is 2 weeks prior to the board
> > > > meeting.
> > > >
> > > > Candidate names should not be made public before people are actually
> > > > elected, so please do not include the names of potential committers or
> > > > PPMC members in your report.
> > > >
> > > > Thanks,
> > > >
> > > > The Apache Incubator PMC
> > > >
> > > > Submitting your Report
> > > >
> > > > --
> > > >
> > > > Your report should contain the following:
> > > >
> > > > *   Your project name
> > > > *   A brief description of your project, which assumes no knowledge of
> > > > the project or necessarily of its field
> > > > *   A list of the three most important issues to address in the move
> > > > towards graduation.
> > > > *   Any issues that the Incubator PMC or ASF Board might wish/need to
> > be
> > > > aware of
> > > > *   How has the community developed since the last report
> > > > *   How has the project developed since the last report.
> > > > *   How does the podling rate their own maturity.
> > > >
> > > > This should be appended to the Incubator Wiki page at:
> > > >
> > > > https://cwiki.apache.org/confluence/display/INCUBATOR/January2020
> > > >
> > > > Note: This is manually populated. You may need to wait a little before
> > > > this page is created from a template.
> > > >
> > > > Note: The format of the report has changed to use markdown.
> > > >
> > > > Mentors
> > > > ---
> > > >
> > > > Mentors should review reports for their project(s) and sign them off on
> > > > the Incubator wiki page. Signing off reports shows that you are
> > > > following the project - projects that are not signed may raise alarms
> > > > for the Incubator PMC.
> > > >
> > > > Incubator PMC
> > > >
> > >
> >
> 

Re: [apache/incubator-mxnet] [RFC] Apache MXNet 2.0 Roadmap (#16167)

[Sheng Zha]

@TaoLv one promising direction that the community is converging to is the 
interface based on packed function (motivation as described by @tqchen in 
https://github.com/apache/incubator-mxnet/issues/17097#issuecomment-56760). 
What this means to the project is that the existing c API will be updated to 
follow the packed function interface.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/16167#issuecomment-569857628

Re: [apache/incubator-mxnet] [RFC][mxnet 2.0][item 10.1] MXNet Imperative Op Invocation Overhead (#17097)

[Sheng Zha]

Test

On Fri, Dec 27, 2019 at 11:54 AM Pedro Larroy 
wrote:

> Thanks for the explanation. I'm not so concerned about complexity of
> dispatching. If I understood you correctly the main benefit that you
> explain for the TVM project was not having to change the C API, but still
> you need to do type checking in both ends, or at least on the receiving end
> of the API, correct? I think we have discussed similar things in the past
> and we might have different views on strongly typed vs dynamic typed. A
> priori I prefer to see an API which can be evolved and changed, I find it
> more explicit and clearer that what I think you do with PackedFun which I
> have looked at briefly but not used extensively.  If one is going to call
> into the C API using pybind, does it make sense to layer a C++ API on top
> of the C API for this?
>
> Also these microbenchmarks are nice, but we also need to consider the
> overhead in typical workloads and see if it's still significant.
>
> CFFI is also another alternative.
>
> I couldn't access your pointers like:
>
> https://github.com/tqchen/tvm/tree/pyffi
>
> On Thu, Dec 26, 2019 at 2:00 PM Tianqi Chen 
> wrote:
>
>> @larroy indeed every solution has trade-offs, and these tradeoffs are
>> discussed in the above posts when we compare solutions, and they are backed
>> by benchmarks :) it would be great if you can also suggest potential
>> tradeoffs here.
>>
>> When you expose an API from typed language(c++) to a dynamic
>> language(python), you have to type erase it, given that the python
>> functions don't have the type, and you have to pass the information along.
>>
>> The only difference is where you do the type checking(that the python
>> type corresponds to the right c++ type), and translation(translating to the
>> c++ type).
>>
>> For example, in the case of pybind, the erasure is done implicitly when
>> you call the python function, then checking and translation happens when
>> you call into the c++ function.
>>
>> In the case of creating a C API for each feature and wrap things in the
>> python side, the type checking is done in the python side, and translation
>> as well.
>>
>> In the case of tvm ffi, the type translation is done in the python/cython
>> side,  while the type checking is done in the c++.
>>
>> To dive deeper into the tradeoffs for PackedFunc calling convention. The
>> convention erases the type by having the type code stored into the
>> arguments. This brings additional cost of passing arguments into heap, as
>> opposed to registers. So they might not be designed for inline functions
>> that needs to happen at the order of 1e-9s, however, for API functions that
>> needs to run around 1e-7 or even 1e-8 level, this convention is pretty good.
>>
>> In terms of the calling cost, it really depends on whether the caller and
>> callee are strongly typed.
>> - If caller is strongly typed, then assigning type code is O(1)
>> - If caller is a dynamic type(like python) then we need to have a
>> dispatcher to dispatch and select the right type code
>> - If callee is strongly typed, then the cost of checking is O(1) by just
>> check the code to be the correct one
>> - If the callee is dynamic type, then a dispatching need to happen, which
>> have another level of hashtable lookup O(1)
>>
>> As we can see, the only place where dispatching is necessary is the
>> dynamic type handling case. Even in these cases, if there is a strong need
>> of specialization, we can directly force the type by running checking on
>> the caller, and pass in the right type code (the engineering burden is the
>> same as wrapping the C API). However, the benchmark suggests that the
>> dynamic dispatching cost is reasonable, and satisfies the API speed.
>>
>> Coming back to the tradeoff, the main tradeoff here is the engineering
>> burden to keep an hourglass design(with fixed set of API) vs efficiency.
>> While my post did not suggest that TVM's ffi is a silver bullet, it does
>> works pretty well for our use cases. hope it helps
>>
>>
>> --
>> You are receiving this because you are subscribed to this thread.
>> Reply to this email directly or view it on GitHub:
>>
>> https://github.com/apache/incubator-mxnet/issues/17097#issuecomment-569139957
>
>


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/17097#issuecomment-569358961

Re: [apache/incubator-mxnet] [RFC][mxnet 2.0][item 10.1] MXNet Imperative Op Invocation Overhead (#17097)

[Sheng Zha]

Thanks for the explanation. I'm not so concerned about complexity of
dispatching. If I understood you correctly the main benefit that you
explain for the TVM project was not having to change the C API, but still
you need to do type checking in both ends, or at least on the receiving end
of the API, correct? I think we have discussed similar things in the past
and we might have different views on strongly typed vs dynamic typed. A
priori I prefer to see an API which can be evolved and changed, I find it
more explicit and clearer that what I think you do with PackedFun which I
have looked at briefly but not used extensively.  If one is going to call
into the C API using pybind, does it make sense to layer a C++ API on top
of the C API for this?

Also these microbenchmarks are nice, but we also need to consider the
overhead in typical workloads and see if it's still significant.

CFFI is also another alternative.

I couldn't access your pointers like:

https://github.com/tqchen/tvm/tree/pyffi

On Thu, Dec 26, 2019 at 2:00 PM Tianqi Chen 
wrote:

> @larroy indeed every solution has trade-offs, and these tradeoffs are
> discussed in the above posts when we compare solutions, and they are backed
> by benchmarks :) it would be great if you can also suggest potential
> tradeoffs here.
>
> When you expose an API from typed language(c++) to a dynamic
> language(python), you have to type erase it, given that the python
> functions don't have the type, and you have to pass the information along.
>
> The only difference is where you do the type checking(that the python type
> corresponds to the right c++ type), and translation(translating to the c++
> type).
>
> For example, in the case of pybind, the erasure is done implicitly when
> you call the python function, then checking and translation happens when
> you call into the c++ function.
>
> In the case of creating a C API for each feature and wrap things in the
> python side, the type checking is done in the python side, and translation
> as well.
>
> In the case of tvm ffi, the type translation is done in the python/cython
> side,  while the type checking is done in the c++.
>
> To dive deeper into the tradeoffs for PackedFunc calling convention. The
> convention erases the type by having the type code stored into the
> arguments. This brings additional cost of passing arguments into heap, as
> opposed to registers. So they might not be designed for inline functions
> that needs to happen at the order of 1e-9s, however, for API functions that
> needs to run around 1e-7 or even 1e-8 level, this convention is pretty good.
>
> In terms of the calling cost, it really depends on whether the caller and
> callee are strongly typed.
> - If caller is strongly typed, then assigning type code is O(1)
> - If caller is a dynamic type(like python) then we need to have a
> dispatcher to dispatch and select the right type code
> - If callee is strongly typed, then the cost of checking is O(1) by just
> check the code to be the correct one
> - If the callee is dynamic type, then a dispatching need to happen, which
> have another level of hashtable lookup O(1)
>
> As we can see, the only place where dispatching is necessary is the
> dynamic type handling case. Even in these cases, if there is a strong need
> of specialization, we can directly force the type by running checking on
> the caller, and pass in the right type code (the engineering burden is the
> same as wrapping the C API). However, the benchmark suggests that the
> dynamic dispatching cost is reasonable, and satisfies the API speed.
>
> Coming back to the tradeoff, the main tradeoff here is the engineering
> burden to keep an hourglass design(with fixed set of API) vs efficiency.
> While my post did not suggest that TVM's ffi is a silver bullet, it does
> works pretty well for our use cases. hope it helps
>
>
> --
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly or view it on GitHub:
>
> https://github.com/apache/incubator-mxnet/issues/17097#issuecomment-569139957


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/17097#issuecomment-569335211

Re: [VOTE] Release Apache MXNet (incubating) version 1.6.0.rc0

[Sheng Zha]

Reading these issues it’s pretty clear to me that these are fixes for broken 
builds. I think we do consider broken builds to be release blockers.

Lin, am I missing something on which you base your suggestion for delaying 
these changes?

-sz

> On Dec 27, 2019, at 10:30 AM, Lin Yuan  wrote:
> 
> Are these release blocker? It's very risky to make such last-minute big
> change after code freeze.
> 
> Can we do this in the next release?
> 
> Lin
> 
>> On Fri, Dec 27, 2019 at 7:37 AM Lausen, Leonard 
>> wrote:
>> 
>> In case of backporting #17012, also
>> https://github.com/apache/incubator-mxnet/pull/17098 must be backported.
>> The
>> updated OpenMP added a new target which is not used by MXNet but breaks the
>> build on some systems with nvptx. #17098 disables building this unused and
>> broken feature.
>> 
>>> On Thu, 2019-12-26 at 12:55 -0800, Pedro Larroy wrote:
>>> https://github.com/apache/incubator-mxnet/pull/17012  should be also
>> ported
>>> to the release branch.
>>> 
>>> On Fri, Dec 20, 2019 at 1:39 PM Przemysław Trędak 
>>> wrote:
>>> 
 That issue is now fixed in master, I am in the process of
>> cherry-picking
 the fix to v1.6.x branch. I will prepare the RC1 once that is ready.
 
 Thanks
 Przemek
 
 On 2019/12/20 20:07:36, Lin Yuan  wrote:
> What's the next step for the release? Should we continue testing
>> this and
> vote or wait until the
> https://github.com/apache/incubator-mxnet/issues/17105 is fixed?
> 
> Thanks!
> 
> Lin
> 
> On Wed, Dec 18, 2019 at 12:55 AM Lausen, Leonard
 
> wrote:
> 
>> Thanks Przemysław for managing this release and everyone who
 contributed
>> to it.
>> 
>> Unfortunately Zechen Wang just discovered another issue with GPU
 Pointwise
>> Fusion: https://github.com/apache/incubator-mxnet/issues/17105
>> 
>> Thus, -1.
>> 
>> Unfortunately, as the nightly release pipeline was broken until
 recently
>> (and
>> still isn't re-set up completely yet), the issue hasn't been
>> discovered
>> earlier.
>> 
>> Przemysław may have a quick fix for the issue. Another option
>> would be
 to
>> release 1.6 with MXNET_USE_FUSION default to 0.
>> 
>> Best regards
>> Leonard
>> 
>> On Wed, 2019-12-18 at 05:30 +, Chen, Ciyong wrote:
>>> Appreciate Tredak to push out voting for 1.6 release.
>>> 
>>> +1 as we've done lots of tests with expected performance in many
>> different
>>> scenarios including both single-node and multi-node (horovod
>> based),
>> both FP32
>>> and INT8 precision on many topologies.
>>> 
>>> -Ciyong
>>> 
>>> -Original Message-
>>> From: Zhao, Patric 
>>> Sent: Tuesday, December 17, 2019 8:51 AM
>>> To: dev@mxnet.incubator.apache.org; d...@mxnet.apache.org
>>> Subject: RE: [VOTE] Release Apache MXNet (incubating) version
 1.6.0.rc0
>>> Thanks, Tredak, I will add some words for the new feature in the
 release
>> note.
>>> +1 for voting because we have ran multiple time of tests in
>> local and
>> got the
>>> expected performance boost.
>>> 
>>> --Patric
>>> 
 -Original Message-
 From: Przemysław Trędak 
 Sent: Tuesday, December 17, 2019 4:49 AM
 To: d...@mxnet.apache.org
 Subject: [VOTE] Release Apache MXNet (incubating) version
>> 1.6.0.rc0
 
 Dear MXNet community,
 
 This is the vote to release Apache MXNet (incubating) version
 1.6.0.
 Voting starts now and will close on Friday, 20th December 2019
>> 23:59:59 PST.
 Link to release notes:
 
 https://cwiki.apache.org/confluence/display/MXNET/1.6.0+Release+notes
 Link to release candidate:
 
>> https://github.com/apache/incubator-mxnet/releases/tag/1.6.0.rc0
 
 Link to source and signatures on apache dist server:
 
>> https://dist.apache.org/repos/dist/dev/incubator/mxnet/1.6.0.rc0/
 
 Please remember to TEST first before voting accordingly:
 +1 = approve
 +0 = no opinion
 -1 = disapprove (provide reason)
 
 Additional notes:
 - There was an issue[1] raised that 1.6.0.rc0 does not build
>> with
 clang on FreeBSD - I decided to not block the voting for this
>> and
 instead let the Community decide whether this is a blocker for
>> the
>> release.
 - Patric Zhao and Tao Lv - could you help preparing a
>> paragraph on
 MKLDNN
 1.0 update in the New features section in the release notes?
 
 [1] https://github.com/apache/incubator-mxnet/issues/17076
 
 Best regards,
 Przemyslaw Tredak
>> 

Re: [apache/incubator-mxnet] [RFC] Apache MXNet 2.0 Roadmap (#16167)

[Sheng Zha]

Once 1.6 release is complete, we will create a branch for MXNet 1.x for future 
releases and start using master branch for 2.0 development.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/16167#issuecomment-565851523

Re: [apache/incubator-mxnet] [RFC] Apache MXNet 2.0 Roadmap (#16167)

[Sheng Zha]

The status of MXNet 2.0 project is tracked at: 
https://github.com/apache/incubator-mxnet/projects/18. The status for each 
project will be updated by the contributor who's driving it. If you have more 
projects that you intend to drive please first discuss here.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/16167#issuecomment-565851392

Re: [apache/incubator-mxnet] [RFC] Apache MXNet 2.0 Roadmap (#16167)

[Sheng Zha]

@stereomatchingkiss good point. What are you using c/c++ api for?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/16167#issuecomment-563883331

Re: [apache/incubator-mxnet] [RFC] Deferred compute in imperative interface to unify imperative and symbolic interface (#16376)

[Sheng Zha]

How's this project going?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/16376#issuecomment-562906794

Re: Stopping nightly releases to Pypi

[Sheng Zha]

> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu92mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu100mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet_cu101mkl-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> 
> You can easily install these pip wheels in your system either by downloading 
> them to your machine first and then installing by doing:
> 
> pip install /path/to/downloaded/wheel.whl
> 
> Or you can install directly by just giving the link to pip like this:
> 
> pip install 
> https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-07/dist/mxnet-1.6.0b20191207-py2.py3-none-manylinux1_x86_64.whl
> 
> Credit goes to everyone involved (in no particular order)
> Rakesh Vasudevan
> Zach Kimberg
> Manu Seth
> Sheng Zha
> Jun Wu
> Pedro Larroy
> Chaitanya Bapat
> 
> Thanks!
> Sam
> 
> 
> On Dec 5, 2019, at 1:16 AM, Lausen, Leonard 
> mailto:lau...@amazon.com.INVALID>> wrote:
> 
> We don't loose pip by hosting on S3. We just don't host nightly releases on 
> Pypi
> servers and mirror them to several hundred mirrors immediately after each 
> build
> is published which is very expensive for the Pypi project.. People can still
> install the nightly builds with pip by specifying the -f option.
> 
> Uploading weekly releases to Pypi will reduce the cost for Pypi by ~75% [1]. 
> It
> may be acceptable to Pypi, but does it make sense for us? I'm not convinced
> weekly release on Pypi is a good idea. Consider one release is buggy, users 
> will
> need to wait for 7 days for a fix. It doesn't provide good user experience.
> If someone has a stronger conviction about the value of weekly releases on 
> Pypi,
> that person shall please go ahead and propose it in a separate discussion
> thread.
> 
> Currently we don't have generally working nightly builds to Pypi and as a 
> matter
> of fact we know that we can't have them due to Pypi's policy and our apparent
> need for large binaries. Given this fact and that no objection was raised by
> 2019-12-05 at 05:42 UTC, I conclude we have lazy consensus on stopping upload
> attempts of nightly builds to Pypi.
> 
> With consensus established, we can change the CI job to stop trying to upload
> the nightly builds and then request Pypi to increase the limit. Then we have 
> one
> less blocker for the 1.6 release.
> 
> Best regards
> Leonard
> 
> [1]: Lower cost due to less releases, but higher cost due to 500MB -> 800MB
> limit increase. Assuming that the limit increase translates into actually 
> larger
> binaries.
> 
> 
> On Wed, 2019-12-04 at 22:20 +0100, Marco de Abreu wrote:
> Are weekly releases an option? It was brought up as concern that we might
> lose pip as a pretty common distribution channel where people consume
> nightly builds. I don't feel like that concern has been properly addressed
> so far.
> 
> -Marco
> 
> Lausen, Leonard mailto:lau...@amazon.com.invalid>> 
> schrieb am Mi., 4. Dez. 2019,
> 04:09:
> 
> As a simple POC to test distribution, you can try installing MXNet based on
> these 3 URLs:
> 
> pip install --no-cache-dir
> 
> https://mxnet-dev.s3.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> pip install --no-cache-dir
> 
> https://mxnet-dev.s3-accelerate.amazonaws.com/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> pip install --no-cache-dir https://d19zq12jzu4w95.cloudfront.net/
> mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> <
> https://d19zq12jzu4w95.cloudfront.net/mxnet_cu101-1.5.1.post0-py2.py3-none-manylinux1_x86_64.whl
> 
> 
> where --no-cache-dir prevents caching the downloaded file, for the purpose
> of
> testing. (cu101 chosen based on large size)
> 
> The first URL uses standard S3 bucket in US. The second uses S3 Accelerate
> based
> on CloudFront CDN. And the third uses CloudFront CDN. I'm adding the third
> URL,
> as S3 Accelerate may or may not use all new CloudFront endpoints yet.
> 
> Regarding voting: Uploading to Pypi is currently impossible, which is a
> reality
> (so there is no option to continue as we do currently). Pypi folks
> indicated
> they will unblock our uploads to Pypi once we stop uploading nightly
> releases
> and taking up 20% of their ressources [1].
> 
> If there are any shortcomings or problems identified with uploading to S3,
> we
> can work to address them. But for now, status quo is broken and this seems
> the
> only solution addressing Pypi's problem.
> 
> I don't mind if you state tha

Re: Stopping nightly releases to Pypi

[Sheng Zha]

This has the following effect:
- the unfortunate group of user whose GPUs' SMs were removed would be 
sacrificed so that import of mxnet on their machine would take quite some time 
on the order of hours. we don't have the usage information to guide which SMs 
to drop.
- it would cut down our binary size by only a corresponding proportion 
(currently there is 11 SMs total), and if we offer it as the only substitute 
for continuation of nightly releases on pypi it likely won't help us get the 
size limit increase.

We chose to publish nightly to pypi out of convenience instead of necessity.

-sz

On 2019/12/03 19:52:42, Marco de Abreu  wrote: 
> What about cutting down on SMs as recommended by Kellen?
> 
> Sheng Zha  schrieb am Di., 3. Dez. 2019, 20:15:
> 
> > This is certainly one way to do it. However, the binary size limits our
> > ability to publish pypi. So assuming that we want to have our binary on
> > pypi still, we'd have to convince pypa to raise our limits. Thus, it seems
> > to me that this hypothetical vote with respect to stopping nightly publish
> > to pypi would likely only have one acceptable outcome.
> >
> > This is more of an emergency situation as an essential distribution
> > channel is currently broken so I'm focusing on the POC for now.
> >
> > -sz
> >
> > On 2019/12/03 18:28:44, Marco de Abreu  wrote:
> > > Excellent! Could we maybe come up with a POC and a quick writeup and then
> > > start a proper vote after everyone verified that it covers their
> > use-cases?
> > >
> > > -Marco
> > >
> > > Sheng Zha  schrieb am Di., 3. Dez. 2019, 19:24:
> > >
> > > > Yes, there is. We can also make it easier to access by using a
> > > > geo-location based DNS server so that China users are directed to that
> > > > local mirror. The rest of the world is already covered by the global
> > > > cloudfront.
> > > >
> > > > -sz
> > > >
> > > > On 2019/12/03 18:22:22, Marco de Abreu 
> > wrote:
> > > > > Isn't there an s3 endpoint in Beijing?
> > > > >
> > > > > It seems like this topic still warrants some discussion and thus I'd
> > > > prefer
> > > > > if we don't move forward with lazy consensus.
> > > > >
> > > > > -Marco
> > > > >
> > > > > Tao Lv  schrieb am Di., 3. Dez. 2019, 14:31:
> > > > >
> > > > > > * For pypi, we can use mirrors.
> > > > > >
> > > > > > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv  wrote:
> > > > > >
> > > > > > > As we have many users in China, I'm considering the
> > accessibility of
> > > > S3.
> > > > > > > For pip, we can mirrors.
> > > > > > >
> > > > > > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
> > > >  > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> I would like to remind everyone that lazy consensus is assumed
> > if no
> > > > > > >> objections
> > > > > > >> are raised before 2019-12-05 at 05:42 UTC. There has been some
> > > > > > discussion
> > > > > > >> about
> > > > > > >> the proposal, but to my understanding no objections were raised.
> > > > > > >>
> > > > > > >> If the proposal is accepted, MXNet releases would be installed
> > via
> > > > > > >>
> > > > > > >>pip install mxnet
> > > > > > >>
> > > > > > >> And release candidates via
> > > > > > >>
> > > > > > >>   pip install --pre mxnet
> > > > > > >>
> > > > > > >> (or with the respective cuda version specifier appended etc.)
> > > > > > >>
> > > > > > >> To obtain releases built automatically from the master branch,
> > users
> > > > > > >> would need
> > > > > > >> to specify something like "-f
> > > > > > >> http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html; option to
> > pip.
> > > > > > >>
> > > > > > >> Best regards
> > > > > > >> Leonard
> > > > > > >>
> > > > > > >> On Mon, 2019-12-02 at 05:42 +, L

Re: Stopping nightly releases to Pypi

[Sheng Zha]

This is certainly one way to do it. However, the binary size limits our ability 
to publish pypi. So assuming that we want to have our binary on pypi still, 
we'd have to convince pypa to raise our limits. Thus, it seems to me that this 
hypothetical vote with respect to stopping nightly publish to pypi would likely 
only have one acceptable outcome.

This is more of an emergency situation as an essential distribution channel is 
currently broken so I'm focusing on the POC for now.

-sz

On 2019/12/03 18:28:44, Marco de Abreu  wrote: 
> Excellent! Could we maybe come up with a POC and a quick writeup and then
> start a proper vote after everyone verified that it covers their use-cases?
> 
> -Marco
> 
> Sheng Zha  schrieb am Di., 3. Dez. 2019, 19:24:
> 
> > Yes, there is. We can also make it easier to access by using a
> > geo-location based DNS server so that China users are directed to that
> > local mirror. The rest of the world is already covered by the global
> > cloudfront.
> >
> > -sz
> >
> > On 2019/12/03 18:22:22, Marco de Abreu  wrote:
> > > Isn't there an s3 endpoint in Beijing?
> > >
> > > It seems like this topic still warrants some discussion and thus I'd
> > prefer
> > > if we don't move forward with lazy consensus.
> > >
> > > -Marco
> > >
> > > Tao Lv  schrieb am Di., 3. Dez. 2019, 14:31:
> > >
> > > > * For pypi, we can use mirrors.
> > > >
> > > > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv  wrote:
> > > >
> > > > > As we have many users in China, I'm considering the accessibility of
> > S3.
> > > > > For pip, we can mirrors.
> > > > >
> > > > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard
> >  > > > >
> > > > > wrote:
> > > > >
> > > > >> I would like to remind everyone that lazy consensus is assumed if no
> > > > >> objections
> > > > >> are raised before 2019-12-05 at 05:42 UTC. There has been some
> > > > discussion
> > > > >> about
> > > > >> the proposal, but to my understanding no objections were raised.
> > > > >>
> > > > >> If the proposal is accepted, MXNet releases would be installed via
> > > > >>
> > > > >>pip install mxnet
> > > > >>
> > > > >> And release candidates via
> > > > >>
> > > > >>   pip install --pre mxnet
> > > > >>
> > > > >> (or with the respective cuda version specifier appended etc.)
> > > > >>
> > > > >> To obtain releases built automatically from the master branch, users
> > > > >> would need
> > > > >> to specify something like "-f
> > > > >> http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html; option to pip.
> > > > >>
> > > > >> Best regards
> > > > >> Leonard
> > > > >>
> > > > >> On Mon, 2019-12-02 at 05:42 +, Lausen, Leonard wrote:
> > > > >> > Hi MXNet Community,
> > > > >> >
> > > > >> > since more than 2 months our binary Python nightly releases
> > published
> > > > >> on Pypi
> > > > >> > are broken. The problem is that our binaries exceed Pypi's size
> > limit.
> > > > >> > Decreasing the binary size by adding compression breaks
> > third-party
> > > > >> libraries
> > > > >> > loading libmxnet.so
> > > > >> https://github.com/apache/incubator-mxnet/issues/16193
> > > > >> >
> > > > >> > Sheng requested Pypi to increase their size limit:
> > > > >> > https://github.com/pypa/pypi-support/issues/50
> > > > >> >
> > > > >> > Currently "the biggest cost for PyPI from [the many MXNet binaries
> > > > with
> > > > >> > nightly
> > > > >> > release to Pypi] is the bandwidth consumed when several hundred
> > > > mirrors
> > > > >> > attempt
> > > > >> > to mirror each release immediately after it's published". So Pypi
> > is
> > > > not
> > > > >> > inclined to allow us to upload even larger binaries on a nightly
> > > > >> schedule.
> > > > >> > Their compromise is to allow it 

Re: Stopping nightly releases to Pypi

[Sheng Zha]

Yes, there is. We can also make it easier to access by using a geo-location 
based DNS server so that China users are directed to that local mirror. The 
rest of the world is already covered by the global cloudfront.

-sz

On 2019/12/03 18:22:22, Marco de Abreu  wrote: 
> Isn't there an s3 endpoint in Beijing?
> 
> It seems like this topic still warrants some discussion and thus I'd prefer
> if we don't move forward with lazy consensus.
> 
> -Marco
> 
> Tao Lv  schrieb am Di., 3. Dez. 2019, 14:31:
> 
> > * For pypi, we can use mirrors.
> >
> > On Tue, Dec 3, 2019 at 9:28 PM Tao Lv  wrote:
> >
> > > As we have many users in China, I'm considering the accessibility of S3.
> > > For pip, we can mirrors.
> > >
> > > On Tue, Dec 3, 2019 at 3:24 PM Lausen, Leonard  > >
> > > wrote:
> > >
> > >> I would like to remind everyone that lazy consensus is assumed if no
> > >> objections
> > >> are raised before 2019-12-05 at 05:42 UTC. There has been some
> > discussion
> > >> about
> > >> the proposal, but to my understanding no objections were raised.
> > >>
> > >> If the proposal is accepted, MXNet releases would be installed via
> > >>
> > >>pip install mxnet
> > >>
> > >> And release candidates via
> > >>
> > >>   pip install --pre mxnet
> > >>
> > >> (or with the respective cuda version specifier appended etc.)
> > >>
> > >> To obtain releases built automatically from the master branch, users
> > >> would need
> > >> to specify something like "-f
> > >> http://mxnet.s3.amazonaws.com/mxnet-X/nightly.html; option to pip.
> > >>
> > >> Best regards
> > >> Leonard
> > >>
> > >> On Mon, 2019-12-02 at 05:42 +, Lausen, Leonard wrote:
> > >> > Hi MXNet Community,
> > >> >
> > >> > since more than 2 months our binary Python nightly releases published
> > >> on Pypi
> > >> > are broken. The problem is that our binaries exceed Pypi's size limit.
> > >> > Decreasing the binary size by adding compression breaks third-party
> > >> libraries
> > >> > loading libmxnet.so
> > >> https://github.com/apache/incubator-mxnet/issues/16193
> > >> >
> > >> > Sheng requested Pypi to increase their size limit:
> > >> > https://github.com/pypa/pypi-support/issues/50
> > >> >
> > >> > Currently "the biggest cost for PyPI from [the many MXNet binaries
> > with
> > >> > nightly
> > >> > release to Pypi] is the bandwidth consumed when several hundred
> > mirrors
> > >> > attempt
> > >> > to mirror each release immediately after it's published". So Pypi is
> > not
> > >> > inclined to allow us to upload even larger binaries on a nightly
> > >> schedule.
> > >> > Their compromise is to allow it on a weekly cadence.
> > >> >
> > >> > However, I would like the community to revisit the necessity of
> > >> releasing pre-
> > >> > release binaries to Pypi on a nightly (or weekly) cadence. Instead, we
> > >> can
> > >> > release nightly binaries ONLY to a public S3 bucket and instruct users
> > >> to
> > >> > install from there. On our side, we only need to prepare a html
> > >> document that
> > >> > contains links to all released nightly binaries.
> > >> > Finally users will install the nightly releases via
> > >> >
> > >> >   pip install --pre mxnet-cu101 -f
> > >> http://mxnet.s3.amazonaws.com/mxnet-cu101/
> > >> > nightly.html
> > >> >
> > >> > Instead of
> > >> >
> > >> >   pip install --pre mxnet-cu101
> > >> >
> > >> > Of course proper releases and release candidates should still be made
> > >> > available
> > >> > via Pypi. Thus releases would be installed via
> > >> >
> > >> >   pip install mxnet-cu101
> > >> >
> > >> > And release candidates via
> > >> >
> > >> >   pip install --pre mxnet-cu101
> > >> >
> > >> > This will substantially reduce the costs of the Pypi project and in
> > fact
> > >> > matches
> > >> > the installation experience provided by PyTorch. I don't think the
> > >> benefit of
> > >> > not including "-f
> > >> http://mxnet.s3.amazonaws.com/mxnet-cu101/nightly.html;
> > >> > matches the costs we currently externalize to the Pypi team.
> > >> >
> > >> > This suggestion seems uncontroversial to me. Thus I would like to
> > start
> > >> lazy
> > >> > consensus. If there are no objections, I will assume lazy consensus on
> > >> > stopping
> > >> > nightly releases to Pypi in 72hrs.
> > >> >
> > >> > Best regards
> > >> > Leonard
> > >>
> > >
> >
> 

Re: [apache/incubator-mxnet] [RFC] RFC Issue Mirroring to d...@mxnet.apache.org (#15749)

[Sheng Zha]

RFC issue mirroring has been up and running. The mirroring on the email side 
requires replying all to the thread on dev@.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/15749#issuecomment-557633070

Re: [apache/incubator-mxnet] [RFC] RFC Issue Mirroring to d...@mxnet.apache.org (#15749)

[Sheng Zha]

Closed #15749.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/15749#event-2824101270

Re: [apache/incubator-mxnet] [RFC] Deferred compute in imperative interface to unify imperative and symbolic interface (#16376)

[Sheng Zha]

Thanks for the proposal, @leezu. Since this is a major change, I have some 
questions regarding the plan.

First, should we restrict this mode to only apply to the new numpy arrays? 
Since the deferred compute mode won't support reverse shape inference, new 
blocks that implement the forward interface will not work without implementing 
the parameter shape inference logic in `infer_shape`. This also applies when 
migrating the existing Gluon blocks in our API. Since we have plan to adopt 
numpy array in Gluon, the two changes can potentially happen at the same time.

Also, could you elaborate on what the changes are to the `infer_shape`, 
especially on how and when it's invoked during deferred initialization?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/16376#issuecomment-538706564

Revisit Maturity Model for MXNet

[Sheng Zha]

Hi,

It's time to revisit the Apache maturity model for MXNet and see where we are 
with respect to graduation. Qing and I updated the maturity model in the wiki 
[1]. Comments are welcome.

-sz

[1] https://cwiki.apache.org/confluence/x/lQqQBQ

Re: Podling Report Reminder - October 2019

[Sheng Zha]

Indeed, good catch. I fixed the link. Thanks!

-sz

On 2019/10/01 20:32:54, Marco de Abreu  wrote: 
> Thanks Sheng!
> 
> The following section might have a copy and paste mistake with the links
> since they're identical:
> 
> Our community is planning for the long-term development of MXNet. RFC:
> https://github.com/apache/incubator-mxnet/issues/16167 Discussion:
> https://github.com/apache/incubator-mxnet/issues/16167
> 
> Sheng Zha  schrieb am Di., 1. Okt. 2019, 22:29:
> 
> > I posted the draft report at
> > https://cwiki.apache.org/confluence/display/INCUBATOR/October2019#mxnet
> >
> > -sz
> >
> > On 2019/09/29 05:08:18, Sheng Zha  wrote:
> > > I'm working on a draft now. Once ready, I will post the draft to the
> > wiki page and reply back here.
> > >
> > > -sz
> > >
> > > On 2019/09/29 00:44:27, jmcl...@apache.org wrote:
> > > > Dear podling,
> > > >
> > > > This email was sent by an automated system on behalf of the Apache
> > > > Incubator PMC. It is an initial reminder to give you plenty of time to
> > > > prepare your quarterly board report.
> > > >
> > > > The board meeting is scheduled for Wed, 16 October 2019, 10:30 am PDT.
> > > > The report for your podling will form a part of the Incubator PMC
> > > > report. The Incubator PMC requires your report to be submitted 2 weeks
> > > > before the board meeting, to allow sufficient time for review and
> > > > submission (Wed, October 02).
> > > >
> > > > Please submit your report with sufficient time to allow the Incubator
> > > > PMC, and subsequently board members to review and digest. Again, the
> > > > very latest you should submit your report is 2 weeks prior to the board
> > > > meeting.
> > > >
> > > > Candidate names should not be made public before people are actually
> > > > elected, so please do not include the names of potential committers or
> > > > PPMC members in your report.
> > > >
> > > > Thanks,
> > > >
> > > > The Apache Incubator PMC
> > > >
> > > > Submitting your Report
> > > >
> > > > --
> > > >
> > > > Your report should contain the following:
> > > >
> > > > *   Your project name
> > > > *   A brief description of your project, which assumes no knowledge of
> > > > the project or necessarily of its field
> > > > *   A list of the three most important issues to address in the move
> > > > towards graduation.
> > > > *   Any issues that the Incubator PMC or ASF Board might wish/need to
> > be
> > > > aware of
> > > > *   How has the community developed since the last report
> > > > *   How has the project developed since the last report.
> > > > *   How does the podling rate their own maturity.
> > > >
> > > > This should be appended to the Incubator Wiki page at:
> > > >
> > > > https://cwiki.apache.org/confluence/display/INCUBATOR/October2019
> > > >
> > > > Note: This is manually populated. You may need to wait a little before
> > > > this page is created from a template.
> > > >
> > > > Note: The format of the report has changed to use markdown.
> > > >
> > > > Mentors
> > > > ---
> > > >
> > > > Mentors should review reports for their project(s) and sign them off on
> > > > the Incubator wiki page. Signing off reports shows that you are
> > > > following the project - projects that are not signed may raise alarms
> > > > for the Incubator PMC.
> > > >
> > > > Incubator PMC
> > > >
> > >
> >
> 

Re: Podling Report Reminder - October 2019

[Sheng Zha]

I posted the draft report at 
https://cwiki.apache.org/confluence/display/INCUBATOR/October2019#mxnet

-sz

On 2019/09/29 05:08:18, Sheng Zha  wrote: 
> I'm working on a draft now. Once ready, I will post the draft to the wiki 
> page and reply back here.
> 
> -sz
> 
> On 2019/09/29 00:44:27, jmcl...@apache.org wrote: 
> > Dear podling,
> > 
> > This email was sent by an automated system on behalf of the Apache
> > Incubator PMC. It is an initial reminder to give you plenty of time to
> > prepare your quarterly board report.
> > 
> > The board meeting is scheduled for Wed, 16 October 2019, 10:30 am PDT.
> > The report for your podling will form a part of the Incubator PMC
> > report. The Incubator PMC requires your report to be submitted 2 weeks
> > before the board meeting, to allow sufficient time for review and
> > submission (Wed, October 02).
> > 
> > Please submit your report with sufficient time to allow the Incubator
> > PMC, and subsequently board members to review and digest. Again, the
> > very latest you should submit your report is 2 weeks prior to the board
> > meeting.
> > 
> > Candidate names should not be made public before people are actually
> > elected, so please do not include the names of potential committers or
> > PPMC members in your report.
> > 
> > Thanks,
> > 
> > The Apache Incubator PMC
> > 
> > Submitting your Report
> > 
> > --
> > 
> > Your report should contain the following:
> > 
> > *   Your project name
> > *   A brief description of your project, which assumes no knowledge of
> > the project or necessarily of its field
> > *   A list of the three most important issues to address in the move
> > towards graduation.
> > *   Any issues that the Incubator PMC or ASF Board might wish/need to be
> > aware of
> > *   How has the community developed since the last report
> > *   How has the project developed since the last report.
> > *   How does the podling rate their own maturity.
> > 
> > This should be appended to the Incubator Wiki page at:
> > 
> > https://cwiki.apache.org/confluence/display/INCUBATOR/October2019
> > 
> > Note: This is manually populated. You may need to wait a little before
> > this page is created from a template.
> > 
> > Note: The format of the report has changed to use markdown.
> > 
> > Mentors
> > ---
> > 
> > Mentors should review reports for their project(s) and sign them off on
> > the Incubator wiki page. Signing off reports shows that you are
> > following the project - projects that are not signed may raise alarms
> > for the Incubator PMC.
> > 
> > Incubator PMC
> > 
> 

Re: Update for 1.5.1 patch release

[Sheng Zha]

Here's what I did for the past few releases:
1. For 3.1.1
- check out the rc version tag locally in git and tag it with the release 
version. (i.e. duplicate 1.5.1.rc0 and name it 1.5.1)
- edit the last release candidate pre-release on github. update the title and 
tag to reflect the new release, and uncheck this is a pre-release checkbox.
- update the release artifact in the release with the artifacts from 3.1.2.
- delete the old rc tag locally and on github.

2. For 3.1.2
- Since folder name needs to be changed inside the archive, the signature and 
hash need to be produced again. I usually just untar the old file, rename the 
folder name to remove mentions of "rc", and follow step 1.10 to tar, sign, and 
digest.
- SVN repo also needs to be updated with the correct folder name and path, and 
the updated artifacts. The end result should be that the old rc artifacts are 
removed from the dev repo, and the release artifacts are uploaded to the dist 
repo.

3. Aaron has been helping with 3.2.

-sz

On 2019/09/28 20:47:37, Lin Yuan  wrote: 
> Ping @Sheng and @Lai who released 1.5.0 for help.
> 
> Could you please update the Release Process doc after you find the right
> answers to these questions?
> 
> Thanks,
> 
> Lin
> 
> On Sat, Sep 28, 2019 at 7:49 AM Tao Lv  wrote:
> 
> > Hi dev,
> >
> > I'm glad to say that the rc0 of 1.5.1 patch release has passed the vote on
> > general@. Please find the voting thread at:
> >
> > https://lists.apache.org/thread.html/282f7911768dab61ddf8f70adcce34ef0afb285046093b3ff0bafb7e@%3Cgeneral.incubator.apache.org%3E
> >
> >
> > Now I'm proceeding the release process and have several questions there.
> > Hope someone can help to answer:
> >
> > 1. Change the 1.5.1.rc0 tag to formal 1.5.1. Seems the step 3.1.1 on the
> > cwiki page [1] doesn't work. It says:
> >
> > "Go to the GitHub repo’s “releases” tab
> >
> > Click “Draft a new release”
> >
> > Provide the release tag in the form of “..”
> > Select the commit by clicking Target: master > the passing release
> > candidate tag"
> >
> > But I cannot find "the passing release candidate tag" in the drop list.
> > There're branches and recent commits. Once I create a new tag, how about
> > the old tag of 1.5.1.rc0?
> >
> > 2. Step 3.1.2 is also confusing to me. Not sure what should be done at step
> > 3 and what need to be uploaded at step 4.
> >
> > 3. As we have a new website now, I guess there are some changes for the
> > step 3.2. Can anyone help to clarify this?
> >
> > Thanks,
> > -tao
> >
> > [1] https://cwiki.apache.org/confluence/display/MXNET/Release+Process
> >
> 

Re: Enable Timestamp in CI Logging

[Sheng Zha]

Thanks. And sorry for the confusion. I meant bootstrap instead of reboot as I 
was assuming that the creation of the master server is automated. Marco 
clarified with me that in fact currently the server is created manually, so 
this isn't really a problem.

-sz

On 2019/09/27 21:55:30, Pedro Larroy  wrote: 
> Sheng, you should have admin access to Jenkins as of now.
> 
> Why wouldn't be persistent through reboots?
> 
> Pedro.
> 
> On Sat, Sep 14, 2019 at 10:07 PM Sheng Zha  wrote:
> 
> > Thank you, Philip. Looks like xgboost is using the same plugin for the
> > timestamps.
> >
> > Unfortunately, I don't have admin access to the CI right now so I cannot
> > add the plugin or view whether the plugin is already added. What's also
> > unclear to me is how to add plugins to the CI in a persistent way that
> > lasts through the next reboot.
> >
> > It would be great if someone could help in these aspects.
> >
> > -sz
> >
> > On 2019/09/15 04:27:47, Philip Cho  wrote:
> > > Hi Sheng:
> > >
> > > Take a look at
> > >
> > https://github.com/dmlc/xgboost/blob/c89bcc4de5368b3f8a7fa170d8348287dab44caf/Jenkinsfile#L21
> > > .
> > >
> > > Philip.
> > >
> > > On Sat, Sep 14, 2019 at 9:26 PM Sheng Zha  wrote:
> > >
> > > > Hi,
> > > >
> > > > There have been timeouts in the build step of CI in PRs. To help
> > identify
> > > > the steps where most time is taken, I suggest that we enable timestamp
> > in
> > > > the CI logging. With the help of a simple Jenkins plugin [1], we
> > should be
> > > > able to turn it on with some simple changes in our Jenkinsfiles.
> > > >
> > > > If you're already familiar with how to proceed, help would be much
> > > > appreciated. Otherwise, I will start looking into how to proceed.
> > > >
> > > > -sz
> > > >
> > > > [1] http://wiki.jenkins.io/display/JENKINS/Timestamper
> > > >
> > >
> >
> 

Re: [DISCUSS] Remove amalgamation

[Sheng Zha]

As an alternative to amalgamation, could we simply ask users to statically link 
to libmxnet.a, and then prune the symbol table to get rid of the binary of 
unused functions? Though I don't know the full context of amalgamation, based 
on my knowledge on this feature I'm not aware of any difference in the end 
result, compared to the code-inlining approach that amalgamation takes.

-sz

On 2019/09/12 17:29:02, Naveen Swamy  wrote: 
> so the original email suggesting to remove was after all self-serving :)
> 
> let's encourage if someone wants to maintain and make use of the original
> work and make it better.
> 
> -1 to remove at this point
> 
> P.S: I suggest to do some due diligence before bringing topics up for
> discussion.
> 
> On Wed, Sep 11, 2019 at 8:10 AM Lv, Tao A  wrote:
> 
> > Sorry to chime in.
> >
> > There is a PR to fix amalgamation. I was pinged several times to merge it
> > but I don't think I have enough knowledge to do that. So it would be great
> > if someone from this thread can help to review.
> >
> > https://github.com/apache/incubator-mxnet/pull/15303
> >
> > thanks,
> > -tao
> >
> > -Original Message-
> > From: Marco de Abreu 
> > Sent: Wednesday, September 11, 2019 9:38 PM
> > To: dev@mxnet.incubator.apache.org
> > Subject: Re: [DISCUSS] Remove amalgamation
> >
> > Is Amalgamation only used on Android though? Are there any other use cases?
> >
> > -Marco
> >
> > Pedro Larroy  schrieb am Mi., 11. Sep. 2019,
> > 11:57:
> >
> > > Hi Anirudh
> > >
> > > Appreciate your feedback and sorry if my email came across that way to
> > > you, I think you might miss some context. I don't think calling
> > > something hacky is anything bad and isn't supposed to be the topic of
> > > the discussion. It was reported as not working by users, hence the
> > > original thread. It was a request for opinions from people who might
> > > actually have tried to work in Mxnet on Android.
> > >
> > > Thanks.
> > >
> > > Pedro.
> > >
> > >
> > > On Tuesday, September 10, 2019, Anirudh Subramanian
> > >  > > >
> > > wrote:
> > > > Hi Pedro,
> > > >
> > > > I don't see anything "destructive" with Chris asking for
> > > > justification
> > > for
> > > > you calling something "hacky". The only email in this thread where I
> > > > see
> > > ad
> > > > hominems and disrespectful comments is your email.
> > > >
> > > > On Sat, Sep 7, 2019, 10:18 PM Pedro Larroy
> > > >  > > >
> > > > wrote:
> > > >
> > > >> Apache mentors should have a look at these reincident harassment
> > > >> and destructive behaviors which demotivate contributions and take
> > > >> action. It takes only one bad apple to ruin a community.
> > > >>
> > > >> The mobile solution that is known to work as of know is cross
> > > >> compiling with "ci/build.py -p build.android_armv8" or
> > > >> "build.android_armv7". The only advantage of amalgamation is to
> > > >> provide a smaller binary that we
> > > could
> > > >> accomplish with the C preprocessor.
> > > >>
> > > >> My technical contributions speak for themselves, including porting
> > > >> MXNet
> > > to
> > > >> Android and ARM and helping many users run MXNet in Jetson,
> > > >> Raspberry Pi and Android amongst many other topics. I have never
> > > >> been disrespectful
> > > to
> > > >> anyone. I'm entitled to my own technical opinions about
> > > >> amalgamation or
> > > any
> > > >> other piece of code whatsoever, that's no personal disrespect to
> > > >> anyone
> > > and
> > > >> perfectly valid. If you are not interested in this project anymore,
> > > >> do
> > > us
> > > >> all a favor and stop trolling and being toxic. If you want my
> > > >> respect,
> > > step
> > > >> up your technical contributions, be positive and encourage others,
> > > >> this including commits, I haven't seen for many months, please be
> > > >> positive
> > > and
> > > >> constructive. This scorched-earth attitude is only reflecting bad
> > > >> on
> > > you.
> > > >> I'm certainly not interested in your ad-hominems or unasked for
> > > technical
> > > >> advice, which to be honest,  showing poor judgment and ignorance.
> > > >> Myself and others have come up with numbers, graphs, metrics and
> > > >> arguments and have been met with dismissal, trolling and
> > > >> sea-lioning. I have recieved your insults via public and private
> > > >> channels (such as linkedin) as have others. This is not ok and has
> > > >> to stop. If you have something personal against me or against your
> > > >> former employer, this is not the right place
> > > or
> > > >> forum.
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> On Fri, Sep 6, 2019 at 3:56 PM Chris Olivier
> > > >> 
> > > >> wrote:
> > > >>
> > > >> > Hi Pedro,
> > > >> >
> > > >> > While I was not involved with amalgamation or its development in
> > > >> > any
> > > way,
> > > >> > can you please refrain from referring to the work of others as a
> > > "hacky
> > > 

[Announcement] New PPMC Member - Tao Lv

[Sheng Zha]

Hi all,

Please join me in welcoming Tao Lv as a new PPMC member of Apache MXNet 
(incubating)!

Tao has been a committer of our project since Nov. 2018, and has remained very 
active
in not only maintaining MKLDNN backend, but many other areas. Over time he and 
the Intel team has greatly helped the project on CPU performance.

Welcome, Tao!

-sz

Re: [DISCUSS] CI Access Control

[Sheng Zha]

; a bare minimum in the form of admins.
> >
> > So while I certainly understand and encourage to distribute the access, I
> > don't feel comfortable widening the access to such a critical productive
> > system. It being down means that the GitHub development is fully halted,
> > which is really problematic since we don't have rollback mechanisms.
> >
> > Best regards,
> > marco
> >
> > On Sun, Sep 15, 2019 at 6:40 AM Sheng Zha  wrote:
> >
> >> Hi,
> >>
> >> I'd like to initiate discussion on how access control should be managed
> >> for the CI system. The hope is that we can present the conclusion of this
> >> discussion as the recommendation and request to the donors of the CI system
> >> from Amazon.
> >>
> >> The specific aspects I'd like to discuss are the abilities to:
> >> 1. trigger PR validation and nightly jobs.
> >> 2. trigger continuous delivery jobs, such as for binary releases in pip,
> >> maven, and dockerhub.
> >> 3. add jobs to the CI system.
> >> 4. maintain and manage the CI system, such as system upgrades and jenkins
> >> plugin installation.
> >>
> >> Given that we already have GitHub SSO enabled on the Jenkins CI, I
> >> suggest the following authentication levels for these items:
> >> 1. all authenticated GitHub users.
> >> 2-4. all MXNet committers
> >>
> >> What do you think? If you have more aspects that you wish to discuss,
> >> feel free to propose.
> >>
> >> -sz
> >>
> >
> 

Re: [apache/incubator-mxnet] [RFC] Apache MXNet 2.0 Roadmap (#16167)

[Sheng Zha]

@pengzhao-intel a tentative target date is by end of Q1 2020.

@zachgk we will create a branch for 2.0. Initially we will keep master to be 
1.x and have 2.0 in a new branch. After 1.6 release we will revisit how to make 
the 2.0 branch the master.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-mxnet/issues/16167#issuecomment-532066487

Re: Enable Timestamp in CI Logging

[Sheng Zha]

Thank you, Philip. Looks like xgboost is using the same plugin for the 
timestamps.

Unfortunately, I don't have admin access to the CI right now so I cannot add 
the plugin or view whether the plugin is already added. What's also unclear to 
me is how to add plugins to the CI in a persistent way that lasts through the 
next reboot.

It would be great if someone could help in these aspects.

-sz

On 2019/09/15 04:27:47, Philip Cho  wrote: 
> Hi Sheng:
> 
> Take a look at
> https://github.com/dmlc/xgboost/blob/c89bcc4de5368b3f8a7fa170d8348287dab44caf/Jenkinsfile#L21
> .
> 
> Philip.
> 
> On Sat, Sep 14, 2019 at 9:26 PM Sheng Zha  wrote:
> 
> > Hi,
> >
> > There have been timeouts in the build step of CI in PRs. To help identify
> > the steps where most time is taken, I suggest that we enable timestamp in
> > the CI logging. With the help of a simple Jenkins plugin [1], we should be
> > able to turn it on with some simple changes in our Jenkinsfiles.
> >
> > If you're already familiar with how to proceed, help would be much
> > appreciated. Otherwise, I will start looking into how to proceed.
> >
> > -sz
> >
> > [1] http://wiki.jenkins.io/display/JENKINS/Timestamper
> >
> 

[DISCUSS] CI Access Control

[Sheng Zha]

Hi,

I'd like to initiate discussion on how access control should be managed for the 
CI system. The hope is that we can present the conclusion of this discussion as 
the recommendation and request to the donors of the CI system from Amazon.

The specific aspects I'd like to discuss are the abilities to:
1. trigger PR validation and nightly jobs.
2. trigger continuous delivery jobs, such as for binary releases in pip, maven, 
and dockerhub.
3. add jobs to the CI system.
4. maintain and manage the CI system, such as system upgrades and jenkins 
plugin installation.

Given that we already have GitHub SSO enabled on the Jenkins CI, I suggest the 
following authentication levels for these items:
1. all authenticated GitHub users.
2-4. all MXNet committers

What do you think? If you have more aspects that you wish to discuss, feel free 
to propose.

-sz

Enable Timestamp in CI Logging

[Sheng Zha]

Hi,

There have been timeouts in the build step of CI in PRs. To help identify the 
steps where most time is taken, I suggest that we enable timestamp in the CI 
logging. With the help of a simple Jenkins plugin [1], we should be able to 
turn it on with some simple changes in our Jenkinsfiles.

If you're already familiar with how to proceed, help would be much appreciated. 
Otherwise, I will start looking into how to proceed.

-sz

[1] http://wiki.jenkins.io/display/JENKINS/Timestamper

[apache/incubator-mxnet] [RFC] Apache MXNet 2.0 Roadmap (#16167)

[Sheng Zha]

# Overview

The purpose of this RFC is to organize and present the roadmap towards 2.0. As 
2.0 will be a major release, changes that would break backward compatibility 
are permissible.

The proposed changes in this RFC are either collected from past roadmap 
discussions such as #9686, or are based on various common issues from the past. 
This RFC organizes these changes into self-contained projects to facilitate 
clear definition of project, captures the risks and status quo to the best of 
our knowledge. To help navigate, the projects are further divided into several 
high-level areas. Some of the listed projects are already in progress, and are 
included to provide a clear overview.

The objectives of Apache MXNet 2.0 include:
- Improve expressiveness and usability of user-facing API.
- Improve expressiveness and usability of the technical stack for lower 
development cost and maintainability.

In terms of frontend, this roadmap focuses mostly on Python-frontend since 
MXNet has been taking a Python-first approach. The expectation with respect to 
other language bindings is that they would evolve along with the backend 
evolution and make use of the improvements. Given that breaking changes can 
occur, maintainers of different language bindings are expected to participate 
in related interface definition discussions.


## N1. NumPy

NumPy has long been established as the standard math library in Python, the 
most prevalent language for the deep learning community. With this library as 
the cornerstone, there are now the largest ecosystem and community for 
scientific computing. The popularity of NumPy comes from its flexibility and 
generality.

In #14253, the MXNet community reached consensus on moving towards a 
NumPy-compatible programing experience and committed to a major endeavor on 
providing NumPy compatible operators.

The primary goal of the projects below is to provide the equivalent usability 
and expressiveness of NumPy in MXNet to facilitate Deep Learning model 
development, which not only helps existing deep learning practitioners but also 
provides people in the existing NumPy community with a shortcut for getting 
started in Deep Learning. The efforts towards this goal would also help a 
secondary goal, which is to enable the existing NumPy ecosystem to utilize GPUs 
and accelerators to speed up large scale computation.

cc @apache/mxnet-committers 

### NumPy Operator Testing

Scope:
1. adopt __array_function__ and numpy existing tests.
2. extend testing to GPU
3. investigate numpy testing strategies
4. decide correctness criteria for acceptance


### NumPy Operator performance profiling

Scope:
1. Automatically profile the performance of NumPy operators


### NumPy operator coverage

Scope:
1. improve operator until full NumPy coverage, with prioritization towards 
operators used in the ecosystem and deep learning in general

Operator coverage as of 07/03/2019

```
|module | NumPy | deepNumPy |   jax |  cupy |
|---|---|---|---|---|
|np |   603 |89 |   445 |   321 |
|   ndarray |71 |32 |71 |56 |
|random |63 | 5 |15 |49 |
|linalg |31 | 2 | 8 |15 |
```

### NumPy Extension Operator Reorganization and Renaming

Scope:
1. consistent type usage for index input and return values from sort, topk 
#11031 #11134, #12197
2. array creation operators with flexible dtype definition #12290. (dtype=None)
3. moving_mean/moving_var in batchnorm
4. consistent usage of axis vs dim
5. promote or deprecate contrib operators


### NumPy ndarray type extension

Scope:
1. bfloat16 support (not in NumPy yet but useful for deep learning) (low 
priority — Intel)
2. boolean type support
3. complex (for FFT)


### NumPy ndarray boolean indexing

Scope:
1. allow boolean masks in NumPy ndarray indexing by adding the operator, 
potentially through extending op.where


### Hybridizable basic (and advanced) indexing

Scope:

1. Allow operations such as y = x[1:3, 2, ...] to be hybridizable

Note: Preliminary work: https://github.com/apache/incubator-mxnet/pull/15663


## Graph Enhancement and 3rdparty support

The objective of the following projects is to enable easier development of 
third-party extensions without requiring changes to be checked in the MXNet 
project. Examples of such extensions include third-party operator library and 
accelerators.

### Graph Partitioning for Dynamic Shape Operators

Scope:
1. partition inside control flow operators (and all cached ops)
2. partition on operators with dynamic shapes for partial memory planning and 
caching.

### Improved Third-party Operator Support

Scope:
1. allow registering custom operators by exposing C API (and frontend API) to 
register NNVM op at runtime.
2. verify serialization, deserialization, and graph passes for graphs with 
these operators are working properly.

### 

[Announcement] New Committer - Junru Shao

[Sheng Zha]

Hi all,

Please join me in welcoming Junru Shao as a new committer of Apache MXNet 
(incubating)!

Junru made a number of contributions to this project such as cond and 
while_loop control-flow
operators, enabling dynamic shape in control-flow ops, and zero-copy array 
creation from numpy
array. He's also been actively working on numpy-compatible programming 
experience and
has helped the community recently by driving the 1.4.1 patch release.

Welcome, Junru!

-sz

Re: [DISCUSS] PPMC member: Tao Lv

[Sheng Zha]

I sincerely apologize for the mistake that I sent this email to the dev@. This 
discussion thread is not intended for dev@ and I withdraw the discussion from 
here.

-sz

On 2019/08/27 18:07:09, Sheng Zha  wrote: 
> Dear PPMC members,
> 
> I'd like to start a discussion on inviting Tao Lv as a PPMC member.
> He has been a full committer of our project since Nov. 2018, and has remained 
> very active
> in not only maintaining MKLDNN backend, but many other areas. Over time he 
> has developed
> good knowledge of MXNet and has been helping the project on CPU performance. 
> He has also been
> quite responsive in any issues related to CPU/MKLDNN backend in MXNet.
> Recently he has been driving the 1.5.1 release.
> 
> PRs authored -
> https://github.com/apache/incubator-mxnet/pulls?utf8=%E2%9C%93=is%3Apr+author%3Ataolv
> 
> PR engagement -
> https://github.com/apache/incubator-mxnet/pulls?utf8=%E2%9C%93=is%3Apr+involves%3Ataolv+
> 
> Issue engagement -
> https://github.com/apache/incubator-mxnet/issues?utf8=%E2%9C%93=is%3Aissue+involves%3ATaoLv+
> 
> dev mailing list -
> https://lists.apache.org/list.html?d...@mxnet.apache.org:lte=11M:Tao%20Lv
> 
> I think he would be a great addition to the PPMC and it would be great to 
> hear what you think.
> 
> Best,
> -sz
> 
> 

[DISCUSS] PPMC member: Tao Lv

[Sheng Zha]

Dear PPMC members,

I'd like to start a discussion on inviting Tao Lv as a PPMC member.
He has been a full committer of our project since Nov. 2018, and has remained 
very active
in not only maintaining MKLDNN backend, but many other areas. Over time he has 
developed
good knowledge of MXNet and has been helping the project on CPU performance. He 
has also been
quite responsive in any issues related to CPU/MKLDNN backend in MXNet.
Recently he has been driving the 1.5.1 release.

PRs authored -
https://github.com/apache/incubator-mxnet/pulls?utf8=%E2%9C%93=is%3Apr+author%3Ataolv

PR engagement -
https://github.com/apache/incubator-mxnet/pulls?utf8=%E2%9C%93=is%3Apr+involves%3Ataolv+

Issue engagement -
https://github.com/apache/incubator-mxnet/issues?utf8=%E2%9C%93=is%3Aissue+involves%3ATaoLv+

dev mailing list -
https://lists.apache.org/list.html?d...@mxnet.apache.org:lte=11M:Tao%20Lv

I think he would be a great addition to the PPMC and it would be great to hear 
what you think.

Best,
-sz