Re: Apache MXNet (incubating) Python Docker Images

[Steffen Rochel]

+1 on Kellen’s comments and suggestion.
On Wed, Oct 17, 2018 at 12:39 PM kellen sunderland <
kellen.sunderl...@gmail.com> wrote:

> This feels like something we should get a little data on before making a
> decision, but I also don't have a strong opinion.  I would bias towards
> pushing something that might be imperfect and moving on to develop other
> improvements for users rather than determining a 'perfect' solution.
>
> The questions/tradeoffs I see are (1) should we support multiple python
> versions in the first place, which requires extra work on our part but
> supports more users and (2) should we favor forwards or backwards
> compatibility, i.e. should we prioritize supporting existing users or
> prioritize making something that won't cause future problems for new and
> exiting users.
>
> The best data I can find with a quick google is the annual Jetbrains survey
> which shows python2 went from 47% in 2017 to 25% in 2018:
> https://www.jetbrains.com/research/devecosystem-2017/python/
> https://www.jetbrains.com/research/devecosystem-2018/python/
>
> So python2 usage is trending sharply down but is not yet low enough to
> ignore which I think means we should try and support both on Dockerhub (1).
>
> I don't see backwards compatibility with existing Docker users is a major
> concern given these Dockerfiles haven't been supported for a long time.  I
> would prioritize forwards compatibility (2) and assume we want to create
> something that will remain compatible for as long as possible.
>
> So I would push both python2 and python3 images, but make python 3.5 the
> default version, and python2 a version with a postfixed py2 tag in
> Dockerhub.
>
> Thanks to Mu (and others?) for original creating this Dockerhub images, I
> used to use them and found them very convenient, and to you Meghna for
> updating them.  I think basing them on the pip packages is also a good way
> to lower maintenance burden and make sure we leverage the great work Sheng
> has done to create those packages.
>
> On Wed, Oct 17, 2018 at 11:52 AM Meghna Baijal  >
> wrote:
>
> > Hi All,
> >
> > I am currently in the process of updating the python docker images for
> > Apache MXNet such that they are built on top of the pip binaries.
> > Until now these were built to use python 2.7 but with an upcoming PR I am
> > also adding python 3.5 docker images. I would like to know the
> community’s
> > preference on whether I should keep the *Python 2.7 Docker image as the
> > default or should I move to Python 3.5 as the default version*?
> >
> > [1] The new python2 dockerfiles and build script can be found here.
> > <
> >
> https://github.com/apache/incubator-mxnet/tree/master/docker/docker-python
> > >
> > [2] The PR for python3 images is in progress and is here.
> > 
> >
> > Thanks,
> > Meghna Baijal
> >
>

Re: [Discussion] Separating PMC and Committership

[Chris Olivier]

I believe the assumption has always been that current PMC members will
remain PMC members.

On Wed, Oct 17, 2018 at 3:51 PM Michael Wall  wrote:

> I too think separating committers from PMC is a good idea for your project
> given the desire to grow committers and the concerns I have seen trying to
> add new committers.  I saw at least one other mentor, Jim on this thread
> too.
>
> Is the plan to leave all current PMC members in the PMC?  If that is not
> the plan, perhaps more discussion is required before moving on.
>
> Assuming you feel the discussion is done, someone needs to start a vote.
> This would be a procedural change as outlined on
> https://www.apache.org/foundation/voting.html
>
> If I were doing it, I would announce on this thread I am starting a vote on
> this matter tomorrow or some specified time.  I might even outline what the
> vote will be.  This give people a chance to speak up if they think more
> discussion is needed.  Assuming no more discussion, start a [VOTE] thread
> on the dev list.
>
> I am used to seeing [VOTE] and [DISCUSS] in the subject line of such emails
> but I didn't find any official guidance on that.  Maybe it is a project by
> project decision, I did find
> https://cwiki.apache.org/confluence/display/EDGENT/Sample+process+emails.
> I totally parsed right over the [Discussion] in the subject this thread but
> I'll be on the look out for it in the future.
>
> Thanks
>
> Mike
>
> On Wed, Oct 17, 2018 at 6:05 PM Carin Meier  wrote:
>
> > Let me rephrase the question.
> >
> > Since I'm new to the committer/PMC process, I wondering what the next
> step
> > is in a proposed change of process like this.
> >
> > If we gauge that there is a significant enough interest do we propose a
> > vote? Is there enough interest and information to have a vote in this
> case?
> >
> > (Anyone feel free to answer the question - mentor or not :) )
> >
> > - Carin
> >
> > On Tue, Oct 16, 2018 at 7:48 PM Carin Meier 
> wrote:
> >
> > > This has been a very interesting discussion and I think it underlined a
> > > desire to increase the committer pool and community for the project.
> I'm
> > > wondering now what the next steps would look like?
> > >
> > > Do any mentors have any advice on how to proceed?
> > >
> > > - Carin
> > >
> > > On Thu, Oct 11, 2018 at 1:23 PM Jim Jagielski  wrote:
> > >
> > >> In my experience, and in my opinion, it makes sense to distinguish and
> > >> differentiate between a committer and a PMC member. The latter shows
> > just a
> > >> bit more "investment" in the project and has obtained a bit more merit
> > due
> > >> to their continued efforts.
> > >>
> > >> Of course, what we also need is some public governance model that
> shows
> > >> what these levels are, what they mean and how to obtain them. The
> > following
> > >> is the normal setup for Apache projects:
> > >>
> > >> https://www.apache.org/foundation/how-it-works.html#roles
> > >>
> > >> The nice this is that this also allows for a very low-bar-to-entry for
> > >> committer-ship while still maintain a somewhat higher bar for the
> PPMC,
> > >> which is great for community building.
> > >>
> > >> > On Oct 9, 2018, at 2:11 PM, Haibin Lin 
> > >> wrote:
> > >> >
> > >> > Dear MXNet community,
> > >> >
> > >> > In the past when we invite a person to become a committer, he/she is
> > >> > automatically made a PMC member. However, a lot of communities keep
> a
> > >> small
> > >> > PMC, and a bigger and more diverse committers to enrich the
> community.
> > >> This
> > >> > has the benefit of having two opportunities to encourage
> contribution.
> > >> This
> > >> > can also help lower the bar for inviting committers, which helps
> build
> > >> > consensus in our already large PMC. I'd like to propose the
> following:
> > >> >
> > >> > For active contributors we first invite them to become our
> committers.
> > >> > Later on as they make significant contribution, we can invite them
> to
> > >> PMC.
> > >> >
> > >> >
> > >> > ===
> > >> > Comments from Marco:
> > >> >
> > >> > That's a great idea!
> > >> >
> > >> > The hard question is how to differentiate between a committer and a
> > PMC
> > >> > member and where we set the bar for each. If I understand you right,
> > you
> > >> > are proposing to honor active contributions by volume (or another
> > >> similar
> > >> > metric). While I think that's a good idea in general, I have a few
> > >> thoughts:
> > >> >
> > >> > We definitely have a lot of active people in the project, but let's
> > say
> > >> > that they contribute a substantial amount, but their contributions
> > >> can't go
> > >> > in as-is because they lack quality, consistency, testing or they
> don't
> > >> > match with the overall style and best practices. For a
> code-committer,
> > >> this
> > >> > would still be a no-go in my opinion. That person would still
> require
> > >> some
> > >> > guidance and mentoring until they are 

Re: [Discussion] Separating PMC and Committership

[Michael Wall]

I too think separating committers from PMC is a good idea for your project
given the desire to grow committers and the concerns I have seen trying to
add new committers.  I saw at least one other mentor, Jim on this thread
too.

Is the plan to leave all current PMC members in the PMC?  If that is not
the plan, perhaps more discussion is required before moving on.

Assuming you feel the discussion is done, someone needs to start a vote.
This would be a procedural change as outlined on
https://www.apache.org/foundation/voting.html

If I were doing it, I would announce on this thread I am starting a vote on
this matter tomorrow or some specified time.  I might even outline what the
vote will be.  This give people a chance to speak up if they think more
discussion is needed.  Assuming no more discussion, start a [VOTE] thread
on the dev list.

I am used to seeing [VOTE] and [DISCUSS] in the subject line of such emails
but I didn't find any official guidance on that.  Maybe it is a project by
project decision, I did find
https://cwiki.apache.org/confluence/display/EDGENT/Sample+process+emails.
I totally parsed right over the [Discussion] in the subject this thread but
I'll be on the look out for it in the future.

Thanks

Mike

On Wed, Oct 17, 2018 at 6:05 PM Carin Meier  wrote:

> Let me rephrase the question.
>
> Since I'm new to the committer/PMC process, I wondering what the next step
> is in a proposed change of process like this.
>
> If we gauge that there is a significant enough interest do we propose a
> vote? Is there enough interest and information to have a vote in this case?
>
> (Anyone feel free to answer the question - mentor or not :) )
>
> - Carin
>
> On Tue, Oct 16, 2018 at 7:48 PM Carin Meier  wrote:
>
> > This has been a very interesting discussion and I think it underlined a
> > desire to increase the committer pool and community for the project. I'm
> > wondering now what the next steps would look like?
> >
> > Do any mentors have any advice on how to proceed?
> >
> > - Carin
> >
> > On Thu, Oct 11, 2018 at 1:23 PM Jim Jagielski  wrote:
> >
> >> In my experience, and in my opinion, it makes sense to distinguish and
> >> differentiate between a committer and a PMC member. The latter shows
> just a
> >> bit more "investment" in the project and has obtained a bit more merit
> due
> >> to their continued efforts.
> >>
> >> Of course, what we also need is some public governance model that shows
> >> what these levels are, what they mean and how to obtain them. The
> following
> >> is the normal setup for Apache projects:
> >>
> >> https://www.apache.org/foundation/how-it-works.html#roles
> >>
> >> The nice this is that this also allows for a very low-bar-to-entry for
> >> committer-ship while still maintain a somewhat higher bar for the PPMC,
> >> which is great for community building.
> >>
> >> > On Oct 9, 2018, at 2:11 PM, Haibin Lin 
> >> wrote:
> >> >
> >> > Dear MXNet community,
> >> >
> >> > In the past when we invite a person to become a committer, he/she is
> >> > automatically made a PMC member. However, a lot of communities keep a
> >> small
> >> > PMC, and a bigger and more diverse committers to enrich the community.
> >> This
> >> > has the benefit of having two opportunities to encourage contribution.
> >> This
> >> > can also help lower the bar for inviting committers, which helps build
> >> > consensus in our already large PMC. I'd like to propose the following:
> >> >
> >> > For active contributors we first invite them to become our committers.
> >> > Later on as they make significant contribution, we can invite them to
> >> PMC.
> >> >
> >> >
> >> > ===
> >> > Comments from Marco:
> >> >
> >> > That's a great idea!
> >> >
> >> > The hard question is how to differentiate between a committer and a
> PMC
> >> > member and where we set the bar for each. If I understand you right,
> you
> >> > are proposing to honor active contributions by volume (or another
> >> similar
> >> > metric). While I think that's a good idea in general, I have a few
> >> thoughts:
> >> >
> >> > We definitely have a lot of active people in the project, but let's
> say
> >> > that they contribute a substantial amount, but their contributions
> >> can't go
> >> > in as-is because they lack quality, consistency, testing or they don't
> >> > match with the overall style and best practices. For a code-committer,
> >> this
> >> > would still be a no-go in my opinion. That person would still require
> >> some
> >> > guidance and mentoring until they are aligned with the project style
> and
> >> > guidelines as otherwise they might accept low-quality PRs. I know we
> can
> >> > revert that, but let's avoid confrontation as much as possible.
> >> >
> >> > The minimum bar for a code committer would then be:
> >> > - (almost) unaltered acceptance of their PRs (of course, some PRs are
> >> > intentionally made for discussions and those would even be a 

Re: Include MKLDNN into default mxnet pip package

[Jun Wu]

If my understanding is correct about the context, it should be acknowledged
that the significant performance improvement comes from the Intel MKLDNN
team's contribution in this PR:
https://github.com/apache/incubator-mxnet/pull/12530.

On Wed, Oct 17, 2018 at 3:12 PM kellen sunderland <
kellen.sunderl...@gmail.com> wrote:

> First of all thanks to Intel for these improvements, really a great effort.
>
> What would the compatibility story look like for users that don't have
> these AVX instructions?  Would there be any negative affect for AMD users?
>
> Regarding TensorRT: It's a possibility but not planned in the short term. A
> few considerations would be the limits on PyPi package sizes and the bloat
> incurred with TRT, the requirements of TRT to be installed on the user
> side, and the TRT engine build times which are non-trivial.  We can work
> towards fixing or working around these issues in the future if default TRT
> is something the user community would like to see for Cuda packages.  While
> the feature is experimental we'll likely continue to use
> 'mxnet-tensorrt-cu92' and 'mxnet-tensorrt-cu90'.
>
> On Wed, Oct 17, 2018 at 2:12 PM Alfredo Luque
>  wrote:
>
> > This is huge. Thanks for working on this. Is there a similar plan with
> eg;
> > tensor-rt support being ported into the main cuda-9.x packages?
> >
> > On October 17, 2018 at 2:10:20 PM, Alex Zai (aza...@gmail.com) wrote:
> >
> > Hey all,
> > We have been working hard these past few months to integrate and
> stabilize
> > Intel’s MKLDNN deep learning CPU accelerator into Mxnet and have made
> > incredible progress. On CPUs with AVX512 instructions (such as c5.18x) we
> > have seen performance increase up to 12x and on other platforms (Macs,
> > AVX2) we seen a speedup of 1.5+. Full list of benchmarks can be found
> here
> > (
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95650764
> > and https://github.com/apache/incubator-mxnet/pull/12591).
> >
> > Currently, using this accelerator requires the developer to either pip
> > install the mxnet-mkl version of mxnet or to build it themselves from
> > source. Given that we should try to provide the best performance "out of
> > the box” with mxnet we should include this in the default build. The
> mkldnn
> > library is included with in the pip package build so it does not require
> an
> > external dependency.
> >
> > There were concerns that MKLDNN could cause regressions on certain
> > platforms (as it did with the tensorflow version a while back); but we
> > added a env flag (MXNET_MKLDNN_ENABLED) that allows users to turn of this
> > feature during runtime. Please bring up any other concerns you may have
> and
> > your thoughts on including this accelerator in the default build.
> >
> > Best,
> > Alex
> >
> > —
> > Alfredo Luque
> > Software Engineer
> > Machine Learning Infrastructure
> > Airbnb
> > San Francisco, CA
> >
>

Re: Storing PGP Key for Publishing packages

[Pedro Larroy]

Do nightly artifacts need to be signed? For releases what you wrote and what 
Apache recommends makes total sense. Thus artifacts from cd can’t be signed 
manually.

Pedro

> On 17. Oct 2018, at 22:29, Naveen Swamy  wrote:
> 
> I am collaborating with Zach Kimberg and Qing to work on automatic (
> currently its very tedious and time consuming) publishing the MXNet-Scala
> maven package to Apache Snapshot repo(either as nightly or weekly), for
> publishing the package the artifacts need to be signed with a committer's
> key, however Zach found Apache seems to strictly advise against storing the
> PGP Keys, so I suggested to look at what Spark is doing and he found that
> they are releasing to Apache Snapshots as a nightly job so they got to be
> storing the credentials on the host.
> I am looking for advise from Mentors on how to proceed with this?
> 
> One option(not preferable) is to publish to a private Repo or an S3 bucket
> and only during the release and the keys continue to remain in the
> committers control.
> 
> -- Advise on PGP Key storage on Apache website--
> 
> 
> “It is recommended that you create a PGP key for your apache.org address
> now (or add that address to an existing key, if you have one). *DO NOT* create
> this key on any machine to which multiple users have access and *DO NOT*,
> ever, copy your private key to any other shared machine. Release managers
> need to take particular care of keys used to sign releases
> .“ (
> https://www.apache.org/dev/new-committers-guide.html#set-up-security-and-pgp-keys
> )
> 
> “Strictly speaking, releases must be *verified
> *
> on
> hardware owned and controlled by the committer. That means hardware the
> committer has physical possession and control of and exclusively full
> administrative/superuser access to. That's because only such hardware is
> qualified to hold a PGP private key, and the release should be verified on
> the machine the private key lives on or on a machine as trusted as that.” (
> https://www.apache.org/legal/release-policy.html#release-signing)
> 
> ---
> 
> 
> Thanks, Naveen

Re: Include MKLDNN into default mxnet pip package

[kellen sunderland]

First of all thanks to Intel for these improvements, really a great effort.

What would the compatibility story look like for users that don't have
these AVX instructions?  Would there be any negative affect for AMD users?

Regarding TensorRT: It's a possibility but not planned in the short term. A
few considerations would be the limits on PyPi package sizes and the bloat
incurred with TRT, the requirements of TRT to be installed on the user
side, and the TRT engine build times which are non-trivial.  We can work
towards fixing or working around these issues in the future if default TRT
is something the user community would like to see for Cuda packages.  While
the feature is experimental we'll likely continue to use
'mxnet-tensorrt-cu92' and 'mxnet-tensorrt-cu90'.

On Wed, Oct 17, 2018 at 2:12 PM Alfredo Luque
 wrote:

> This is huge. Thanks for working on this. Is there a similar plan with eg;
> tensor-rt support being ported into the main cuda-9.x packages?
>
> On October 17, 2018 at 2:10:20 PM, Alex Zai (aza...@gmail.com) wrote:
>
> Hey all,
> We have been working hard these past few months to integrate and stabilize
> Intel’s MKLDNN deep learning CPU accelerator into Mxnet and have made
> incredible progress. On CPUs with AVX512 instructions (such as c5.18x) we
> have seen performance increase up to 12x and on other platforms (Macs,
> AVX2) we seen a speedup of 1.5+. Full list of benchmarks can be found here
> (
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95650764
> and https://github.com/apache/incubator-mxnet/pull/12591).
>
> Currently, using this accelerator requires the developer to either pip
> install the mxnet-mkl version of mxnet or to build it themselves from
> source. Given that we should try to provide the best performance "out of
> the box” with mxnet we should include this in the default build. The mkldnn
> library is included with in the pip package build so it does not require an
> external dependency.
>
> There were concerns that MKLDNN could cause regressions on certain
> platforms (as it did with the tensorflow version a while back); but we
> added a env flag (MXNET_MKLDNN_ENABLED) that allows users to turn of this
> feature during runtime. Please bring up any other concerns you may have and
> your thoughts on including this accelerator in the default build.
>
> Best,
> Alex
>
> —
> Alfredo Luque
> Software Engineer
> Machine Learning Infrastructure
> Airbnb
> San Francisco, CA
>

Re: [Discussion] Separating PMC and Committership

[Carin Meier]

Let me rephrase the question.

Since I'm new to the committer/PMC process, I wondering what the next step
is in a proposed change of process like this.

If we gauge that there is a significant enough interest do we propose a
vote? Is there enough interest and information to have a vote in this case?

(Anyone feel free to answer the question - mentor or not :) )

- Carin

On Tue, Oct 16, 2018 at 7:48 PM Carin Meier  wrote:

> This has been a very interesting discussion and I think it underlined a
> desire to increase the committer pool and community for the project. I'm
> wondering now what the next steps would look like?
>
> Do any mentors have any advice on how to proceed?
>
> - Carin
>
> On Thu, Oct 11, 2018 at 1:23 PM Jim Jagielski  wrote:
>
>> In my experience, and in my opinion, it makes sense to distinguish and
>> differentiate between a committer and a PMC member. The latter shows just a
>> bit more "investment" in the project and has obtained a bit more merit due
>> to their continued efforts.
>>
>> Of course, what we also need is some public governance model that shows
>> what these levels are, what they mean and how to obtain them. The following
>> is the normal setup for Apache projects:
>>
>> https://www.apache.org/foundation/how-it-works.html#roles
>>
>> The nice this is that this also allows for a very low-bar-to-entry for
>> committer-ship while still maintain a somewhat higher bar for the PPMC,
>> which is great for community building.
>>
>> > On Oct 9, 2018, at 2:11 PM, Haibin Lin 
>> wrote:
>> >
>> > Dear MXNet community,
>> >
>> > In the past when we invite a person to become a committer, he/she is
>> > automatically made a PMC member. However, a lot of communities keep a
>> small
>> > PMC, and a bigger and more diverse committers to enrich the community.
>> This
>> > has the benefit of having two opportunities to encourage contribution.
>> This
>> > can also help lower the bar for inviting committers, which helps build
>> > consensus in our already large PMC. I'd like to propose the following:
>> >
>> > For active contributors we first invite them to become our committers.
>> > Later on as they make significant contribution, we can invite them to
>> PMC.
>> >
>> >
>> > ===
>> > Comments from Marco:
>> >
>> > That's a great idea!
>> >
>> > The hard question is how to differentiate between a committer and a PMC
>> > member and where we set the bar for each. If I understand you right, you
>> > are proposing to honor active contributions by volume (or another
>> similar
>> > metric). While I think that's a good idea in general, I have a few
>> thoughts:
>> >
>> > We definitely have a lot of active people in the project, but let's say
>> > that they contribute a substantial amount, but their contributions
>> can't go
>> > in as-is because they lack quality, consistency, testing or they don't
>> > match with the overall style and best practices. For a code-committer,
>> this
>> > would still be a no-go in my opinion. That person would still require
>> some
>> > guidance and mentoring until they are aligned with the project style and
>> > guidelines as otherwise they might accept low-quality PRs. I know we can
>> > revert that, but let's avoid confrontation as much as possible.
>> >
>> > The minimum bar for a code committer would then be:
>> > - (almost) unaltered acceptance of their PRs (of course, some PRs are
>> > intentionally made for discussions and those would even be a plus!)
>> > - following mxnets community guidelines, rules and styles
>> > - giving useful reviews (in order to see how they would be as reviewers
>> if
>> > they were a committer)
>> > The would be weighted differently on a case by case base, but this
>> could be
>> > a starting point to describe what we are looking for.
>> >
>> > From committer to PMC on the other hand, the difference is quite small.
>> > Something I personally would be looking for are three things:
>> > - judgement
>> > - community engagement
>> > - Apache way
>> > While a committer might be chosen due to their contributions, they
>> wouldn't
>> > be evaluated that strictly for the above points. A PMC member is a
>> > representative of the project who steers the long term development of
>> it.
>> > Thus, they should be active on our channels like dev@, make good
>> reviews on
>> > GitHub (if applicable), express good judgement and reasoning during
>> votes
>> > and generally show that they are generally helpful to the project on a
>> > non-code level.
>> >
>> > These are just some thoughts of mine to help start of this discussions.
>> It
>> > would be good to hear what other people are looking for while evaluating
>> > candidates and if there's anything they would like to highlight.
>> >
>> > ==
>> >
>> > Comments from Carin:
>> >
>> > I think it is a good idea. Here is a bit of reasoning behind my
>> thoughts.
>> >
>> > *Pros 

Storing PGP Key for Publishing packages

[Naveen Swamy]

I am collaborating with Zach Kimberg and Qing to work on automatic (
currently its very tedious and time consuming) publishing the MXNet-Scala
maven package to Apache Snapshot repo(either as nightly or weekly), for
publishing the package the artifacts need to be signed with a committer's
key, however Zach found Apache seems to strictly advise against storing the
PGP Keys, so I suggested to look at what Spark is doing and he found that
they are releasing to Apache Snapshots as a nightly job so they got to be
storing the credentials on the host.
I am looking for advise from Mentors on how to proceed with this?

One option(not preferable) is to publish to a private Repo or an S3 bucket
and only during the release and the keys continue to remain in the
committers control.

-- Advise on PGP Key storage on Apache website--


“It is recommended that you create a PGP key for your apache.org address
now (or add that address to an existing key, if you have one). *DO NOT* create
this key on any machine to which multiple users have access and *DO NOT*,
ever, copy your private key to any other shared machine. Release managers
need to take particular care of keys used to sign releases
.“ (
https://www.apache.org/dev/new-committers-guide.html#set-up-security-and-pgp-keys
)

“Strictly speaking, releases must be *verified
*
on
hardware owned and controlled by the committer. That means hardware the
committer has physical possession and control of and exclusively full
administrative/superuser access to. That's because only such hardware is
qualified to hold a PGP private key, and the release should be verified on
the machine the private key lives on or on a machine as trusted as that.” (
https://www.apache.org/legal/release-policy.html#release-signing)

 ---


Thanks, Naveen

Re: Apache MXNet (incubating) Python Docker Images

[kellen sunderland]

This feels like something we should get a little data on before making a
decision, but I also don't have a strong opinion.  I would bias towards
pushing something that might be imperfect and moving on to develop other
improvements for users rather than determining a 'perfect' solution.

The questions/tradeoffs I see are (1) should we support multiple python
versions in the first place, which requires extra work on our part but
supports more users and (2) should we favor forwards or backwards
compatibility, i.e. should we prioritize supporting existing users or
prioritize making something that won't cause future problems for new and
exiting users.

The best data I can find with a quick google is the annual Jetbrains survey
which shows python2 went from 47% in 2017 to 25% in 2018:
https://www.jetbrains.com/research/devecosystem-2017/python/
https://www.jetbrains.com/research/devecosystem-2018/python/

So python2 usage is trending sharply down but is not yet low enough to
ignore which I think means we should try and support both on Dockerhub (1).

I don't see backwards compatibility with existing Docker users is a major
concern given these Dockerfiles haven't been supported for a long time.  I
would prioritize forwards compatibility (2) and assume we want to create
something that will remain compatible for as long as possible.

So I would push both python2 and python3 images, but make python 3.5 the
default version, and python2 a version with a postfixed py2 tag in
Dockerhub.

Thanks to Mu (and others?) for original creating this Dockerhub images, I
used to use them and found them very convenient, and to you Meghna for
updating them.  I think basing them on the pip packages is also a good way
to lower maintenance burden and make sure we leverage the great work Sheng
has done to create those packages.

On Wed, Oct 17, 2018 at 11:52 AM Meghna Baijal 
wrote:

> Hi All,
>
> I am currently in the process of updating the python docker images for
> Apache MXNet such that they are built on top of the pip binaries.
> Until now these were built to use python 2.7 but with an upcoming PR I am
> also adding python 3.5 docker images. I would like to know the community’s
> preference on whether I should keep the *Python 2.7 Docker image as the
> default or should I move to Python 3.5 as the default version*?
>
> [1] The new python2 dockerfiles and build script can be found here.
> <
> https://github.com/apache/incubator-mxnet/tree/master/docker/docker-python
> >
> [2] The PR for python3 images is in progress and is here.
> 
>
> Thanks,
> Meghna Baijal
>

Re: [LAZY VOTE]: rename dockerfiles s/.build.//

[kellen sunderland]

May be of interest to people that we're trying get a good set of
production-ready Dockerfiles (which I'm referring to as runtime Dockerfiles
in this thread) with a PR open here:
https://github.com/apache/incubator-mxnet/pull/12791 (thanks for updating
these Meghna).

On Wed, Oct 17, 2018 at 12:00 PM Naveen Swamy  wrote:

> I agree with Kellen on not renaming the CI docker files (by renaming - i
> think its implicit you can use these for production) i don't think we
> should telling our users go use these bloated docker files, you could
> create lean separate docker files for production use-case with only
> necessary runtime packages.
>
> -1
>
> On Wed, Oct 17, 2018 at 11:48 AM kellen sunderland <
> kellen.sunderl...@gmail.com> wrote:
>
> > Hey Pedro, sorry I still don't see a good reason to justify changing the
> > filenames.  Renaming them to be less specific isn't going to explain to
> > users what the purpose of the files is, and it could cause breakages with
> > any system that refer to these files including external company's CI
> > systems.  If I think of the benefits versus potential errors introduced
> by
> > making the change I see more potential risk than obvious benefits.  I
> also
> > feel that this change will make the difference between the runtime docker
> > files and the CI docker files less clear to users, not more clear.  In
> > general I think adding a descriptive README.md would server our purposed
> > better here.  Happy to hear what others think.
> >
> > On Wed, Oct 17, 2018 at 6:45 AM Pedro Larroy <
> pedro.larroy.li...@gmail.com
> > >
> > wrote:
> >
> > > Hi Kellen, thank you for your response.
> > >
> > > Maybe I didn't explain myself correctly. The purpose of this
> > infrastructure
> > > is not changed.
> > >
> > > I'm not planning to use these Dockerfiles as MXNet docker containers
> for
> > > users to run MXNet, that is a separate concern.
> > >
> > > It is just that some of this Dockerfiles we use in CI to build, test
> and
> > > generate documentation, so are used as a runtime container as well.
> Thus
> > > i'm just changing the pathing for semantic reasons and remove the
> .build.
> > > which is just noise.
> > >
> > > As an example I would like to explain that we are about to merge the PR
> > > which uses QEMU to run the unit tests, so there's an associated
> > Dockerfile
> > > which hosts the QEMU runtime environment used to execute the unit tests
> > in
> > > an ARM emulated machine. Thus makes little sense that these Dockerfiles
> > are
> > > called "build".  I don't know if my explanation changes your vote.
> Either
> > > way please let me know. Separating this change in a different PR was
> > > suggested by several MXNet contributors during review.
> > >
> > > Pedro.
> > >
> > > On Wed, Oct 17, 2018 at 11:21 AM kellen sunderland <
> > > kellen.sunderl...@gmail.com> wrote:
> > >
> > > > -1. (non-binding)
> > > >
> > > > These Dockerfiles are very bloated and imo only useful for creating a
> > > build
> > > > environment or running tests.  Just as you wouldn't setup a server
> for
> > a
> > > > service and then install 200 packages that may or may not be used for
> > the
> > > > service I wouldn't recommend using these Dockerfiles at runtime.
> > Runtime
> > > > Dockerfiles should in my opinion be as lightweight and suited to
> their
> > > task
> > > > as possible.
> > > >
> > > > On Wed, Oct 17, 2018, 1:58 AM Hagay Lupesko 
> wrote:
> > > >
> > > > > The PR provides a good explanation of this change and all code
> > updates.
> > > > > LGTM.
> > > > >
> > > > > On Tue, Oct 16, 2018 at 8:41 AM Pedro Larroy <
> > > > pedro.larroy.li...@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi
> > > > > >
> > > > > > I would like to rename the dockerfiles since they are used as a
> > > runtime
> > > > > > environment and not only as build as they were initially
> intended.
> > > > > >
> > > > > > More info about the change in this PR:
> > > > > > https://github.com/apache/incubator-mxnet/pull/12423/files
> > > > > >
> > > > > >
> > > > > > Pedro.
> > > > > >
> > > > >
> > > >
> > >
> >
>

Slack

[Holger Kohr]

Hi!

I'd like to join the MXNet Slack channel. My GH handle is @kohr-h.

Cheers,
Holger

Re: [LAZY VOTE]: rename dockerfiles s/.build.//

[Naveen Swamy]

I agree with Kellen on not renaming the CI docker files (by renaming - i
think its implicit you can use these for production) i don't think we
should telling our users go use these bloated docker files, you could
create lean separate docker files for production use-case with only
necessary runtime packages.

-1

On Wed, Oct 17, 2018 at 11:48 AM kellen sunderland <
kellen.sunderl...@gmail.com> wrote:

> Hey Pedro, sorry I still don't see a good reason to justify changing the
> filenames.  Renaming them to be less specific isn't going to explain to
> users what the purpose of the files is, and it could cause breakages with
> any system that refer to these files including external company's CI
> systems.  If I think of the benefits versus potential errors introduced by
> making the change I see more potential risk than obvious benefits.  I also
> feel that this change will make the difference between the runtime docker
> files and the CI docker files less clear to users, not more clear.  In
> general I think adding a descriptive README.md would server our purposed
> better here.  Happy to hear what others think.
>
> On Wed, Oct 17, 2018 at 6:45 AM Pedro Larroy  >
> wrote:
>
> > Hi Kellen, thank you for your response.
> >
> > Maybe I didn't explain myself correctly. The purpose of this
> infrastructure
> > is not changed.
> >
> > I'm not planning to use these Dockerfiles as MXNet docker containers for
> > users to run MXNet, that is a separate concern.
> >
> > It is just that some of this Dockerfiles we use in CI to build, test and
> > generate documentation, so are used as a runtime container as well. Thus
> > i'm just changing the pathing for semantic reasons and remove the .build.
> > which is just noise.
> >
> > As an example I would like to explain that we are about to merge the PR
> > which uses QEMU to run the unit tests, so there's an associated
> Dockerfile
> > which hosts the QEMU runtime environment used to execute the unit tests
> in
> > an ARM emulated machine. Thus makes little sense that these Dockerfiles
> are
> > called "build".  I don't know if my explanation changes your vote. Either
> > way please let me know. Separating this change in a different PR was
> > suggested by several MXNet contributors during review.
> >
> > Pedro.
> >
> > On Wed, Oct 17, 2018 at 11:21 AM kellen sunderland <
> > kellen.sunderl...@gmail.com> wrote:
> >
> > > -1. (non-binding)
> > >
> > > These Dockerfiles are very bloated and imo only useful for creating a
> > build
> > > environment or running tests.  Just as you wouldn't setup a server for
> a
> > > service and then install 200 packages that may or may not be used for
> the
> > > service I wouldn't recommend using these Dockerfiles at runtime.
> Runtime
> > > Dockerfiles should in my opinion be as lightweight and suited to their
> > task
> > > as possible.
> > >
> > > On Wed, Oct 17, 2018, 1:58 AM Hagay Lupesko  wrote:
> > >
> > > > The PR provides a good explanation of this change and all code
> updates.
> > > > LGTM.
> > > >
> > > > On Tue, Oct 16, 2018 at 8:41 AM Pedro Larroy <
> > > pedro.larroy.li...@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > Hi
> > > > >
> > > > > I would like to rename the dockerfiles since they are used as a
> > runtime
> > > > > environment and not only as build as they were initially intended.
> > > > >
> > > > > More info about the change in this PR:
> > > > > https://github.com/apache/incubator-mxnet/pull/12423/files
> > > > >
> > > > >
> > > > > Pedro.
> > > > >
> > > >
> > >
> >
>

Apache MXNet (incubating) Python Docker Images

[Meghna Baijal]

Hi All,

I am currently in the process of updating the python docker images for
Apache MXNet such that they are built on top of the pip binaries.
Until now these were built to use python 2.7 but with an upcoming PR I am
also adding python 3.5 docker images. I would like to know the community’s
preference on whether I should keep the *Python 2.7 Docker image as the
default or should I move to Python 3.5 as the default version*?

[1] The new python2 dockerfiles and build script can be found here.

[2] The PR for python3 images is in progress and is here.


Thanks,
Meghna Baijal

Re: [LAZY VOTE]: rename dockerfiles s/.build.//

[kellen sunderland]

Hey Pedro, sorry I still don't see a good reason to justify changing the
filenames.  Renaming them to be less specific isn't going to explain to
users what the purpose of the files is, and it could cause breakages with
any system that refer to these files including external company's CI
systems.  If I think of the benefits versus potential errors introduced by
making the change I see more potential risk than obvious benefits.  I also
feel that this change will make the difference between the runtime docker
files and the CI docker files less clear to users, not more clear.  In
general I think adding a descriptive README.md would server our purposed
better here.  Happy to hear what others think.

On Wed, Oct 17, 2018 at 6:45 AM Pedro Larroy 
wrote:

> Hi Kellen, thank you for your response.
>
> Maybe I didn't explain myself correctly. The purpose of this infrastructure
> is not changed.
>
> I'm not planning to use these Dockerfiles as MXNet docker containers for
> users to run MXNet, that is a separate concern.
>
> It is just that some of this Dockerfiles we use in CI to build, test and
> generate documentation, so are used as a runtime container as well. Thus
> i'm just changing the pathing for semantic reasons and remove the .build.
> which is just noise.
>
> As an example I would like to explain that we are about to merge the PR
> which uses QEMU to run the unit tests, so there's an associated Dockerfile
> which hosts the QEMU runtime environment used to execute the unit tests in
> an ARM emulated machine. Thus makes little sense that these Dockerfiles are
> called "build".  I don't know if my explanation changes your vote. Either
> way please let me know. Separating this change in a different PR was
> suggested by several MXNet contributors during review.
>
> Pedro.
>
> On Wed, Oct 17, 2018 at 11:21 AM kellen sunderland <
> kellen.sunderl...@gmail.com> wrote:
>
> > -1. (non-binding)
> >
> > These Dockerfiles are very bloated and imo only useful for creating a
> build
> > environment or running tests.  Just as you wouldn't setup a server for a
> > service and then install 200 packages that may or may not be used for the
> > service I wouldn't recommend using these Dockerfiles at runtime.  Runtime
> > Dockerfiles should in my opinion be as lightweight and suited to their
> task
> > as possible.
> >
> > On Wed, Oct 17, 2018, 1:58 AM Hagay Lupesko  wrote:
> >
> > > The PR provides a good explanation of this change and all code updates.
> > > LGTM.
> > >
> > > On Tue, Oct 16, 2018 at 8:41 AM Pedro Larroy <
> > pedro.larroy.li...@gmail.com
> > > >
> > > wrote:
> > >
> > > > Hi
> > > >
> > > > I would like to rename the dockerfiles since they are used as a
> runtime
> > > > environment and not only as build as they were initially intended.
> > > >
> > > > More info about the change in this PR:
> > > > https://github.com/apache/incubator-mxnet/pull/12423/files
> > > >
> > > >
> > > > Pedro.
> > > >
> > >
> >
>

Re: New MXNet CI guide: Managing and accessing credentials

[Marco de Abreu]

Yes, definitely! I have added it to the list of examples. Thanks :)

-Marco

Aaron Markham  schrieb am Mi., 17. Okt. 2018,
16:15:

> Thanks Marco, this is a great feature. I didn't see git listed under "when
> to use this guide". Can we use this to access git credentials like for when
> we publish the website?
> Cheers,
> Aaron
>
> On Wed, Oct 17, 2018, 06:19 Marco de Abreu
>  wrote:
>
> > Hello,
> >
> > I have just published a new guide how to manage and access credentials in
> > MXNet CI. It's available at
> >
> >
> https://cwiki.apache.org/confluence/display/MXNET/Managing+and+accessing+credentials
> > .
> >
> > This is was part of the preparation for the Scala Maven Publish job that
> > will run as Continuous Deployment.
> >
> > Best regards,
> > Marco
> >
>

Re: New MXNet CI guide: Managing and accessing credentials

[Aaron Markham]

Thanks Marco, this is a great feature. I didn't see git listed under "when
to use this guide". Can we use this to access git credentials like for when
we publish the website?
Cheers,
Aaron

On Wed, Oct 17, 2018, 06:19 Marco de Abreu
 wrote:

> Hello,
>
> I have just published a new guide how to manage and access credentials in
> MXNet CI. It's available at
>
> https://cwiki.apache.org/confluence/display/MXNET/Managing+and+accessing+credentials
> .
>
> This is was part of the preparation for the Scala Maven Publish job that
> will run as Continuous Deployment.
>
> Best regards,
> Marco
>

Re: [LAZY VOTE]: rename dockerfiles s/.build.//

[Pedro Larroy]

Hi Kellen, thank you for your response.

Maybe I didn't explain myself correctly. The purpose of this infrastructure
is not changed.

I'm not planning to use these Dockerfiles as MXNet docker containers for
users to run MXNet, that is a separate concern.

It is just that some of this Dockerfiles we use in CI to build, test and
generate documentation, so are used as a runtime container as well. Thus
i'm just changing the pathing for semantic reasons and remove the .build.
which is just noise.

As an example I would like to explain that we are about to merge the PR
which uses QEMU to run the unit tests, so there's an associated Dockerfile
which hosts the QEMU runtime environment used to execute the unit tests in
an ARM emulated machine. Thus makes little sense that these Dockerfiles are
called "build".  I don't know if my explanation changes your vote. Either
way please let me know. Separating this change in a different PR was
suggested by several MXNet contributors during review.

Pedro.

On Wed, Oct 17, 2018 at 11:21 AM kellen sunderland <
kellen.sunderl...@gmail.com> wrote:

> -1. (non-binding)
>
> These Dockerfiles are very bloated and imo only useful for creating a build
> environment or running tests.  Just as you wouldn't setup a server for a
> service and then install 200 packages that may or may not be used for the
> service I wouldn't recommend using these Dockerfiles at runtime.  Runtime
> Dockerfiles should in my opinion be as lightweight and suited to their task
> as possible.
>
> On Wed, Oct 17, 2018, 1:58 AM Hagay Lupesko  wrote:
>
> > The PR provides a good explanation of this change and all code updates.
> > LGTM.
> >
> > On Tue, Oct 16, 2018 at 8:41 AM Pedro Larroy <
> pedro.larroy.li...@gmail.com
> > >
> > wrote:
> >
> > > Hi
> > >
> > > I would like to rename the dockerfiles since they are used as a runtime
> > > environment and not only as build as they were initially intended.
> > >
> > > More info about the change in this PR:
> > > https://github.com/apache/incubator-mxnet/pull/12423/files
> > >
> > >
> > > Pedro.
> > >
> >
>

New MXNet CI guide: Managing and accessing credentials

[Marco de Abreu]

Hello,

I have just published a new guide how to manage and access credentials in
MXNet CI. It's available at
https://cwiki.apache.org/confluence/display/MXNET/Managing+and+accessing+credentials
.

This is was part of the preparation for the Scala Maven Publish job that
will run as Continuous Deployment.

Best regards,
Marco

Re: [LAZY VOTE]: rename dockerfiles s/.build.//

[kellen sunderland]

-1. (non-binding)

These Dockerfiles are very bloated and imo only useful for creating a build
environment or running tests.  Just as you wouldn't setup a server for a
service and then install 200 packages that may or may not be used for the
service I wouldn't recommend using these Dockerfiles at runtime.  Runtime
Dockerfiles should in my opinion be as lightweight and suited to their task
as possible.

On Wed, Oct 17, 2018, 1:58 AM Hagay Lupesko  wrote:

> The PR provides a good explanation of this change and all code updates.
> LGTM.
>
> On Tue, Oct 16, 2018 at 8:41 AM Pedro Larroy  >
> wrote:
>
> > Hi
> >
> > I would like to rename the dockerfiles since they are used as a runtime
> > environment and not only as build as they were initially intended.
> >
> > More info about the change in this PR:
> > https://github.com/apache/incubator-mxnet/pull/12423/files
> >
> >
> > Pedro.
> >
>

Re: [LAZY VOTE]: rename dockerfiles s/.build.//

[Hagay Lupesko]

The PR provides a good explanation of this change and all code updates.
LGTM.

On Tue, Oct 16, 2018 at 8:41 AM Pedro Larroy 
wrote:

> Hi
>
> I would like to rename the dockerfiles since they are used as a runtime
> environment and not only as build as they were initially intended.
>
> More info about the change in this PR:
> https://github.com/apache/incubator-mxnet/pull/12423/files
>
>
> Pedro.
>