Re: [ANNOUNCEMENT] New Committer: Qing Lan

2018-11-20 Thread Qing Lan
Thanks everyone! Looking forward to contribute more to the community!

Thanks,
Qing

On 11/20/18, 6:31 PM, "Steffen Rochel"  wrote:

Congratulation Qing!

On Tue, Nov 20, 2018 at 6:29 PM Hagay Lupesko  wrote:

> Congrats Qing! Awesome to see you become a committer!
>
> On Tue, Nov 20, 2018 at 4:26 PM Marco de Abreu
>  wrote:
>
> > Great to have your on board, Qing!
> >
> > Am Mi., 21. Nov. 2018, 01:24 hat Naveen Swamy 
> > geschrieben:
> >
> > > The Project Podling Management Committee (PPMC) for Apache MXNet has
> > > invited Qing Lan based on his contribution to MXNet Scala to become a
> > > committer and we are pleased to announce that he has accepted.
> > >
> > > Qing, thanks a lot for your contribution and continued effort to
> support
> > > MXNet community.
> > >
> > > Please join me in welcoming Qing to the project!
> > >
> > > Thanks, Naveen
> > > (on behalf of Apache MXNet PPMC)
> > >
> >
>




CI impaired

2018-11-20 Thread Marco de Abreu
Hello,

I'd like to let you know that our CI was impaired and down for the last few
hours. After getting the CI back up, I noticed that our auto scaling broke
due to a silent update of Jenkins which broke our upscale-detection. Manual
scaling is currently not possible and stopping the scaling won't help
either because there are currently no p3 instances available, which means
that all jobs will fail none the less. In a few hours, the auto scaling
will have recycled all slaves through the down-scale mechanism and we will
be out of capacity. This will lead to resource starvation and thus timeouts.

Your PRs will be properly registered by Jenkins, but please expect the jobs
to time out and thus fail your PRs.

I will fix the auto scaling as soon as I'm awake again.

Sorry for the caused inconveniences.

Best regards,
Marco


P.S. Sorry for the brief email and my lack of further fixes, but it's
5:30AM now and I've been working for 17 hours.


Re: [Anouncement] New Committer: Kellen Sunderland

2018-11-20 Thread Marco de Abreu
Welcome Kellen !

Am Mi., 21. Nov. 2018, 03:28 hat Steffen Rochel 
geschrieben:

> Congratulation Kellen, well deserved!
> Steffen
>
> On Tue, Nov 20, 2018 at 4:02 PM Tianqi Chen  wrote:
>
> > We are pleased to announce Kellen Sunderland as a new committer of Apache
> > MXNet. Kellen has a  sustained effort to the project both in the
> discussion
> > and code contributions.
> >
> > PRs
> >
> >
> https://github.com/apache/incubator-mxnet/pulls?utf8=%E2%9C%93=is%3Apr+author%3AKellenSunderland+
> > reviews
> >
> >
> https://github.com/apache/incubator-mxnet/pulls?utf8=%E2%9C%93=is%3Apr+reviewed-by%3AKellenSunderland
> > dev@
> > https://lists.apache.org/list.html?d...@mxnet.apache.org:lte=3y:Kellen
> > %20Sunderland
> > <
> https://lists.apache.org/list.html?d...@mxnet.apache.org:lte=3y:Kellen%20Sunderland
> >
> >
> > Please join me to welcome Kellen to the team!
> >
> > Tianqi
> > - On behalf of Apache MXNet(incubating) PMC
> >
>


Re: [Announce] Upcoming Apache MXNet (incubating) 1.4.0 release

2018-11-20 Thread kellen sunderland
Spoke too soon[1], looks like others have been adding Turing support as
well (thanks to those helping with this).  I believe there's still a few
changes we'd have to make to claim support though (mshadow CMake changes,
PyPi package creation tweaks).

1:
https://github.com/apache/incubator-mxnet/commit/2c3357443ec3d49a11e93c89f278264ce10c2f08

On Tue, Nov 20, 2018 at 7:00 PM kellen sunderland <
kellen.sunderl...@gmail.com> wrote:

> Hey Steffen, I'd like to be able to merge this PR for version 1.4:
> https://github.com/apache/incubator-mxnet/pull/13310 . It fixes a
> regression in master which causes incorrect feature vectors to be output
> when using the TensorRT feature.  (Thanks to Nathalie for helping me track
> down the root cause of the issue).   I'm currently blocked on a CI issue I
> haven't seen before, but hope to have it resolved by EOW.
>
> One call-out I would make is that we currently don't support Turing
> architecture (sm_75).  I've been slowly trying to add support, but I don't
> think I'd have capacity to do this done by EOW.  Does anyone feel strongly
> we need this in the 1.4 release?  From my perspective this will already be
> a strong release without it.
>
> On Tue, Nov 20, 2018 at 6:42 PM Steffen Rochel 
> wrote:
>
>> Thanks Patrick, lets target to get the PR's merged this week.
>>
>> Call for contributions from the community: Right now we have 10 PR
>> awaiting
>> merge
>> <
>> https://github.com/apache/incubator-mxnet/pulls?utf8=%E2%9C%93=is%3Apr+is%3Aopen+label%3Apr-awaiting-merge+
>> >
>> and
>> we have 61 open PR awaiting review.
>> <
>> https://github.com/apache/incubator-mxnet/pulls?utf8=%E2%9C%93=is%3Apr+is%3Aopen+label%3Apr-awaiting-review
>> >
>> I would appreciate if you all can help to review the open PR and the
>> committers can drive the merge before code freeze for 1.4.0.
>>
>> The contributors on the Java API are making progress, but not all
>> performance issues are resolved. With some luck it should be possible to
>> code freeze towards end of this week.
>>
>> Are there other critical features/bugs/PR you think need to be included in
>> 1.4.0? If so, please communicate as soon as possible.
>>
>> Regards,
>> Steffen
>>
>> On Mon, Nov 19, 2018 at 8:26 PM Zhao, Patric 
>> wrote:
>>
>> > Thanks, Steffen. I think there is NO open issue to block the MKLDNN to
>> GA
>> > now.
>> >
>> > BTW, several quantization related PRs (#13297,#13260) are under the
>> review
>> > and I think it can be merged in this week.
>> >
>> > Thanks,
>> >
>> > --Patric
>> >
>> >
>> > > -Original Message-
>> > > From: Steffen Rochel [mailto:steffenroc...@gmail.com]
>> > > Sent: Tuesday, November 20, 2018 2:57 AM
>> > > To: dev@mxnet.incubator.apache.org
>> > > Subject: Re: [Announce] Upcoming Apache MXNet (incubating) 1.4.0
>> release
>> > >
>> > > On Friday the contributors working on Java API discovered a potential
>> > > performance problem with inference using Java API vs. Python.
>> > Investigation
>> > > is ongoing.
>> > > As the Java API is one of the main features for the upcoming release,
>> I
>> > > suggest to post-pone the code freeze towards end of this week.
>> > >
>> > > Please provide feedback and concern about the change in dates for code
>> > > freeze and 1.4.0 release. I will provide updates on progress resolving
>> > the
>> > > potential performance problem.
>> > >
>> > > Patrick - do you think it is possible to resolve the remaining issues
>> on
>> > MKL-
>> > > DNN this week, so we can consider GA for MKL-DNN with 1.4.0?
>> > >
>> > > Regards,
>> > > Steffen
>> > >
>> > > On Thu, Nov 15, 2018 at 5:26 AM Anton Chernov 
>> > > wrote:
>> > >
>> > > > I'd like to remind everyone that 'code freeze' would mean cutting a
>> > > > v1.4.x release branch and all following fixes would need to be
>> > backported.
>> > > > Development on master can be continued as usual.
>> > > >
>> > > > Best
>> > > > Anton
>> > > >
>> > > > ср, 14 нояб. 2018 г. в 6:04, Steffen Rochel <
>> steffenroc...@gmail.com>:
>> > > >
>> > > > > Dear MXNet community,
>> > > > > the agreed plan was to establish code freeze for 1.4.0 release
>> > > > > today. As the 1.3.1 patch release is still ongoing I suggest to
>> > > > > post-pone the code freeze to Friday 16th November 2018.
>> > > > >
>> > > > > Sergey Kolychev has agreed to act as co-release manager for all
>> > > > > tasks
>> > > > which
>> > > > > require committer privileges. If anybody is interested to
>> volunteer
>> > > > > as release manager - now is the time to speak up. Otherwise I will
>> > > > > manage
>> > > > the
>> > > > > release.
>> > > > >
>> > > > > Regards,
>> > > > > Steffen
>> > > > >
>> > > >
>> >
>>
>


Re: [Announce] Upcoming Apache MXNet (incubating) 1.4.0 release

2018-11-20 Thread kellen sunderland
Hey Steffen, I'd like to be able to merge this PR for version 1.4:
https://github.com/apache/incubator-mxnet/pull/13310 . It fixes a
regression in master which causes incorrect feature vectors to be output
when using the TensorRT feature.  (Thanks to Nathalie for helping me track
down the root cause of the issue).   I'm currently blocked on a CI issue I
haven't seen before, but hope to have it resolved by EOW.

One call-out I would make is that we currently don't support Turing
architecture (sm_75).  I've been slowly trying to add support, but I don't
think I'd have capacity to do this done by EOW.  Does anyone feel strongly
we need this in the 1.4 release?  From my perspective this will already be
a strong release without it.

On Tue, Nov 20, 2018 at 6:42 PM Steffen Rochel 
wrote:

> Thanks Patrick, lets target to get the PR's merged this week.
>
> Call for contributions from the community: Right now we have 10 PR awaiting
> merge
> <
> https://github.com/apache/incubator-mxnet/pulls?utf8=%E2%9C%93=is%3Apr+is%3Aopen+label%3Apr-awaiting-merge+
> >
> and
> we have 61 open PR awaiting review.
> <
> https://github.com/apache/incubator-mxnet/pulls?utf8=%E2%9C%93=is%3Apr+is%3Aopen+label%3Apr-awaiting-review
> >
> I would appreciate if you all can help to review the open PR and the
> committers can drive the merge before code freeze for 1.4.0.
>
> The contributors on the Java API are making progress, but not all
> performance issues are resolved. With some luck it should be possible to
> code freeze towards end of this week.
>
> Are there other critical features/bugs/PR you think need to be included in
> 1.4.0? If so, please communicate as soon as possible.
>
> Regards,
> Steffen
>
> On Mon, Nov 19, 2018 at 8:26 PM Zhao, Patric 
> wrote:
>
> > Thanks, Steffen. I think there is NO open issue to block the MKLDNN to GA
> > now.
> >
> > BTW, several quantization related PRs (#13297,#13260) are under the
> review
> > and I think it can be merged in this week.
> >
> > Thanks,
> >
> > --Patric
> >
> >
> > > -Original Message-
> > > From: Steffen Rochel [mailto:steffenroc...@gmail.com]
> > > Sent: Tuesday, November 20, 2018 2:57 AM
> > > To: dev@mxnet.incubator.apache.org
> > > Subject: Re: [Announce] Upcoming Apache MXNet (incubating) 1.4.0
> release
> > >
> > > On Friday the contributors working on Java API discovered a potential
> > > performance problem with inference using Java API vs. Python.
> > Investigation
> > > is ongoing.
> > > As the Java API is one of the main features for the upcoming release, I
> > > suggest to post-pone the code freeze towards end of this week.
> > >
> > > Please provide feedback and concern about the change in dates for code
> > > freeze and 1.4.0 release. I will provide updates on progress resolving
> > the
> > > potential performance problem.
> > >
> > > Patrick - do you think it is possible to resolve the remaining issues
> on
> > MKL-
> > > DNN this week, so we can consider GA for MKL-DNN with 1.4.0?
> > >
> > > Regards,
> > > Steffen
> > >
> > > On Thu, Nov 15, 2018 at 5:26 AM Anton Chernov 
> > > wrote:
> > >
> > > > I'd like to remind everyone that 'code freeze' would mean cutting a
> > > > v1.4.x release branch and all following fixes would need to be
> > backported.
> > > > Development on master can be continued as usual.
> > > >
> > > > Best
> > > > Anton
> > > >
> > > > ср, 14 нояб. 2018 г. в 6:04, Steffen Rochel  >:
> > > >
> > > > > Dear MXNet community,
> > > > > the agreed plan was to establish code freeze for 1.4.0 release
> > > > > today. As the 1.3.1 patch release is still ongoing I suggest to
> > > > > post-pone the code freeze to Friday 16th November 2018.
> > > > >
> > > > > Sergey Kolychev has agreed to act as co-release manager for all
> > > > > tasks
> > > > which
> > > > > require committer privileges. If anybody is interested to volunteer
> > > > > as release manager - now is the time to speak up. Otherwise I will
> > > > > manage
> > > > the
> > > > > release.
> > > > >
> > > > > Regards,
> > > > > Steffen
> > > > >
> > > >
> >
>


Re: [Announce] Upcoming Apache MXNet (incubating) 1.4.0 release

2018-11-20 Thread Steffen Rochel
Thanks Patrick, lets target to get the PR's merged this week.

Call for contributions from the community: Right now we have 10 PR awaiting
merge

and
we have 61 open PR awaiting review.

I would appreciate if you all can help to review the open PR and the
committers can drive the merge before code freeze for 1.4.0.

The contributors on the Java API are making progress, but not all
performance issues are resolved. With some luck it should be possible to
code freeze towards end of this week.

Are there other critical features/bugs/PR you think need to be included in
1.4.0? If so, please communicate as soon as possible.

Regards,
Steffen

On Mon, Nov 19, 2018 at 8:26 PM Zhao, Patric  wrote:

> Thanks, Steffen. I think there is NO open issue to block the MKLDNN to GA
> now.
>
> BTW, several quantization related PRs (#13297,#13260) are under the review
> and I think it can be merged in this week.
>
> Thanks,
>
> --Patric
>
>
> > -Original Message-
> > From: Steffen Rochel [mailto:steffenroc...@gmail.com]
> > Sent: Tuesday, November 20, 2018 2:57 AM
> > To: dev@mxnet.incubator.apache.org
> > Subject: Re: [Announce] Upcoming Apache MXNet (incubating) 1.4.0 release
> >
> > On Friday the contributors working on Java API discovered a potential
> > performance problem with inference using Java API vs. Python.
> Investigation
> > is ongoing.
> > As the Java API is one of the main features for the upcoming release, I
> > suggest to post-pone the code freeze towards end of this week.
> >
> > Please provide feedback and concern about the change in dates for code
> > freeze and 1.4.0 release. I will provide updates on progress resolving
> the
> > potential performance problem.
> >
> > Patrick - do you think it is possible to resolve the remaining issues on
> MKL-
> > DNN this week, so we can consider GA for MKL-DNN with 1.4.0?
> >
> > Regards,
> > Steffen
> >
> > On Thu, Nov 15, 2018 at 5:26 AM Anton Chernov 
> > wrote:
> >
> > > I'd like to remind everyone that 'code freeze' would mean cutting a
> > > v1.4.x release branch and all following fixes would need to be
> backported.
> > > Development on master can be continued as usual.
> > >
> > > Best
> > > Anton
> > >
> > > ср, 14 нояб. 2018 г. в 6:04, Steffen Rochel :
> > >
> > > > Dear MXNet community,
> > > > the agreed plan was to establish code freeze for 1.4.0 release
> > > > today. As the 1.3.1 patch release is still ongoing I suggest to
> > > > post-pone the code freeze to Friday 16th November 2018.
> > > >
> > > > Sergey Kolychev has agreed to act as co-release manager for all
> > > > tasks
> > > which
> > > > require committer privileges. If anybody is interested to volunteer
> > > > as release manager - now is the time to speak up. Otherwise I will
> > > > manage
> > > the
> > > > release.
> > > >
> > > > Regards,
> > > > Steffen
> > > >
> > >
>


Re: [ANNOUNCEMENT] New Committer: Qing Lan

2018-11-20 Thread Steffen Rochel
Congratulation Qing!

On Tue, Nov 20, 2018 at 6:29 PM Hagay Lupesko  wrote:

> Congrats Qing! Awesome to see you become a committer!
>
> On Tue, Nov 20, 2018 at 4:26 PM Marco de Abreu
>  wrote:
>
> > Great to have your on board, Qing!
> >
> > Am Mi., 21. Nov. 2018, 01:24 hat Naveen Swamy 
> > geschrieben:
> >
> > > The Project Podling Management Committee (PPMC) for Apache MXNet has
> > > invited Qing Lan based on his contribution to MXNet Scala to become a
> > > committer and we are pleased to announce that he has accepted.
> > >
> > > Qing, thanks a lot for your contribution and continued effort to
> support
> > > MXNet community.
> > >
> > > Please join me in welcoming Qing to the project!
> > >
> > > Thanks, Naveen
> > > (on behalf of Apache MXNet PPMC)
> > >
> >
>


Re: [Announcement] New committer: Thomas Delteil

2018-11-20 Thread Steffen Rochel
Congratulation Thomas!

On Tue, Nov 20, 2018 at 2:54 PM Thomas DELTEIL 
wrote:

> Thanks Marco and PMC members, looking forward to the road ahead.
>
> All the best,
>
> Thomas Delteil
>
> Le mar. 20 nov. 2018 à 14:40, Marco de Abreu
>  a écrit :
>
> > The Project Management Committee (PMC) for Apache MXNet (incubating)
> > has invited Thomas Delteil to become a committer and we are pleased
> > to announce that he has accepted.
> >
> > Thomas, thanks a lot for your helpful contributions and we are looking
> > forward to
> > working with you on further projects!
> >
> > Best regards,
> > Marco de Abreu
> > - On behalf of the Apache MXNet (incubating) PMC
> >
>


Re: [ANNOUNCEMENT] New Committer: Qing Lan

2018-11-20 Thread Hagay Lupesko
Congrats Qing! Awesome to see you become a committer!

On Tue, Nov 20, 2018 at 4:26 PM Marco de Abreu
 wrote:

> Great to have your on board, Qing!
>
> Am Mi., 21. Nov. 2018, 01:24 hat Naveen Swamy 
> geschrieben:
>
> > The Project Podling Management Committee (PPMC) for Apache MXNet has
> > invited Qing Lan based on his contribution to MXNet Scala to become a
> > committer and we are pleased to announce that he has accepted.
> >
> > Qing, thanks a lot for your contribution and continued effort to support
> > MXNet community.
> >
> > Please join me in welcoming Qing to the project!
> >
> > Thanks, Naveen
> > (on behalf of Apache MXNet PPMC)
> >
>


Re: [Anouncement] New Committer: Kellen Sunderland

2018-11-20 Thread Steffen Rochel
Congratulation Kellen, well deserved!
Steffen

On Tue, Nov 20, 2018 at 4:02 PM Tianqi Chen  wrote:

> We are pleased to announce Kellen Sunderland as a new committer of Apache
> MXNet. Kellen has a  sustained effort to the project both in the discussion
> and code contributions.
>
> PRs
>
> https://github.com/apache/incubator-mxnet/pulls?utf8=%E2%9C%93=is%3Apr+author%3AKellenSunderland+
> reviews
>
> https://github.com/apache/incubator-mxnet/pulls?utf8=%E2%9C%93=is%3Apr+reviewed-by%3AKellenSunderland
> dev@
> https://lists.apache.org/list.html?d...@mxnet.apache.org:lte=3y:Kellen
> %20Sunderland
> 
>
> Please join me to welcome Kellen to the team!
>
> Tianqi
> - On behalf of Apache MXNet(incubating) PMC
>


Re: [ANNOUNCEMENT] New Committer: Qing Lan

2018-11-20 Thread Marco de Abreu
Great to have your on board, Qing!

Am Mi., 21. Nov. 2018, 01:24 hat Naveen Swamy 
geschrieben:

> The Project Podling Management Committee (PPMC) for Apache MXNet has
> invited Qing Lan based on his contribution to MXNet Scala to become a
> committer and we are pleased to announce that he has accepted.
>
> Qing, thanks a lot for your contribution and continued effort to support
> MXNet community.
>
> Please join me in welcoming Qing to the project!
>
> Thanks, Naveen
> (on behalf of Apache MXNet PPMC)
>


[ANNOUNCEMENT] New Committer: Qing Lan

2018-11-20 Thread Naveen Swamy
The Project Podling Management Committee (PPMC) for Apache MXNet has
invited Qing Lan based on his contribution to MXNet Scala to become a
committer and we are pleased to announce that he has accepted.

Qing, thanks a lot for your contribution and continued effort to support
MXNet community.

Please join me in welcoming Qing to the project!

Thanks, Naveen
(on behalf of Apache MXNet PPMC)


[Anouncement] New Committer: Kellen Sunderland

2018-11-20 Thread Tianqi Chen
We are pleased to announce Kellen Sunderland as a new committer of Apache
MXNet. Kellen has a  sustained effort to the project both in the discussion
and code contributions.

PRs
https://github.com/apache/incubator-mxnet/pulls?utf8=%E2%9C%93=is%3Apr+author%3AKellenSunderland+
reviews
https://github.com/apache/incubator-mxnet/pulls?utf8=%E2%9C%93=is%3Apr+reviewed-by%3AKellenSunderland
dev@  https://lists.apache.org/list.html?d...@mxnet.apache.org:lte=3y:Kellen
%20Sunderland

Please join me to welcome Kellen to the team!

Tianqi
- On behalf of Apache MXNet(incubating) PMC


soft relu gradient, is it correct?

2018-11-20 Thread Pedro Larroy
I bumped into the definition of the softrelu gradient:

https://github.com/apache/incubator-mxnet/blob/master/src/operator/mshadow_op.h#L170

Which is defined as  1- exp(-x)

As we define the forward of the softrelu as the softplus function,
shouldn't the gradient be the logistic function?

Is my understanding that the gradient of the softrelu should go down
to zero as Lim x -> -Inf  Which is not the case with the above
definition which goes to -Inf as Lim x- > -Inf

https://en.wikipedia.org/wiki/Rectifier_(neural_networks)


Pedro.


Re: [Announcement] New committer: Thomas Delteil

2018-11-20 Thread Thomas DELTEIL
Thanks Marco and PMC members, looking forward to the road ahead.

All the best,

Thomas Delteil

Le mar. 20 nov. 2018 à 14:40, Marco de Abreu
 a écrit :

> The Project Management Committee (PMC) for Apache MXNet (incubating)
> has invited Thomas Delteil to become a committer and we are pleased
> to announce that he has accepted.
>
> Thomas, thanks a lot for your helpful contributions and we are looking
> forward to
> working with you on further projects!
>
> Best regards,
> Marco de Abreu
> - On behalf of the Apache MXNet (incubating) PMC
>


[Announcement] New committer: Thomas Delteil

2018-11-20 Thread Marco de Abreu
The Project Management Committee (PMC) for Apache MXNet (incubating)
has invited Thomas Delteil to become a committer and we are pleased
to announce that he has accepted.

Thomas, thanks a lot for your helpful contributions and we are looking
forward to
working with you on further projects!

Best regards,
Marco de Abreu
- On behalf of the Apache MXNet (incubating) PMC


Re: Splitting Jenkins pipelines - stop changes to Jenkinsfiles!

2018-11-20 Thread Marco de Abreu
I have just submitted my PR at
https://github.com/apache/incubator-mxnet/pull/13344. Test jobs are
available at
http://jenkins.mxnet-ci-dev.amazon-ml.com/view/test-marco-mxnet/.

As soon as I'm done with my tests, I will mark it as ready for review.

Best regards,
Marco

On Tue, Nov 20, 2018 at 9:09 PM Marco de Abreu 
wrote:

> Thanks, Pedro!
>
> I have also been looking into that issue, but it seems like this would
> require changes in the groovy interpreter of Jenkins. From what I can tell,
> a refactor will give us multiple benefits (clarity and speed) aside from
> resolving this issue.
>
> Best regards,
> Marco
>
> Am Di., 20. Nov. 2018, 19:54 hat Pedro Larroy <
> pedro.larroy.li...@gmail.com> geschrieben:
>
>> I think this is a big problem, which has blocked us before. I want to
>> point out that you are doing a great thing by avoiding everyone
>> getting blocked by refactoring the pipelines.
>>
>> My concern is that we are kicking the can down the road and not
>> addressing the root cause of the problem with is known
>> https://issues.jenkins-ci.org/browse/JENKINS-37984
>>
>> Pedro.
>>
>>
>> On Tue, Nov 20, 2018 at 6:08 PM Marco de Abreu
>>  wrote:
>> >
>> > Hello Steffen,
>> >
>> > no, there won't be any impact on the PR process or nightly regressions.
>> > Only the reporting will have to be updated with the new job links, but
>> that
>> > should be a minor issue. To avoid any outage, I have been thinking about
>> > running both versions in parallel.
>> >
>> > Best regards,
>> > Marco
>> >
>> >
>> >
>> > On Tue, Nov 20, 2018 at 5:53 PM Steffen Rochel > >
>> > wrote:
>> >
>> > > Hi Marco - is there any impact on reporting, the PR process or nightly
>> > > regression beside reduction in TAT?  If yes, please elaborate.
>> > > Steffen
>> > >
>> > > On Tue, Nov 20, 2018 at 8:05 AM Marco de Abreu
>> > >  wrote:
>> > >
>> > > > Hello,
>> > > >
>> > > > we ran into issues around the maximum filesize of the Jenkinsfile a
>> few
>> > > > times already. In order to resolve this issue, I'd like to combine
>> this
>> > > > with some refactors I have planned for quite some time.
>> > > >
>> > > > The idea is basically to move away from one big Jenkinsfile and
>> instead
>> > > > split it into separate jobs that run in parallel and report their
>> status
>> > > > individually. Besides avoiding the size restriction, this will
>> greatly
>> > > > speed up the PR validation process by reducing the critical path.
>> Instead
>> > > > of having to wait for every single step within a stage to finish
>> before
>> > > the
>> > > > next stage (e.g. tests) is getting executed, these pipelines would
>> now be
>> > > > able to move forward individually. I'm still in the process of
>> > > refactoring
>> > > > and can't provide any numbers or documentation at this time, but I
>> would
>> > > > like to announce this early on to avoid conflicts:
>> > > >
>> > > > Since I will remove the original Jenkinsfile, this might cause
>> conflicts
>> > > > with ongoing efforts that try to change the Jenkinsfile. This poses
>> the
>> > > > risk that I might forget to port a change. Thus, I'd like to ask all
>> > > > contributors to wait with changes of Jenkinsfile and would like to
>> > > request
>> > > > fellow-committers to wait with merging any Jenkinsfile-related PRs
>> until
>> > > > further notice.
>> > > >
>> > > > I expect to finish this refactor until the end of the week. Please
>> don't
>> > > > hesitate to ask if you've got further questions.
>> > > >
>> > > > Please excuse any caused inconveniences.
>> > > >
>> > > > Best regards,
>> > > > Marco
>> > > >
>> > >
>>
>


Re: MXNet - Gluon - Audio

2018-11-20 Thread Sheng Zha
Hi Gaurav,

The performance concerns is not just around librosa, but also the way to 
integrate it. librosa as a python library requires holding GIL when calling it, 
which makes it hard for asynchronous data preprocessing during training. Also, 
the API design hasn't been verified on the more full-fledged use cases that you 
outlined. That, and based on the lack of expertise of audio processing 
reviewing the design doc, my suggestion is to continue the work as a Gluon 
example, until other use cases are adopted, which is what you started in 
https://github.com/apache/incubator-mxnet/pull/13325. Once you make more 
progress and become more familiar with Gluon design, please report back to this 
thread and I'd be happy to help more on the review.

-sz

On 2018/11/20 19:20:18, Gaurav Gireesh  wrote: 
> Hi All!
> Following up on this PR:
> https://github.com/apache/incubator-mxnet/pull/13241
> I would need some comments or feedback regarding the API design :
> https://cwiki.apache.org/confluence/display/MXNET/Gluon+-+Audio
> 
> The comments on the PR were mostly around *librosa *and its performance
> being a blocker if and when the designed API can be tested with bigger ASR
> models DeepSpeech 2, DeepSpeech 3.
> I would appreciate if the community provides their expertise/knowledge on
> loading audio data and feature extraction used currently with bigger ARS
> models.
> If there is anything in design which may be changed/improved that will
> improve the performance, I ll be happy to look into this.
> 
> Thanks and regards,
> Gaurav Gireesh
> 
> On Thu, Nov 15, 2018 at 10:47 AM Gaurav Gireesh 
> wrote:
> 
> > Hi Lai!
> > Thank you for your comments!
> > Below are the answers to your comments/queries:
> > 1) That's a good suggestion. However, I have added an example in the Pull
> > request related to this:
> > https://github.com/apache/incubator-mxnet/pull/13241/commits/eabb68256d8fd603a0075eafcd8947d92e7df27f
> > .
> > I would be happy to include a dataset similar to MNIST to support that. I
> > have come across an example dataset used in tensor flow speech
> > related example here
> > . This
> > could be included.
> >
> > 2) Thank you for the suggestion, I shall look into the FFT operator that
> > you have pointed out. However, there are other kind of features like, mfcc,
> > mels and so on which are popular in audio data feature extraction, which
> > will find utility if implemented. I am not sure if we have operators for
> > this.
> >
> > 3) The references look good too. I shall look into them. Thank you for
> > bringing them into my notice.
> >
> > Regards,
> > Gaurav
> >
> > On Tue, Nov 13, 2018 at 11:22 AM Lai Wei  wrote:
> >
> >> Hi Gaurav,
> >>
> >> Thanks for starting this. I see the PR is out
> >> , left some initial
> >> reviews, good work!
> >>
> >> In addition to Sandeep's queries, I have the following:
> >> 1. Can we include some simple classic audio dataset for users to directly
> >> import and try out? like MNIST in vision. (e.g.:
> >> http://pytorch.org/audio/datasets.html#yesno)
> >> 2. Librosa provides some good audio feature extractions, we can use it for
> >> now. But it's slow as you have to do conversions between ndarray and
> >> numpy.
> >> In the long term, can we make transforms to use mxnet operators and change
> >> your transforms to hybrid blocks? For example, mxnet FFT
> >> <
> >> https://mxnet.apache.org/api/python/ndarray/contrib.html?highlight=fft#mxnet.ndarray.contrib.fft
> >> >
> >> operator
> >> can be used in a hybrid block transformer, which will be a lot faster.
> >>
> >> Some additional references on users already using mxnet on audio, we
> >> should
> >> aim to make it easier and automate the file load/preprocess/transform
> >> process.
> >> 1. https://github.com/chen0040/mxnet-audio
> >> 2. https://github.com/shuokay/mxnet-wavenet
> >>
> >> Looking forward to seeing this feature out.
> >> Thanks!
> >>
> >> Best Regards
> >>
> >> Lai
> >>
> >>
> >> On Tue, Nov 13, 2018 at 9:09 AM sandeep krishnamurthy <
> >> sandeep.krishn...@gmail.com> wrote:
> >>
> >> > Thanks, Gaurav for starting this initiative. The design document is
> >> > detailed and gives all the information.
> >> > Starting to add this in "Contrib" is a good idea while we expect a few
> >> > rough edges and cleanups to follow.
> >> >
> >> > I had the following queries:
> >> > 1. Is there any analysis comparing LibROSA with other libraries? w.r.t
> >> > features, performance, community usage in audio data domain.
> >> > 2. What is the recommendation of LibROSA dependency? Part of MXNet PyPi
> >> or
> >> > ask the user to install if required? I prefer the latter, similar to
> >> > protobuf in ONNX-MXNet.
> >> > 3. I see LibROSA is a fully Python-based library. Are we getting
> >> blocked on
> >> > the dependency for future use cases when we want to make
> >> transformations as
> >> > 

Re: Splitting Jenkins pipelines - stop changes to Jenkinsfiles!

2018-11-20 Thread Marco de Abreu
Thanks, Pedro!

I have also been looking into that issue, but it seems like this would
require changes in the groovy interpreter of Jenkins. From what I can tell,
a refactor will give us multiple benefits (clarity and speed) aside from
resolving this issue.

Best regards,
Marco

Am Di., 20. Nov. 2018, 19:54 hat Pedro Larroy 
geschrieben:

> I think this is a big problem, which has blocked us before. I want to
> point out that you are doing a great thing by avoiding everyone
> getting blocked by refactoring the pipelines.
>
> My concern is that we are kicking the can down the road and not
> addressing the root cause of the problem with is known
> https://issues.jenkins-ci.org/browse/JENKINS-37984
>
> Pedro.
>
>
> On Tue, Nov 20, 2018 at 6:08 PM Marco de Abreu
>  wrote:
> >
> > Hello Steffen,
> >
> > no, there won't be any impact on the PR process or nightly regressions.
> > Only the reporting will have to be updated with the new job links, but
> that
> > should be a minor issue. To avoid any outage, I have been thinking about
> > running both versions in parallel.
> >
> > Best regards,
> > Marco
> >
> >
> >
> > On Tue, Nov 20, 2018 at 5:53 PM Steffen Rochel 
> > wrote:
> >
> > > Hi Marco - is there any impact on reporting, the PR process or nightly
> > > regression beside reduction in TAT?  If yes, please elaborate.
> > > Steffen
> > >
> > > On Tue, Nov 20, 2018 at 8:05 AM Marco de Abreu
> > >  wrote:
> > >
> > > > Hello,
> > > >
> > > > we ran into issues around the maximum filesize of the Jenkinsfile a
> few
> > > > times already. In order to resolve this issue, I'd like to combine
> this
> > > > with some refactors I have planned for quite some time.
> > > >
> > > > The idea is basically to move away from one big Jenkinsfile and
> instead
> > > > split it into separate jobs that run in parallel and report their
> status
> > > > individually. Besides avoiding the size restriction, this will
> greatly
> > > > speed up the PR validation process by reducing the critical path.
> Instead
> > > > of having to wait for every single step within a stage to finish
> before
> > > the
> > > > next stage (e.g. tests) is getting executed, these pipelines would
> now be
> > > > able to move forward individually. I'm still in the process of
> > > refactoring
> > > > and can't provide any numbers or documentation at this time, but I
> would
> > > > like to announce this early on to avoid conflicts:
> > > >
> > > > Since I will remove the original Jenkinsfile, this might cause
> conflicts
> > > > with ongoing efforts that try to change the Jenkinsfile. This poses
> the
> > > > risk that I might forget to port a change. Thus, I'd like to ask all
> > > > contributors to wait with changes of Jenkinsfile and would like to
> > > request
> > > > fellow-committers to wait with merging any Jenkinsfile-related PRs
> until
> > > > further notice.
> > > >
> > > > I expect to finish this refactor until the end of the week. Please
> don't
> > > > hesitate to ask if you've got further questions.
> > > >
> > > > Please excuse any caused inconveniences.
> > > >
> > > > Best regards,
> > > > Marco
> > > >
> > >
>


Re: [RESULTS] [VOTE] Release Apache MXNet (incubating) version 1.3.1.rc0

2018-11-20 Thread Hagay Lupesko
Great - congrats!

On Tue, Nov 20, 2018 at 8:51 AM Anton Chernov  wrote:

> Dear MXNet community,
>
> I'm happy to announce the results of the vote.
>
> This vote passes with 8 +1 votes (4 binding) and no 0 or -1 votes.
>
> +1 votes
>
> * Carin / binding
> * Indhu / binding
> * Sandeep / binding
> * Jim / binding
> * Kellen
> * Steffen
> * Roshani
> * Aaron
>
> 0 votes
> * No votes
>
> -1 votes
> * No votes
>
> Vote thread can be found here [1]. The list of members can be found here
> [2].
>
> I'll continue with the release process and the release announcement will
> follow in the next few days.
>
>
> Best
> Anton
>
> [1]
>
> https://lists.apache.org/thread.html/32ab13b6d2d80fd75dbc2ec62151d12d09f6e0ca89799ae0aa26894b@%3Cdev.mxnet.apache.org%3E
> [2] http://incubator.apache.org/projects/mxnet.html
>


Re: MXNet - Gluon - Audio

2018-11-20 Thread Gaurav Gireesh
Hi All!
Following up on this PR:
https://github.com/apache/incubator-mxnet/pull/13241
I would need some comments or feedback regarding the API design :
https://cwiki.apache.org/confluence/display/MXNET/Gluon+-+Audio

The comments on the PR were mostly around *librosa *and its performance
being a blocker if and when the designed API can be tested with bigger ASR
models DeepSpeech 2, DeepSpeech 3.
I would appreciate if the community provides their expertise/knowledge on
loading audio data and feature extraction used currently with bigger ARS
models.
If there is anything in design which may be changed/improved that will
improve the performance, I ll be happy to look into this.

Thanks and regards,
Gaurav Gireesh

On Thu, Nov 15, 2018 at 10:47 AM Gaurav Gireesh 
wrote:

> Hi Lai!
> Thank you for your comments!
> Below are the answers to your comments/queries:
> 1) That's a good suggestion. However, I have added an example in the Pull
> request related to this:
> https://github.com/apache/incubator-mxnet/pull/13241/commits/eabb68256d8fd603a0075eafcd8947d92e7df27f
> .
> I would be happy to include a dataset similar to MNIST to support that. I
> have come across an example dataset used in tensor flow speech
> related example here
> . This
> could be included.
>
> 2) Thank you for the suggestion, I shall look into the FFT operator that
> you have pointed out. However, there are other kind of features like, mfcc,
> mels and so on which are popular in audio data feature extraction, which
> will find utility if implemented. I am not sure if we have operators for
> this.
>
> 3) The references look good too. I shall look into them. Thank you for
> bringing them into my notice.
>
> Regards,
> Gaurav
>
> On Tue, Nov 13, 2018 at 11:22 AM Lai Wei  wrote:
>
>> Hi Gaurav,
>>
>> Thanks for starting this. I see the PR is out
>> , left some initial
>> reviews, good work!
>>
>> In addition to Sandeep's queries, I have the following:
>> 1. Can we include some simple classic audio dataset for users to directly
>> import and try out? like MNIST in vision. (e.g.:
>> http://pytorch.org/audio/datasets.html#yesno)
>> 2. Librosa provides some good audio feature extractions, we can use it for
>> now. But it's slow as you have to do conversions between ndarray and
>> numpy.
>> In the long term, can we make transforms to use mxnet operators and change
>> your transforms to hybrid blocks? For example, mxnet FFT
>> <
>> https://mxnet.apache.org/api/python/ndarray/contrib.html?highlight=fft#mxnet.ndarray.contrib.fft
>> >
>> operator
>> can be used in a hybrid block transformer, which will be a lot faster.
>>
>> Some additional references on users already using mxnet on audio, we
>> should
>> aim to make it easier and automate the file load/preprocess/transform
>> process.
>> 1. https://github.com/chen0040/mxnet-audio
>> 2. https://github.com/shuokay/mxnet-wavenet
>>
>> Looking forward to seeing this feature out.
>> Thanks!
>>
>> Best Regards
>>
>> Lai
>>
>>
>> On Tue, Nov 13, 2018 at 9:09 AM sandeep krishnamurthy <
>> sandeep.krishn...@gmail.com> wrote:
>>
>> > Thanks, Gaurav for starting this initiative. The design document is
>> > detailed and gives all the information.
>> > Starting to add this in "Contrib" is a good idea while we expect a few
>> > rough edges and cleanups to follow.
>> >
>> > I had the following queries:
>> > 1. Is there any analysis comparing LibROSA with other libraries? w.r.t
>> > features, performance, community usage in audio data domain.
>> > 2. What is the recommendation of LibROSA dependency? Part of MXNet PyPi
>> or
>> > ask the user to install if required? I prefer the latter, similar to
>> > protobuf in ONNX-MXNet.
>> > 3. I see LibROSA is a fully Python-based library. Are we getting
>> blocked on
>> > the dependency for future use cases when we want to make
>> transformations as
>> > operators and allow for cross-language support?
>> > 4. In performance design considerations, with lazy=True / False the
>> > performance difference is too scary ( 8 minutes to 4 hours!!) This
>> requires
>> > some more analysis. If we known turning a flag off/on has 24X
>> performance
>> > degradation, should we need to provide that control to user? What is the
>> > impact of this on Memory usage?
>> > 5. I see LibROSA has ISC license (
>> > https://github.com/librosa/librosa/blob/master/LICENSE.md) which says
>> free
>> > to use with same license notification. I am not sure if this is ok. I
>> > request other committers/mentors to suggest.
>> >
>> > Best,
>> > Sandeep
>> >
>> > On Fri, Nov 9, 2018 at 5:45 PM Gaurav Gireesh > >
>> > wrote:
>> >
>> > > Dear MXNet Community,
>> > >
>> > > I recently started looking into performing some simple sound
>> multi-class
>> > > classification tasks with Audio Data and realized that as a user, I
>> would
>> > > like MXNet to have an out of the box feature which allows us 

Re: Splitting Jenkins pipelines - stop changes to Jenkinsfiles!

2018-11-20 Thread Pedro Larroy
I think this is a big problem, which has blocked us before. I want to
point out that you are doing a great thing by avoiding everyone
getting blocked by refactoring the pipelines.

My concern is that we are kicking the can down the road and not
addressing the root cause of the problem with is known
https://issues.jenkins-ci.org/browse/JENKINS-37984

Pedro.


On Tue, Nov 20, 2018 at 6:08 PM Marco de Abreu
 wrote:
>
> Hello Steffen,
>
> no, there won't be any impact on the PR process or nightly regressions.
> Only the reporting will have to be updated with the new job links, but that
> should be a minor issue. To avoid any outage, I have been thinking about
> running both versions in parallel.
>
> Best regards,
> Marco
>
>
>
> On Tue, Nov 20, 2018 at 5:53 PM Steffen Rochel 
> wrote:
>
> > Hi Marco - is there any impact on reporting, the PR process or nightly
> > regression beside reduction in TAT?  If yes, please elaborate.
> > Steffen
> >
> > On Tue, Nov 20, 2018 at 8:05 AM Marco de Abreu
> >  wrote:
> >
> > > Hello,
> > >
> > > we ran into issues around the maximum filesize of the Jenkinsfile a few
> > > times already. In order to resolve this issue, I'd like to combine this
> > > with some refactors I have planned for quite some time.
> > >
> > > The idea is basically to move away from one big Jenkinsfile and instead
> > > split it into separate jobs that run in parallel and report their status
> > > individually. Besides avoiding the size restriction, this will greatly
> > > speed up the PR validation process by reducing the critical path. Instead
> > > of having to wait for every single step within a stage to finish before
> > the
> > > next stage (e.g. tests) is getting executed, these pipelines would now be
> > > able to move forward individually. I'm still in the process of
> > refactoring
> > > and can't provide any numbers or documentation at this time, but I would
> > > like to announce this early on to avoid conflicts:
> > >
> > > Since I will remove the original Jenkinsfile, this might cause conflicts
> > > with ongoing efforts that try to change the Jenkinsfile. This poses the
> > > risk that I might forget to port a change. Thus, I'd like to ask all
> > > contributors to wait with changes of Jenkinsfile and would like to
> > request
> > > fellow-committers to wait with merging any Jenkinsfile-related PRs until
> > > further notice.
> > >
> > > I expect to finish this refactor until the end of the week. Please don't
> > > hesitate to ask if you've got further questions.
> > >
> > > Please excuse any caused inconveniences.
> > >
> > > Best regards,
> > > Marco
> > >
> >


Re: Splitting Jenkins pipelines - stop changes to Jenkinsfiles!

2018-11-20 Thread Marco de Abreu
Hello Steffen,

no, there won't be any impact on the PR process or nightly regressions.
Only the reporting will have to be updated with the new job links, but that
should be a minor issue. To avoid any outage, I have been thinking about
running both versions in parallel.

Best regards,
Marco



On Tue, Nov 20, 2018 at 5:53 PM Steffen Rochel 
wrote:

> Hi Marco - is there any impact on reporting, the PR process or nightly
> regression beside reduction in TAT?  If yes, please elaborate.
> Steffen
>
> On Tue, Nov 20, 2018 at 8:05 AM Marco de Abreu
>  wrote:
>
> > Hello,
> >
> > we ran into issues around the maximum filesize of the Jenkinsfile a few
> > times already. In order to resolve this issue, I'd like to combine this
> > with some refactors I have planned for quite some time.
> >
> > The idea is basically to move away from one big Jenkinsfile and instead
> > split it into separate jobs that run in parallel and report their status
> > individually. Besides avoiding the size restriction, this will greatly
> > speed up the PR validation process by reducing the critical path. Instead
> > of having to wait for every single step within a stage to finish before
> the
> > next stage (e.g. tests) is getting executed, these pipelines would now be
> > able to move forward individually. I'm still in the process of
> refactoring
> > and can't provide any numbers or documentation at this time, but I would
> > like to announce this early on to avoid conflicts:
> >
> > Since I will remove the original Jenkinsfile, this might cause conflicts
> > with ongoing efforts that try to change the Jenkinsfile. This poses the
> > risk that I might forget to port a change. Thus, I'd like to ask all
> > contributors to wait with changes of Jenkinsfile and would like to
> request
> > fellow-committers to wait with merging any Jenkinsfile-related PRs until
> > further notice.
> >
> > I expect to finish this refactor until the end of the week. Please don't
> > hesitate to ask if you've got further questions.
> >
> > Please excuse any caused inconveniences.
> >
> > Best regards,
> > Marco
> >
>


Re: Splitting Jenkins pipelines - stop changes to Jenkinsfiles!

2018-11-20 Thread Steffen Rochel
Hi Marco - is there any impact on reporting, the PR process or nightly
regression beside reduction in TAT?  If yes, please elaborate.
Steffen

On Tue, Nov 20, 2018 at 8:05 AM Marco de Abreu
 wrote:

> Hello,
>
> we ran into issues around the maximum filesize of the Jenkinsfile a few
> times already. In order to resolve this issue, I'd like to combine this
> with some refactors I have planned for quite some time.
>
> The idea is basically to move away from one big Jenkinsfile and instead
> split it into separate jobs that run in parallel and report their status
> individually. Besides avoiding the size restriction, this will greatly
> speed up the PR validation process by reducing the critical path. Instead
> of having to wait for every single step within a stage to finish before the
> next stage (e.g. tests) is getting executed, these pipelines would now be
> able to move forward individually. I'm still in the process of refactoring
> and can't provide any numbers or documentation at this time, but I would
> like to announce this early on to avoid conflicts:
>
> Since I will remove the original Jenkinsfile, this might cause conflicts
> with ongoing efforts that try to change the Jenkinsfile. This poses the
> risk that I might forget to port a change. Thus, I'd like to ask all
> contributors to wait with changes of Jenkinsfile and would like to request
> fellow-committers to wait with merging any Jenkinsfile-related PRs until
> further notice.
>
> I expect to finish this refactor until the end of the week. Please don't
> hesitate to ask if you've got further questions.
>
> Please excuse any caused inconveniences.
>
> Best regards,
> Marco
>


[RESULTS] [VOTE] Release Apache MXNet (incubating) version 1.3.1.rc0

2018-11-20 Thread Anton Chernov
Dear MXNet community,

I'm happy to announce the results of the vote.

This vote passes with 8 +1 votes (4 binding) and no 0 or -1 votes.

+1 votes

* Carin / binding
* Indhu / binding
* Sandeep / binding
* Jim / binding
* Kellen
* Steffen
* Roshani
* Aaron

0 votes
* No votes

-1 votes
* No votes

Vote thread can be found here [1]. The list of members can be found here
[2].

I'll continue with the release process and the release announcement will
follow in the next few days.


Best
Anton

[1]
https://lists.apache.org/thread.html/32ab13b6d2d80fd75dbc2ec62151d12d09f6e0ca89799ae0aa26894b@%3Cdev.mxnet.apache.org%3E
[2] http://incubator.apache.org/projects/mxnet.html


Re: Should PR-860 (Use modernized range loops where possible) be reverted?

2018-11-20 Thread Carin Meier
Great. Thanks for the clarifications everyone. I think we are good then :)

- Carin

On Tue, Nov 20, 2018 at 10:43 AM kellen sunderland <
kellen.sunderl...@gmail.com> wrote:

> Hey Carin, I don't think there's any issues merging this PR.  The veto'd
> aspect was around _requiring_ modern loop usage, and failing the build if
> clang tidy detected modern loops could be used but weren't.  The original
> PR included a check for this and would fail any builds not using modern
> loops.  Several people didn't like this aspect so I updated the PR and
> removed that overly-strict check.  The current PR doesn't have anything it
> in that's been vetod.  We're continuing to warn if clang-tidy detects a
> loop could be modernized, but are not causing an error (which was actually
> the behaviour before this PR was merged).
>
> On Tue, Nov 20, 2018 at 7:29 AM Anton Chernov  wrote:
>
> > Hi Carin,
> >
> > The discussion [1] was about whether to enable automatic checks on using
> > old behaviour in new PR's. Kellens PR [2] was about modernizing the
> actual
> > code itself and was not up for voting, thus could not receive any
> technical
> > veto votes.
> >
> > Per the discussion (as I have understood it), we won't get veto votes if
> we
> > would enable the check on CI, if it would be treated as a warning.
> >
> > Thank you for merging the PR in the first place. I see no reason for
> > reverting it.
> >
> > Best
> > Anton
> >
> > [1]
> >
> >
> https://lists.apache.org/thread.html/b47f285a80bef47c5ead6c361614e338a0661f6c0c76196c1e3719c5@%3Cdev.mxnet.apache.org%3E
> > [2] https://github.com/apache/incubator-mxnet/pull/12356
> >
> >
> > вт, 20 нояб. 2018 г. в 15:24, Pedro Larroy  >:
> >
> > > Hi all
> > >
> > > I think we have to make the clear separation between the thread votes
> > > on "uniformly adopting C++11 range loops in the MXNet project" and a
> > > PR which refactored code to be more legible and with improved variable
> > > names.
> > > Merging that PR doesn't imply that we have to uniformly adopt the
> > > previous proposal.  The PR was reviewed and approved by several
> > > people. I would keep the two topics separate, merging this PR doesn't
> > > prescribe any particular idiom for future commits or reviews.
> > >
> > > Pedro.
> > >
> > > On Tue, Nov 20, 2018 at 2:58 PM Carin Meier 
> > wrote:
> > > >
> > > > My intent was to be helpful, but I think I may have merged this PR
> > > > yesterday too soon thinking it was approved and ready to merge
> > > > https://github.com/apache/incubator-mxnet/pull/12356
> > > >
> > > > I didn't see the connected dev discussion
> > > >
> > >
> >
> https://lists.apache.org/thread.html/b47f285a80bef47c5ead6c361614e338a0661f6c0c76196c1e3719c5@%3Cdev.mxnet.apache.org%3E
> > > > where there were -1 votes, which I believe are vetos?
> > > >
> > > > So the question is confirm: should PR should be reverted?
> > > >
> > > > Sorry for any confusion,
> > > > Carin
> > >
> >
>


Re: [VOTE] Release Apache MXNet (incubating) version 1.3.1.rc0

2018-11-20 Thread Anton Chernov
Thank you everyone, the vote is closed. I will send the results in a
separate announcement.

Best
Anton

пн, 19 нояб. 2018 г. в 15:44, Jim Jagielski :

> +1 from me (macOS)
>
> > On Nov 16, 2018, at 2:52 AM, kellen sunderland <
> kellen.sunderl...@gmail.com> wrote:
> >
> > Thanks for organizing the release Anton and for testing Carin and
> Steffen.
> > Lots of great fixes in this release.  As we don't have the required 3
> > committers I'd suggest extending the vote for a few days.
> >
> > I tested the following on MacOS 10.13, High Sierra:
> >
> > INCUBATING IN RELEASE FILE: check.
> > LICENSE check.
> > NOTICE check.
> > SIGNATURE check.
> > HASH check.
> > DISCLAIMER check.
> > SOURCE COMPILES VIA MAKEFILE check.
> > SOURCE COMPILES VIA CMAKE check.
> > C++ TESTS PASS fail
> > Two tests failing for me.
> > Build with flags: cmake -DUSE_CUDA=0 -DUSE_CUDNN=0 -DUSE_OPENMP=0
> > -DUSE_OPENCV=0 ..
> > Ran c++ tests with exclusions: ./tests/mxnet_unit_tests
> > --gtest_filter=-GpuTopology.*
> > Result:
> > [  FAILED  ] 2 tests, listed below:
> > [  FAILED  ] ACTIVATION_PERF.ExecuteBidirectional
> > [  FAILED  ] ACTIVATION_PERF.TimingCPU
> >
> > PYHTON UNIT TESTS PASS check.
> >
> > Not sure if the test failures are a regression so I'm +0 (non-binding)
> >
> > On Thu, Nov 15, 2018 at 5:43 PM Steffen Rochel 
> > wrote:
> >
> >> +1 build on MacOS Sierra following instructions on
> >>
> >>
> https://cwiki.apache.org/confluence/display/MXNET/MXNet+Developer+Setup+on+Mac
> >> and run one training test.
> >>
> >> On Tue, Nov 13, 2018 at 2:34 PM Carin Meier 
> wrote:
> >>
> >>> +1 - Clojure package tested fine with Scala jars
> >>>
> >>> On Mon, Nov 12, 2018 at 6:53 PM Anton Chernov 
> >> wrote:
> >>>
>  Dear MXNet community,
> 
>  This is the vote to release Apache MXNet (incubating) version 1.3.1.
> >>> Voting
>  will start now, on Monday the 12th of November 2018 and close on 14:00
>  Thursday the 15th of November 2018, Pacific Time (PT).
> 
>  Link to release notes:
>  https://cwiki.apache.org/confluence/x/eZGzBQ
> 
>  Link to release candidate 1.3.1.rc0:
>  https://github.com/apache/incubator-mxnet/releases/tag/1.3.1.rc0
> 
>  Link to source and signatures on apache dist server:
>  https://dist.apache.org/repos/dist/dev/incubator/mxnet/1.3.1.rc0/
> 
>  Link to scala packages on the staging repo:
> 
>  * CPU
> 
> 
> >>>
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/mxnet/mxnet-full_2.11-osx-x86_64-cpu/1.3.1-SNAPSHOT/
> 
>  * GPU
> 
> 
> >>>
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/mxnet/mxnet-full_2.11-linux-x86_64-gpu/1.3.1-SNAPSHOT/
> 
>  Please remember to TEST first before voting accordingly:
>  +1 = approve
>  +0 = no opinion
>  -1 = disapprove (provide reason)
> 
> 
>  Best regards,
>  Anton
> 
> >>>
> >>
>
>


Splitting Jenkins pipelines - stop changes to Jenkinsfiles!

2018-11-20 Thread Marco de Abreu
Hello,

we ran into issues around the maximum filesize of the Jenkinsfile a few
times already. In order to resolve this issue, I'd like to combine this
with some refactors I have planned for quite some time.

The idea is basically to move away from one big Jenkinsfile and instead
split it into separate jobs that run in parallel and report their status
individually. Besides avoiding the size restriction, this will greatly
speed up the PR validation process by reducing the critical path. Instead
of having to wait for every single step within a stage to finish before the
next stage (e.g. tests) is getting executed, these pipelines would now be
able to move forward individually. I'm still in the process of refactoring
and can't provide any numbers or documentation at this time, but I would
like to announce this early on to avoid conflicts:

Since I will remove the original Jenkinsfile, this might cause conflicts
with ongoing efforts that try to change the Jenkinsfile. This poses the
risk that I might forget to port a change. Thus, I'd like to ask all
contributors to wait with changes of Jenkinsfile and would like to request
fellow-committers to wait with merging any Jenkinsfile-related PRs until
further notice.

I expect to finish this refactor until the end of the week. Please don't
hesitate to ask if you've got further questions.

Please excuse any caused inconveniences.

Best regards,
Marco


Re: Should PR-860 (Use modernized range loops where possible) be reverted?

2018-11-20 Thread kellen sunderland
Hey Carin, I don't think there's any issues merging this PR.  The veto'd
aspect was around _requiring_ modern loop usage, and failing the build if
clang tidy detected modern loops could be used but weren't.  The original
PR included a check for this and would fail any builds not using modern
loops.  Several people didn't like this aspect so I updated the PR and
removed that overly-strict check.  The current PR doesn't have anything it
in that's been vetod.  We're continuing to warn if clang-tidy detects a
loop could be modernized, but are not causing an error (which was actually
the behaviour before this PR was merged).

On Tue, Nov 20, 2018 at 7:29 AM Anton Chernov  wrote:

> Hi Carin,
>
> The discussion [1] was about whether to enable automatic checks on using
> old behaviour in new PR's. Kellens PR [2] was about modernizing the actual
> code itself and was not up for voting, thus could not receive any technical
> veto votes.
>
> Per the discussion (as I have understood it), we won't get veto votes if we
> would enable the check on CI, if it would be treated as a warning.
>
> Thank you for merging the PR in the first place. I see no reason for
> reverting it.
>
> Best
> Anton
>
> [1]
>
> https://lists.apache.org/thread.html/b47f285a80bef47c5ead6c361614e338a0661f6c0c76196c1e3719c5@%3Cdev.mxnet.apache.org%3E
> [2] https://github.com/apache/incubator-mxnet/pull/12356
>
>
> вт, 20 нояб. 2018 г. в 15:24, Pedro Larroy :
>
> > Hi all
> >
> > I think we have to make the clear separation between the thread votes
> > on "uniformly adopting C++11 range loops in the MXNet project" and a
> > PR which refactored code to be more legible and with improved variable
> > names.
> > Merging that PR doesn't imply that we have to uniformly adopt the
> > previous proposal.  The PR was reviewed and approved by several
> > people. I would keep the two topics separate, merging this PR doesn't
> > prescribe any particular idiom for future commits or reviews.
> >
> > Pedro.
> >
> > On Tue, Nov 20, 2018 at 2:58 PM Carin Meier 
> wrote:
> > >
> > > My intent was to be helpful, but I think I may have merged this PR
> > > yesterday too soon thinking it was approved and ready to merge
> > > https://github.com/apache/incubator-mxnet/pull/12356
> > >
> > > I didn't see the connected dev discussion
> > >
> >
> https://lists.apache.org/thread.html/b47f285a80bef47c5ead6c361614e338a0661f6c0c76196c1e3719c5@%3Cdev.mxnet.apache.org%3E
> > > where there were -1 votes, which I believe are vetos?
> > >
> > > So the question is confirm: should PR should be reverted?
> > >
> > > Sorry for any confusion,
> > > Carin
> >
>


Re: Should PR-860 (Use modernized range loops where possible) be reverted?

2018-11-20 Thread Anton Chernov
Hi Carin,

The discussion [1] was about whether to enable automatic checks on using
old behaviour in new PR's. Kellens PR [2] was about modernizing the actual
code itself and was not up for voting, thus could not receive any technical
veto votes.

Per the discussion (as I have understood it), we won't get veto votes if we
would enable the check on CI, if it would be treated as a warning.

Thank you for merging the PR in the first place. I see no reason for
reverting it.

Best
Anton

[1]
https://lists.apache.org/thread.html/b47f285a80bef47c5ead6c361614e338a0661f6c0c76196c1e3719c5@%3Cdev.mxnet.apache.org%3E
[2] https://github.com/apache/incubator-mxnet/pull/12356


вт, 20 нояб. 2018 г. в 15:24, Pedro Larroy :

> Hi all
>
> I think we have to make the clear separation between the thread votes
> on "uniformly adopting C++11 range loops in the MXNet project" and a
> PR which refactored code to be more legible and with improved variable
> names.
> Merging that PR doesn't imply that we have to uniformly adopt the
> previous proposal.  The PR was reviewed and approved by several
> people. I would keep the two topics separate, merging this PR doesn't
> prescribe any particular idiom for future commits or reviews.
>
> Pedro.
>
> On Tue, Nov 20, 2018 at 2:58 PM Carin Meier  wrote:
> >
> > My intent was to be helpful, but I think I may have merged this PR
> > yesterday too soon thinking it was approved and ready to merge
> > https://github.com/apache/incubator-mxnet/pull/12356
> >
> > I didn't see the connected dev discussion
> >
> https://lists.apache.org/thread.html/b47f285a80bef47c5ead6c361614e338a0661f6c0c76196c1e3719c5@%3Cdev.mxnet.apache.org%3E
> > where there were -1 votes, which I believe are vetos?
> >
> > So the question is confirm: should PR should be reverted?
> >
> > Sorry for any confusion,
> > Carin
>


Re: Should PR-860 (Use modernized range loops where possible) be reverted?

2018-11-20 Thread Pedro Larroy
Hi all

I think we have to make the clear separation between the thread votes
on "uniformly adopting C++11 range loops in the MXNet project" and a
PR which refactored code to be more legible and with improved variable
names.
Merging that PR doesn't imply that we have to uniformly adopt the
previous proposal.  The PR was reviewed and approved by several
people. I would keep the two topics separate, merging this PR doesn't
prescribe any particular idiom for future commits or reviews.

Pedro.

On Tue, Nov 20, 2018 at 2:58 PM Carin Meier  wrote:
>
> My intent was to be helpful, but I think I may have merged this PR
> yesterday too soon thinking it was approved and ready to merge
> https://github.com/apache/incubator-mxnet/pull/12356
>
> I didn't see the connected dev discussion
> https://lists.apache.org/thread.html/b47f285a80bef47c5ead6c361614e338a0661f6c0c76196c1e3719c5@%3Cdev.mxnet.apache.org%3E
> where there were -1 votes, which I believe are vetos?
>
> So the question is confirm: should PR should be reverted?
>
> Sorry for any confusion,
> Carin


Should PR-860 (Use modernized range loops where possible) be reverted?

2018-11-20 Thread Carin Meier
My intent was to be helpful, but I think I may have merged this PR
yesterday too soon thinking it was approved and ready to merge
https://github.com/apache/incubator-mxnet/pull/12356

I didn't see the connected dev discussion
https://lists.apache.org/thread.html/b47f285a80bef47c5ead6c361614e338a0661f6c0c76196c1e3719c5@%3Cdev.mxnet.apache.org%3E
where there were -1 votes, which I believe are vetos?

So the question is confirm: should PR should be reverted?

Sorry for any confusion,
Carin