[GitHub] nihui commented on issue #9960: add label augmentation support in image recordio2

[GitBox]

nihui commented on issue #9960: add label augmentation support in image 
recordio2
URL: https://github.com/apache/incubator-mxnet/pull/9960#issuecomment-370686696
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nihui commented on issue #9960: add label augmentation support in image recordio2

[GitBox]

nihui commented on issue #9960: add label augmentation support in image 
recordio2
URL: https://github.com/apache/incubator-mxnet/pull/9960#issuecomment-370686696
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] GoodJoey commented on issue #951: run two machine with two gpu is slower than one machine with one gpu

[GitBox]

GoodJoey commented on issue #951: run two machine with two gpu is slower than 
one machine with one gpu
URL: https://github.com/apache/incubator-mxnet/issues/951#issuecomment-370685150
 
 
   @freepjf  
   i met the same issue. what's the bandwidth of your IB net? 56G or 100G?
   Adding more parameter server may help, but i think bandwidth is the main 
bottleneck here


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] wkcn commented on issue #9939: add multi proposal operator (cpu version) and fix the bug in proposal op (gpu version)

[GitBox]

wkcn commented on issue #9939: add multi proposal operator (cpu version) and 
fix the bug in proposal op (gpu version)
URL: https://github.com/apache/incubator-mxnet/pull/9939#issuecomment-370681494
 
 
   @marcoabreu 
   Hello! Could you please retrigger the test?
   It seems that test_operator_gpu.test_deconv has some problem.
   And I have removed  'USE_STABLE_SORT_FOR_PROPOSAL'  building flag.
   
   Thank you!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sxjscience commented on a change in pull request #10000: fix average pooling kernel size assignment error

[GitBox]

sxjscience commented on a change in pull request #1: fix average pooling 
kernel size assignment error
URL: https://github.com/apache/incubator-mxnet/pull/1#discussion_r172421833
 
 

 ##
 File path: src/operator/nn/pooling-inl.h
 ##
 @@ -178,6 +178,11 @@ class PoolingOp {
 template
 PoolingOp (const PoolingParam ) {
   static thread_local PoolingOp op;
+  // check if filter size assigned correctly
+  if (param.global_pool == false) {
+CHECK_GT(param.kernel.ndim(), 0U)
+<< "A positive number must be assigned as filter size";
 
 Review comment:
   I think we can use "You need to set the kernel size if global pooling is not 
used".


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sxjscience commented on issue #10000: fix average pooling kernel size assignment error

[GitBox]

sxjscience commented on issue #1: fix average pooling kernel size 
assignment error
URL: https://github.com/apache/incubator-mxnet/pull/1#issuecomment-370679259
 
 
   The error code show that the change somehow breaks the cpp package. I need 
to investigate the status of the CPP package.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sergeykolychev commented on a change in pull request #9988: [Perl] Sparse feature.

[GitBox]

sergeykolychev commented on a change in pull request #9988: [Perl] Sparse 
feature.
URL: https://github.com/apache/incubator-mxnet/pull/9988#discussion_r172412780
 
 

 ##
 File path: perl-package/AI-MXNetCAPI/mxnet_typemaps.i
 ##
 @@ -112,6 +112,32 @@
 }
 }
 
+%typemap(in,numinputs=0) (const int **out_stypes) (int* temp)
+{
+temp = NULL;
+$1 = 
+}
+
+%typemap(argout) (const int **out_stypes)
+{
+if(av_len((AV*)SvRV(ST(3))) == -1 && !result)
+{
+AV *myav;
+SV **svs;
+int i = 0;
+svs = (SV **)safemalloc(*arg4*sizeof(SV *));
+for (i = 0; i < *arg4 ; i++) {
+svs[i] = newSViv((*$1)[i]);
+sv_2mortal(svs[i]);
+}
+myav = av_make(*arg4, svs);
 
 Review comment:
   I went with this solution because I needed to solve a peculiar problem of 
making a decision dependent on a particular input state.
   You helped me in the past to clean up the swig usage, could you please see 
if you can come up with a solution later in separate pull that is more stable 
(I used this technique in another place as well in the past).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sergeykolychev commented on a change in pull request #9988: [Perl] Sparse feature.

[GitBox]

sergeykolychev commented on a change in pull request #9988: [Perl] Sparse 
feature.
URL: https://github.com/apache/incubator-mxnet/pull/9988#discussion_r172412780
 
 

 ##
 File path: perl-package/AI-MXNetCAPI/mxnet_typemaps.i
 ##
 @@ -112,6 +112,32 @@
 }
 }
 
+%typemap(in,numinputs=0) (const int **out_stypes) (int* temp)
+{
+temp = NULL;
+$1 = 
+}
+
+%typemap(argout) (const int **out_stypes)
+{
+if(av_len((AV*)SvRV(ST(3))) == -1 && !result)
+{
+AV *myav;
+SV **svs;
+int i = 0;
+svs = (SV **)safemalloc(*arg4*sizeof(SV *));
+for (i = 0; i < *arg4 ; i++) {
+svs[i] = newSViv((*$1)[i]);
+sv_2mortal(svs[i]);
+}
+myav = av_make(*arg4, svs);
 
 Review comment:
   I went with this solution because I needed to solve a peculiar problem of 
making a decision dependent on a particular input state.
   You helped me in the past to clean up the swig usage, could you please see 
if you can come up with a solution that is more stable (I used this technique 
in another place as well in the past).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sergeykolychev commented on a change in pull request #9988: [Perl] Sparse feature.

[GitBox]

sergeykolychev commented on a change in pull request #9988: [Perl] Sparse 
feature.
URL: https://github.com/apache/incubator-mxnet/pull/9988#discussion_r172412268
 
 

 ##
 File path: perl-package/AI-MXNet/lib/AI/MXNet/Base.pm
 ##
 @@ -365,4 +452,20 @@ $PDL::toolongtoprint = 1000_000_000;
 *PDL::dtype  = sub { DTYPE_MX_TO_STR->{ DTYPE_PDL_TO_MX->{ 
shift->type->numval } } };
 }
 
+sub digitize
+{
+my ($d, $bins) = @_;
+for(my $i = 0; $i < @$bins; $i++)
+{
+return $i if $d < $bins->[$i];
+}
+return scalar(@$bins);
+}
+
+use B;
+sub hash { hex(B::hash(shift)) }
 
 Review comment:
   yeah I know, it's the same with Python's hash() call, so it's a feature.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sergeykolychev commented on a change in pull request #9988: [Perl] Sparse feature.

[GitBox]

sergeykolychev commented on a change in pull request #9988: [Perl] Sparse 
feature.
URL: https://github.com/apache/incubator-mxnet/pull/9988#discussion_r172412182
 
 

 ##
 File path: perl-package/AI-MXNet/lib/AI/MXNet/NDArray/Sparse.pm
 ##
 @@ -0,0 +1,1342 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+package AI::MXNet::NDArray::Sparse;
+use strict;
+use warnings;
+use AI::MXNet::Base;
+use AI::MXNet::Function::Parameters;
+use Mouse;
+extends 'AI::MXNet::NDArray';
+
+=head1 NAME
+
+AI::MXNet::NDArray::Sparse - Sparse NDArray API of MXNet
+=cut
+
+=head1 DESCRIPTION
+
+The base class of an NDArray stored in a sparse storage format.
+See AI::MXNet::NDArray::CSR and AI::MXNet::NDArray::RowSparse for more 
details.
+=cut
+
+method _new_alloc_handle(
+Stype$stype,
+Shape$shape,
+AI::MXNet::Context   $ctx,
+Bool $delay_alloc,
+Dtype$dtype,
+AuxTypes $aux_types,
+Maybe[ArrayRef[Shape]]   $aux_shapes=
+)
+{
+confess("only int64 is supported for aux types")
+if (grep { $_ ne 'int64' } @$aux_types);
+my $aux_type_ids = [map { DTYPE_STR_TO_MX->{$_} } @$aux_types];
+$aux_shapes //= [map { [0] } @$aux_types];
+my $aux_shape_lens = [map { scalar(@$_) } @$aux_shapes];
+@$aux_shapes = map { @$_ } @$aux_shapes;
+my $num_aux = @{ $aux_types };
+my $handle = check_call(
+AI::MXNetCAPI::NDArrayCreateSparseEx(
+STORAGE_TYPE_STR_TO_ID->{$stype},
+$shape,
+scalar(@$shape),
+$ctx->device_type_id,
+$ctx->device_id,
+$delay_alloc,
+DTYPE_STR_TO_MX->{$dtype},
+scalar(@$aux_types),
+$aux_type_ids,
+$aux_shape_lens,
+$aux_shapes
+)
+);
+}
+
+method _class_name()
+{
+my $class = ref $self || $self;
+$class;
+}
+
+sub not_implemented { confess "Not implemented" }
 
 Review comment:
   please elaborate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sergeykolychev commented on a change in pull request #9988: [Perl] Sparse feature.

[GitBox]

sergeykolychev commented on a change in pull request #9988: [Perl] Sparse 
feature.
URL: https://github.com/apache/incubator-mxnet/pull/9988#discussion_r172412124
 
 

 ##
 File path: perl-package/AI-MXNet/lib/AI/MXNet/IO.pm
 ##
 @@ -459,20 +459,33 @@ sub BUILD
 my $self  = shift;
 my $data  = AI::MXNet::IO->init_data($self->data,  allow_empty => 0, 
default_name => 'data');
 my $label = AI::MXNet::IO->init_data($self->label, allow_empty => 1, 
default_name => $self->label_name);
+if (
+((blessed $data and $data->isa('AI::MXNet::NDArray::CSR'))
+or
+(blessed $label and $label->isa('AI::MXNet::NDArray::CSR')))
+and
+($self->last_batch_handle != 'discard')
+)
 
 Review comment:
   yep, wrong indentation, will fix.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sergeykolychev commented on a change in pull request #9988: [Perl] Sparse feature.

[GitBox]

sergeykolychev commented on a change in pull request #9988: [Perl] Sparse 
feature.
URL: https://github.com/apache/incubator-mxnet/pull/9988#discussion_r172411854
 
 

 ##
 File path: perl-package/AI-MXNet/lib/AI/MXNet/Optimizer.pm
 ##
 @@ -398,6 +423,461 @@ method update(
 
 __PACKAGE__->register;
 
+=head1 NAME
+
+AI::MXNet::Signum - The Signum optimizer that takes the sign of gradient 
or momentum.
+=cut
+
+=head1 DESCRIPTION
+
+The optimizer updates the weight by:
+
+rescaled_grad = rescale_grad * clip(grad, clip_gradient) + wd * weight
+state = momentum * state + (1-momentum)*rescaled_grad
+weight = (1 - lr * wd_lh) * weight - lr * sign(state)
+
+See the original paper at: 
https://jeremybernste.in/projects/amazon/signum.pdf
+
+For details of the update algorithm see
+:class:`~mxnet.ndarray.signsgd_update` and 
:class:`~mxnet.ndarray.signum_update`.
+
+This optimizer accepts the following parameters in addition to those 
accepted
+by :class:`.Optimizer`.
+
+Parameters
+--
+momentum : float, optional
+   The momentum value.
+wd_lh : float, optional
+   The amount of decoupled weight decay regularization, see details in the 
original paper at:\
+   https://arxiv.org/abs/1711.05101
+=cut
+
+package AI::MXNet::Signum;
+use Mouse;
+extends 'AI::MXNet::Optimizer';
+
+has 'momentum' => (is => "rw", isa => "Num", default => 0.9);
+has 'wd_lh'=> (is => "rw", isa => "Num", default => 0);
+
+method create_state(Index $index, AI::MXNet::NDArray $weight)
+{
+
+my $momentum;
+if($self->momentum != 0)
+{
+$momentum = AI::MXNet::NDArray->zeros(
+$weight->shape,
+ctx => $weight->context,
+dtype=>$weight->dtype,
+stype=>$weight->stype
+);
+}
+return $momentum;
+}
+
+method update(
+Index $index,
+AI::MXNet::NDArray$weight,
+AI::MXNet::NDArray$grad,
+Maybe[AI::MXNet::NDArray|ArrayRef[Maybe[AI::MXNet::NDArray]]] $state
+)
+{
+$self->_update_count($index);
+my $lr = $self->_get_lr($index);
+my $wd = $self->_get_wd($index);
+my %kwargs = (
+out => $weight,
+lr  => $lr,
+wd  => $wd,
+rescale_grad => $self->rescale_grad,
+);
+if($self->momentum > 0)
+{
+$kwargs{momentum} = $self->momentum;
+}
+if($self->clip_gradient)
+{
+$kwargs{clip_gradient} = $self->clip_gradient;
+}
+if($self->wd_lh)
+{
+$kwargs{wd_lh} = $self->wd_lh;
+}
+if(defined $state)
+{
+AI::MXNet::NDArray->signum_update(
+$weight, $grad, $state, %kwargs
+);
+}
+else
+{
+AI::MXNet::NDArray->signsgd_update(
+$weight, $grad, %kwargs
+);
+}
+}
+
+__PACKAGE__->register;
+
+=head1 NAME
+
+AI::MXNet::FTML - The FTML optimizer.
+=cut
+
+=head1 DESCRIPTION
+
+This class implements the optimizer described in
+*FTML - Follow the Moving Leader in Deep Learning*,
+available at http://proceedings.mlr.press/v70/zheng17a/zheng17a.pdf.
+
+This optimizer accepts the following parameters in addition to those 
accepted
+by AI::MXNet::Optimizer
+
+Parameters
+--
+beta1 : float, optional
+0 < beta1 < 1. Generally close to 0.5.
+beta2 : float, optional
+0 < beta2 < 1. Generally close to 1.
+epsilon : float, optional
+Small value to avoid division by 0.
+=cut
+
+package AI::MXNet::FTML;
+use Mouse;
+extends 'AI::MXNet::Optimizer';
+
+has 'beta1'   => (is => "rw", isa => "Num", default => 0.6);
+has 'beta2'   => (is => "rw", isa => "Num", default => 0.999);
+has 'epsilon' => (is => "rw", isa => "Num", default => 1e-8);
+
+method create_state(Index $index, AI::MXNet::NDArray $weight)
+{
+return [
+AI::MXNet::NDArray->zeros($weight->shape, ctx => $weight->context, 
dtype=>$weight->dtype), # d_0
+AI::MXNet::NDArray->zeros($weight->shape, ctx => $weight->context, 
dtype=>$weight->dtype), # v_0
+AI::MXNet::NDArray->zeros($weight->shape, ctx => $weight->context, 
dtype=>$weight->dtype), # z_0
+];
+}
+
+method update(
+Index $index,
+AI::MXNet::NDArray$weight,
+AI::MXNet::NDArray$grad,
+Maybe[AI::MXNet::NDArray|ArrayRef[Maybe[AI::MXNet::NDArray]]] $state
+)
+{
+my $lr = $self->_get_lr($index);
+my $wd = $self->_get_wd($index);
+my $t = $self->_update_count($index);
+my %kwargs = (
+out => $weight,
+lr  => $lr,
+wd  => $wd,
+t   => $t,
+beta1 => $self->beta1,
+beta2 => $self->beta2,
+epsilon => $self->epsilon,
+rescale_grad => $self->rescale_grad
+);
+use Data::Dumper;
 
 Review comment:
   yes, will remove, thanks for noticing.


This is an automated message from 

[GitHub] sergeykolychev commented on a change in pull request #9988: [Perl] Sparse feature.

[GitBox]

sergeykolychev commented on a change in pull request #9988: [Perl] Sparse 
feature.
URL: https://github.com/apache/incubator-mxnet/pull/9988#discussion_r172411854
 
 

 ##
 File path: perl-package/AI-MXNet/lib/AI/MXNet/Optimizer.pm
 ##
 @@ -398,6 +423,461 @@ method update(
 
 __PACKAGE__->register;
 
+=head1 NAME
+
+AI::MXNet::Signum - The Signum optimizer that takes the sign of gradient 
or momentum.
+=cut
+
+=head1 DESCRIPTION
+
+The optimizer updates the weight by:
+
+rescaled_grad = rescale_grad * clip(grad, clip_gradient) + wd * weight
+state = momentum * state + (1-momentum)*rescaled_grad
+weight = (1 - lr * wd_lh) * weight - lr * sign(state)
+
+See the original paper at: 
https://jeremybernste.in/projects/amazon/signum.pdf
+
+For details of the update algorithm see
+:class:`~mxnet.ndarray.signsgd_update` and 
:class:`~mxnet.ndarray.signum_update`.
+
+This optimizer accepts the following parameters in addition to those 
accepted
+by :class:`.Optimizer`.
+
+Parameters
+--
+momentum : float, optional
+   The momentum value.
+wd_lh : float, optional
+   The amount of decoupled weight decay regularization, see details in the 
original paper at:\
+   https://arxiv.org/abs/1711.05101
+=cut
+
+package AI::MXNet::Signum;
+use Mouse;
+extends 'AI::MXNet::Optimizer';
+
+has 'momentum' => (is => "rw", isa => "Num", default => 0.9);
+has 'wd_lh'=> (is => "rw", isa => "Num", default => 0);
+
+method create_state(Index $index, AI::MXNet::NDArray $weight)
+{
+
+my $momentum;
+if($self->momentum != 0)
+{
+$momentum = AI::MXNet::NDArray->zeros(
+$weight->shape,
+ctx => $weight->context,
+dtype=>$weight->dtype,
+stype=>$weight->stype
+);
+}
+return $momentum;
+}
+
+method update(
+Index $index,
+AI::MXNet::NDArray$weight,
+AI::MXNet::NDArray$grad,
+Maybe[AI::MXNet::NDArray|ArrayRef[Maybe[AI::MXNet::NDArray]]] $state
+)
+{
+$self->_update_count($index);
+my $lr = $self->_get_lr($index);
+my $wd = $self->_get_wd($index);
+my %kwargs = (
+out => $weight,
+lr  => $lr,
+wd  => $wd,
+rescale_grad => $self->rescale_grad,
+);
+if($self->momentum > 0)
+{
+$kwargs{momentum} = $self->momentum;
+}
+if($self->clip_gradient)
+{
+$kwargs{clip_gradient} = $self->clip_gradient;
+}
+if($self->wd_lh)
+{
+$kwargs{wd_lh} = $self->wd_lh;
+}
+if(defined $state)
+{
+AI::MXNet::NDArray->signum_update(
+$weight, $grad, $state, %kwargs
+);
+}
+else
+{
+AI::MXNet::NDArray->signsgd_update(
+$weight, $grad, %kwargs
+);
+}
+}
+
+__PACKAGE__->register;
+
+=head1 NAME
+
+AI::MXNet::FTML - The FTML optimizer.
+=cut
+
+=head1 DESCRIPTION
+
+This class implements the optimizer described in
+*FTML - Follow the Moving Leader in Deep Learning*,
+available at http://proceedings.mlr.press/v70/zheng17a/zheng17a.pdf.
+
+This optimizer accepts the following parameters in addition to those 
accepted
+by AI::MXNet::Optimizer
+
+Parameters
+--
+beta1 : float, optional
+0 < beta1 < 1. Generally close to 0.5.
+beta2 : float, optional
+0 < beta2 < 1. Generally close to 1.
+epsilon : float, optional
+Small value to avoid division by 0.
+=cut
+
+package AI::MXNet::FTML;
+use Mouse;
+extends 'AI::MXNet::Optimizer';
+
+has 'beta1'   => (is => "rw", isa => "Num", default => 0.6);
+has 'beta2'   => (is => "rw", isa => "Num", default => 0.999);
+has 'epsilon' => (is => "rw", isa => "Num", default => 1e-8);
+
+method create_state(Index $index, AI::MXNet::NDArray $weight)
+{
+return [
+AI::MXNet::NDArray->zeros($weight->shape, ctx => $weight->context, 
dtype=>$weight->dtype), # d_0
+AI::MXNet::NDArray->zeros($weight->shape, ctx => $weight->context, 
dtype=>$weight->dtype), # v_0
+AI::MXNet::NDArray->zeros($weight->shape, ctx => $weight->context, 
dtype=>$weight->dtype), # z_0
+];
+}
+
+method update(
+Index $index,
+AI::MXNet::NDArray$weight,
+AI::MXNet::NDArray$grad,
+Maybe[AI::MXNet::NDArray|ArrayRef[Maybe[AI::MXNet::NDArray]]] $state
+)
+{
+my $lr = $self->_get_lr($index);
+my $wd = $self->_get_wd($index);
+my $t = $self->_update_count($index);
+my %kwargs = (
+out => $weight,
+lr  => $lr,
+wd  => $wd,
+t   => $t,
+beta1 => $self->beta1,
+beta2 => $self->beta2,
+epsilon => $self->epsilon,
+rescale_grad => $self->rescale_grad
+);
+use Data::Dumper;
 
 Review comment:
   yes, will remove later. 


This is an automated message from the Apache 

[GitHub] spidyDev closed pull request #10001: Add op avg pool arg max min

[GitBox]

spidyDev closed pull request #10001: Add op avg pool arg max min
URL: https://github.com/apache/incubator-mxnet/pull/10001
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/python/mxnet/contrib/__init__.py b/python/mxnet/contrib/__init__.py
index 36ee21305bf..63cd8ce2664 100644
--- a/python/mxnet/contrib/__init__.py
+++ b/python/mxnet/contrib/__init__.py
@@ -28,5 +28,5 @@
 from . import tensorboard
 
 from . import text
-
+from . import onnx
 from . import io
diff --git a/python/mxnet/contrib/onnx/__init__.py 
b/python/mxnet/contrib/onnx/__init__.py
new file mode 100644
index 000..eff91206298
--- /dev/null
+++ b/python/mxnet/contrib/onnx/__init__.py
@@ -0,0 +1,20 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Module for importing and exporting ONNX models."""
+
+from ._import.import_model import import_model
diff --git a/python/mxnet/contrib/onnx/_import/__init__.py 
b/python/mxnet/contrib/onnx/_import/__init__.py
new file mode 100644
index 000..002cfa92583
--- /dev/null
+++ b/python/mxnet/contrib/onnx/_import/__init__.py
@@ -0,0 +1,21 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+"""ONNX Import module"""
+from . import import_model
+from . import import_onnx
diff --git a/python/mxnet/contrib/onnx/_import/import_helper.py 
b/python/mxnet/contrib/onnx/_import/import_helper.py
new file mode 100644
index 000..2f964bac534
--- /dev/null
+++ b/python/mxnet/contrib/onnx/_import/import_helper.py
@@ -0,0 +1,43 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+# pylint: disable=invalid-name
+"""Operator attributes conversion"""
+from .op_translations import add, absolute, negative, reduce_max, reshape
+from .op_translations import reduce_mean, avg_pooling
+from .op_translations import sigmoid
+from .op_translations import argmax, argmin
+
+# _convert_map defines maps of name to converter functor(callable)
+_convert_map = {
+# Arithmetic Operators
+'Add'   : add,
+'Abs'   : absolute,
+'Neg'   : negative,
+# Basic neural network functions
+'Sigmoid'   : sigmoid,
+# Changing shape and type.
+'Reshape'   : reshape,
+# Reduce Functions
+'ReduceMax' : reduce_max,
+'ReduceMean': reduce_mean,
+'AveragePool'   : avg_pooling,
+# Sorting and Searching
+'ArgMax': argmax,
+'ArgMin': argmin
+}
diff --git a/python/mxnet/contrib/onnx/_import/import_model.py 

[GitHub] spidyDev commented on issue #10001: Add op avg pool arg max min

[GitBox]

spidyDev commented on issue #10001: Add op avg pool arg max min
URL: https://github.com/apache/incubator-mxnet/pull/10001#issuecomment-370662293
 
 
   closing this PR as this is incorrectly generated. Need to generate on top of 
anirudhacharya/onnx1 branch


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] spidyDev opened a new pull request #10001: Add op avg pool arg max min

[GitBox]

spidyDev opened a new pull request #10001: Add op avg pool arg max min
URL: https://github.com/apache/incubator-mxnet/pull/10001
 
 
   ## Description ##
   Added operators:
   - AvgPool
   - ArgMax
   - ArgMin
   - Abs
   
   Minor changes in logic for import_onnx
   
   ## Checklist ##
   ### Essentials ###
   - [ ] Passed code style checking (`make lint`)
   - [ ] Changes are complete (i.e. I finished coding on this PR)
   - [ ] All changes have test coverage:
   - Unit tests are added for small changes to verify correctness (e.g. adding 
a new operator)
   - Nightly tests are added for complicated/long-running ones (e.g. changing 
distributed kvstore)
   - Build tests will be added for build configuration changes (e.g. adding a 
new build option with NCCL)
   - [ ] Code is well-documented: 
   - For user-facing API changes, API doc string has been updated. 
   - For new C++ functions in header files, their functionalities and arguments 
are documented. 
   - For new examples, README.md is added to explain the what the example does, 
the source of the dataset, expected performance on test set and reference to 
the original paper if applicable
   - [ ] To the my best knowledge, examples are either not affected by this 
change, or have been fixed to be compatible with this change
   
   ### Changes ###
   - [ ] Feature1, tests, (and when applicable, API doc)
   - [ ] Feature2, tests, (and when applicable, API doc)
   
   ## Comments ##
   - If this change is a backward incompatible change, why must this change be 
made.
   - Interesting edge cases to note here
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] reminisce commented on issue #9916: precision bug in batchnorm.

[GitBox]

reminisce commented on issue #9916: precision bug in batchnorm.
URL: 
https://github.com/apache/incubator-mxnet/issues/9916#issuecomment-370660560
 
 
   Also saw failure w/o MKL in one of CI jobs: Python3 GPU
   
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/master/429/pipeline/


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] tlby commented on a change in pull request #9988: [Perl] Sparse feature.

[GitBox]

tlby commented on a change in pull request #9988: [Perl] Sparse feature.
URL: https://github.com/apache/incubator-mxnet/pull/9988#discussion_r172401798
 
 

 ##
 File path: perl-package/AI-MXNet/lib/AI/MXNet/Optimizer.pm
 ##
 @@ -398,6 +423,461 @@ method update(
 
 __PACKAGE__->register;
 
+=head1 NAME
+
+AI::MXNet::Signum - The Signum optimizer that takes the sign of gradient 
or momentum.
+=cut
+
+=head1 DESCRIPTION
+
+The optimizer updates the weight by:
+
+rescaled_grad = rescale_grad * clip(grad, clip_gradient) + wd * weight
+state = momentum * state + (1-momentum)*rescaled_grad
+weight = (1 - lr * wd_lh) * weight - lr * sign(state)
+
+See the original paper at: 
https://jeremybernste.in/projects/amazon/signum.pdf
+
+For details of the update algorithm see
+:class:`~mxnet.ndarray.signsgd_update` and 
:class:`~mxnet.ndarray.signum_update`.
+
+This optimizer accepts the following parameters in addition to those 
accepted
+by :class:`.Optimizer`.
+
+Parameters
+--
+momentum : float, optional
+   The momentum value.
+wd_lh : float, optional
+   The amount of decoupled weight decay regularization, see details in the 
original paper at:\
+   https://arxiv.org/abs/1711.05101
+=cut
+
+package AI::MXNet::Signum;
+use Mouse;
+extends 'AI::MXNet::Optimizer';
+
+has 'momentum' => (is => "rw", isa => "Num", default => 0.9);
+has 'wd_lh'=> (is => "rw", isa => "Num", default => 0);
+
+method create_state(Index $index, AI::MXNet::NDArray $weight)
+{
+
+my $momentum;
+if($self->momentum != 0)
+{
+$momentum = AI::MXNet::NDArray->zeros(
+$weight->shape,
+ctx => $weight->context,
+dtype=>$weight->dtype,
+stype=>$weight->stype
+);
+}
+return $momentum;
+}
+
+method update(
+Index $index,
+AI::MXNet::NDArray$weight,
+AI::MXNet::NDArray$grad,
+Maybe[AI::MXNet::NDArray|ArrayRef[Maybe[AI::MXNet::NDArray]]] $state
+)
+{
+$self->_update_count($index);
+my $lr = $self->_get_lr($index);
+my $wd = $self->_get_wd($index);
+my %kwargs = (
+out => $weight,
+lr  => $lr,
+wd  => $wd,
+rescale_grad => $self->rescale_grad,
+);
+if($self->momentum > 0)
+{
+$kwargs{momentum} = $self->momentum;
+}
+if($self->clip_gradient)
+{
+$kwargs{clip_gradient} = $self->clip_gradient;
+}
+if($self->wd_lh)
+{
+$kwargs{wd_lh} = $self->wd_lh;
+}
+if(defined $state)
+{
+AI::MXNet::NDArray->signum_update(
+$weight, $grad, $state, %kwargs
+);
+}
+else
+{
+AI::MXNet::NDArray->signsgd_update(
+$weight, $grad, %kwargs
+);
+}
+}
+
+__PACKAGE__->register;
+
+=head1 NAME
+
+AI::MXNet::FTML - The FTML optimizer.
+=cut
+
+=head1 DESCRIPTION
+
+This class implements the optimizer described in
+*FTML - Follow the Moving Leader in Deep Learning*,
+available at http://proceedings.mlr.press/v70/zheng17a/zheng17a.pdf.
+
+This optimizer accepts the following parameters in addition to those 
accepted
+by AI::MXNet::Optimizer
+
+Parameters
+--
+beta1 : float, optional
+0 < beta1 < 1. Generally close to 0.5.
+beta2 : float, optional
+0 < beta2 < 1. Generally close to 1.
+epsilon : float, optional
+Small value to avoid division by 0.
+=cut
+
+package AI::MXNet::FTML;
+use Mouse;
+extends 'AI::MXNet::Optimizer';
+
+has 'beta1'   => (is => "rw", isa => "Num", default => 0.6);
+has 'beta2'   => (is => "rw", isa => "Num", default => 0.999);
+has 'epsilon' => (is => "rw", isa => "Num", default => 1e-8);
+
+method create_state(Index $index, AI::MXNet::NDArray $weight)
+{
+return [
+AI::MXNet::NDArray->zeros($weight->shape, ctx => $weight->context, 
dtype=>$weight->dtype), # d_0
+AI::MXNet::NDArray->zeros($weight->shape, ctx => $weight->context, 
dtype=>$weight->dtype), # v_0
+AI::MXNet::NDArray->zeros($weight->shape, ctx => $weight->context, 
dtype=>$weight->dtype), # z_0
+];
+}
+
+method update(
+Index $index,
+AI::MXNet::NDArray$weight,
+AI::MXNet::NDArray$grad,
+Maybe[AI::MXNet::NDArray|ArrayRef[Maybe[AI::MXNet::NDArray]]] $state
+)
+{
+my $lr = $self->_get_lr($index);
+my $wd = $self->_get_wd($index);
+my $t = $self->_update_count($index);
+my %kwargs = (
+out => $weight,
+lr  => $lr,
+wd  => $wd,
+t   => $t,
+beta1 => $self->beta1,
+beta2 => $self->beta2,
+epsilon => $self->epsilon,
+rescale_grad => $self->rescale_grad
+);
+use Data::Dumper;
 
 Review comment:
   debugging cruft?


This is an automated message from the Apache Git Service.
To 

[GitHub] tlby commented on a change in pull request #9988: [Perl] Sparse feature.

[GitBox]

tlby commented on a change in pull request #9988: [Perl] Sparse feature.
URL: https://github.com/apache/incubator-mxnet/pull/9988#discussion_r172403304
 
 

 ##
 File path: perl-package/AI-MXNet/lib/AI/MXNet/NDArray/Sparse.pm
 ##
 @@ -0,0 +1,1342 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+package AI::MXNet::NDArray::Sparse;
+use strict;
+use warnings;
+use AI::MXNet::Base;
+use AI::MXNet::Function::Parameters;
+use Mouse;
+extends 'AI::MXNet::NDArray';
+
+=head1 NAME
+
+AI::MXNet::NDArray::Sparse - Sparse NDArray API of MXNet
+=cut
+
+=head1 DESCRIPTION
+
+The base class of an NDArray stored in a sparse storage format.
+See AI::MXNet::NDArray::CSR and AI::MXNet::NDArray::RowSparse for more 
details.
+=cut
+
+method _new_alloc_handle(
+Stype$stype,
+Shape$shape,
+AI::MXNet::Context   $ctx,
+Bool $delay_alloc,
+Dtype$dtype,
+AuxTypes $aux_types,
+Maybe[ArrayRef[Shape]]   $aux_shapes=
+)
+{
+confess("only int64 is supported for aux types")
+if (grep { $_ ne 'int64' } @$aux_types);
+my $aux_type_ids = [map { DTYPE_STR_TO_MX->{$_} } @$aux_types];
+$aux_shapes //= [map { [0] } @$aux_types];
+my $aux_shape_lens = [map { scalar(@$_) } @$aux_shapes];
+@$aux_shapes = map { @$_ } @$aux_shapes;
+my $num_aux = @{ $aux_types };
+my $handle = check_call(
+AI::MXNetCAPI::NDArrayCreateSparseEx(
+STORAGE_TYPE_STR_TO_ID->{$stype},
+$shape,
+scalar(@$shape),
+$ctx->device_type_id,
+$ctx->device_id,
+$delay_alloc,
+DTYPE_STR_TO_MX->{$dtype},
+scalar(@$aux_types),
+$aux_type_ids,
+$aux_shape_lens,
+$aux_shapes
+)
+);
+}
+
+method _class_name()
+{
+my $class = ref $self || $self;
+$class;
+}
+
+sub not_implemented { confess "Not implemented" }
 
 Review comment:
   perhaps 
"[...](https://perldoc.perl.org/perlsyn.html#The-Ellipsis-Statement)" is 
appropriate here?  it's kinda standard for declaring abstract functions.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] tlby commented on a change in pull request #9988: [Perl] Sparse feature.

[GitBox]

tlby commented on a change in pull request #9988: [Perl] Sparse feature.
URL: https://github.com/apache/incubator-mxnet/pull/9988#discussion_r172405674
 
 

 ##
 File path: perl-package/AI-MXNetCAPI/mxnet_typemaps.i
 ##
 @@ -112,6 +112,32 @@
 }
 }
 
+%typemap(in,numinputs=0) (const int **out_stypes) (int* temp)
+{
+temp = NULL;
+$1 = 
+}
+
+%typemap(argout) (const int **out_stypes)
+{
+if(av_len((AV*)SvRV(ST(3))) == -1 && !result)
+{
+AV *myav;
+SV **svs;
+int i = 0;
+svs = (SV **)safemalloc(*arg4*sizeof(SV *));
+for (i = 0; i < *arg4 ; i++) {
+svs[i] = newSViv((*$1)[i]);
+sv_2mortal(svs[i]);
+}
+myav = av_make(*arg4, svs);
 
 Review comment:
   I don't think you can guarantee that "arg4" will be a stable variable name 
for the other parameter in the function that uses this typemap, so this might 
blow up at some time in the future, but I'll pester you elsewhere for more 
details.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] tlby commented on a change in pull request #9988: [Perl] Sparse feature.

[GitBox]

tlby commented on a change in pull request #9988: [Perl] Sparse feature.
URL: https://github.com/apache/incubator-mxnet/pull/9988#discussion_r172404489
 
 

 ##
 File path: perl-package/AI-MXNet/lib/AI/MXNet/Base.pm
 ##
 @@ -365,4 +452,20 @@ $PDL::toolongtoprint = 1000_000_000;
 *PDL::dtype  = sub { DTYPE_MX_TO_STR->{ DTYPE_PDL_TO_MX->{ 
shift->type->numval } } };
 }
 
+sub digitize
+{
+my ($d, $bins) = @_;
+for(my $i = 0; $i < @$bins; $i++)
+{
+return $i if $d < $bins->[$i];
+}
+return scalar(@$bins);
+}
+
+use B;
+sub hash { hex(B::hash(shift)) }
 
 Review comment:
   Careful with this, 
[remember](https://perldoc.perl.org/perl5180delta.html#Hash-overhaul) the hash 
produced will vary from one run to the next.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] tlby commented on a change in pull request #9988: [Perl] Sparse feature.

[GitBox]

tlby commented on a change in pull request #9988: [Perl] Sparse feature.
URL: https://github.com/apache/incubator-mxnet/pull/9988#discussion_r172401998
 
 

 ##
 File path: perl-package/AI-MXNet/lib/AI/MXNet/IO.pm
 ##
 @@ -459,20 +459,33 @@ sub BUILD
 my $self  = shift;
 my $data  = AI::MXNet::IO->init_data($self->data,  allow_empty => 0, 
default_name => 'data');
 my $label = AI::MXNet::IO->init_data($self->label, allow_empty => 1, 
default_name => $self->label_name);
+if (
+((blessed $data and $data->isa('AI::MXNet::NDArray::CSR'))
+or
+(blessed $label and $label->isa('AI::MXNet::NDArray::CSR')))
+and
+($self->last_batch_handle != 'discard')
+)
 
 Review comment:
   "and" and "or" at the same indent depth?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] YujiOshima commented on a change in pull request #9994: Tensorboard: Add histogram callback

[GitBox]

YujiOshima commented on a change in pull request #9994: Tensorboard: Add 
histogram callback
URL: https://github.com/apache/incubator-mxnet/pull/9994#discussion_r172404718
 
 

 ##
 File path: python/mxnet/contrib/tensorboard.py
 ##
 @@ -71,3 +72,43 @@ def __call__(self, param):
 if self.prefix is not None:
 name = '%s-%s' % (self.prefix, name)
 self.summary_writer.add_scalar(name, value)
+
+def node_histogram_visualization(self, prefix=None, node_names=None, 
bins="auto"):
 
 Review comment:
   Thank you for comment.
   This method can be used like below.
   
   ```python
   lm = mx.contrib.tensorboard.LogMetricsCallback("/log")
   model.fit(train,
   ...
   epoch_end_callback=lm.node_histogram_visualization(),
   ...
   )
   ```
   
   I understand it would be nice that histogram callback is defined as a new 
class.
   But if you use both of scalar and histogram callback, you should use the 
same `tensorboard.SummaryWriter` instance.
   Another way, make a new class and give SummaryWriter instance as an argument 
like below.
   
   ```python
   class HistogramCallback(object):
   def __init__(self, logging_dir, prefix=None, summarywriter=None):
   ```
   
   Any ideas?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] pengzhao-intel commented on issue #9916: precision bug in batchnorm.

[GitBox]

pengzhao-intel commented on issue #9916: precision bug in batchnorm.
URL: 
https://github.com/apache/incubator-mxnet/issues/9916#issuecomment-370655977
 
 
   Tested w/o MKL-DNN, the same issue is observed.
   The computation result from FP16 is not matched w/ FP32.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] asitstands commented on issue #9991: Random shuffle implementation

[GitBox]

asitstands commented on issue #9991: Random shuffle implementation
URL: https://github.com/apache/incubator-mxnet/pull/9991#issuecomment-370652954
 
 
   @piiswrong The difference of this operator and numpy's shuffle is not the 
default axis. This performs a batch operation but numpy does not. For example, 
if 2D array is given, numpy shuffles the order of the rows (not individual 
elements), i.e., the performed elementwise permutation is exactly same for all 
columns. The operator in this PR would shuffle the elements in each column 
independently if it shuffled along the first axis. I'm not sure why numpy chose 
such a way. However, I think that the batch operation is more natural and 
consistent with the behavior of other operators, especially with `sort` which 
also reorders the elements. Shuffle along the last axis is also consistent with 
other operators. Moreover, `mx.random` package already has an operator 
(`multinomial`) which behaves differently from the operator with the same name 
in numpy. If we need to follow numpy despite of these, I'll change the behavior 
of the operator in this PR to comply numpy.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] asitstands commented on issue #9991: Random shuffle implementation

[GitBox]

asitstands commented on issue #9991: Random shuffle implementation
URL: https://github.com/apache/incubator-mxnet/pull/9991#issuecomment-370652954
 
 
   @piiswrong The difference of this operator and numpy's shuffle is not the 
default axis. This performs a batch operation but numpy does not. For example, 
if 2D array is given, numpy shuffles the order of the rows (not individual 
elements), i.e., the performed elementwise permutation is exactly same for all 
columns. The operator in this PR would shuffle the elements in each column 
independently if it shuffled along the first axis. I'm not sure why numpy chose 
such a way. However, I think that the batch operation is more natural and 
consistent with the behavior of other operators, especially with `sort` which 
also reorders the elements. Shuffle along the last axis is also consistent with 
other operators. Moreover, `mx.random` package already has an operator 
(`multinomial`) which behaves differently from the operator with the same name 
in numpy. If we need to follow numpy despite of these, I'll change the behavior 
of the operator to comply numpy.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] asitstands commented on issue #9991: Random shuffle implementation

[GitBox]

asitstands commented on issue #9991: Random shuffle implementation
URL: https://github.com/apache/incubator-mxnet/pull/9991#issuecomment-370486626
 
 
   @cjolivier01 `test_bmod` fails in `Python3: MKLDNN-CPU` 
(http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-9991/3/pipeline#step-810-log-756).
 I have no clue why this fails. The test seems to have nothing to do with this 
PR. Is there a way to trigger the test once again?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] asitstands commented on issue #9991: Random shuffle implementation

[GitBox]

asitstands commented on issue #9991: Random shuffle implementation
URL: https://github.com/apache/incubator-mxnet/pull/9991#issuecomment-370652954
 
 
   @piiswrong The difference of this operator and numpy's shuffle is not the 
default axis. This performs a batch operation but numpy does not. For example, 
if 2D array is given, numpy shuffles the order of the rows (not individual 
elements), i.e., the performed elementwise permutation is exactly same for all 
columns. The operator in this PR would shuffle the elements in each column 
independently if it shuffled along the first axis. I'm not sure why numpy chose 
such a way. However, I think that the batch operation is more natural and 
consistent with the behavior of other operators, especially with `sort` which 
also reorders the elements. Shuffle along the last axis is also consistent with 
other operators. Moreover, `mx.random` package already has an operator 
(`multinormial`) which behaves differently from the operator with the same name 
in numpy. If we need to follow numpy despite of these, I'll change the behavior 
of the operator to comply numpy.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] zhanghang1989 commented on issue #9688: Adapt operators from PyTorch, will keep adding

[GitBox]

zhanghang1989 commented on issue #9688: Adapt operators from PyTorch, will keep 
adding
URL: https://github.com/apache/incubator-mxnet/pull/9688#issuecomment-370652764
 
 
   A few issues are related to 'BilinearResize' operator 
(https://github.com/apache/incubator-mxnet/issues/9970, 
https://github.com/onnx/onnx-mxnet/issues/33). It is quite frequently used 
feature. Is that possible to consider merging this PR @piiswrong ?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] wkcn commented on issue #9939: add multi proposal operator (cpu version) and fix the bug in proposal op (gpu version)

[GitBox]

wkcn commented on issue #9939: add multi proposal operator (cpu version) and 
fix the bug in proposal op (gpu version)
URL: https://github.com/apache/incubator-mxnet/pull/9939#issuecomment-370285916
 
 
   @pengzhao-intel @xinyu-intel 
   Thank you! I will have a try.
   The performance table
   
   name|time (ms)
   --|--
   BBoxTransformInv|268
   IoUTransformInv|Not used
   FilterBox|22
   CopyScore|18
   ReverseArgsort(unstable sort)|7303
   ReorderProposals|338
   nms(calculate area)|286
   nms(calcuate nms)|7547
   allocate memory for workspace|1
   copy anchor to workspace_proposal|0
   enumrate all shifted anchors|9
   copy workspace_proposals_base to workspace_proposals|162
   assign foreground scores for each anchor|45
   prepare output|3
   Total|16002
   
   Using stable sort to sort anchors (ReverseArgsort) will increase about 3000 
ms.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] wkcn commented on issue #9939: add multi proposal operator (cpu version) and fix the bug in proposal op (gpu version)

[GitBox]

wkcn commented on issue #9939: add multi proposal operator (cpu version) and 
fix the bug in proposal op (gpu version)
URL: https://github.com/apache/incubator-mxnet/pull/9939#issuecomment-370285916
 
 
   @pengzhao-intel @xinyu-intel 
   Thank you! I will have a try.
   The performance table
   
   name|time (ms)
   --|--
   Total|16002
   BBoxTransformInv|268
   IoUTransformInv|Not used
   FilterBox|22
   CopyScore|18
   ReverseArgsort(unstable sort)|7303
   ReorderProposals|338
   nms(calculate area)|286
   nms(calcuate nms)|7547
   allocate memory for workspace|1
   copy anchor to workspace_proposal|0
   enumrate all shifted anchors|9
   copy workspace_proposals_base to workspace_proposals|162
   assign foreground scores for each anchor|45
   prepare output|3
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] wkcn commented on issue #9939: add multi proposal operator (cpu version) and fix the bug in proposal op (gpu version)

[GitBox]

wkcn commented on issue #9939: add multi proposal operator (cpu version) and 
fix the bug in proposal op (gpu version)
URL: https://github.com/apache/incubator-mxnet/pull/9939#issuecomment-370285916
 
 
   @pengzhao-intel @xinyu-intel 
   Thank you! I will have a try.
   The performance table
   
   name|time (ms)
   --|--
   Total|16002
   BBoxTransformInv|268
   IoUTransformInv|Not used
   FilterBox|22
   CopyScore|18
   ReverseArgsort(stable sort)|7303
   ReorderProposals|338
   nms(calculate area)|286
   nms(calcuate nms)|7547
   allocate memory for workspace|1
   copy anchor to workspace_proposal|0
   enumrate all shifted anchors|9
   copy workspace_proposals_base to workspace_proposals|162
   assign foreground scores for each anchor|45
   prepare output|3
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] vdantu commented on issue #9979: Computing the second derivative

[GitBox]

vdantu commented on issue #9979: Computing the second derivative
URL: 
https://github.com/apache/incubator-mxnet/issues/9979#issuecomment-370639938
 
 
   @aidan-plenert-macdonald : This looks like a question and not an issue in 
MxNet. Could you please post this on  discuss.mxnet.io . You would have a 
larger audience to answer your question.
   
   @sandeep-krishnamurthy : Could you label this issue as question. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] zhanghang1989 commented on a change in pull request #9931: Add axes support to Dropout for variational dropout in NLP

[GitBox]

zhanghang1989 commented on a change in pull request #9931: Add axes support to 
Dropout for variational dropout in NLP
URL: https://github.com/apache/incubator-mxnet/pull/9931#discussion_r172389676
 
 

 ##
 File path: src/operator/nn/dropout-inl.h
 ##
 @@ -228,11 +301,27 @@ class DropoutOp {
 if (!MKLForward(s, pgen, this->pkeep_, in_data, out_data)) {
   const TBlob  = out_data[dropout::kMask];
   CHECK(req[dropout::kOut] != kAddTo);
-  LaunchRNG(s, pgen, out.Size(),
-out.dptr(),
-mask.dptr(),
-in_data[dropout::kData].dptr(),
-this->pkeep_);
+  // initialize the mask
+  LaunchRNG(s, pgen, out.Size(),
+  mask.dptr(),
+  this->pkeep_);
+  if (req[0] != kNullOp) {
+// broardcast mul
+TShape new_lshape, new_rshape, new_oshape;
+int ndim = 
BinaryBroadcastShapeCompact(in_data[dropout::kData].shape_,
+   mask.shape_, out.shape_,
+   _lshape, _rshape, 
_oshape);
+BROADCAST_NDIM_SWITCH(ndim, NDim, {
+  mshadow::Shape oshape = new_oshape.get();
+  mshadow::Shape lstride = 
mxnet_op::calc_stride(new_lshape.get());
+  mshadow::Shape rstride = 
mxnet_op::calc_stride(new_rshape.get());
+  mxnet_op::Kernel, xpu>::
+  template LaunchEx(s, new_oshape.Size(), req[0], lstride, 
rstride, oshape,
+  in_data[dropout::kData].dptr(),
+  mask.dptr(), out.dptr());
+});
+  }
 }
 
 Review comment:
   Thx @cjolivier01 . I added the condition check here 
https://github.com/apache/incubator-mxnet/pull/9931/files#diff-4aea2cc24c0bb4e8e48face9faf4aa26R249


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] zhanghang1989 commented on a change in pull request #9931: Add axes support to Dropout for variational dropout in NLP

[GitBox]

zhanghang1989 commented on a change in pull request #9931: Add axes support to 
Dropout for variational dropout in NLP
URL: https://github.com/apache/incubator-mxnet/pull/9931#discussion_r171937954
 
 

 ##
 File path: src/operator/nn/dropout-inl.h
 ##
 @@ -178,30 +184,17 @@ class DropoutOp {
   /*!
* \brief Dropout kernel, compute dropout tensor
*/
-  struct DropoutKernel {
-/*!
- * \brief Dropout kernel function
- * \param id Thread number (0-based representing count)
- * \param gen Random number generator
- * \param N Total number of items in the output
- * \param step Step between items, related to parallelism
- * \param dropout_out Output dropout values
- * \param mask_out  Output mask (is multiplied to create dropout output, 
may be 0)
- * \param input_data Input data to perform the dropout on
- * \param pkeep Dropout rate (keep when the generated random number is 
less than this value)
- */
+  struct BernoulliKernel {
+/*! \brief Bernoulli kernel for generating mask */
 MSHADOW_XINLINE static void Map(int id,
 RandGenerator gen,
 const int N,
 const int step,
-DType *dropout_out,
 DType *mask_out,
-const DType *input_data,
 const real_t pkeep) {
   RNG_KERNEL_LOOP(xpu, DType, id, gen, N, step, {
 const real_t rand_num = static_cast(genImpl.uniform());
 mask_out[i] = mshadow_op::threshold::Map(rand_num, pkeep) * 
(1.0f / pkeep);
-dropout_out[i] = input_data[i] * mask_out[i];
 
 Review comment:
   Thx @cjolivier01 . I get your point for efficiency. I have added a condition 
check for standard dropout, which has the same efficiency when none-axes 
provided:
   
https://github.com/apache/incubator-mxnet/pull/9931/files#diff-4aea2cc24c0bb4e8e48face9faf4aa26R252


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] zhanghang1989 commented on a change in pull request #9931: Add axes support to Dropout for variational dropout in NLP

[GitBox]

zhanghang1989 commented on a change in pull request #9931: Add axes support to 
Dropout for variational dropout in NLP
URL: https://github.com/apache/incubator-mxnet/pull/9931#discussion_r172388387
 
 

 ##
 File path: src/operator/nn/dropout-inl.h
 ##
 @@ -337,6 +336,7 @@ class DropoutOp {
   real_t pkeep_;
   /*! \brief Dropout mode */
   dropout::DropoutOpMode mode_;
+  TShape axes;
 
 Review comment:
   Thx ? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] cjolivier01 commented on a change in pull request #9931: Add axes support to Dropout for variational dropout in NLP

[GitBox]

cjolivier01 commented on a change in pull request #9931: Add axes support to 
Dropout for variational dropout in NLP
URL: https://github.com/apache/incubator-mxnet/pull/9931#discussion_r172388095
 
 

 ##
 File path: src/operator/nn/dropout-inl.h
 ##
 @@ -337,6 +336,7 @@ class DropoutOp {
   real_t pkeep_;
   /*! \brief Dropout mode */
   dropout::DropoutOpMode mode_;
+  TShape axes;
 
 Review comment:
   nit: member variable name should end in an underscore 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] cjolivier01 commented on issue #9993: cmake cannot build mxnet

[GitBox]

cjolivier01 commented on issue #9993: cmake cannot build mxnet
URL: 
https://github.com/apache/incubator-mxnet/issues/9993#issuecomment-370631191
 
 
   closed on accident


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] CoinCheung commented on a change in pull request #10000: fix average pooling kernel size assignment error

[GitBox]

CoinCheung commented on a change in pull request #1: fix average pooling 
kernel size assignment error
URL: https://github.com/apache/incubator-mxnet/pull/1#discussion_r172385236
 
 

 ##
 File path: src/operator/nn/pooling-inl.h
 ##
 @@ -178,6 +178,11 @@ class PoolingOp {
 template
 PoolingOp (const PoolingParam ) {
   static thread_local PoolingOp op;
+  // check if filter size assigned correctly
+  if (param.global_pool == false) {
+CHECK_GT(param.kernel.ndim(), 0U)
+<< "A positive number must be assigned as filter size";
 
 Review comment:
   How about "Must assign a positive kernel size"?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sxjscience commented on a change in pull request #10000: fix average pooling kernel size assignment error

[GitBox]

sxjscience commented on a change in pull request #1: fix average pooling 
kernel size assignment error
URL: https://github.com/apache/incubator-mxnet/pull/1#discussion_r172384242
 
 

 ##
 File path: src/operator/nn/pooling-inl.h
 ##
 @@ -178,6 +178,11 @@ class PoolingOp {
 template
 PoolingOp (const PoolingParam ) {
   static thread_local PoolingOp op;
+  // check if filter size assigned correctly
+  if (param.global_pool == false) {
+CHECK_GT(param.kernel.ndim(), 0U)
+<< "A positive number must be assigned as filter size";
 
 Review comment:
   Need a better error message.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sxjscience commented on issue #10000: fix average pooling kernel size assignment error

[GitBox]

sxjscience commented on issue #1: fix average pooling kernel size 
assignment error
URL: https://github.com/apache/incubator-mxnet/pull/1#issuecomment-370621396
 
 
   Thanks for submitting this PR! This should fix the issue in 
https://discuss.gluon.ai/t/topic/5015/7. You can remove the `kernel=` phrases 
in the test cases 
(https://github.com/apache/incubator-mxnet/blob/master/tests/python/gpu/test_operator_gpu.py#L914-L935).
 Also, you may run `make pylint` to check the lint locally.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sxjscience commented on issue #10000: fix average pooling kernel size assignment error

[GitBox]

sxjscience commented on issue #1: fix average pooling kernel size 
assignment error
URL: https://github.com/apache/incubator-mxnet/pull/1#issuecomment-370621396
 
 
   Thanks for submitting this fix! You can remove the `kernel=` phrases in the 
test cases 
(https://github.com/apache/incubator-mxnet/blob/master/tests/python/gpu/test_operator_gpu.py#L914-L935).
 Also, you may run `make pylint` to check the lint locally.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] CoinCheung opened a new pull request #10000: fix average pooling kernel size assignment error

[GitBox]

CoinCheung opened a new pull request #1: fix average pooling kernel size 
assignment error
URL: https://github.com/apache/incubator-mxnet/pull/1
 
 
   ## Description ##
   The pooling operator still complains about not assigning the kernel size 
when the "global_pool" parameter is assigned to be "True";
   
   ## Checklist ##
   ### Essentials ###
   - [ ] Passed code style checking (`make lint`)
   - [ ] Changes are complete (i.e. I finished coding on this PR)
   - [ ] All changes have test coverage:
   - Unit tests are added for small changes to verify correctness (e.g. adding 
a new operator)
   - Nightly tests are added for complicated/long-running ones (e.g. changing 
distributed kvstore)
   - Build tests will be added for build configuration changes (e.g. adding a 
new build option with NCCL)
   - [ ] Code is well-documented: 
   - For user-facing API changes, API doc string has been updated. 
   - For new C++ functions in header files, their functionalities and arguments 
are documented. 
   - For new examples, README.md is added to explain the what the example does, 
the source of the dataset, expected performance on test set and reference to 
the original paper if applicable
   - [ ] To the my best knowledge, examples are either not affected by this 
change, or have been fixed to be compatible with this change
   
   ### Changes ###
   - [ ] Feature1, tests, (and when applicable, API doc)
   - [ ] Feature2, tests, (and when applicable, API doc)
   
   ## Comments ##
   - If this change is a backward incompatible change, why must this change be 
made.
   - Interesting edge cases to note here
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] jacky4323 opened a new issue #9993: cmake cannot build mxnet

[GitBox]

jacky4323 opened a new issue #9993: cmake cannot build mxnet
URL: https://github.com/apache/incubator-mxnet/issues/9993
 
 
   Hi,
   sorry,there are some problems with cmake
   
   command I use
   $git clone --recursive https://github.com/apache/incubator-mxnet.git
   $cd incubator-mxnet/
   $mkdir build/Release && cd build/Release
   $cmake ../../
   $make -j8
   
   
   - cmake ../../ output log
   
   -- The C compiler identification is GNU 5.4.0
   -- The CXX compiler identification is GNU 5.4.0
   -- Check for working C compiler: /usr/bin/cc
   -- Check for working C compiler: /usr/bin/cc -- works
   -- Detecting C compiler ABI info
   -- Detecting C compiler ABI info - done
   -- Detecting C compile features
   -- Detecting C compile features - done
   -- Check for working CXX compiler: /usr/bin/c++
   -- Check for working CXX compiler: /usr/bin/c++ -- works
   -- Detecting CXX compiler ABI info
   -- Detecting CXX compiler ABI info - done
   -- Detecting CXX compile features
   -- Detecting CXX compile features - done
   -- CMake version '3.5.1' using generator 'Unix Makefiles'
   -- Performing Test SUPPORT_CXX11
   -- Performing Test SUPPORT_CXX11 - Success
   -- Performing Test SUPPORT_CXX0X
   -- Performing Test SUPPORT_CXX0X - Success
   -- Performing Test SUPPORT_MSSE2
   -- Performing Test SUPPORT_MSSE2 - Success
   -- CMAKE_BUILD_TYPE is unset, defaulting to Release
   -- Detecting Intel(R) MKL: trying mklml_intel
   -- Detecting Intel(R) MKL: trying mklml
   -- Detecting Intel(R) MKL: trying mkl_rt
   CMake Warning at 3rdparty/mkldnn/cmake/MKL.cmake:177 (message):
 Intel(R) MKL not found.  Some performance features may not be available.
 Please run scripts/prepare_mkl.sh to download a minimal set of libraries or
 get a full version from https://software.intel.com/en-us/intel-mkl
   Call Stack (most recent call first):
 3rdparty/mkldnn/cmake/OpenMP.cmake:24 (include)
 3rdparty/mkldnn/CMakeLists.txt:57 (include)
   
   
   -- Try OpenMP C flag = [-fopenmp]
   -- Performing Test OpenMP_FLAG_DETECTED
   -- Performing Test OpenMP_FLAG_DETECTED - Success
   -- Try OpenMP CXX flag = [-fopenmp]
   -- Performing Test OpenMP_FLAG_DETECTED
   -- Performing Test OpenMP_FLAG_DETECTED - Success
   -- Found OpenMP: -fopenmp
   -- Could NOT find Doxygen (missing:  DOXYGEN_EXECUTABLE)
   -- VTune profiling environment is unset
   -- Looking for pthread.h
   -- Looking for pthread.h - found
   -- Looking for pthread_create
   -- Looking for pthread_create - found
   -- Found Threads: TRUE
   -- Found CUDA: /usr/local/cuda-8.0 (found version "8.0")
   -- Found OpenBLAS libraries: /usr/lib/libopenblas.so
   -- Found OpenBLAS include: /usr/include
   -- CUDA detected: 8.0
   -- Found cuDNN (include: /usr/local/cuda-8.0/include, library: 
/usr/local/cuda-8.0/lib64/libcudnn.so)
   -- Running GPU architecture autodetection
   -- Found CUDA arch 5.2 5.2 5.2 5.2
   -- Added CUDA NVCC flags for: sm_52
   -- Could NOT find Gperftools (missing:  GPERFTOOLS_LIBRARIES 
GPERFTOOLS_INCLUDE_DIR)
   -- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1")
   -- Could NOT find Jemalloc (missing:  JEMALLOC_LIBRARY JEMALLOC_INCLUDE_DIR)
   --  OpenCV_LIBS=opencv_core;opencv_highgui;opencv_imgproc;opencv_imgcodecs
   -- OpenCV found (/usr/local/share/OpenCV)
   -- Performing Test LIBOMP_HAVE_STD_CPP11_FLAG
   -- Performing Test LIBOMP_HAVE_STD_CPP11_FLAG - Success
   -- Performing Test LIBOMP_HAVE_FNO_EXCEPTIONS_FLAG
   -- Performing Test LIBOMP_HAVE_FNO_EXCEPTIONS_FLAG - Success
   -- Performing Test LIBOMP_HAVE_FNO_RTTI_FLAG
   -- Performing Test LIBOMP_HAVE_FNO_RTTI_FLAG - Success
   -- Performing Test LIBOMP_HAVE_X_CPP_FLAG
   -- Performing Test LIBOMP_HAVE_X_CPP_FLAG - Success
   -- Performing Test LIBOMP_HAVE_WERROR_FLAG
   -- Performing Test LIBOMP_HAVE_WERROR_FLAG - Success
   -- Performing Test LIBOMP_HAVE_WNO_UNUSED_FUNCTION_FLAG
   -- Performing Test LIBOMP_HAVE_WNO_UNUSED_FUNCTION_FLAG - Success
   -- Performing Test LIBOMP_HAVE_WNO_UNUSED_LOCAL_TYPEDEF_FLAG
   -- Performing Test LIBOMP_HAVE_WNO_UNUSED_LOCAL_TYPEDEF_FLAG - Failed
   -- Performing Test LIBOMP_HAVE_WNO_UNUSED_VALUE_FLAG
   -- Performing Test LIBOMP_HAVE_WNO_UNUSED_VALUE_FLAG - Success
   -- Performing Test LIBOMP_HAVE_WNO_UNUSED_VARIABLE_FLAG
   -- Performing Test LIBOMP_HAVE_WNO_UNUSED_VARIABLE_FLAG - Success
   -- Performing Test LIBOMP_HAVE_WNO_SWITCH_FLAG
   -- Performing Test LIBOMP_HAVE_WNO_SWITCH_FLAG - Success
   -- Performing Test LIBOMP_HAVE_WNO_COVERED_SWITCH_DEFAULT_FLAG
   -- Performing Test LIBOMP_HAVE_WNO_COVERED_SWITCH_DEFAULT_FLAG - Failed
   -- Performing Test LIBOMP_HAVE_WNO_DEPRECATED_REGISTER_FLAG
   -- Performing Test LIBOMP_HAVE_WNO_DEPRECATED_REGISTER_FLAG - Failed
   -- Performing Test LIBOMP_HAVE_WNO_SIGN_COMPARE_FLAG
   -- Performing Test LIBOMP_HAVE_WNO_SIGN_COMPARE_FLAG - Success
   -- Performing Test LIBOMP_HAVE_WNO_GNU_ANONYMOUS_STRUCT_FLAG
   -- Performing Test 

[GitHub] sxjscience commented on issue #9998: [Feature Request] batchwise gather_nd/scatter_nd

[GitBox]

sxjscience commented on issue #9998: [Feature Request] batchwise 
gather_nd/scatter_nd
URL: 
https://github.com/apache/incubator-mxnet/issues/9998#issuecomment-370618047
 
 
   This should be the correct version of batch_take. We need to support this in 
the future.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sxjscience commented on issue #9998: [Feature Request] batchwise gather_nd/scatter_nd

[GitBox]

sxjscience commented on issue #9998: [Feature Request] batchwise 
gather_nd/scatter_nd
URL: 
https://github.com/apache/incubator-mxnet/issues/9998#issuecomment-370618047
 
 
   This should be the correct version of batch_take. We need to support his in 
the future.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] cjolivier01 closed issue #9993: cmake cannot build mxnet

[GitBox]

cjolivier01 closed issue #9993: cmake cannot build mxnet
URL: https://github.com/apache/incubator-mxnet/issues/9993
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] cjolivier01 commented on issue #9993: cmake cannot build mxnet

[GitBox]

cjolivier01 commented on issue #9993: cmake cannot build mxnet
URL: 
https://github.com/apache/incubator-mxnet/issues/9993#issuecomment-370612326
 
 
   I am not sure what was changed for the enormous mkldnn PR, but 
"USE_MKL_IF_AVAILABLE" means use it if it is found (real MKL or MKLML), 
otherwise don't.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] marcoabreu commented on issue #9993: cmake cannot build mxnet

[GitBox]

marcoabreu commented on issue #9993: cmake cannot build mxnet
URL: 
https://github.com/apache/incubator-mxnet/issues/9993#issuecomment-370611181
 
 
   How come that CMake properly detects that MKL is not present
   ```
   -- Detecting Intel(R) MKL: trying mklml_intel
   -- Detecting Intel(R) MKL: trying mklml
   -- Detecting Intel(R) MKL: trying mkl_rt
   CMake Warning at 3rdparty/mkldnn/cmake/MKL.cmake:177 (message):
   Intel(R) MKL not found. Some performance features may not be available.
   Please run scripts/prepare_mkl.sh to download a minimal set of libraries or
   get a full version from https://software.intel.com/en-us/intel-mkl
   Call Stack (most recent call first):
   3rdparty/mkldnn/cmake/OpenMP.cmake:24 (include)
   3rdparty/mkldnn/CMakeLists.txt:57 (include)
   ```
   but still tries to use it?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-mxnet] branch master updated: Exception handling documentation (#9869)

[jxie]

This is an automated email from the ASF dual-hosted git repository.

jxie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/master by this push:
 new 7d08810  Exception handling documentation (#9869)
7d08810 is described below

commit 7d0881036959f383bab7efa22a80a9caa419274c
Author: Anirudh Subramanian 
AuthorDate: Mon Mar 5 15:52:21 2018 -0800

Exception handling documentation (#9869)

* Add tests for Exception Handling in Iterators

* Fixing test_random

* Add documentation for exc handling

* Fix for exc handling doc

* Fix exc handling doc

* Add exception handling documentation

* Correct the seed change

* Fix

* Improve exception handling docs

* Add dmlc-core

* Empty commit

* Add dmlc-core

* Move to architecture design docs

* Add exception handling to index

* Trigger CI
---
 dmlc-core   |   2 +-
 docs/architecture/exception_handling.md | 111 
 docs/architecture/index.md  |   1 +
 tests/python/unittest/test_io.py|  24 +++
 4 files changed, 137 insertions(+), 1 deletion(-)

diff --git a/dmlc-core b/dmlc-core
index a1fd683..282b986 16
--- a/dmlc-core
+++ b/dmlc-core
@@ -1 +1 @@
-Subproject commit a1fd6834c0cd3fd2cc586deec2dc24194924cada
+Subproject commit 282b98663f59df6b26f906580af610dea3046f22
diff --git a/docs/architecture/exception_handling.md 
b/docs/architecture/exception_handling.md
new file mode 100644
index 000..5b4448a
--- /dev/null
+++ b/docs/architecture/exception_handling.md
@@ -0,0 +1,111 @@
+# Exception Handling in MXNet
+
+This tutorial explains the exception handling support in MXNet, 
+and provides examples on how to throw and handle exceptions when in a 
multithreaded context.
+Although, the examples are in Python, they can be easily extended to MXNet
+language bindings.
+
+MXNet exceptions can be thrown from two areas:
+- MXNet main thread. For eg. Infershape and InferType.
+- Spawned threads:
+* By dependency engine for operator execution in parallel
+* By the iterators, during the data loading, text parsing phase etc.
+
+In the first case, the exception is thrown and can be handled in the main 
thread.
+In the second case, the exception is thrown in a spawned thread, caught and 
transported to the
+main thread, where it is rethrown. This tutorial will give more explanation 
and examples on how 
+to handle exceptions for the second case.
+
+## Prerequisites 
+
+To complete this tutorial, we need:
+- MXNet 
[7b24137](https://github.com/apache/incubator-mxnet/commit/7b24137ed45df605defa4ce72ec91554f6e445f0).
 See Instructions in [Setup and 
Installation](http://mxnet.io/install/index.html).
+
+## Exception Handling for Iterators
+
+The below example shows how to handle exceptions for iterators. In this 
example,
+we populate files for data and labels with fewer number of labels compared to 
the
+number of samples. This should throw an exception.
+
+CSVIter uses PrefetcherIter for loading and parsing data.
+The PrefetcherIter spawns a producer thread in the background which prefetches
+the data while the main thread consumes the data. The exception is thrown in 
the spawned
+producer thread during the prefetching, when the label is not found 
corresponding to a specific sample. 
+
+The exception is transported to the main thread, where it is rethrown when 
Next is 
+called as part of the following line: `for batch in iter(data_train)`.
+
+In general, Exception may be rethrown as part of `Next` and `BeforeFirst` 
calls which correspond to `reset()` and `next()` methods in `MXDataIter` for 
Python language bindings.
+
+```python
+import os
+import mxnet as mx
+
+cwd = os.getcwd()
+data_path = os.path.join(cwd, "data.csv")
+label_path = os.path.join(cwd, "label.csv")
+
+with open(data_path, "w") as fout:
+for i in range(8):
+fout.write("1,2,3,4,5,6,7,8,9,10\n")
+
+with open(label_path, "w") as fout:
+for i in range(7):
+fout.write("label"+str(i))
+
+try:
+data_train = mx.io.CSVIter(data_csv=data_path, label_csv=label_path, 
data_shape=(1, 10),
+   batch_size=4)
+
+for batch in iter(data_train):
+print(data_train.getdata().asnumpy())
+except mx.base.MXNetError as ex:
+print("Exception handled")
+print(ex)
+```
+
+### Limitation
+
+There is a race condition when your last `next()` call doesnt reach the batch 
in your dataset where exception occurs. Exception may or may not be thrown in 
this case depending on which thread wins the race. To avoid this situation, you 
should try and iterate through your full dataset if you think it can throw 
exceptions which need to be handled.
+
+
+## Exception Handling for Operators
+
+The below example shows how to handle 

[GitHub] piiswrong closed pull request #9869: Exception handling documentation

[GitBox]

piiswrong closed pull request #9869: Exception handling documentation
URL: https://github.com/apache/incubator-mxnet/pull/9869
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/dmlc-core b/dmlc-core
index a1fd6834c0c..282b98663f5 16
--- a/dmlc-core
+++ b/dmlc-core
@@ -1 +1 @@
-Subproject commit a1fd6834c0cd3fd2cc586deec2dc24194924cada
+Subproject commit 282b98663f59df6b26f906580af610dea3046f22
diff --git a/docs/architecture/exception_handling.md 
b/docs/architecture/exception_handling.md
new file mode 100644
index 000..5b4448a53ab
--- /dev/null
+++ b/docs/architecture/exception_handling.md
@@ -0,0 +1,111 @@
+# Exception Handling in MXNet
+
+This tutorial explains the exception handling support in MXNet, 
+and provides examples on how to throw and handle exceptions when in a 
multithreaded context.
+Although, the examples are in Python, they can be easily extended to MXNet
+language bindings.
+
+MXNet exceptions can be thrown from two areas:
+- MXNet main thread. For eg. Infershape and InferType.
+- Spawned threads:
+* By dependency engine for operator execution in parallel
+* By the iterators, during the data loading, text parsing phase etc.
+
+In the first case, the exception is thrown and can be handled in the main 
thread.
+In the second case, the exception is thrown in a spawned thread, caught and 
transported to the
+main thread, where it is rethrown. This tutorial will give more explanation 
and examples on how 
+to handle exceptions for the second case.
+
+## Prerequisites 
+
+To complete this tutorial, we need:
+- MXNet 
[7b24137](https://github.com/apache/incubator-mxnet/commit/7b24137ed45df605defa4ce72ec91554f6e445f0).
 See Instructions in [Setup and 
Installation](http://mxnet.io/install/index.html).
+
+## Exception Handling for Iterators
+
+The below example shows how to handle exceptions for iterators. In this 
example,
+we populate files for data and labels with fewer number of labels compared to 
the
+number of samples. This should throw an exception.
+
+CSVIter uses PrefetcherIter for loading and parsing data.
+The PrefetcherIter spawns a producer thread in the background which prefetches
+the data while the main thread consumes the data. The exception is thrown in 
the spawned
+producer thread during the prefetching, when the label is not found 
corresponding to a specific sample. 
+
+The exception is transported to the main thread, where it is rethrown when 
Next is 
+called as part of the following line: `for batch in iter(data_train)`.
+
+In general, Exception may be rethrown as part of `Next` and `BeforeFirst` 
calls which correspond to `reset()` and `next()` methods in `MXDataIter` for 
Python language bindings.
+
+```python
+import os
+import mxnet as mx
+
+cwd = os.getcwd()
+data_path = os.path.join(cwd, "data.csv")
+label_path = os.path.join(cwd, "label.csv")
+
+with open(data_path, "w") as fout:
+for i in range(8):
+fout.write("1,2,3,4,5,6,7,8,9,10\n")
+
+with open(label_path, "w") as fout:
+for i in range(7):
+fout.write("label"+str(i))
+
+try:
+data_train = mx.io.CSVIter(data_csv=data_path, label_csv=label_path, 
data_shape=(1, 10),
+   batch_size=4)
+
+for batch in iter(data_train):
+print(data_train.getdata().asnumpy())
+except mx.base.MXNetError as ex:
+print("Exception handled")
+print(ex)
+```
+
+### Limitation
+
+There is a race condition when your last `next()` call doesnt reach the batch 
in your dataset where exception occurs. Exception may or may not be thrown in 
this case depending on which thread wins the race. To avoid this situation, you 
should try and iterate through your full dataset if you think it can throw 
exceptions which need to be handled.
+
+
+## Exception Handling for Operators
+
+The below example shows how to handle exceptions for operators in the 
imperative mode.
+
+For the operator case, the dependency engine spawns a number of threads if it 
is running in the `ThreadedEnginePool` or `ThreadedEnginePerDevice` mode. The 
final operator is executed in one of the spawned threads. 
+
+If an operator throws an exception during execution, this exception is 
propagated
+down the dependency chain. Once there is a synchronizing call i.e. WaitToRead 
for a variable in the dependency chain, the propagated exception is rethrown. 
+
+In the below example, I illustrate how an exception that occured in the first 
line is propagated down the dependency chain, and finally is rethrown when we 
make a synchronizing call to WaitToRead.
+
+```python
+import mxnet as mx
+a = mx.nd.random.normal(0, 1, (2, 2))
+b = mx.nd.random.normal(0, 2, (2, 2))
+c = mx.nd.dot(a, b)
+d = mx.nd.random.normal(0, -1, (2, 2))
+e = mx.nd.dot(c, d)

[GitHub] cjolivier01 commented on issue #9996: add softsign operator to v1.1.0

[GitBox]

cjolivier01 commented on issue #9996: add softsign operator to v1.1.0
URL: https://github.com/apache/incubator-mxnet/pull/9996#issuecomment-370546635
 
 
   LGTM <-- before all of the R changes


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] szha commented on issue #9919: Update PR Template

[GitBox]

szha commented on issue #9919: Update PR Template
URL: https://github.com/apache/incubator-mxnet/pull/9919#issuecomment-370607199
 
 
   @eric-haibin-lin would you add an item to remind the committers to check the 
API doc of the PR? See details in #9928. It should read like:
   Committers should check the API doc at 
http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html.
 (e.g. 
[link](http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-9928/25/index.html))


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] aaronmarkham commented on issue #9948: [Doc] Overlapping between footer and the left column + Strange nav-link highlighting

[GitBox]

aaronmarkham commented on issue #9948: [Doc] Overlapping between footer and the 
left column + Strange nav-link highlighting
URL: 
https://github.com/apache/incubator-mxnet/issues/9948#issuecomment-370606641
 
 
   @thinksanky 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-mxnet] branch master updated: host doc on s3 (#9928)

[marcoabreu]

This is an automated email from the ASF dual-hosted git repository.

marcoabreu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet.git


The following commit(s) were added to refs/heads/master by this push:
 new 5984a5b  host doc on s3 (#9928)
5984a5b is described below

commit 5984a5b002bad27e6dece34bf87f01db8bad93c7
Author: Sheng Zha 
AuthorDate: Mon Mar 5 18:33:24 2018 -0500

host doc on s3 (#9928)

* host doc on s3

* update install
---
 Jenkinsfile|  1 +
 tests/ci_build/deploy/aws  | 29 
 tests/ci_build/deploy/ci_deploy_doc.sh | 35 ++
 3 files changed, 65 insertions(+)

diff --git a/Jenkinsfile b/Jenkinsfile
index 78af1cf..d0295f0 100644
--- a/Jenkinsfile
+++ b/Jenkinsfile
@@ -692,6 +692,7 @@ try {
 init_git()
 sh "make clean"
 sh "make docs"
+sh "tests/ci_build/deploy/ci_deploy_doc.sh ${env.BRANCH_NAME} 
${env.BUILD_NUMBER}"
   }
 }
   }
diff --git a/tests/ci_build/deploy/aws b/tests/ci_build/deploy/aws
new file mode 100755
index 000..810a0b9
--- /dev/null
+++ b/tests/ci_build/deploy/aws
@@ -0,0 +1,29 @@
+#!/usr/bin/env python
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import sys
+import os
+
+if os.environ.get('LC_CTYPE', '') == 'UTF-8':
+os.environ['LC_CTYPE'] = 'en_US.UTF-8'
+import awscli.clidriver
+
+main = awscli.clidriver.main
+
+if __name__ == '__main__':
+sys.exit(main())
diff --git a/tests/ci_build/deploy/ci_deploy_doc.sh 
b/tests/ci_build/deploy/ci_deploy_doc.sh
new file mode 100755
index 000..44c8192
--- /dev/null
+++ b/tests/ci_build/deploy/ci_deploy_doc.sh
@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#
+# Execute command outside a docker container
+#
+# Usage: ci_deploy_doc.sh  
+#
+# PR_ID: the PR number
+#
+# BUILD_ID: the current build ID for the specified PR
+#
+
+# TODO szha@: installation of awscli here should be removed once slave hosts 
have them during
+# bootstrap. The following line along with the "aws" script should both be 
removed then.
+pip install --user awscli
+
+tests/ci_build/deploy/aws s3 sync --delete docs/_build/html/ 
s3://mxnet-ci-doc/$1/$2 \
+&& echo "Doc is hosted at 
http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/$1/$2/index.html;

-- 
To stop receiving notification emails like this one, please contact
marcoab...@apache.org.

[GitHub] nswamy opened a new pull request #9999: R fix

[GitBox]

nswamy opened a new pull request #: R fix
URL: https://github.com/apache/incubator-mxnet/pull/
 
 
   ## Description ##
   Cherry picked commits that resolve R builds
   
   
https://github.com/apache/incubator-mxnet/commit/bce00aaca0cb8e69343b5a09da00ef671cad8815
   
https://github.com/apache/incubator-mxnet/commit/f973fe04b88a8045dae3d0fa063d1f3dd8032341
   
   ## Checklist ##
   ### Essentials ###
   - [ ] Passed code style checking (`make lint`)
   - [ ] Changes are complete (i.e. I finished coding on this PR)
   - [ ] All changes have test coverage:
   - Unit tests are added for small changes to verify correctness (e.g. adding 
a new operator)
   - Nightly tests are added for complicated/long-running ones (e.g. changing 
distributed kvstore)
   - Build tests will be added for build configuration changes (e.g. adding a 
new build option with NCCL)
   - [ ] Code is well-documented: 
   - For user-facing API changes, API doc string has been updated. 
   - For new C++ functions in header files, their functionalities and arguments 
are documented. 
   - For new examples, README.md is added to explain the what the example does, 
the source of the dataset, expected performance on test set and reference to 
the original paper if applicable
   - [ ] To the my best knowledge, examples are either not affected by this 
change, or have been fixed to be compatible with this change
   
   ### Changes ###
   - [ ] Feature1, tests, (and when applicable, API doc)
   - [ ] Feature2, tests, (and when applicable, API doc)
   
   ## Comments ##
   - If this change is a backward incompatible change, why must this change be 
made.
   - Interesting edge cases to note here
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] marcoabreu commented on issue #9975: revert DiagrammeR compatibility to 0.9.0

[GitBox]

marcoabreu commented on issue #9975: revert DiagrammeR compatibility to 0.9.0
URL: https://github.com/apache/incubator-mxnet/pull/9975#issuecomment-370600586
 
 
   @jeremiedb could you please apply these changes to branch 1.0.0 and 1.1.0?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ShownX opened a new issue #9998: [Feature Request] batchwise gather_nd/scatter_nd

[GitBox]

ShownX opened a new issue #9998: [Feature Request] batchwise 
gather_nd/scatter_nd
URL: https://github.com/apache/incubator-mxnet/issues/9998
 
 
   ## Description
   Assuming I have a data tensor X and indices tensor Y: X has a shape 
```(batch, N, 3)``` and Y has a shape ```(batch, 1, M)```, I hope there is a 
new operation can do **batch-wise** gather_nd and scatter_nd using the index 
within the batch and gather the data to new tensor ```(batch, M, 3)```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] anirudh2290 commented on issue #9869: Exception handling documentation

[GitBox]

anirudh2290 commented on issue #9869: Exception handling documentation
URL: https://github.com/apache/incubator-mxnet/pull/9869#issuecomment-370583517
 
 
   Unable to reproduce even with the same seed. Trigerring CI again. Opened an 
issue for the flaky test: #9997


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] anirudh2290 commented on issue #9997: Flaky test: test_sparse_embedding

[GitBox]

anirudh2290 commented on issue #9997: Flaky test: test_sparse_embedding
URL: 
https://github.com/apache/incubator-mxnet/issues/9997#issuecomment-370586871
 
 
   @eric-haibin-lin 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] anirudh2290 opened a new issue #9997: Flaky test: test_sparse_embedding

[GitBox]

anirudh2290 opened a new issue #9997: Flaky test: test_sparse_embedding
URL: https://github.com/apache/incubator-mxnet/issues/9997
 
 
   ## Description
   Please see:  
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-9869/13/pipeline
   
   The test_sparse_embedding fails in assert_almost_equal. At first look, it 
feels like setting specific tolerance values should fix the issue. Even in the 
failed example, the values of a and b at the point of error are the same.
   
   ```
   ==
   
   FAIL: test sparse embedding operator
   
   --
   
   Traceback (most recent call last):
   
 File "/usr/local/lib/python2.7/dist-packages/nose/case.py", line 197, in 
runTest
   
   self.test(*self.arg)
   
 File "/usr/local/lib/python2.7/dist-packages/nose/util.py", line 620, in 
newfunc
   
   return func(*arg, **kw)
   
 File "/workspace/tests/python/gpu/../unittest/common.py", line 157, in 
test_new
   
   orig_test(*args, **kwargs)
   
 File "/workspace/tests/python/gpu/../unittest/test_sparse_operator.py", 
line 1675, in test_sparse_embedding
   
   check_sparse_embedding(in_dim, out_dim, batch, densities, False)
   
 File "/workspace/tests/python/gpu/../unittest/test_sparse_operator.py", 
line 1668, in check_sparse_embedding
   
   assert_almost_equal(grad_map["embed_weight"].asnumpy(), 
np.dot(np_onehot.T, grad.asnumpy()))
   
 File "/workspace/python/mxnet/test_utils.py", line 493, in 
assert_almost_equal
   
   raise AssertionError(msg)
   
   AssertionError: 
   
   Items are not equal:
   
   Error 7.891414 exceeds tolerance rtol=0.10, atol=0.00.  Location of 
maximum error:(24, 1), a=0.000378, b=0.000378
   
a: array([[ 0.,  0.,  0.],
   
  [ 0.,  0.,  0.],
   
  [ 0.,  0.,  0.],...
   
b: array([[ 0.,  0.,  0.],
   
  [ 0.,  0.,  0.],
   
  [ 0.,  0.,  0.],...
   
    >> begin captured logging << 
   
   common: INFO: Setting test np/mx/python random seeds, use 
MXNET_TEST_SEED=1475581309 to reproduce.
   
   - >> end captured logging << -
   
   
   ```
   ## Environment info (Required)
   
   Package used (Python/R/Scala/Julia):
   Python
   
   ## Build info (Required if built from source)
   
   Compiler (gcc/clang/mingw/visual studio):gcc
   
   MXNet commit hash: 02dd89a68f659c2a9b0bff62c54c50dff1151f6b
   
   ## Error Message:
   
   ```
   AssertionError: 
   
   Items are not equal:
   
   Error 7.891414 exceeds tolerance rtol=0.10, atol=0.00.  Location of 
maximum error:(24, 1), a=0.000378, b=0.000378
   
a: array([[ 0.,  0.,  0.],
   
  [ 0.,  0.,  0.],
   
  [ 0.,  0.,  0.],...
   
b: array([[ 0.,  0.,  0.],
   
  [ 0.,  0.,  0.],
   
  [ 0.,  0.,  0.],...
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] anirudh2290 commented on issue #9869: Exception handling documentation

[GitBox]

anirudh2290 commented on issue #9869: Exception handling documentation
URL: https://github.com/apache/incubator-mxnet/pull/9869#issuecomment-370583517
 
 
   Unable to reproduce even with the same seed. Trigerring CI again.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] piiswrong commented on issue #9982: Unary ops logcdf_normal, derivlogcdf_normal for log CDF of standard n?

[GitBox]

piiswrong commented on issue #9982: Unary ops logcdf_normal, derivlogcdf_normal 
for log CDF of standard n?
URL: https://github.com/apache/incubator-mxnet/pull/9982#issuecomment-370580786
 
 
   Still not quite sure about the name...


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] anirudh2290 commented on issue #9869: Exception handling documentation

[GitBox]

anirudh2290 commented on issue #9869: Exception handling documentation
URL: https://github.com/apache/incubator-mxnet/pull/9869#issuecomment-370574891
 
 
   test_sparse_embedding_operator seems to be flaky. Trying to reproduce with 
the seed MXNET_TEST_SEED=1475581309. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] piiswrong commented on a change in pull request #9986: gluon language modeling dataset and text token reader

[GitBox]

piiswrong commented on a change in pull request #9986: gluon language modeling 
dataset and text token reader
URL: https://github.com/apache/incubator-mxnet/pull/9986#discussion_r172327467
 
 

 ##
 File path: python/mxnet/gluon/data/datareader.py
 ##
 @@ -0,0 +1,33 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+# pylint: disable=
+"""Dataset reader."""
+__all__ = ['DataReader']
+
+class DataReader(object):
 
 Review comment:
   It depends on what you want it to do.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] szha commented on a change in pull request #9986: gluon language modeling dataset and text token reader

[GitBox]

szha commented on a change in pull request #9986: gluon language modeling 
dataset and text token reader
URL: https://github.com/apache/incubator-mxnet/pull/9986#discussion_r172323412
 
 

 ##
 File path: python/mxnet/gluon/data/datareader.py
 ##
 @@ -0,0 +1,33 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+# pylint: disable=
+"""Dataset reader."""
+__all__ = ['DataReader']
+
+class DataReader(object):
 
 Review comment:
   Shall we make a data reader a generator? (i.e. define `__iter__`)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] szha commented on a change in pull request #9986: gluon language modeling dataset and text token reader

[GitBox]

szha commented on a change in pull request #9986: gluon language modeling 
dataset and text token reader
URL: https://github.com/apache/incubator-mxnet/pull/9986#discussion_r172315558
 
 

 ##
 File path: python/mxnet/gluon/data/text/_constants.py
 ##
 @@ -0,0 +1,26 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+
+"""Constants relevant to text processing."""
+
+UNK_TOKEN = ''
+
+BOS_TOKEN = ''
 
 Review comment:
   > Isn't this usually ``?
   
   I see more use of ``, though honestly it would just be a preference, as 
long as we can agree on one.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] szha commented on a change in pull request #9986: gluon language modeling dataset and text token reader

[GitBox]

szha commented on a change in pull request #9986: gluon language modeling 
dataset and text token reader
URL: https://github.com/apache/incubator-mxnet/pull/9986#discussion_r172317063
 
 

 ##
 File path: python/mxnet/gluon/data/text/lm.py
 ##
 @@ -0,0 +1,157 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+# pylint: disable=
+"""Language model datasets."""
+
+__all__ = ['LanguageModelDataset', 'WikiText2', 'WikiText103']
+
+import os
+import zipfile
+import shutil
+
+from .base import TextDataset, TextTokenReader
+from . import _constants as C
+from ...utils import download, check_sha1, get_repo_file_url
+
+
+class LanguageModelDataset(TextDataset):
 
 Review comment:
   It's common for language modeling to preprocess arbitrary corpus into 
sequences of fixed number of tokens (which I call `collate` here). I put this 
preprocessing step in LangaugeModelDataset as it's not common for other 
datasets.
   
   An alternative is to treat such preprocessing function as util function. 
Would that be preferred?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] szha commented on a change in pull request #9986: gluon language modeling dataset and text token reader

[GitBox]

szha commented on a change in pull request #9986: gluon language modeling 
dataset and text token reader
URL: https://github.com/apache/incubator-mxnet/pull/9986#discussion_r172316324
 
 

 ##
 File path: python/mxnet/gluon/data/text/base.py
 ##
 @@ -0,0 +1,89 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+# pylint: disable=
+
+"""Base classes for text datasets and readers."""
+
+__all__ = ['TextTokenReader']
+
+import io
+import os
+
+from ..dataset import Dataset
+from ..datareader import DataReader
+
+class TextTokenReader(DataReader):
+"""Text reader that produces lists of tokens.
+
+Parameters
+--
+filename : str
+Path to the input text file.
+encoding : str, default 'utf8'
+File encoding format.
+sample_splitter : function, default str.splitlines
+A function that splits the dataset string into samples.
+tokenizer : function, default str.split
+A function that splits each sample string into list of tokens.
+bos : str or None, default None
+The token to add at the begining of each sentence. If None, nothing is 
added.
+eos : str or None, default None
+The token to add at the end of each sentence. If None, nothing is 
added.
+"""
+def __init__(self, filename, encoding='utf8', sample_splitter=lambda s: 
s.splitlines(),
+ tokenizer=lambda s: s.split(), bos=None, eos=None):
+filename = os.path.expanduser(filename)
+with io.open(filename, 'r', encoding=encoding) as fin:
+content = fin.read()
+
+samples = [s for s in [tokenizer(x) for x in sample_splitter(content)] 
if s]
+if bos or eos:
+for tokens in samples:
+if bos:
+tokens.insert(0, bos)
+if eos:
+tokens.append(eos)
+
+self._samples = samples
+
+def __len__(self):
+return self._samples.__len__
+
+def __getitem__(self, idx):
+return self._samples[idx]
+
+class TextDataset(Dataset):
+"""Abstract dataset class for text data.
+
+Subclasses need to override `__getitem__`, which returns the i-th
+element, and `__len__`, which returns the total number elements."""
+
+def __getitem__(self, idx):
+raise NotImplementedError
+
+def __len__(self):
+raise NotImplementedError
+
+def _flatten(self, samples):
 
 Review comment:
   I will organize these as utils and get rid of TextDataset then.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] szha commented on a change in pull request #9986: gluon language modeling dataset and text token reader

[GitBox]

szha commented on a change in pull request #9986: gluon language modeling 
dataset and text token reader
URL: https://github.com/apache/incubator-mxnet/pull/9986#discussion_r172316070
 
 

 ##
 File path: python/mxnet/gluon/data/text/base.py
 ##
 @@ -0,0 +1,89 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+# pylint: disable=
+
+"""Base classes for text datasets and readers."""
+
+__all__ = ['TextTokenReader']
+
+import io
+import os
+
+from ..dataset import Dataset
+from ..datareader import DataReader
+
+class TextTokenReader(DataReader):
+"""Text reader that produces lists of tokens.
 
 Review comment:
   For this dataset, word tokenization happens in the data reader because the 
result of tokenization could decide the sample boundary. It's also possible to 
defer the chunking-by-length to the dataset class. I will change it to the 
later and see how it is.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] szha commented on a change in pull request #9986: gluon language modeling dataset and text token reader

[GitBox]

szha commented on a change in pull request #9986: gluon language modeling 
dataset and text token reader
URL: https://github.com/apache/incubator-mxnet/pull/9986#discussion_r172315558
 
 

 ##
 File path: python/mxnet/gluon/data/text/_constants.py
 ##
 @@ -0,0 +1,26 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+
+"""Constants relevant to text processing."""
+
+UNK_TOKEN = ''
+
+BOS_TOKEN = ''
 
 Review comment:
   > Isn't this usually ``?
   I see more use of ``, though honestly it would just be a preference, as 
long as we can agree on one.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] szha commented on a change in pull request #9986: gluon language modeling dataset and text token reader

[GitBox]

szha commented on a change in pull request #9986: gluon language modeling 
dataset and text token reader
URL: https://github.com/apache/incubator-mxnet/pull/9986#discussion_r172314994
 
 

 ##
 File path: python/mxnet/gluon/data/text/__init__.py
 ##
 @@ -0,0 +1,24 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+# pylint: disable=wildcard-import
+"""Text utilities."""
+
+from .base import *
+
+from . import lm
 
 Review comment:
   I thought it would be a good idea to separate the datasets by application. 
The reason is that some dataset names are common among more than one tasks, 
such as semeval datasets (e.g. semeval 2017 dataset consists of three tasks on 
the same set of data).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] szha commented on a change in pull request #9986: gluon language modeling dataset and text token reader

[GitBox]

szha commented on a change in pull request #9986: gluon language modeling 
dataset and text token reader
URL: https://github.com/apache/incubator-mxnet/pull/9986#discussion_r172314482
 
 

 ##
 File path: python/mxnet/gluon/data/datareader.py
 ##
 @@ -0,0 +1,33 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+# pylint: disable=
+"""Dataset reader."""
+__all__ = ['DataReader']
+
+class DataReader(object):
 
 Review comment:
   (e.g. SemEval2017 reader can read the dataset, while different dataset 
classes can treat them as datasets for different tasks, such as POS tagging and 
keyphrase extraction)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] szha commented on a change in pull request #9986: gluon language modeling dataset and text token reader

[GitBox]

szha commented on a change in pull request #9986: gluon language modeling 
dataset and text token reader
URL: https://github.com/apache/incubator-mxnet/pull/9986#discussion_r172314053
 
 

 ##
 File path: python/mxnet/gluon/data/datareader.py
 ##
 @@ -0,0 +1,33 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+# pylint: disable=
+"""Dataset reader."""
+__all__ = ['DataReader']
+
+class DataReader(object):
 
 Review comment:
   OK. At first I separated those classes by division of concern instead of 
interface. I agree that it would be better if the interfaces differ too.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] szha commented on issue #9986: gluon language modeling dataset and text token reader

[GitBox]

szha commented on issue #9986: gluon language modeling dataset and text token 
reader
URL: https://github.com/apache/incubator-mxnet/pull/9986#issuecomment-370548139
 
 
   Sounds good. I created a branch called `nlp_toolkit`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] hetong007 commented on issue #9930: Support single array input for metric

[GitBox]

hetong007 commented on issue #9930: Support single array input for metric
URL: https://github.com/apache/incubator-mxnet/pull/9930#issuecomment-370547926
 
 
   @piiswrong The for loop is fixed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] cjolivier01 commented on issue #9996: add softsign operator to v1.1.0

[GitBox]

cjolivier01 commented on issue #9996: add softsign operator to v1.1.0
URL: https://github.com/apache/incubator-mxnet/pull/9996#issuecomment-370546635
 
 
   LGTM


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] nswamy opened a new pull request #9996: add softsign operator to v1.1.0

[GitBox]

nswamy opened a new pull request #9996: add softsign operator to v1.1.0
URL: https://github.com/apache/incubator-mxnet/pull/9996
 
 
   ## Description ##
   One of the customer is upgrading MXNet from v0.11.0 to v1.1.0 and would like 
to this operator to be ported on v1.1.0 on the master branch
   
   ## Checklist ##
   ### Essentials ###
   - [ ] Passed code style checking (`make lint`)
   - [ ] Changes are complete (i.e. I finished coding on this PR)
   - [ ] All changes have test coverage:
   - Unit tests are added for small changes to verify correctness (e.g. adding 
a new operator)
   - Nightly tests are added for complicated/long-running ones (e.g. changing 
distributed kvstore)
   - Build tests will be added for build configuration changes (e.g. adding a 
new build option with NCCL)
   - [ ] Code is well-documented: 
   - For user-facing API changes, API doc string has been updated. 
   - For new C++ functions in header files, their functionalities and arguments 
are documented. 
   - For new examples, README.md is added to explain the what the example does, 
the source of the dataset, expected performance on test set and reference to 
the original paper if applicable
   - [ ] To the my best knowledge, examples are either not affected by this 
change, or have been fixed to be compatible with this change
   
   ### Changes ###
   - [ ] Feature1, tests, (and when applicable, API doc)
   - [ ] Feature2, tests, (and when applicable, API doc)
   
   ## Comments ##
   - If this change is a backward incompatible change, why must this change be 
made.
   - Interesting edge cases to note here
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[incubator-mxnet-site] 01/01: Merge pull request #56 from thinksanky/touched_file_to_refresh_site

[liuyizhi]

This is an automated email from the ASF dual-hosted git repository.

liuyizhi pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet-site.git

commit 70839430eb9823e531bf12f87d1f9a6b8ee2850d
Merge: aec9cfa 81517a4
Author: Yizhi Liu 
AuthorDate: Mon Mar 5 11:46:46 2018 -0800

Merge pull request #56 from thinksanky/touched_file_to_refresh_site

touched faq bucketing.html to trigger a refresh

 faq/bucketing.html | 1 +
 1 file changed, 1 insertion(+)

-- 
To stop receiving notification emails like this one, please contact
liuyi...@apache.org.

[incubator-mxnet-site] branch asf-site updated (aec9cfa -> 7083943)

[liuyizhi]

This is an automated email from the ASF dual-hosted git repository.

liuyizhi pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/incubator-mxnet-site.git.


from aec9cfa  Merge pull request #55 from thinksanky/fix_how_to_dir
 add 81517a4  touched faq bucketing.html to trigger a refresh
 new 7083943  Merge pull request #56 from 
thinksanky/touched_file_to_refresh_site

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 faq/bucketing.html | 1 +
 1 file changed, 1 insertion(+)

-- 
To stop receiving notification emails like this one, please contact
liuyi...@apache.org.

[GitHub] yzhliu closed pull request #56: touched faq bucketing.html to trigger a refresh

[GitBox]

yzhliu closed pull request #56: touched faq bucketing.html to trigger a refresh
URL: https://github.com/apache/incubator-mxnet-site/pull/56
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/faq/bucketing.html b/faq/bucketing.html
index 856e62214..a2ab50d32 100644
--- a/faq/bucketing.html
+++ b/faq/bucketing.html
@@ -3,6 +3,7 @@
 
 
 
+
 
 
 Bucketing in MXNet ? mxnet  documentation


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] thinksanky opened a new pull request #56: touched faq bucketing.html to trigger a refresh

[GitBox]

thinksanky opened a new pull request #56: touched faq bucketing.html to trigger 
a refresh
URL: https://github.com/apache/incubator-mxnet-site/pull/56
 
 
   ## Description ##
   Touched a file by adding a comment just to trigger web-site refresh
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] anirudhacharya commented on a change in pull request #9963: Onnx Module to import onnx models into mxnet

[GitBox]

anirudhacharya commented on a change in pull request #9963: Onnx Module to 
import onnx models into mxnet
URL: https://github.com/apache/incubator-mxnet/pull/9963#discussion_r172301381
 
 

 ##
 File path: python/mxnet/contrib/onnx/_import/translation_utils/__init__.py
 ##
 @@ -0,0 +1,141 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+# pylint: 
disable=invalid-name,no-self-use,too-many-branches,too-few-public-methods,too-many-arguments
+from __future__ import absolute_import as _abs
+from base import string_types
+
+def _fix_attribute_names(attrs, change_map):
+new_attr = {}
+for k in attrs.keys():
+if k in change_map:
+new_attr[change_map[k]] = attrs[k]
+else:
+new_attr[k] = attrs[k]
+return new_attr
+
+def _fix_pooling(op_name, inputs, new_attr):
+"""onnx pooling operator supports asymmetrical padding
+Adding pad operator before pooling in mxnet to work with onnx"""
+pool_type = 'avg' if op_name == 'AveragePool' else 'max'
+stride = new_attr.get('strides')
+kernel = new_attr.get('kernel_shape')
+padding = new_attr.get('pads')
+pad_width = (0, 0, 0, 0) + _pad_sequence_fix(padding, len(kernel))
+new_pad_op = symbol.pad(inputs[0], mode='constant', pad_width=pad_width)
+new_pooling_op = symbol.Pooling(new_pad_op, pool_type=pool_type,
+stride=stride, kernel=kernel)
+return new_pooling_op
+
+def _fix_slice(inputs, new_attr):
+"""onnx slice provides slicing on multiple axis. Adding multiple 
slice_axis operator
+for multiple axes from mxnet"""
+begin = new_attr.get('begin')
+end = new_attr.get('end')
+axes = new_attr.get('axis', tuple(range(len(begin
+slice_op = symbol.slice_axis(inputs[0], axis=axes[0], begin=begin[0], 
end=end[0])
+if len(axes) > 1:
+for i, axis in enumerate(axes):
+slice_op = symbol.slice_axis(slice_op, axis=axis, begin=begin[i], 
end=end[i])
+return slice_op
+
+def _fix_squeeze(inputs, new_attr):
+"""
+MXNet doesnt have a squeeze operator.
+Using "split" to perform similar operation.
+"split" can be slower compared to "reshape".
+ This can have performance impact.
+ TODO: Remove this implementation once mxnet adds the support.
 
 Review comment:
   will create an issue and add it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] anirudhacharya commented on a change in pull request #9963: Onnx Module to import onnx models into mxnet

[GitBox]

anirudhacharya commented on a change in pull request #9963: Onnx Module to 
import onnx models into mxnet
URL: https://github.com/apache/incubator-mxnet/pull/9963#discussion_r172301079
 
 

 ##
 File path: python/mxnet/contrib/onnx/_import/import_onnx.py
 ##
 @@ -0,0 +1,166 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+# pylint: disable=invalid-name,too-many-locals,no-self-use
+""" Support import export formats."""
+from __future__ import absolute_import as _abs
+from  import symbol
+from  import ndarray as nd
+from .import_helper import _convert_map, _pad_sequence_fix
+
+def _convert_operator(op_name, attrs, inputs, convert_map=None):
+"""Convert from onnx operator to mxnet operator.
+The converter must specify conversions explicitly for incompatible name, 
and
+apply handlers to operator attributes.
+
+Parameters
+--
+op_name : str
+Operator name, such as Convolution, FullyConnected
+attrs : dict
+Dict of operator attributes
+inputs: list
+list of inputs to the operator
+convert_map : dict
+Dict of name : callable, where name is the op's name that
+require conversion to mxnet, callable are functions which
+take attrs and return (new_op_name, new_attrs, inputs)
+
+Returns
+---
+(op_name, attrs)
+Converted (op_name, attrs) for mxnet.
+"""
+convert_map = convert_map if convert_map else _convert_map
+if op_name in convert_map:
+op_name, attrs, inputs = convert_map[op_name](op_name, attrs, inputs)
+else:
+raise NotImplementedError("Operator {} not 
implemented.".format(op_name))
+op = getattr(symbol, op_name, None)
+if not op:
+raise RuntimeError("Unable to map op_name {} to sym".format(op_name))
+return op, attrs, inputs
+
+class GraphProto(object): # pylint: disable=too-few-public-methods
+"""A helper class for handling mxnet symbol copying from pb2.GraphProto.
+Definition: https://github.com/onnx/onnx/blob/master/onnx/onnx.proto
+"""
+def __init__(self):
+self._nodes = {}
+self._params = {}
+self._renames = {}
+self._num_input = 0
+self._num_param = 0
+
+def from_onnx(self, graph):
+"""Construct symbol from onnx graph.
+The inputs from onnx graph is vague, only providing "1", "2"...
+For convenience, we rename the `real` input names to "input_0",
+"input_1"... And renaming parameters to "param_0", "param_1"...
+
+Parameters
+--
+graph : onnx protobuf object
+The loaded onnx graph
+
+Returns
+---
+sym :symbol.Symbol
+The returned mxnet symbol
+params : dict
+A dict of name: nd.array pairs, used as pretrained weights
+"""
+# parse network inputs, aka parameters
+for init_tensor in graph.initializer:
+if not init_tensor.name.strip():
+raise ValueError("Tensor's name is required.")
+self._params[init_tensor.name] = self._parse_array(init_tensor)
+
+# converting GraphProto message
+for i in graph.input:
+if i.name in self._params:
+# i is a param instead of input
+name_param = 'param_{}'.format(self._num_param)
+self._num_param += 1
+self._params[name_param] = self._params.pop(i.name)
+self._nodes[name_param] = symbol.Variable(name=name_param,
+  
shape=self._params[name_param].shape)
+self._renames[i.name] = name_param
+else:
+name_input = 'input_{}'.format(self._num_input)
+self._num_input += 1
+self._nodes[name_input] = symbol.Variable(name=name_input)
+self._renames[i.name] = name_input
+
+# constructing nodes, nodes are stored as directed acyclic graph
+# converting NodeProto message
+for node in graph.node:
+op_name = node.op_type
+node_name = node.name.strip()
+node_name = node_name if 

[GitHub] anirudhacharya commented on a change in pull request #9963: Onnx Module to import onnx models into mxnet

[GitBox]

anirudhacharya commented on a change in pull request #9963: Onnx Module to 
import onnx models into mxnet
URL: https://github.com/apache/incubator-mxnet/pull/9963#discussion_r172300946
 
 

 ##
 File path: python/mxnet/contrib/onnx/_import/import_onnx.py
 ##
 @@ -0,0 +1,166 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+# pylint: disable=invalid-name,too-many-locals,no-self-use
+""" Support import export formats."""
+from __future__ import absolute_import as _abs
+from  import symbol
+from  import ndarray as nd
+from .import_helper import _convert_map, _pad_sequence_fix
+
+def _convert_operator(op_name, attrs, inputs, convert_map=None):
+"""Convert from onnx operator to mxnet operator.
+The converter must specify conversions explicitly for incompatible name, 
and
+apply handlers to operator attributes.
+
+Parameters
+--
+op_name : str
+Operator name, such as Convolution, FullyConnected
+attrs : dict
+Dict of operator attributes
+inputs: list
+list of inputs to the operator
+convert_map : dict
+Dict of name : callable, where name is the op's name that
+require conversion to mxnet, callable are functions which
+take attrs and return (new_op_name, new_attrs, inputs)
+
+Returns
+---
+(op_name, attrs)
+Converted (op_name, attrs) for mxnet.
+"""
+convert_map = convert_map if convert_map else _convert_map
+if op_name in convert_map:
+op_name, attrs, inputs = convert_map[op_name](op_name, attrs, inputs)
+else:
+raise NotImplementedError("Operator {} not 
implemented.".format(op_name))
+op = getattr(symbol, op_name, None)
+if not op:
+raise RuntimeError("Unable to map op_name {} to sym".format(op_name))
+return op, attrs, inputs
+
+class GraphProto(object): # pylint: disable=too-few-public-methods
+"""A helper class for handling mxnet symbol copying from pb2.GraphProto.
+Definition: https://github.com/onnx/onnx/blob/master/onnx/onnx.proto
+"""
+def __init__(self):
+self._nodes = {}
+self._params = {}
+self._renames = {}
+self._num_input = 0
+self._num_param = 0
+
+def from_onnx(self, graph):
+"""Construct symbol from onnx graph.
+The inputs from onnx graph is vague, only providing "1", "2"...
+For convenience, we rename the `real` input names to "input_0",
+"input_1"... And renaming parameters to "param_0", "param_1"...
+
+Parameters
+--
+graph : onnx protobuf object
+The loaded onnx graph
+
+Returns
+---
+sym :symbol.Symbol
+The returned mxnet symbol
+params : dict
+A dict of name: nd.array pairs, used as pretrained weights
+"""
+# parse network inputs, aka parameters
+for init_tensor in graph.initializer:
+if not init_tensor.name.strip():
+raise ValueError("Tensor's name is required.")
+self._params[init_tensor.name] = self._parse_array(init_tensor)
+
+# converting GraphProto message
+for i in graph.input:
+if i.name in self._params:
+# i is a param instead of input
+name_param = 'param_{}'.format(self._num_param)
+self._num_param += 1
+self._params[name_param] = self._params.pop(i.name)
+self._nodes[name_param] = symbol.Variable(name=name_param,
+  
shape=self._params[name_param].shape)
+self._renames[i.name] = name_param
+else:
+name_input = 'input_{}'.format(self._num_input)
+self._num_input += 1
+self._nodes[name_input] = symbol.Variable(name=name_input)
+self._renames[i.name] = name_input
+
+# constructing nodes, nodes are stored as directed acyclic graph
+# converting NodeProto message
+for node in graph.node:
+op_name = node.op_type
+node_name = node.name.strip()
+node_name = node_name if 

[GitHub] anirudhacharya commented on a change in pull request #9963: Onnx Module to import onnx models into mxnet

[GitBox]

anirudhacharya commented on a change in pull request #9963: Onnx Module to 
import onnx models into mxnet
URL: https://github.com/apache/incubator-mxnet/pull/9963#discussion_r172300559
 
 

 ##
 File path: python/mxnet/contrib/onnx/_import/import_onnx.py
 ##
 @@ -0,0 +1,166 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+# pylint: disable=invalid-name,too-many-locals,no-self-use
+""" Support import export formats."""
+from __future__ import absolute_import as _abs
+from  import symbol
+from  import ndarray as nd
+from .import_helper import _convert_map, _pad_sequence_fix
+
+def _convert_operator(op_name, attrs, inputs, convert_map=None):
+"""Convert from onnx operator to mxnet operator.
+The converter must specify conversions explicitly for incompatible name, 
and
+apply handlers to operator attributes.
+
+Parameters
+--
+op_name : str
+Operator name, such as Convolution, FullyConnected
+attrs : dict
+Dict of operator attributes
+inputs: list
+list of inputs to the operator
+convert_map : dict
+Dict of name : callable, where name is the op's name that
+require conversion to mxnet, callable are functions which
+take attrs and return (new_op_name, new_attrs, inputs)
+
+Returns
+---
+(op_name, attrs)
+Converted (op_name, attrs) for mxnet.
+"""
+convert_map = convert_map if convert_map else _convert_map
+if op_name in convert_map:
+op_name, attrs, inputs = convert_map[op_name](op_name, attrs, inputs)
+else:
+raise NotImplementedError("Operator {} not 
implemented.".format(op_name))
+op = getattr(symbol, op_name, None)
+if not op:
+raise RuntimeError("Unable to map op_name {} to sym".format(op_name))
+return op, attrs, inputs
+
+class GraphProto(object): # pylint: disable=too-few-public-methods
+"""A helper class for handling mxnet symbol copying from pb2.GraphProto.
+Definition: https://github.com/onnx/onnx/blob/master/onnx/onnx.proto
+"""
+def __init__(self):
+self._nodes = {}
+self._params = {}
+self._renames = {}
+self._num_input = 0
+self._num_param = 0
+
+def from_onnx(self, graph):
+"""Construct symbol from onnx graph.
+The inputs from onnx graph is vague, only providing "1", "2"...
+For convenience, we rename the `real` input names to "input_0",
+"input_1"... And renaming parameters to "param_0", "param_1"...
+
+Parameters
+--
+graph : onnx protobuf object
+The loaded onnx graph
+
+Returns
+---
+sym :symbol.Symbol
+The returned mxnet symbol
+params : dict
+A dict of name: nd.array pairs, used as pretrained weights
+"""
+# parse network inputs, aka parameters
+for init_tensor in graph.initializer:
+if not init_tensor.name.strip():
+raise ValueError("Tensor's name is required.")
+self._params[init_tensor.name] = self._parse_array(init_tensor)
+
+# converting GraphProto message
+for i in graph.input:
+if i.name in self._params:
+# i is a param instead of input
+name_param = 'param_{}'.format(self._num_param)
+self._num_param += 1
+self._params[name_param] = self._params.pop(i.name)
+self._nodes[name_param] = symbol.Variable(name=name_param,
 
 Review comment:
   okay. will make the change.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] anirudhacharya commented on a change in pull request #9963: Onnx Module to import onnx models into mxnet

[GitBox]

anirudhacharya commented on a change in pull request #9963: Onnx Module to 
import onnx models into mxnet
URL: https://github.com/apache/incubator-mxnet/pull/9963#discussion_r172300457
 
 

 ##
 File path: python/mxnet/contrib/onnx/_import/import_helper.py
 ##
 @@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+# pylint: disable=invalid-name
+"""Operator attributes conversion"""
+from . import *
 
 Review comment:
   will do.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sandeep-krishnamurthy commented on a change in pull request #9963: Onnx Module to import onnx models into mxnet

[GitBox]

sandeep-krishnamurthy commented on a change in pull request #9963: Onnx Module 
to import onnx models into mxnet
URL: https://github.com/apache/incubator-mxnet/pull/9963#discussion_r172272632
 
 

 ##
 File path: python/mxnet/contrib/onnx/_import/import_onnx.py
 ##
 @@ -0,0 +1,166 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+# pylint: disable=invalid-name,too-many-locals,no-self-use
+""" Support import export formats."""
+from __future__ import absolute_import as _abs
+from  import symbol
+from  import ndarray as nd
+from .import_helper import _convert_map, _pad_sequence_fix
+
+def _convert_operator(op_name, attrs, inputs, convert_map=None):
+"""Convert from onnx operator to mxnet operator.
+The converter must specify conversions explicitly for incompatible name, 
and
+apply handlers to operator attributes.
+
+Parameters
+--
+op_name : str
+Operator name, such as Convolution, FullyConnected
+attrs : dict
+Dict of operator attributes
+inputs: list
+list of inputs to the operator
+convert_map : dict
+Dict of name : callable, where name is the op's name that
+require conversion to mxnet, callable are functions which
+take attrs and return (new_op_name, new_attrs, inputs)
+
+Returns
+---
+(op_name, attrs)
+Converted (op_name, attrs) for mxnet.
+"""
+convert_map = convert_map if convert_map else _convert_map
+if op_name in convert_map:
+op_name, attrs, inputs = convert_map[op_name](op_name, attrs, inputs)
+else:
+raise NotImplementedError("Operator {} not 
implemented.".format(op_name))
+op = getattr(symbol, op_name, None)
+if not op:
+raise RuntimeError("Unable to map op_name {} to sym".format(op_name))
+return op, attrs, inputs
+
+class GraphProto(object): # pylint: disable=too-few-public-methods
+"""A helper class for handling mxnet symbol copying from pb2.GraphProto.
+Definition: https://github.com/onnx/onnx/blob/master/onnx/onnx.proto
+"""
+def __init__(self):
+self._nodes = {}
+self._params = {}
+self._renames = {}
+self._num_input = 0
+self._num_param = 0
+
+def from_onnx(self, graph):
+"""Construct symbol from onnx graph.
+The inputs from onnx graph is vague, only providing "1", "2"...
+For convenience, we rename the `real` input names to "input_0",
+"input_1"... And renaming parameters to "param_0", "param_1"...
+
+Parameters
+--
+graph : onnx protobuf object
+The loaded onnx graph
+
+Returns
+---
+sym :symbol.Symbol
+The returned mxnet symbol
+params : dict
+A dict of name: nd.array pairs, used as pretrained weights
+"""
+# parse network inputs, aka parameters
+for init_tensor in graph.initializer:
+if not init_tensor.name.strip():
+raise ValueError("Tensor's name is required.")
+self._params[init_tensor.name] = self._parse_array(init_tensor)
+
+# converting GraphProto message
+for i in graph.input:
+if i.name in self._params:
+# i is a param instead of input
+name_param = 'param_{}'.format(self._num_param)
+self._num_param += 1
+self._params[name_param] = self._params.pop(i.name)
+self._nodes[name_param] = symbol.Variable(name=name_param,
+  
shape=self._params[name_param].shape)
+self._renames[i.name] = name_param
+else:
+name_input = 'input_{}'.format(self._num_input)
+self._num_input += 1
+self._nodes[name_input] = symbol.Variable(name=name_input)
+self._renames[i.name] = name_input
+
+# constructing nodes, nodes are stored as directed acyclic graph
+# converting NodeProto message
+for node in graph.node:
+op_name = node.op_type
+node_name = node.name.strip()
+node_name = 

[GitHub] sandeep-krishnamurthy commented on a change in pull request #9963: Onnx Module to import onnx models into mxnet

[GitBox]

sandeep-krishnamurthy commented on a change in pull request #9963: Onnx Module 
to import onnx models into mxnet
URL: https://github.com/apache/incubator-mxnet/pull/9963#discussion_r172272317
 
 

 ##
 File path: python/mxnet/contrib/onnx/_import/import_onnx.py
 ##
 @@ -0,0 +1,166 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+# pylint: disable=invalid-name,too-many-locals,no-self-use
+""" Support import export formats."""
+from __future__ import absolute_import as _abs
+from  import symbol
+from  import ndarray as nd
+from .import_helper import _convert_map, _pad_sequence_fix
+
+def _convert_operator(op_name, attrs, inputs, convert_map=None):
+"""Convert from onnx operator to mxnet operator.
+The converter must specify conversions explicitly for incompatible name, 
and
+apply handlers to operator attributes.
+
+Parameters
+--
+op_name : str
+Operator name, such as Convolution, FullyConnected
+attrs : dict
+Dict of operator attributes
+inputs: list
+list of inputs to the operator
+convert_map : dict
+Dict of name : callable, where name is the op's name that
+require conversion to mxnet, callable are functions which
+take attrs and return (new_op_name, new_attrs, inputs)
+
+Returns
+---
+(op_name, attrs)
+Converted (op_name, attrs) for mxnet.
+"""
+convert_map = convert_map if convert_map else _convert_map
+if op_name in convert_map:
+op_name, attrs, inputs = convert_map[op_name](op_name, attrs, inputs)
+else:
+raise NotImplementedError("Operator {} not 
implemented.".format(op_name))
+op = getattr(symbol, op_name, None)
+if not op:
+raise RuntimeError("Unable to map op_name {} to sym".format(op_name))
+return op, attrs, inputs
+
+class GraphProto(object): # pylint: disable=too-few-public-methods
+"""A helper class for handling mxnet symbol copying from pb2.GraphProto.
+Definition: https://github.com/onnx/onnx/blob/master/onnx/onnx.proto
+"""
+def __init__(self):
+self._nodes = {}
+self._params = {}
+self._renames = {}
+self._num_input = 0
+self._num_param = 0
+
+def from_onnx(self, graph):
+"""Construct symbol from onnx graph.
+The inputs from onnx graph is vague, only providing "1", "2"...
+For convenience, we rename the `real` input names to "input_0",
+"input_1"... And renaming parameters to "param_0", "param_1"...
+
+Parameters
+--
+graph : onnx protobuf object
+The loaded onnx graph
+
+Returns
+---
+sym :symbol.Symbol
+The returned mxnet symbol
+params : dict
+A dict of name: nd.array pairs, used as pretrained weights
+"""
+# parse network inputs, aka parameters
+for init_tensor in graph.initializer:
+if not init_tensor.name.strip():
+raise ValueError("Tensor's name is required.")
+self._params[init_tensor.name] = self._parse_array(init_tensor)
+
+# converting GraphProto message
+for i in graph.input:
+if i.name in self._params:
+# i is a param instead of input
+name_param = 'param_{}'.format(self._num_param)
+self._num_param += 1
+self._params[name_param] = self._params.pop(i.name)
+self._nodes[name_param] = symbol.Variable(name=name_param,
+  
shape=self._params[name_param].shape)
+self._renames[i.name] = name_param
+else:
+name_input = 'input_{}'.format(self._num_input)
+self._num_input += 1
+self._nodes[name_input] = symbol.Variable(name=name_input)
+self._renames[i.name] = name_input
+
+# constructing nodes, nodes are stored as directed acyclic graph
+# converting NodeProto message
+for node in graph.node:
+op_name = node.op_type
+node_name = node.name.strip()
+node_name = 

[GitHub] sandeep-krishnamurthy commented on a change in pull request #9963: Onnx Module to import onnx models into mxnet

[GitBox]

sandeep-krishnamurthy commented on a change in pull request #9963: Onnx Module 
to import onnx models into mxnet
URL: https://github.com/apache/incubator-mxnet/pull/9963#discussion_r172270960
 
 

 ##
 File path: python/mxnet/contrib/onnx/_import/import_onnx.py
 ##
 @@ -0,0 +1,166 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+# pylint: disable=invalid-name,too-many-locals,no-self-use
+""" Support import export formats."""
+from __future__ import absolute_import as _abs
+from  import symbol
+from  import ndarray as nd
+from .import_helper import _convert_map, _pad_sequence_fix
+
+def _convert_operator(op_name, attrs, inputs, convert_map=None):
+"""Convert from onnx operator to mxnet operator.
+The converter must specify conversions explicitly for incompatible name, 
and
+apply handlers to operator attributes.
+
+Parameters
+--
+op_name : str
+Operator name, such as Convolution, FullyConnected
+attrs : dict
+Dict of operator attributes
+inputs: list
+list of inputs to the operator
+convert_map : dict
+Dict of name : callable, where name is the op's name that
+require conversion to mxnet, callable are functions which
+take attrs and return (new_op_name, new_attrs, inputs)
+
+Returns
+---
+(op_name, attrs)
+Converted (op_name, attrs) for mxnet.
+"""
+convert_map = convert_map if convert_map else _convert_map
+if op_name in convert_map:
+op_name, attrs, inputs = convert_map[op_name](op_name, attrs, inputs)
+else:
+raise NotImplementedError("Operator {} not 
implemented.".format(op_name))
+op = getattr(symbol, op_name, None)
+if not op:
+raise RuntimeError("Unable to map op_name {} to sym".format(op_name))
+return op, attrs, inputs
+
+class GraphProto(object): # pylint: disable=too-few-public-methods
+"""A helper class for handling mxnet symbol copying from pb2.GraphProto.
+Definition: https://github.com/onnx/onnx/blob/master/onnx/onnx.proto
+"""
+def __init__(self):
+self._nodes = {}
+self._params = {}
+self._renames = {}
+self._num_input = 0
+self._num_param = 0
+
+def from_onnx(self, graph):
+"""Construct symbol from onnx graph.
+The inputs from onnx graph is vague, only providing "1", "2"...
+For convenience, we rename the `real` input names to "input_0",
+"input_1"... And renaming parameters to "param_0", "param_1"...
+
+Parameters
+--
+graph : onnx protobuf object
+The loaded onnx graph
+
+Returns
+---
+sym :symbol.Symbol
+The returned mxnet symbol
+params : dict
+A dict of name: nd.array pairs, used as pretrained weights
+"""
+# parse network inputs, aka parameters
+for init_tensor in graph.initializer:
+if not init_tensor.name.strip():
+raise ValueError("Tensor's name is required.")
+self._params[init_tensor.name] = self._parse_array(init_tensor)
+
+# converting GraphProto message
+for i in graph.input:
+if i.name in self._params:
+# i is a param instead of input
+name_param = 'param_{}'.format(self._num_param)
+self._num_param += 1
+self._params[name_param] = self._params.pop(i.name)
+self._nodes[name_param] = symbol.Variable(name=name_param,
 
 Review comment:
   Do we have dtype information? If yes, it would be good idea to use dtype as 
well than assuming fp32 default. Helps in fp16 and other type support later. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sandeep-krishnamurthy commented on a change in pull request #9963: Onnx Module to import onnx models into mxnet

[GitBox]

sandeep-krishnamurthy commented on a change in pull request #9963: Onnx Module 
to import onnx models into mxnet
URL: https://github.com/apache/incubator-mxnet/pull/9963#discussion_r172269243
 
 

 ##
 File path: python/mxnet/contrib/onnx/_import/import_helper.py
 ##
 @@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+# pylint: disable=invalid-name
+"""Operator attributes conversion"""
+from . import *
 
 Review comment:
   nit: Instead of * , can we explicitly list all imported modules


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sandeep-krishnamurthy commented on a change in pull request #9963: Onnx Module to import onnx models into mxnet

[GitBox]

sandeep-krishnamurthy commented on a change in pull request #9963: Onnx Module 
to import onnx models into mxnet
URL: https://github.com/apache/incubator-mxnet/pull/9963#discussion_r172298550
 
 

 ##
 File path: python/mxnet/contrib/onnx/_import/op_translations/reduce_max.py
 ##
 @@ -0,0 +1,23 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+from ..translation_utils import _fix_attribute_names
+
+def reduce_max(op_name, attrs, inputs):
+new_attr = _fix_attrbute_names(attrs, {'axes':'axis'})
+return 'max', attrs, inputs
 
 Review comment:
   nit: new line


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sandeep-krishnamurthy commented on a change in pull request #9963: Onnx Module to import onnx models into mxnet

[GitBox]

sandeep-krishnamurthy commented on a change in pull request #9963: Onnx Module 
to import onnx models into mxnet
URL: https://github.com/apache/incubator-mxnet/pull/9963#discussion_r172293282
 
 

 ##
 File path: python/mxnet/contrib/onnx/_import/translation_utils/__init__.py
 ##
 @@ -0,0 +1,141 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+# pylint: 
disable=invalid-name,no-self-use,too-many-branches,too-few-public-methods,too-many-arguments
+from __future__ import absolute_import as _abs
+from base import string_types
+
+def _fix_attribute_names(attrs, change_map):
+new_attr = {}
+for k in attrs.keys():
+if k in change_map:
+new_attr[change_map[k]] = attrs[k]
+else:
+new_attr[k] = attrs[k]
+return new_attr
+
+def _fix_pooling(op_name, inputs, new_attr):
+"""onnx pooling operator supports asymmetrical padding
+Adding pad operator before pooling in mxnet to work with onnx"""
+pool_type = 'avg' if op_name == 'AveragePool' else 'max'
+stride = new_attr.get('strides')
+kernel = new_attr.get('kernel_shape')
+padding = new_attr.get('pads')
+pad_width = (0, 0, 0, 0) + _pad_sequence_fix(padding, len(kernel))
+new_pad_op = symbol.pad(inputs[0], mode='constant', pad_width=pad_width)
+new_pooling_op = symbol.Pooling(new_pad_op, pool_type=pool_type,
+stride=stride, kernel=kernel)
+return new_pooling_op
+
+def _fix_slice(inputs, new_attr):
+"""onnx slice provides slicing on multiple axis. Adding multiple 
slice_axis operator
+for multiple axes from mxnet"""
+begin = new_attr.get('begin')
+end = new_attr.get('end')
+axes = new_attr.get('axis', tuple(range(len(begin
+slice_op = symbol.slice_axis(inputs[0], axis=axes[0], begin=begin[0], 
end=end[0])
+if len(axes) > 1:
+for i, axis in enumerate(axes):
+slice_op = symbol.slice_axis(slice_op, axis=axis, begin=begin[i], 
end=end[i])
+return slice_op
+
+def _fix_squeeze(inputs, new_attr):
+"""
+MXNet doesnt have a squeeze operator.
+Using "split" to perform similar operation.
+"split" can be slower compared to "reshape".
+ This can have performance impact.
+ TODO: Remove this implementation once mxnet adds the support.
 
 Review comment:
   Related GitHub issue link?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] DickJC123 commented on issue #9543: Variable Length Support for cuDNN RNN

[GitBox]

DickJC123 commented on issue #9543: Variable Length Support for cuDNN RNN
URL: 
https://github.com/apache/incubator-mxnet/issues/9543#issuecomment-370532401
 
 
   I've been remiss at commenting because frankly new cuDNN features are 
generally advertised only upon general availability.  The cuDNN team is 
continually improving the feature set, and it's clear to us that sorting and 
packing of sequences is a customer pain point.  As such, we are working on 
feature improvements in this area for one of our future releases.  I'll point 
out that cudNN v7.1 is newly available for download, although this minor 
release has no new features to address variable-length sequences.  If there's 
more detail you are able to share about your application, whether publicly or 
privately, that would be very helpful.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] piiswrong commented on a change in pull request #9986: gluon language modeling dataset and text token reader

[GitBox]

piiswrong commented on a change in pull request #9986: gluon language modeling 
dataset and text token reader
URL: https://github.com/apache/incubator-mxnet/pull/9986#discussion_r172284803
 
 

 ##
 File path: python/mxnet/gluon/data/text/__init__.py
 ##
 @@ -0,0 +1,24 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+# pylint: disable=wildcard-import
+"""Text utilities."""
+
+from .base import *
+
+from . import lm
 
 Review comment:
   why?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] piiswrong commented on a change in pull request #9986: gluon language modeling dataset and text token reader

[GitBox]

piiswrong commented on a change in pull request #9986: gluon language modeling 
dataset and text token reader
URL: https://github.com/apache/incubator-mxnet/pull/9986#discussion_r172287707
 
 

 ##
 File path: python/mxnet/gluon/utils.py
 ##
 @@ -218,3 +220,25 @@ def download(url, path=None, overwrite=False, 
sha1_hash=None):
   'the default repo.'.format(fname))
 
 return fname
+
+def get_repo_url():
 
 Review comment:
   should this be hidden?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] piiswrong commented on a change in pull request #9986: gluon language modeling dataset and text token reader

[GitBox]

piiswrong commented on a change in pull request #9986: gluon language modeling 
dataset and text token reader
URL: https://github.com/apache/incubator-mxnet/pull/9986#discussion_r172284891
 
 

 ##
 File path: python/mxnet/gluon/data/text/_constants.py
 ##
 @@ -0,0 +1,26 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+
+"""Constants relevant to text processing."""
+
+UNK_TOKEN = ''
+
+BOS_TOKEN = ''
 
 Review comment:
   Isn't this usually ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] piiswrong commented on a change in pull request #9986: gluon language modeling dataset and text token reader

[GitBox]

piiswrong commented on a change in pull request #9986: gluon language modeling 
dataset and text token reader
URL: https://github.com/apache/incubator-mxnet/pull/9986#discussion_r172285568
 
 

 ##
 File path: python/mxnet/gluon/data/text/base.py
 ##
 @@ -0,0 +1,89 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+# pylint: disable=
+
+"""Base classes for text datasets and readers."""
+
+__all__ = ['TextTokenReader']
+
+import io
+import os
+
+from ..dataset import Dataset
+from ..datareader import DataReader
+
+class TextTokenReader(DataReader):
+"""Text reader that produces lists of tokens.
+
+Parameters
+--
+filename : str
+Path to the input text file.
+encoding : str, default 'utf8'
+File encoding format.
+sample_splitter : function, default str.splitlines
+A function that splits the dataset string into samples.
+tokenizer : function, default str.split
+A function that splits each sample string into list of tokens.
+bos : str or None, default None
+The token to add at the begining of each sentence. If None, nothing is 
added.
+eos : str or None, default None
+The token to add at the end of each sentence. If None, nothing is 
added.
+"""
+def __init__(self, filename, encoding='utf8', sample_splitter=lambda s: 
s.splitlines(),
+ tokenizer=lambda s: s.split(), bos=None, eos=None):
+filename = os.path.expanduser(filename)
+with io.open(filename, 'r', encoding=encoding) as fin:
+content = fin.read()
+
+samples = [s for s in [tokenizer(x) for x in sample_splitter(content)] 
if s]
+if bos or eos:
+for tokens in samples:
+if bos:
+tokens.insert(0, bos)
+if eos:
+tokens.append(eos)
+
+self._samples = samples
+
+def __len__(self):
+return self._samples.__len__
+
+def __getitem__(self, idx):
+return self._samples[idx]
+
+class TextDataset(Dataset):
+"""Abstract dataset class for text data.
+
+Subclasses need to override `__getitem__`, which returns the i-th
+element, and `__len__`, which returns the total number elements."""
+
+def __getitem__(self, idx):
+raise NotImplementedError
+
+def __len__(self):
+raise NotImplementedError
+
+def _flatten(self, samples):
 
 Review comment:
   These are more like util functions than class methods.
   So far it doesn't look like there is anything special about TextDataset to 
make a separate class


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] piiswrong commented on a change in pull request #9986: gluon language modeling dataset and text token reader

[GitBox]

piiswrong commented on a change in pull request #9986: gluon language modeling 
dataset and text token reader
URL: https://github.com/apache/incubator-mxnet/pull/9986#discussion_r172285145
 
 

 ##
 File path: python/mxnet/gluon/data/text/base.py
 ##
 @@ -0,0 +1,89 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# coding: utf-8
+# pylint: disable=
+
+"""Base classes for text datasets and readers."""
+
+__all__ = ['TextTokenReader']
+
+import io
+import os
+
+from ..dataset import Dataset
+from ..datareader import DataReader
+
+class TextTokenReader(DataReader):
+"""Text reader that produces lists of tokens.
 
 Review comment:
   why combine tokenizer with reader?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] piiswrong commented on a change in pull request #9986: gluon language modeling dataset and text token reader

[GitBox]

piiswrong commented on a change in pull request #9986: gluon language modeling 
dataset and text token reader
URL: https://github.com/apache/incubator-mxnet/pull/9986#discussion_r172287751
 
 

 ##
 File path: python/mxnet/gluon/utils.py
 ##
 @@ -218,3 +220,25 @@ def download(url, path=None, overwrite=False, 
sha1_hash=None):
   'the default repo.'.format(fname))
 
 return fname
+
+def get_repo_url():
+"""Return the base URL for Gluon dataset and model repository."""
+default_repo = 
'https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/'
+repo_url = os.environ.get('MXNET_GLUON_REPO', default_repo)
+if repo_url[-1] != '/':
+repo_url = repo_url+'/'
+return repo_url
+
+def get_repo_file_url(namespace, filename):
 
 Review comment:
   hidden?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services